@bongos/core 1.17.0

package/docs/recipes/multi-builder-merge.md

@@ -0,0 +1,78 @@
+# Multi-builder merge safety — audit + serialization model
+
+> **Deploy-mode note (current state).** This recipe documents the **`laptop`** merge path (operator merges `main` + SSHes the droplet). Since the 2026-06-13 cutover `config/deploy.json` is **`mode: ci`** — the merge + deploy run **server-side** (PR auto-merge + `.github/workflows/deploy-prod.yml`), and credential-less dev boxes ship via server-mediated publish ([ADR 0055](../adr/<redacted>.md)). The local-`main`-worktree + merge-lock model below applies only if `deploy.json` is reverted to `mode: laptop`.
+
+> **Task:** V3.R50 / [#269](https://example.com/builders#/task/269) (criterion **C6 — parallel-safe-under-load**).
+> **Question it answers:** when N>1 builders ship and merge concurrently, where do `ship.js` and `/merge-mode` assume "I am the only builder," and what serializes the shared state so they don't collide?
+
+This recipe is the audit deliverable for [#269](https://example.com/builders#/task/269). It enumerates every shared-state surface in the merge/ship path, marks which assume serialized access, names every git operation that could race, and documents the mutex that now covers them.
+
+---
+
+## TL;DR
+
+- **Claim side is already deterministic.** `claimTask()` runs in a single transaction with `SELECT … FOR UPDATE` on the task row, a touches-overlap check, and the `uniq_active_claim_per_task` partial unique index. A lost race surfaces as `ALREADY_CLAIMED` (Postgres `23505`), never as corruption. No change needed.
+- **Merge side is the risk, and it has ONE shared resource:** the single `main` worktree (`REPO_ROOT`) on the build machine. Every ship's auto-merge and every `/merge-mode` run checks out `main`, fetches, merges, pushes, and deploys *there*.
+- **[#462](https://example.com/builders#/task/462) (criterion C4)** added a lockfile so two concurrent **ships** couldn't collide — but the lock lived *inside* `ship.js`, so `/merge-mode` (which mutates the same tree by hand) never took it. That left two races open: **ship-vs-merge-mode** and **merge-mode-vs-merge-mode**.
+- **[#269](https://example.com/builders#/task/269) (this task)** extracts the lock into [`src/bongos/merge-lock.js`](../../src/bongos/merge-lock.js) and has **both** surfaces acquire the **same** mutex. `/merge-mode` takes it via the [`scripts/gds/merge-lock.js`](../../scripts/gds/merge-lock.js) CLI before it touches `main`, and releases it after deploy.
+
+---
+
+## The shared-state surfaces
+
+| Surface | Shared across builders? | Serialized by | Notes |
+|---|---|---|---|
+| **`main` worktree** (`REPO_ROOT`): checkout, `git fetch`, `git merge`, `git push origin main`, post-merge smoke, `~/deploy.sh` | **Yes — the single hot resource.** One checkout, one HEAD. | **`merge-lock` mutex** (`.git/otb-automerge.lock`) — held by `ship.js` auto-merge AND `/merge-mode`. | This is the surface [#269](https://example.com/builders#/task/269) closes end-to-end. |
+| **`claims` table** (claim/release) | Yes | Postgres txn + `uniq_active_claim_per_task` unique index + touches-overlap check (`db.js claimTask`) | Deterministic. `ALREADY_CLAIMED` / `TOUCHES_CONFLICT` are clean refusals, not corruption. |
+| **`creator_session_id` on claims** | Yes | `CLAIM_SESSION_MISMATCH` (migration 013) | Only the session that created a claim may ship/release it — prevents cross-session interference. |
+| **`worktree_name` uniqueness** | Yes | Server-side disambiguation in `claimTask` (V3.R51 / [#270](https://example.com/builders#/task/270)) | Two claims with the same worktree name → second gets a suffix. |
+| **`migration_counter`** (migration number allocation) | Yes | `UPDATE … RETURNING` inside the claim txn | Atomic; each claim that signals `migrations/` gets a distinct number. |
+| **Feature worktree** (`.claude/worktrees/<name>`) | No — per-builder | n/a | Each builder's feature branch work is isolated. The collision only appears at merge time, on `main`. |
+| **`origin/main` on GitHub** | Yes | `git push origin main` is atomic server-side; a stale push is rejected (non-fast-forward) | The local mutex makes the local merge serialized so the push is fast-forward; a rejected push bails the ship to `confirmed`. |
+| **The droplet deploy** (`ssh … ~/deploy.sh`) | Yes | Runs *inside* the merge-lock critical section | Two deploys can't overlap because each holds the mutex through merge→push→deploy. |
+
+---
+
+## Every place the merge path assumed "I am the only builder"
+
+1. **`ship.js autoMerge()` — `git merge --ff-only origin/main` on `REPO_ROOT`.**
+   If another session left `main` mid-merge or moved it, this failed and bailed with the misleading *"main is ahead of origin somehow."* — **Fixed in [#462](https://example.com/builders#/task/462)** with the in-`ship.js` lock + `mainWorktreeBusy()` guard (refuses to mutate `REPO_ROOT` when `MERGE_HEAD` exists or the tree is dirty).
+
+2. **`/merge-mode` — `git checkout main; git pull; git merge …` on `REPO_ROOT`, with no lock.** ← **the gap [#269](https://example.com/builders#/task/269) closes.**
+   `/merge-mode` mutated the exact same worktree `ship.js` does, but never acquired [#462](https://example.com/builders#/task/462)'s lock (the lock was a private function inside `ship.js`). A ship auto-merging while an operator ran `/merge-mode` could interleave checkouts/merges on one HEAD. **Fixed in [#269](https://example.com/builders#/task/269):** `/merge-mode` now runs `node scripts/gds/merge-lock.js acquire` before Step 2 and `release` in Step 5c.
+
+3. **macOS Finder/iCloud `* 2` duplicate refs.**
+   Both surfaces clean these before fetching (`cleanupMacFinderDupRefs` / `find .git -name "* 2" -delete`). Not a concurrency bug, but a shared-filesystem hazard worth listing — it broke every auto-merge on 2026-05-10.
+
+---
+
+## The mutex, precisely
+
+[`src/bongos/merge-lock.js`](../../src/bongos/merge-lock.js) is a portable, advisory file lock at `<REPO_ROOT>/.git/otb-automerge.lock`. **Not `flock(2)`** — ships run on macOS over an iCloud-synced filesystem where `flock` is unreliable, and the manual `/merge-mode` holder is a short-lived CLI process with no long-lived owner.
+
+Lockfile format (3 lines): `pid` · `epoch-ms timestamp` · `mode` (`auto` | `manual`).
+
+| | `auto` (a ship — process stays alive through its merge) | `manual` (`/merge-mode` via the CLI — no long-lived owner) |
+|---|---|---|
+| Recorded pid | `process.pid` of the ship | `0` (the CLI process exits immediately) |
+| Reclaimed when | holder pid is dead **OR** stamp older than `AUTO_STALE_MS` (10 min) | stamp older than `MANUAL_STALE_MS` (30 min) — **time only** |
+| Released by | the owning ship process (pid match), in a `finally` | whoever runs `merge-lock.js release` (any process, since mode=manual) |
+
+A ship that can't get the lock within 30s bails cleanly to `confirmed` (the task is safe; `/merge-mode` lands it later). An operator who sees `acquire` exit `3` is told to wait — never to force past a live holder. The two stale windows differ on purpose: an automated ship that crashes should free up fast (10 min), while a human resolving conflicts mid-merge gets a generous 30 min before the lock is presumed abandoned.
+
+Behavior is pinned by [`tests/merge_lock.mjs`](../../tests/merge_lock.mjs) (auto-vs-auto wait, dead-pid + time-based reclaim, manual-window survival, owner-guarded release, two-line legacy-lockfile back-compat).
+
+---
+
+## Known residual limitation (filed, not blocking C6)
+
+**The mutex is machine-local.** `.git/otb-automerge.lock` serializes everything that mutates the *one* `main` worktree on the *one* build machine. Today that is correct and complete: all ships and merges run from worktrees under the single shared checkout on Lars's Mac. If a future version lets builders ship from *different physical machines* against *different* checkouts of `main`, a filesystem lock won't serialize them — that needs a server-side lease (e.g. an advisory lock row / `pg_advisory_lock` keyed on `main`, handed out by the API at merge time). Captured as an idea for GDS-V4; out of scope for C6, which targets parallel safety on the current single-machine, multi-session model.
+
+---
+
+## Related
+
+- [ADR 0024](../adr/<redacted>.md) — multi-agent system (the reason N>1 builders exist).
+- [#462](https://example.com/builders#/task/462) — the C4 auto-merge lock this task generalizes.
+- [#270](https://example.com/builders#/task/270) (V3.R51) — server-side worktree-name uniqueness (a sibling parallel-safety fix).
+- [#272](https://example.com/builders#/task/272) — the parallel-load smoke test that *exercises* this serialization under 5 simulated builders (the other open C6 task). Its **DB rank is R53** (the DB is king); the original planning-seed cross-ref text called it "R54", but the live rank renumbering settled it at R53/[#272](https://example.com/builders#/task/272).

package/docs/recipes/ops-gotchas.md

@@ -0,0 +1,212 @@
+---
+track: internal
+scope: shared-infra (production droplet, Caddy, Postgres, deploy chain) — touches both tracks but the *recipes themselves* are builder-facing
+---
+
+# Ops gotchas — read before deploying or touching infra
+
+Things we've already paid for once. Don't pay again.
+
+Organized by topic, not chronology. New entries go in the matching section; create a new section for a new topic.
+
+---
+
+## PostgreSQL
+
+### `pg` defaults to TCP localhost, not the Unix socket
+
+- **Symptom:** `new Pool()` with no `host` fails with `SASL: SCRAM-SERVER-FIRST-MESSAGE: client password must be a string`. The Node process appears to start, but every database call silently fails (or noisily fails depending on where it happens).
+- **Root cause:** The `pg` library defaults to TCP `localhost`. With Postgres configured for **peer authentication** (matching the OS user to the DB user), TCP doesn't qualify — TCP requires a password, peer auth only works over the Unix socket. So `pg` tries TCP, Postgres demands a password, no password is provided, SASL bombs.
+- **Fix:** Pass the socket directory explicitly:
+  ```js
+  new Pool({ host: '/var/run/postgresql' })
+  ```
+  (Or set `PGHOST=/var/run/postgresql` in the environment.) That directory is the Debian/Ubuntu default Postgres socket location; on other distros it may differ.
+- **Why it bites:** The `pg` library's defaults assume a standard TCP+password setup, which is the most common configuration but not the cheapest one for a single-machine deploy. Peer auth is more secure and frictionless once you know the socket path. The error message points at SASL, not at the host setting, which buries the real cause.
+- **Where this lives in the codebase:** [`src/db.js`](../../src/db.js) has the explicit socket host.
+
+---
+
+## Colyseus
+
+### Server/client version skew between 0.16 and 0.17
+
+- **Symptom:** `npm install colyseus` picks the latest 0.17.x server, but `colyseus.js` (the browser client) is still 0.16.x. Install fails with an `eresolve` peer-dependency error on `@colyseus/schema` because 0.17 wants schema v4 and 0.16 wants schema v3.
+- **Root cause:** Colyseus releases server-side first; the browser client lags by several months. The two must be on the same major.minor.
+- **Fix:** Pin all four to the same major.minor in `package.json`:
+  ```json
+  "colyseus": "^0.16",
+  "@colyseus/ws-transport": "^0.16",
+  "@colyseus/schema": "^3",
+  "colyseus.js": "^0.16"
+  ```
+- **Why it bites:** `npm install <pkg>` pulls the latest version unless told otherwise, and there's no warning that the matching client doesn't exist yet. `colyseus.js` 0.17 will eventually ship — when it does, the pins can be lifted in lockstep.
+
+### Colyseus 0.16 ignores `static maxClients`
+
+- **Symptom:** Setting `maxClients` as a static class field on a Room subclass (the documented pattern in some online examples) has no effect. The room accepts unlimited clients.
+- **Fix:** Set `this.maxClients = N` inside `onCreate()`. It is an instance property, not a static one, in 0.16.
+- **Why it bites:** The docs and examples drift between Colyseus versions. The older static-field pattern is still findable on the web.
+
+---
+
+## Caddy + Cloudflare
+
+### Let's Encrypt auto-renewal silently breaks under Cloudflare proxy (resolved)
+
+- **Symptom:** Caddy obtained the initial cert fine, then failed to renew at the 90-day mark — site went down.
+- **Root cause:** With Cloudflare proxy ON (orange cloud), CF intercepts port 80 at the edge and Caddy never sees the HTTP-01 challenge from Let's Encrypt. Renewal silently fails.
+- **Fix (already in place):** Replaced Let's Encrypt with a Cloudflare Origin Certificate (15-year cert at `/etc/caddy/origin.crt` + `.key`, explicit `tls` directive in Caddyfile). Eliminates the auto-renewal dependency for the CF↔origin leg. See [ADR 0003](../adr/<redacted>.md).
+- **Why it bites:** The failure mode is silent for ~89 days, then catastrophic. Easy to think the site is "set and forget" when it is actually approaching a cliff.
+
+### Deploying configs to `/etc/caddy/` requires staging through `~/`
+
+- **Symptom:** `scp` directly to `/etc/caddy/Caddyfile` fails with permission denied because `lars` doesn't own `/etc/caddy/`.
+- **Fix:** `scp` to `~/staging/Caddyfile` first, then `sudo mv ~/staging/Caddyfile /etc/caddy/Caddyfile && sudo systemctl reload caddy`. Same pattern for the systemd unit at `/etc/systemd/system/example.service`.
+- **Why it bites:** The deploy script handles this for normal app code, but config files in `/etc/` need the staging step. Easy to forget when you haven't touched infra in a while.
+
+### Cloudflare hides real client IPs from the origin
+
+- **Symptom:** Caddy logs show Cloudflare IPs (162.158.x.x, 172.69.x.x, etc.) instead of real visitor IPs.
+- **Fix when needed:** Wire `trusted_proxies cloudflare` into the Caddyfile so Caddy honors `X-Forwarded-For` / `CF-Connecting-IP`. CF maintains the IP-range list — the directive auto-pulls it.
+- **Why it bites:** Not yet biting (we're not rate-limiting or doing abuse detection). It will bite the moment we try to identify a single misbehaving client. Documented here so when that day comes the fix is one line.
+
+---
+
+## SSH + droplet
+
+### Don't `apt-get upgrade` in a plain SSH session
+
+- **Symptom:** Mid-upgrade, sshd restarts (because openssh-server was in the upgrade set). The SSH session drops, leaving the upgrade half-run and the operator locked out until they can reach the DigitalOcean web console.
+- **Recovery (already once):** Power-cycle from DO web console, log back in, verify the upgrade completed. It usually has — apt is good at finishing what it started — but this is luck not design.
+- **Fix going forward:** Skip routine `apt-get upgrade` entirely. Let `unattended-upgrades` handle security patches in the background — it's already enabled. If a manual upgrade is genuinely needed, wrap it in `tmux` or `screen` so it survives the session drop.
+- **Why it bites:** sshd is the lifeline. Anything that touches it from inside an SSH session is dangerous.
+
+### SSH is key-only, root login disabled
+
+- **Reminder, not a gotcha:** `PermitRootLogin no`, `PasswordAuthentication no` in `/etc/ssh/sshd_config`. Only `lars` can log in, and only with the registered ed25519 key. If the key is ever lost, recovery is via DigitalOcean's web console (or by attaching the volume to a recovery droplet).
+
+### SSH silently times out — fail2ban banned your IP
+
+- **Symptom:** `ssh lars@REDACTED_IP` hangs and times out. `ping REDACTED_IP` works. `curl https://example.com/healthz` returns 200. Only port 22 from your specific public IP is dead. Looks identical to a firewall outage.
+- **Root cause:** fail2ban (running on the droplet, watches `/var/log/auth.log`) saw N auth failures from your IP inside the rolling window and inserted an iptables `DROP` rule. Default policy in this distro: 5 failures in 10 minutes triggers a 10-minute ban. The drop is silent — no RST, no Mac-side log, no obvious error — which is why diagnosis is slow the first time.
+- **Immediate fix:** DigitalOcean web console (it bypasses port 22 because it doesn't go through SSH at all). Log into the droplet via the console, then:
+  ```bash
+  sudo fail2ban-client status sshd            # confirm the IP is in the Banned list
+  sudo fail2ban-client unban <your-public-IP>
+  ```
+  Then verify from the Mac: `ssh -o ConnectTimeout=5 lars@REDACTED_IP "uptime"` should respond in <1s.
+- **Permanent fix:** add your IP to `/etc/fail2ban/jail.local`'s `ignoreip` list and reload fail2ban. `jail.local` overrides the package's `jail.conf` so apt upgrades won't trample the whitelist. Example file:
+  ```ini
+  [DEFAULT]
+  ignoreip = 127.0.0.1/8 ::1 <your-public-IP>
+  bantime = 10m
+  findtime = 10m
+  maxretry = 5
+
+  [sshd]
+  enabled = true
+  ```
+  Then `sudo systemctl reload fail2ban`. Confirm with `sudo fail2ban-client status sshd | grep -i ignor` — the IP should appear in the Ignored list.
+- **Why it bites:** the failure mode is silent. There's no log on the Mac side, no log from the operating shell, and the symptom is indistinguishable from "port 22 is closed" or "the droplet is down." You only realize it's fail2ban after the DO console reveals the iptables DROP. First-time diagnosis is ~20 minutes; once you know the signature it's 30 seconds.
+- **For new builders:** most builders (Metic and below) don't need direct droplet SSH — they work through the public API. The whitelist is primarily an Archon-comfort item. Don't reflexively add every new builder's IP; only add IPs for people who actually deploy.
+
+---
+
+## GitHub OAuth (web flow)
+
+### Authorized callback URL must include the full /api/gds path
+
+- **Symptom:** Visitor clicks "Sign in with GitHub" on /builders, GitHub renders its red "Be careful!" warning page or returns `?error=redirect_uri_mismatch`, and the visitor never lands back on the hall.
+- **Root cause:** The GitHub OAuth app's **Authorization callback URL** field must match exactly what the server tells GitHub. Our server sends `${publicOrigin}/api/gds/auth/web/callback` — easy to misconfigure as just `https://example.com/` or `/auth/callback` when registering a fresh OAuth app.
+- **Fix:** In github.com → Settings → Developer settings → OAuth Apps → (the Off the Boats app) → **Authorization callback URL**, set:
+  ```
+  https://example.com/api/gds/auth/web/callback
+  ```
+  No trailing slash, full path. If staging gets provisioned ([#260](https://example.com/builders#/task/260)) it will need a second OAuth app or the same app with multiple callback URLs (GitHub now supports up to 10).
+- **Where this lives in the codebase:** [`src/bongos/routes/auth.js`](../../src/bongos/routes/auth.js) — `/auth/web/start` computes `redirectUri` from `publicOrigin(req)` and the `/auth/web/callback` route reads back to it. As of [#371](https://example.com/builders#/task/371) the error branches render a styled HTML page (with a retry link) instead of `text/plain`, so the visitor at least sees something coherent.
+- **Why it bites:** GitHub's warning page is intentionally scary and doesn't tell you *which* URL it expected; the only diagnostic is the `error=redirect_uri_mismatch` query param on the redirect back, which our server now surfaces in the styled error page.
+
+---
+
+## Mac / local dev
+
+### TextEdit defaults to RTF and silently corrupts plain-text files
+
+- **Symptom:** Saved a Cloudflare Origin Cert to `~/Desktop/origin.crt` from TextEdit. The file ended up wrapped in RTF formatting codes (and TextEdit added `.rtf` to the filename).
+- **Fix:** `textutil -convert txt -output X.crt X.crt.rtf` strips the RTF wrapper and writes plain PEM.
+- **Going forward:** When editing certs, keys, configs, or any plain-text file on this Mac, either Format → Make Plain Text in TextEdit first, or use VS Code (plain text by default), or use `nano`/`vim` from the terminal.
+- **Why it bites:** TextEdit's default makes sense for word-processing but is hostile to anything that needs to be byte-exact.
+
+### `.env.local` is auto-provisioned into new worktrees (don't `cp` it by hand)
+
+- **Behavior (V3.R59 [#278](https://example.com/builders#/task/278)):** `.env.local` is gitignored, so `git worktree add` does not carry it over. The `SessionStart` hook [`.claude/hooks/session-start.js`](../../.claude/hooks/session-start.js) copies it from the main checkout into the worktree on first session — no manual `cp` from a sibling needed.
+- **If the main checkout has no `.env.local`:** the hook tells you to run `/builder-setup`.
+- **The Google AI Studio key is separate:** it resolves from `~/.config/otb/env`, which already survives worktree churn — see [`art/pipeline/gen_api.py`](../../art/pipeline/gen_api.py).
+- **All hooks run via `node`, not `bash` ([#671](https://example.com/builders#/task/671)):** `SessionStart`, `UserPromptSubmit`, and `Stop` run identically on Windows/macOS/Linux. `session-start.js` chains the env-copy + onboarding + rank-change greeting + memory pull that the old `sync-env-local.sh` shell chain ran.
+
+---
+
+## GDS build pipeline
+
+### Two deterministic routines run as in-process server timers, not external cron
+
+- **What:** `peer-vote-tally` (weekly) runs inside the always-on Node process via the autonomy module's poller `modules/autonomy/pollers/routine-timers.js` (ADR 0078 amendment, task 1382; carved into `modules/autonomy/` at BV1.R83) — no `claude -p`, no external cron, no token. The loader's `startModulePollers` starts it (on the game/droplet boot only); it reaches `tallyPeerVotes` through the doorway. It's a no-op anywhere the DB is unreachable (builder machines), so it only does work on the prod droplet.
+- **Disable / move it out-of-process:** set `ROUTINE_TIMERS_DISABLED=1` in `/etc/example/web-source.env` and restart. Also set it on any **second** always-on instance that shares the `example` DB so the tally can't double-fold (the `FOR UPDATE OF pv` row lock is the hard guard if you forget).
+- **Run it manually now:** `node scripts/gds/run-routine.js peer-vote-tally` (the dispatcher front door still works). The cadence is boot-anchored ("~90s after boot, then weekly"), not a wall-clock Monday job — frequent deploys re-anchor it, which is harmless (the tally is idempotent).
+
+### A ship was graded FAIL but the work is genuinely good (grader false-negative)
+
+- **Symptom:** `/builder-ship` runs the subagent grader and it hard-FAILs with blockers that are *factually wrong* — e.g. "No PNG committed" / "no evidence the review ran" / "empty diff — no files changed" — even though the work is committed on the branch and passed its own review. The task is stuck at `completed` with 0 credits, and the builder is below Archon so every fix route (`PATCH touches`, `POST /confirm`, override-decide) is gated.
+- **Root cause (one known class, fixed):** the grader read a `null` baseline because the task had empty `touches[]`, so it diffed against nothing and "saw" zero files. Fixed in [#536](https://example.com/builders#/task/536) — the grader baseline is now derived from `git merge-base` independent of `touches[]`. If you hit a *different* false-negative class, treat the below as the general escape hatch.
+- **The process (canonical — works at any rank):**
+  1. The **stuck builder** (any rank, incl. Xenos) files an override request — this is the one builder-rank-available escape hatch:
+     `POST /api/gds/tasks/:id/override-request` with `{ "rationale": "<≥20 chars: why the grade is wrong>" }`.
+     This is *not* routing around the trust boundary — Xenos are allowed to file; only the decision is privileged.
+  2. An **Archon** reviews the open queue (`GET /api/gds/override-requests?status=open`) and decides:
+     `POST /api/gds/override-requests/:id/decide` with `{ "decision": "approve", "note": "..." }`.
+  3. On approve, the task flips `completed → confirmed` and **both the credit AND the ship attribution (`shipped_by`) land with the original requester**, never the approving Archon ([#538](https://example.com/builders#/task/538)). The whole exchange is permanently audited in `task_override_requests` (requester, decider, rationale, decision note, timestamps).
+- **Do NOT** use `POST /tasks/:id/confirm` to unstick *another* builder's task — it credits + attributes to the *caller* (the Archon), not the original builder. `/confirm` is for the shipping builder's own `--skip-grade` path. The override-request → approve flow is the correct tool when the work was someone else's.
+- **Why it bites:** the grader defaults to fail (by design — it's adversarial), so a false-negative looks identical to a real rejection. The fix is a documented, audited human-in-the-loop appeal, not a grader bypass — and the appeal pays the right person.
+
+---
+
+## Status page + monitoring
+
+### Cloudflare 522 with the whole site dark — the droplet itself is down (often billing)
+
+- **Symptom:** `example.com` shows Cloudflare **522 (connection timed out)** and the status page is dark too. From your machine, `ping` + `ssh` (port 22) + `curl https://…` (443) **all time out** — not "connection refused" — yet `dig example.com` still returns Cloudflare IPs and status.digitalocean.com is green.
+- **Root cause:** the droplet is powered off / hung / firewalled — **not** an app crash. A *timeout* (black-hole) on every port means packets are dropped, the signature of the box being down or network-blocked; *refused* would mean the box is up but the service isn't listening. On 2026-06-03 the cause was a **failed DigitalOcean payment** — DO powers off droplets when a charge fails.
+- **Fix:** every lever is in the **DigitalOcean web panel** (SSH is dead and we keep no `doctl`/API token locally): check for a billing banner, check the droplet's power state, then Power → Turn on / Power Cycle. The recovery console bypasses SSH entirely. Confirm with `ssh … uptime` (a fresh boot) once it answers; systemd auto-starts caddy/node/postgres. Prevent recurrence with a **backup payment method + balance alerts** on DO.
+- **Why it bites:** a billing power-off looks identical to a crash or a firewall block, and the status page can't tell you which — it's served by the same droplet (see next entry).
+
+### Your status page must not share fate with the product (and cannot measure itself)
+
+- **Symptom:** the site is down for hours, but The Watch still shows 100% / "standing".
+- **Root cause:** `status.example.com` is served by the same Node process on the same droplet as the game, and the in-process poller (`src/bongos/uptime-poller.js`) probes `127.0.0.1`. When the droplet dies, the page dies with it and the poller writes **zero rows** — the gap renders as green. A monitor that lives inside the monitored thing cannot witness its own death.
+- **Fix:** measure availability from **outside** (the external GitHub-Actions prober — `.github/workflows/status-probe.yml`, task [#565](https://example.com/builders#/task/565)) and host the status page **off-box** (follow-up [#566](https://example.com/builders#/task/566)). See [ADR 0029](../adr/<redacted>.md) + GDS learning `#35`.
+- **Why it bites:** a status page that lies is worse than none — it actively reassures you while the product is on fire.
+
+### The live status page is NOT served by the droplet — editing the status surface + deploying does nothing
+
+> **Path note (BV1.R84):** the status surface moved from `public-status/` to `modules/status-ui/public/` (carved to the `status-ui` web-surface module). Everything below applies at the new path; `scripts/status-mirror-sync.js` was re-pointed. **Open follow-up:** `.github/workflows/sync-status-mirror.yml` still triggers on `public-status/**` and must be re-pointed at `modules/status-ui/public/**` by an owner/Archon (the GitHub App can't push `.github/workflows/*`, learning 107) — until then the auto-sync won't fire and you must run the script by hand.
+
+- **Symptom:** you change `modules/status-ui/public/`, deploy to the droplet, the origin serves the new page (verified via `curl` on the box), but `status.example.com` keeps showing the old version. Purging Cloudflare doesn't help.
+- **Root cause:** the off-box move (ADR 0029) repointed `status.example.com` DNS at **GitHub Pages from the separate repo `example-owner/example-status`** (served by Fastly — headers say `server: GitHub.com`, `via: varnish`, **no `cf-*` headers** → Cloudflare isn't even in the path). The droplet still serves `modules/status-ui/public/` as a same-origin fallback, but the public DNS goes to Pages. There was no sync, so droplet edits never reached the live page.
+- **Fix:** publish to the mirror with `node scripts/status-mirror-sync.js --repo-dir <mirror-checkout> --push` — or (once the workflow trigger is re-pointed, see the path note above) let `.github/workflows/sync-status-mirror.yml` do it on push (needs the `STATUS_MIRROR_TOKEN` repo secret). `status.js`/`style.css`/`cursors/` are byte-identical copies (the one off-box difference, the API base, is injected via `window.STATUS_API_BASE` in the mirror's `index.html`); the sync injects the off-box banner + API-base script into `index.html` at fixed anchors and **throws if an anchor is missing** rather than mis-publishing. GitHub Pages purges its own Fastly cache on deploy, so the live page updates ~1 min after the Pages build goes green (`gh api repos/example-owner/example-status/pages/builds/latest`). ([#737](https://example.com/builders#/task/737))
+- **Why it bites:** the origin looks correct at every layer you can `curl` from the box, so you chase a phantom cache on the wrong CDN. The tell is the response headers — `server: GitHub.com` + Fastly, not `cloudflare`.
+
+---
+
+## Dev box
+
+### `bypassPermissions` "still not available" even though the box was provisioned for it (task 2022)
+
+- **Symptom:** you reconnect to the browser terminal (or SSH) on a dev box and `claude` still stops for permission prompts, even though tasks [#1251](https://example.com/builders#/task/1251) and [#1314](https://example.com/builders#/task/1314) default every box to `bypassPermissions`.
+- **Root cause:** `otb-box-launch.sh` (behind `box-terminal.service`) only ever runs `claude` once — the first time it creates the tmux `otb` session. Every later connect just reattaches to that **same running process**. The `defaultMode=bypassPermissions` setting is written into `~/.claude/settings.json` by `box-source-fetch.sh`'s cron, on its own `*/10` schedule (and, on an already-provisioned box, only after that box's baked copy of the script has self-updated to even contain the fix). If the `otb` tmux session got created before that write landed — an older box, or a fresh box whose very first connect raced the first cron tick — `claude` started once in normal `auto` mode and is stuck there for the life of the box; reconnecting never re-launches it, so the fix never takes effect even once the setting is correct on disk. (Flagged as an open follow-up in [ADR 0104](../adr/<redacted>.md).)
+- **Fix:** `otb-box-launch.sh` now asserts `defaultMode=bypassPermissions` itself, synchronously, right before it creates a **new** `otb` tmux session — so a fresh session never depends on cron timing. This prevents the race going forward but does **not** repair a box whose `otb` session is already running in the wrong mode: for that, from an SSH/browser-terminal shell run `tmux kill-session -t otb` then reconnect (the launcher creates a fresh session and picks up the corrected setting immediately). Killing the session only drops the terminal pane — it does not touch `/workspace` or in-flight git state.
+- **Why it bites:** every layer *looks* fixed — `~/.claude/settings.json` has the right value, `IS_SANDBOX=1` is set — because you're inspecting the box's filesystem, not the already-running process that never re-read it.
+
+## When to add a new entry here
+
+Anytime you spend more than ~10 minutes diagnosing something that turned out to have a non-obvious cause, add a section. Include symptom, root cause, fix, and why-it-bites. The cost of a 5-minute write-up is much less than the cost of the next session re-discovering the same thing.

package/docs/recipes/overnight-builder.md

@@ -0,0 +1,53 @@
+# Recipe — running the autonomous overnight builder ("session polling")
+
+> Added under GDS-V3 task [#358](https://example.com/builders#/task/358). The routine itself is `.claude/scheduled-tasks/overnight-builder/SKILL.md`; this page is the operator's runbook. Archon-only.
+
+## What it does
+
+Starts one session that works the GDS queue **by priority, on its own**. It claims the highest-priority safe task, works it in a fresh sub-agent, ships it, then re-polls the claimable list — the promote-on-ship DB trigger (migration 020) self-feeds the queue, so as soon as one task ships the next unblocked one is waiting. It loops until nothing claimable remains, the time budget runs out, or the cost cap is hit.
+
+It only ever touches tasks the GDS marks safe to run unattended (`automation_tag = 'overnight-safe'`, `manual_degree <= 2`, unblocked). Nothing broken reaches `main`: the quality grader gates credits and the auto-merge re-runs smoke tests and bails (parking the task) on any conflict or failure.
+
+## Start it (serial — the proven path)
+
+1. **Keep the Mac awake.** Deploys run from your machine, so it must stay on. In a terminal:
+   ```bash
+   caffeinate -i &
+   ```
+   `caffeinate` is a built-in macOS command that stops the Mac from sleeping; `-i` prevents idle sleep until you kill it (`kill %1` or just close the terminal in the morning). Nothing to install.
+2. **Open a Claude Code session** in the repo and launch the routine, e.g.:
+   > "Run the overnight-builder routine. Serial, 6-hour budget, stop at $40."
+3. It checks you're **Archon** (refuses otherwise), then loops. Walk away.
+
+## Go parallel (advanced — opt in only with credit headroom)
+
+Parallel runs several tasks at once in **separate isolated worktrees**. It gets more done but spends Claude credits several times faster and has more ways to get stuck unattended. Only use it when you know you have the credit headroom for the night.
+
+> "Run overnight-builder, **parallel 3**, 6-hour budget, stop at $80."
+
+The runner only fills parallel slots with tasks explicitly flagged **`parallel_safe = true`** (and, as always, with non-overlapping file footprints). Flag a task parallel-safe at creation (`parallel_safe: true` in `POST /api/gds/tasks`) or after the fact:
+
+```bash
+TOKEN=$(node -e "console.log(require(require('os').homedir()+'/.config/otb/pms-session.json').token)")
+curl -sS -X PATCH -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" \
+  -d '{"parallel_safe": true}' https://example.com/api/gds/tasks/<id>
+```
+
+In parallel mode each worker stops at `confirmed` (credits land) and the runner lands the merges **one at a time** — concurrent merges to `main` are never allowed.
+
+## Reading the morning report
+
+The end-of-run report buckets everything; the only things needing you are:
+
+- **Parked — needs merge** (`confirmed`): a ship passed quality review but hasn't landed yet. The server reconciler sweeps these to `shipped` on its own within ~5 min — re-check the report before acting. Only if one is *still* parked after that does it need **`/merge-mode`** by hand (a conflict the server couldn't auto-resolve).
+- **Parked — needs grade review** (`completed`): the quality grader failed or flaked (it flakes ~1 in 3). Eyeball the change; if it's good, manually confirm it (`POST /api/gds/tasks/:id/confirm`, archon-only).
+- **Released with feedback**: the runner couldn't finish these and put them back in the queue with a note (a captured learning) explaining why — so the next builder, you or it, doesn't start from zero. Nothing to do unless you want to unblock them.
+
+Everything under **Shipped** is already merged and deployed.
+
+## Gotchas
+
+- **It's archon-only** by policy (only `archon` and `thetes` ranks are live; claim/ship are open to any builder at the API, so the rank check lives in the routine). When middle ranks go live, widen the gate there.
+- **No more `--expand-touches` / drift scanner (task 879).** ship.js no longer scans for files outside the declared touches[] or offers to expand it — a wrong up-front prediction isn't ship-blocking; `git merge` at land time is the authoritative collision detector. (Older notes referenced `--expand-touches auto`; that flag is gone.)
+- **`.env.local` doesn't propagate to fresh worktrees** — if a parallel worker's task hits the Google API (art pipeline), copy it in first. See `docs/recipes/pixel-art-pipeline.md`.
+- **Start serial first.** Prove a clean single-task loop before trusting a parallel night.

package/docs/recipes/packaging-the-core.md

@@ -0,0 +1,49 @@
+# Recipe — packaging the Cloud Bongos core as a pinned, installable artifact
+
+> Implements [ADR 0100 §1](../adr/<redacted>.md) (the two-repo instance model). This is **R84**, the first step of the `R84 → R85 → R86` spine: package the core → stand a downstream instance up on a pinned core → make `bongos upgrade` a real update channel. Tooling: [`scripts/gds/package-core.js`](../../scripts/gds/package-core.js) (`bongos package-core`).
+>
+> **Real installable npm package ([ADR 0108](../adr/<redacted>.md) §2, task 1882/W2, shipped):** the artifact is `package/`-rooted (the npm convention `npm install ./x.tgz` requires) and carries a synthesized `package.json` — `name: "@cloudbongos/core"`, `version` = `CORE_VERSION`, `main: "src/platform-server.js"`, `exports` for the default entry (`buildPlatformApp`) plus `./serve-internal` (`mountInternalSurfaces`) and `./module-api`, `bin: "bongos"` — and a **pruned** `package-lock.json` (express + pg's own transitive closure only, resolved offline from the source lockfile — no colyseus/phaser/sharp/discord.js). Both files are synthesized fresh, never copied verbatim from the source repo's game-heavy `package.json`. `resolveCoreRoot()`/`resolveInstanceRoot()` ([`src/instance-config.js`](../../src/instance-config.js), W1/task 1881) are what a consumer's own code routes through once installed.
+
+## What the artifact is
+
+A Cloud Bongos *instance* can be its **own repository** that installs a **pinned version** of the Cloud Bongos **core** — the portable methodology + platform + neutral branding — and layers its own content, branding, and modules on top. "The core" is not re-defined here: it is exactly the subtree [`isPublishable()`](../../scripts/gds/publish-manifest.js) already defines ([ADR 0098](../adr/<redacted>.md)).
+
+`bongos package-core` produces three things in `dist/` (git-ignored — the artifact is a build product, not committed):
+
+| Output | What it is |
+|---|---|
+| `cloudbongos-core-<version>.tgz` | the installable unit — a gzip'd tar of the redacted core tree, rooted at `package/` (the npm convention `npm install ./x.tgz` requires), carrying a synthesized `@cloudbongos/core` `package.json` + pruned `package-lock.json` (task 1882) |
+| `cloudbongos-core-<version>.manifest.json` | the **pin** — version, source commit, per-file hashes, and the tree hash a consumer verifies against |
+| `cloudbongos-core-<version>/` | the same redacted tree unpacked, for inspection |
+
+The version is the **platform version** — `CORE_VERSION` from [`src/module-api.js`](../../src/module-api.js), the same coordinate the module loader and `bongos upgrade` compare against ([#1203](https://example.com/builders#/task/1203)). That shared coordinate is what keeps the spine coherent: a module pinned to `^core_contract` is guaranteed to load against the packaged core.
+
+Every file is **redacted** through the same fail-closed pipeline the public mirror uses ([ADR 0099](../adr/<redacted>.md)): instance literals + public IPs are stripped, a secret backstop runs, and if *any* residual leak survives the packager **produces nothing** (`dist/` is never touched). So a downstream repo never inherits the source instance's identity or secrets — it gets a neutral core.
+
+## Produce it
+
+```sh
+bongos package-core                 # from HEAD, version = platform CORE_VERSION
+bongos package-core --ref origin/main --version 1.13.0
+node scripts/gds/package-core.js --droplet-ip <ip>   # add the prod IP as an exact gate literal
+```
+
+`--ref` chooses the source commit (default `HEAD`), `--out-dir` the output dir (default `dist`), `--version` overrides the stamped version. `stdout` carries a one-line JSON pin (`{ core_version, source_commit, tree_sha256, tarball }`) for CI/scripts to capture; human detail goes to `stderr`.
+
+If the no-leak gate fails, it names the file + the residual literal. Fix the redaction rules ([`scripts/gds/mirror-redact.js`](../../scripts/gds/mirror-redact.js)) or the manifest exclusions ([`scripts/gds/publish-manifest.js`](../../scripts/gds/publish-manifest.js)) and re-run — do not weaken the gate to force a pass.
+
+## Consume + pin it (the R85 side)
+
+A downstream instance repo:
+
+1. **Installs** the `.tgz` — `npm install ./cloudbongos-core-<version>.tgz` (or a fetched-in-CI path) — which lands it at `node_modules/@cloudbongos/core/`, a directory the consumer does not hand-edit.
+2. **Pins** by recording `core_version` + `source_commit` + `tree_sha256` from the manifest.
+3. **Verifies** the install by recomputing `tree_sha256` from the unpacked tree — for each core file, `sha256(content)` with a canonical mode (`0000644`/`0000755`), then `sha256` over the sorted `"<path> <mode> <filesha>"` lines. It must equal the manifest's `tree_sha256`. The tree hash is **compression-independent**, so it is a stable pin even though `.tgz` bytes are an implementation detail (`tarball.sha256` separately verifies the exact download).
+
+The manifest's `files[]` carries the same `{ path, mode, sha256 }` triples, so the pin also recomputes from the manifest alone — the two must agree.
+
+## Boundaries + relation to the mirror
+
+- **Not the public mirror.** The delayed, redacted, ~6-month-lagged public mirror ([ADR 0099](../adr/<redacted>.md)) is the open-source surface. This artifact is a *current*, versioned unit for **dependency consumption** by a sister repo — no lag. Both reuse the same `isPublishable()` selection and the same redaction; `package-core.js` imports the mirror's `redactBlob` so the redaction logic is defined once.
+- **Per-repo GDS.** Each instance runs its own database, tasks, builders, and credits ([ADR 0100 §3](../adr/<redacted>.md)). Packaging the core moves *code*, never work-tracking state.
+- **The single-checkout instance model is not deprecated** — it stays the cheaper shape for a quick reskin (cloudbongos.com). Two-repo is the destination for a project that wants to be its own thing and pull core updates on its own schedule.

package/docs/recipes/parallel-load-results-2026-05-30.md

@@ -0,0 +1,63 @@
+# Parallel-load test results — 2026-05-30 ([#272](https://example.com/builders#/task/272), criterion C6)
+
+> **Harness:** [`tests/parallel-load.mjs`](../../tests/parallel-load.mjs) · **Procedure:** [parallel-load-test.md](parallel-load-test.md) · **Target:** seeded staging ([#262](https://example.com/builders#/task/262)) at `http://127.0.0.1:3001` on the droplet (direct, bypassing Cloudflare).
+
+This is the live run that closes **[#272](https://example.com/builders#/task/272) (V3.R53)** and satisfies criterion **C6 — parallel-safe-under-load**. It exercises the deterministic claim concurrency audited in [#269](https://example.com/builders#/task/269) (the `claimTask()` transaction + `uniq_active_claim_per_task` index) against a real running server under concurrent load.
+
+## Bottom line
+
+**Across both runs: ZERO `server_error` (5xx), ZERO races, ZERO anomalies.** Every claim resolved to exactly one of: `success`, `collision` (409 — exactly-one-wins), or `throttled` (429 — the rate limiter cleanly shedding excess writes). Staging's journal logged **0** 5xx responses over the full window. C6's bar — "all claims succeed or 409 cleanly; no 500s, no race conditions" — holds, with the added finding that excess write load is shed cleanly via 429 rather than ever degrading into a race or crash.
+
+## Run A — sustained 30-minute run (the [#272](https://example.com/builders#/task/272) acceptance run)
+
+`--workers 5 --duration 1800 --work-ms 150` (natural claim→work→release→repeat loop across distinct tasks):
+
+| metric | value |
+|---|---|
+| wall | 1800.1s (30 min) |
+| claim attempts | 144 |
+| success | 13 |
+| collision (409, exactly-one-wins) | 8 (5.6%) |
+| throttled (429) | 123 |
+| **server_error (5xx)** | **0** |
+| client_error / other | 0 / 0 |
+| anomalies | 0 |
+| staging 5xx in journal | 0 |
+
+The system stayed healthy (`/healthz` 200) for the entire 30 minutes with no degradation. Low attempt volume is expected: the per-IP rate limiter (below) caps writes, and releases that get throttled leave tasks `active`, draining the claimable pool — so workers spend most cycles idling on an empty queue rather than hammering. That's the system behaving correctly under its own backpressure.
+
+## Run B — high-intensity contention burst (forces collisions)
+
+`--workers 5 --duration 25 --work-ms 100 --contend` (all 5 workers race the SAME task — the canonical "two builders, one task" case):
+
+| metric | value |
+|---|---|
+| claim attempts | 2733 |
+| success | 8 |
+| collision (409, exactly-one-wins) | 17 |
+| throttled (429) | 2708 |
+| **server_error (5xx)** | **0** |
+| anomalies | 0 |
+
+This is the direct demonstration of the C6 invariant: when multiple workers race the same task row, **exactly one wins and the rest get a clean 409 `ALREADY_CLAIMED`** — never two winners, never a 5xx.
+
+## The rate limiter (a relevant, healthy finding)
+
+`src/bongos/middleware/rate-limit.js` enforces **30 writes / 60s per-IP** and **60 writes / 60s per-session** on write methods (GET is unlimited). Because all simulated workers run from the droplet's single IP, the per-IP limit binds first, so most write attempts under load return **429** (`throttled` above). For a parallel-safety test this is a *positive* result: under sustained overload the server sheds excess writes cleanly (429 + `Retry-After`) instead of racing or 500ing. To exercise the claim logic at higher volume in a future run, raise the limit on staging or distribute workers across IPs.
+
+## Harness fixes made while running this (committed under [#272](https://example.com/builders#/task/272))
+
+Running against the real server surfaced live-path bugs the self-test (mock) couldn't:
+
+- **claimable parsing:** the harness read `.data.tasks`, but `GET /tasks/claimable` returns `{ claimable: [...] }` → it saw 0 tasks and never claimed. Fixed to read `.claimable` (with `.tasks` fallback).
+- **429 classification:** 429 was bucketed as `other` / flagged as a release anomaly. Added a dedicated `throttled` outcome (clean) so the report and the pass/fail conclusion are accurate.
+- **`--contend` mode:** added so workers can be pointed at one task to demonstrate exactly-one-wins collisions on demand (Run B).
+
+## Reproduce
+
+```bash
+# On the droplet (staging seeded per #262, service running, alphanumeric token minted):
+cd /home/lars/example-staging
+BASE_URL=http://127.0.0.1:3001 TOKEN=<redacted> node tests/parallel-load.mjs --workers 5 --duration 1800 --work-ms 150
+# add --contend to force same-task collisions
+```

package/docs/recipes/parallel-load-test.md

@@ -0,0 +1,32 @@
+# Parallel-load test — exercising claim concurrency under N builders
+
+> **Harness:** [`tests/parallel-load.mjs`](../../tests/parallel-load.mjs) (built under prep task [#472](https://example.com/builders#/task/472)).
+> **Live run:** criterion **C6** task **[#272](https://example.com/builders#/task/272)** (V3.R53) — a 30-min, 5-builder run against **staging**, gated on **[#262](https://example.com/builders#/task/262)** (staging data seeding).
+
+The harness spawns N concurrent simulated "builder" loops that hammer the GDS claim lifecycle (claim → simulated work → release → repeat) and classifies every claim outcome into `success` / `collision` (clean 409/refusal) / `server_error` (the anomaly we hunt) / `client_error` / `other`. It validates the deterministic claim concurrency C6 requires — the same `claimTask()` transaction + `uniq_active_claim_per_task` index audited in [multi-builder-merge.md](multi-builder-merge.md).
+
+## Self-test (CI-safe, no network)
+
+```bash
+node tests/parallel-load.mjs                      # defaults: 5 workers, 60s
+node tests/parallel-load.mjs --workers 5 --duration 3 --work-ms 50
+```
+
+With no `BASE_URL`, the harness drives an **in-process mock server** whose claim semantics mirror the DB (one active claim per task; touches-overlap → conflict). The board deliberately has two tasks sharing a path so collisions are *forced*, then asserts: ≥1 success, ≥1 collision, **zero** server_error, no invariant violation, and that attempts == sum of classes. Exit 0 = harness accounting proven correct. This runs in CI without a droplet.
+
+## Live run against staging ([#272](https://example.com/builders#/task/272) — once [#262](https://example.com/builders#/task/262) has seeded staging)
+
+```bash
+# Get builder tokens (one or more simulated builders share the load).
+# NEVER point this at production — it mutates real claim/task state. The
+# harness refuses a prod example.com host unless you pass --i-know.
+BASE_URL=https://staging.example.com \
+TOKENS=$tokA,$tokB,$tokC,$tokD,$tokE \
+  node tests/parallel-load.mjs --workers 5 --duration 1800
+```
+
+`#272` acceptance: run 30 min (`--duration 1800`); **every claim resolves to `success` or a clean `collision` — zero `server_error`, no races**; the report's throughput + collision-rate are the deliverable. A single anomaly (any 5xx, thrown error, or invariant violation) exits 1 and fails the run.
+
+## Why the split (prep [#472](https://example.com/builders#/task/472) vs live [#272](https://example.com/builders#/task/272))
+
+`#272`'s done-when explicitly requires a seeded staging environment, which `#262` (backlog) provides. Rather than block the runnable harness behind infra, [#472](https://example.com/builders#/task/472) built + self-validated the harness now; [#272](https://example.com/builders#/task/272) stays the live-staging run. When [#262](https://example.com/builders#/task/262) ships, point the harness at staging per above to close [#272](https://example.com/builders#/task/272) and satisfy C6.