@bongos/core 1.17.0

package/docs/recipes/managed-settings-remote-control.md

@@ -0,0 +1,416 @@
+# Recipe - managed-settings SSH + browser-terminal on-ramps
+
+> **Task [#599](https://example.com/builders#/task/599) · [ADR 0031](../adr/<redacted>.md) §1.** This is the operator guide for distributing a locked Claude Desktop SSH connection to builders and for serving the browser web-terminal on the box. Read [builder-box-lifecycle.md](builder-box-lifecycle.md) first — this recipe assumes a box is already provisioned and running.
+
+> **Chromebook/browser path (ADR 0038 → [ADR 0040](../adr/<redacted>.md), [#760](https://example.com/builders#/task/760) / [#824](https://example.com/builders#/task/824)):** the box serves its own web terminal - `ttyd` bound to localhost, exposed via an outbound-only Cloudflare Tunnel - and the builder runs a NORMAL interactive `claude` inside it from any browser. Because there is a real human at a real browser-backed terminal, `/login`, workspace-trust, and Trusted-Devices enrollment all succeed (the things that made headless `claude rc` unworkable). **ADR 0040 narrows that terminal to a one-time launchpad:** the builder logs in there once, runs the in-session **`/remote-control`** slash command, and then drives the box from the Claude web/mobile app — the terminal is no longer their daily surface. **The SSH path (Mac/Windows) below is unaffected and works.**
+
+---
+
+## The two on-ramps
+
+| Builder's device | Path | What this recipe wires |
+|---|---|---|
+| **Mac / Windows** | Claude Desktop → SSH → box | Pre-distributed `managed-settings.json` (locked `sshConfigs`) |
+| **Chromebook / any browser** | Browser -> `ttyd` web terminal over an outbound Cloudflare Tunnel -> box (reached at `GET /box/terminal`) -> log in once, then `/remote-control` -> drive the box from the Claude web/mobile app (ADR 0040) | `box-terminal.service` (ttyd, localhost-only) + `cloudflared.service` (outbound tunnel) + `box-report-terminal` cron |
+
+Both paths land the builder on the same box and the same `/workspace`. The choice is which GUI they prefer, not which machine they reach.
+
+---
+
+## Part 0 — the one-command flow ([#685](https://example.com/builders#/task/685), the default)
+
+> **This is the path to use now.** Parts 1, 3 and 4 below document the underlying
+> pieces (and the manual fallbacks); [#685](https://example.com/builders#/task/685) collapses the builder's side to a single
+> command and the operator's side to a single command. Nobody has to `scp` a JSON
+> file, paste a public key into `authorized_keys`, or `sudo cp` anything by hand.
+
+**The split:**
+
+| Who | Runs | What it does |
+|---|---|---|
+| **Builder** (own Mac/Windows) | `/builder-connect` — or, on a bare laptop, the hosted bootstrap (below) | GitHub sign-in → create+register their SSH key → install `managed-settings.json` |
+| **Operator** (control plane) | `node scripts/gds/box.js onboard <login> --apply` | provision the box (bakes the builder's registered key) + print the welcome packet |
+
+**Builder side** — guided in Claude Desktop (`/builder-connect`), or paste the hosted bootstrap on a bare laptop:
+
+```bash
+# macOS / Linux
+curl -fsSL https://example.com/api/gds/box/connect.sh | bash
+# Windows (PowerShell)
+irm https://example.com/api/gds/box/connect.ps1 | iex
+# Plain-text guide (no auth):  https://example.com/api/gds/box/connect
+```
+
+The bootstrap (`infra/box-connect.sh` / `.ps1`, served by `GET /box/connect.sh|.ps1`)
+is self-contained — only `curl` + `ssh-keygen` (or PowerShell). It device-flow signs
+the builder in, creates `~/.ssh/otb_builder` if absent, registers the **public** key
+(`POST /box/ssh-key`), and downloads + installs `managed-settings.json`
+(`GET /box/managed-settings`) at the OS path. The private key never leaves the machine;
+the token is held only in memory for the two authenticated calls (passed to `curl` via a
+0600 config file, never on the command line).
+
+> **Operator note (staging):** the served script's `GDS_API_BASE` is pinned to
+> `GDS_PUBLIC_ORIGIN` (default `https://example.com`) — the SAME env var the OAuth
+> `redirect_uri` uses — and is **never** derived from a request header, so a spoofed
+> `Host` can't redirect a victim's `curl … | bash`. On staging, set
+> `GDS_PUBLIC_ORIGIN=https://staging.example.com` in the server env (you already do,
+> for OAuth) and the connect scripts target staging automatically.
+
+**Operator side:**
+
+```bash
+node scripts/gds/box.js onboard <login>          # dry-run: shows the plan + key count
+node scripts/gds/box.js onboard <login> --apply  # provision + print the welcome packet
+```
+
+`onboard` refuses if the builder has **no registered SSH key** (the box would have no
+way in) — the canonical order is **builder connects first, then operator onboards**.
+Override with `--allow-no-keys` only if you will re-provision once they register a key.
+
+**How the key reaches the box without an operator SSH round-trip:** `box.js` reads the
+builder's registered keys from the DB and **bakes** them into the cloud-init managed
+block at provision (so the first login works before the box has any GDS token). From
+then on, `infra/<redacted>.sh` (a 10-min cron on the box, wired by the
+cloud-init) keeps `authorized_keys` in sync via `GET /box/authorized-keys` — which is
+**gated on the builder's LIVE rank+status** exactly like source access ([#600](https://example.com/builders#/task/600)). A
+demoted/deactivated builder gets a 403 and the cron empties the managed block — instant
+SSH-login revocation — **without touching the operator's own key** (only the lines
+between the `otb-managed` markers are rewritten). The droplet teardown follows via
+`box.js reconcile`.
+
+The rest of this recipe is the manual decomposition + fallbacks.
+
+---
+
+## Part 1 — Claude Desktop SSH (Mac / Windows)
+
+### How it works
+
+The builder installs Claude Desktop (free; no paid plan needed for the app shell). You pre-distribute a `managed-settings.json` that contains the SSH connection for their box. When they open Claude Desktop → Code → Environment, they see exactly one entry: **"Example Dev Box"** — no configuration required on their end.
+
+Claude Desktop SSHes to the box, auto-installs Claude Code on the host (on first connect), and starts a Code session directly in `/workspace`. The builder is ready to work.
+
+### Step 1 — Provision the box
+
+```bash
+node scripts/gds/box.js provision <login> --apply
+```
+
+This creates the DigitalOcean droplet, runs the cloud-init, and registers the box. The builder's hostname is `box-<login>.dev.example.com`.
+
+### Step 2 — Generate the managed-settings.json
+
+```bash
+bash infra/gen-managed-settings.sh <login> > /tmp/<login>-managed-settings.json
+cat /tmp/<login>-managed-settings.json
+```
+
+The output is a small JSON file, e.g.:
+
+```json
+{
+  "sshConfigs": [
+    {
+      "id": "otb-alice",
+      "name": "Example Dev Box",
+      "sshHost": "root@box-alice.dev.example.com",
+      "sshIdentityFile": "~/.ssh/otb_builder",
+      "startDirectory": "/workspace"
+    }
+  ],
+  "sshHostAllowlist": ["*.dev.example.com"]
+}
+```
+
+### Step 3 — Give it to the builder
+
+Send them the file and these instructions (copy-paste friendly):
+
+> **Mac:** Open Terminal and run:
+> ```bash
+> sudo mkdir -p "/Library/Application Support/ClaudeCode"
+> sudo cp ~/Downloads/<login>-managed-settings.json "/Library/Application Support/ClaudeCode/managed-settings.json"
+> ```
+> Then restart Claude Desktop. You'll see "Example Dev Box" in Code → Environment.
+>
+> **Windows:** Save the file to `C:\Program Files\ClaudeCode\managed-settings.json` (create the folder if it doesn't exist; needs Administrator). Restart Claude Desktop.
+
+> ⚠️ **The directory is `ClaudeCode`, not `Claude`.** Claude Code reads managed settings from `/Library/Application Support/ClaudeCode/` (macOS) and `%ProgramFiles%\ClaudeCode\` (Windows). A file under `…/Claude/` or `%ProgramData%\Claude\` is silently ignored and the box never appears in the Environment picker (the [#808](https://example.com/builders#/task/808) onboarding bug). The legacy `%ProgramData%\ClaudeCode\` Windows path was dropped in v2.1.75.
+
+### Step 4 — Builder's SSH key
+
+The `managed-settings.json` points to `~/.ssh/otb_builder` as the identity file. The builder needs this key pair generated on their machine, and you need to add the **public key** to the box.
+
+**Builder runs (once):**
+```bash
+ssh-keygen -t ed25519 -f ~/.ssh/otb_builder -C "otb-builder"
+cat ~/.ssh/otb_builder.pub
+```
+
+**Operator adds the public key to the box:**
+```bash
+ssh root@box-<login>.dev.example.com \
+  "echo '<public key here>' >> ~/.ssh/authorized_keys"
+```
+
+Or, if the builder's key is already registered as a DO SSH key, pass it via `BOX_SSH_KEY_IDS` at provision time so it's injected automatically.
+
+### What `sshHostAllowlist` does
+
+The `sshHostAllowlist: ["*.dev.example.com"]` entry prevents the builder from using Claude Desktop SSH to connect to arbitrary hosts. Their SSH picker only shows the pre-configured OTB box — they can't misconfigure or wander off it. This is a `managed-settings`-only field; user-level settings cannot override it.
+
+---
+
+## Part 2 - Browser on-ramp: terminal launchpad -> Remote Control (Chromebook / any browser) - ADR 0038 -> [ADR 0040](../adr/<redacted>.md), [#760](https://example.com/builders#/task/760) / [#824](https://example.com/builders#/task/824)
+
+### How it works
+
+The box serves its own web terminal. Two systemd units do the work:
+
+- **`box-terminal.service`** runs `ttyd` (a web-terminal daemon) bound to `127.0.0.1:7681`, basic-auth protected with a per-box generated password, dropping whoever connects into a shell in `/workspace`. It listens on localhost only - the box opens **no inbound port**.
+- **`cloudflared.service`** (via `infra/cloudflared-run.sh`) runs a Cloudflare Tunnel that makes that localhost terminal reachable from a browser. The tunnel is **outbound-only**: the box dials out to Cloudflare, so nothing has to be opened in the firewall.
+
+The builder opens the tunnel URL in any browser, authenticates at the basic-auth prompt, and then runs a **normal interactive `claude`** in the terminal. The first `claude` run prompts a normal browser `/login`; because there is a real human at a real browser-backed terminal, `/login`, workspace-trust, and Trusted-Devices enrollment all succeed - the exact steps that headless `claude rc` could never perform.
+
+**The terminal is a one-time launchpad, not the daily surface (ADR 0040).** Once the builder is signed in to that interactive `claude` session, they run the in-session **`/remote-control`** slash command. It prints a pairing link; opening that link drives the box from the Claude web/mobile app. From there the builder lives in the Claude app — the ttyd terminal is just where they logged in and kicked RC off.
+
+> ⚠️ **Use the in-session `/remote-control` slash command — NOT the standalone `claude rc` shell command.** The standalone command mis-checks workspace trust ([anthropics/claude-code#35817](https://github.com/anthropics/claude-code/issues/35817)): it reports `Workspace not trusted` even when the directory is trusted, and a plain shell can't accept the trust dialog. Starting RC from *inside* an already-running, already-trusted `claude` session is the working path (verified live 2026-06-07). Closing the terminal window ends the remote session and closes the box — a property of the session lifecycle, stated up front so builders aren't surprised.
+
+> **Workspace trust is pre-seeded ([#824](https://example.com/builders#/task/824)).** The cloud-init writes `/root/.claude.json` with `<redacted>` for `/root` + `/workspace`, so the first `claude` launch usually skips the trust dialog entirely. This is **non-load-bearing** (ADR 0040): the seed is best-effort and only written when the file is absent (upstream pre-seeding is flaky, [#9113](https://github.com/anthropics/claude-code/issues/9113)). If it doesn't take, the builder just accepts trust once — the flow works either way.
+
+The cloud-init wires both units automatically (and `infra/box-source-fetch.sh` re-installs them on its post-clone bring-up, so a box provisioned before this shipped picks them up on its next refresh). It copies `infra/box-terminal.service` and `infra/cloudflared.service` to `/etc/systemd/system/`, enables them with `systemctl enable --now`, and wires `infra/box-report-terminal.sh` as a 1-minute cron that publishes the terminal URL + credential **up** to the GDS (`POST /box/terminal`) so the builder's self-serve flow can read them back.
+
+**Tunnel modes:**
+
+- **Quick tunnel (default, zero operator secrets).** `cloudflared` opens a tokenless quick tunnel and gets a random `https://<id>.trycloudflare.com` hostname. Nothing to configure - works out of the box.
+- **Named tunnel (production upgrade, stable hostname).** When `/etc/otb/box.env` carries `BOX_TUNNEL_TOKEN` + `BOX_TERMINAL_HOSTNAME`, `cloudflared` runs a named tunnel instead and the terminal is reachable at a stable `term-<login>.example.com` host. As of [#771](https://example.com/builders#/task/771) the control plane **creates and bakes those automatically** on `box.js provision` (and tears them down on `deprovision`) — see the [named-tunnel automation](#<redacted>) section below.
+
+### Getting onto the terminal (builder self-serve)
+
+No SSH and no pairing script - two API calls:
+
+```bash
+# 1. Get a box (provisions or wakes it; see Part 0 / #701):
+POST /api/gds/box/ensure
+
+# 2. Poll until the box has reported its terminal up:
+GET /api/gds/box/terminal
+#    -> {"url": "https://<id>.trycloudflare.com", "credential": "<redacted>"}
+```
+
+Then:
+
+1. Open the returned `url` in a browser.
+2. At the browser's basic-auth prompt, enter username **`otb`** and the password (the `credential` value).
+3. In the terminal, type `claude` and press enter. On first run it walks the normal browser `/login` (workspace trust is pre-seeded, [#824](https://example.com/builders#/task/824), so the trust dialog usually doesn't appear; accept once if it does). You're now in an interactive Claude Code session in `/workspace`.
+4. Inside that session, run the **`/remote-control`** slash command. It prints a pairing link.
+5. Open that link to drive the box from the Claude web/mobile app. The terminal was the launchpad — work from the Claude app from here. (Closing the terminal window ends the remote session and closes the box.)
+
+### Service lifecycle
+
+| State | What happens |
+|---|---|
+| Box first boots | `box-terminal.service` (ttyd) + `cloudflared.service` start; `box-report-terminal` cron publishes the URL + credential to the GDS within ~1 min |
+| Builder connects (first time) | Opens the tunnel URL, basic-auths as `otb`, runs `claude` (one-time browser `/login`), then `/remote-control` to get a pairing link and drive the box from the Claude app (ADR 0040) |
+| Builder works (each session) | Lives in the Claude web/mobile app via the RC link; the ttyd terminal is only the launchpad they logged in through |
+| Session ends | Builder closes the browser tab; the terminal + tunnel stay up for the next visit |
+| Box is parked (idle-suspend) | Both units stop naturally when the droplet is destroyed; they restart clean on wake and the cron re-publishes a fresh URL |
+
+
+### Checking the services
+
+```bash
+# On the box:
+sudo systemctl status box-terminal      # the ttyd web terminal
+sudo systemctl status cloudflared        # the outbound tunnel
+journalctl -u box-terminal -f            # live tail
+journalctl -u cloudflared -f             # tunnel logs - quick-tunnel URL appears here
+journalctl -u cloudflared -n 100         # last 100 lines
+sudo systemctl restart box-terminal cloudflared
+```
+
+The quick-tunnel hostname is printed in the `cloudflared` journal; the same value is published to the GDS by the `box-report-terminal` cron (verify with `crontab -l | grep box-report-terminal`).
+
+### Named-tunnel automation (control plane, [#771](https://example.com/builders#/task/771))
+
+The named-tunnel upgrade is automated on the control plane (`scripts/gds/box.js` + `src/bongos/cf-tunnel.js`). It is **OPT-IN, default OFF** — leave it off and every box uses the zero-secret quick tunnel.
+
+**Turning it on (operator, in the control-plane `box.env` — the same file that holds `DO_API_TOKEN`, NEVER the web tier):**
+
+```bash
+BOX_NAMED_TUNNEL=1
+CLOUDFLARE_API_TOKEN=<token with Account:Argo Tunnel:Edit + Zone:DNS:Edit>
+CLOUDFLARE_ACCOUNT_ID=<the Cloudflare account id>
+# optional (defaults shown):
+CLOUDFLARE_ZONE=example.com      # the DNS zone the record lives in
+BOX_TERMINAL_ZONE=example.com    # hostname suffix → term-<login>.example.com
+```
+
+This reuses the **same `CLOUDFLARE_API_TOKEN`** as `box-dns-cloudflare.sh` (the [#725](https://example.com/builders#/task/725) `BOX_DNS_HOOK`) — just add the Tunnel scope to it. In the Cloudflare token UI the permission is **Account → Argo Tunnel → Edit** (the older label for the Cloudflare Tunnel / `cfd_tunnel` API; "Cloudflare Tunnel" on newer accounts). Keep the existing **Zone → DNS → Edit**.
+
+> ⚠️ **TLS coverage — why the host is at the apex, not `dev.*` ([#771](https://example.com/builders#/task/771)).** The terminal rides a **proxied** tunnel (Cloudflare terminates HTTPS at its edge), so the hostname needs an edge certificate. Free **Universal SSL covers `example.com` and `*.example.com` (one label) but NOT a two-label host** like `term-x.dev.example.com` — a `dev.*` host fails the TLS handshake at the edge (verified live). So `BOX_TERMINAL_ZONE` defaults to the **apex** (`term-<login>.example.com`, one label, covered for free). Only override it to `dev.example.com` if you've bought an **Advanced Certificate** for `*.dev.example.com`. (The SSH A-record path stays on `dev.*` — it's `proxied=false`/direct-IP and needs no edge cert.)
+
+**What the lifecycle does when it's on:**
+
+| Op | Tunnel action |
+|---|---|
+| `provision` | create-or-reuse a per-builder named tunnel (`otb-term-<login>`), point its ingress at `127.0.0.1:7681`, route a proxied CNAME `term-<login>.example.com → <tunnel-id>.cfargotunnel.com`, fetch the connector token, and **bake** `BOX_TUNNEL_TOKEN` + `BOX_TERMINAL_HOSTNAME` into the box's `/etc/otb/box.env` via cloud-init. The tunnel is created **before** the droplet so its token is present at first boot. |
+| `park` → `wake` | nothing — the tunnel + DNS are IP-independent and durable, and `box.env` rides the snapshot, so the stable hostname survives park/wake with no control-plane call. |
+| `deprovision` (incl. `sweep-dormant` / `reconcile` clawback) | best-effort teardown: delete the CNAME and the tunnel (the droplet is already gone, so connections have dropped). Never blocks the deprovision. |
+
+`box.js provision` (dry-run, the default) prints the tunnel plan without touching Cloudflare; pass `--apply` to create it for real. If `BOX_NAMED_TUNNEL=1` but the token/account id are missing, provision **fails loudly** rather than silently falling back.
+
+**Per-box manual override (no automation):** you can still set the two values by hand on a single box and restart — `cloudflared-run.sh` detects the token and switches modes:
+
+```bash
+# In /etc/otb/box.env on the box:
+BOX_TUNNEL_TOKEN=<redacted> named-tunnel token>
+BOX_TERMINAL_HOSTNAME=term-<login>.example.com
+sudo systemctl restart cloudflared
+```
+
+### Verifying the browser on-ramp end-to-end ([#771](https://example.com/builders#/task/771))
+
+The build is unit- and smoke-covered, but the loop crosses real hardware + a human at a browser. Run this once per change to `ttyd`/`cloudflared`/the terminal routes. **Operator deps:** a box provision (small DO cost) and, for the named-tunnel leg, the CF token with Tunnel scope.
+
+**Leg A — quick tunnel (zero secrets).**
+
+1. Provision a box for a test builder: `node scripts/gds/box.js onboard <login> --apply` (builder must have a registered SSH key, or `--allow-no-keys`). Confirm cloud-init finished: SSH in and `cloud-init status --wait` → `done`; `systemctl is-active box-terminal cloudflared` → `active`.
+2. Confirm the box published its terminal: `tail /tmp/box-report-terminal.log` should show `terminal access reported`. Then as the builder: `GET /api/gds/box/terminal` (or `/builder-box`) → returns `{ url: https://<id>.trycloudflare.com, credential }`.
+3. Open the `url` in a browser. Basic-auth as **`otb`** + the `credential`. You should land in a `/workspace` shell.
+4. In that terminal run `claude`. Complete the browser `/login` (trust is pre-seeded, [#824](https://example.com/builders#/task/824) — accept once if still prompted) and Trusted-Devices enrollment. Confirm an interactive session starts. **This is the leg `claude rc` could never do** ([#745](https://example.com/builders#/task/745)).
+5. Inside that session, run **`/remote-control`** (the slash command, NOT `claude rc`). Confirm it prints a pairing link; open the link and confirm the Claude web/mobile app drives the box (ADR 0040, verified live 2026-06-07).
+6. Full Chromebook loop: from a browser-only device, `POST /box/ensure` → poll `GET /box/terminal` → open → `claude` → `/remote-control` → drive from the Claude app. No SSH, no Desktop.
+
+**Leg B — named tunnel (stable hostname).**
+
+1. With the CF token (Tunnel + DNS scope) in the control-plane `box.env`, set `BOX_NAMED_TUNNEL=1` + `CLOUDFLARE_ACCOUNT_ID`. Dry-run first: `node scripts/gds/box.js provision <login>` — confirm it prints the `term-<login>.example.com` plan and the CF calls without touching anything.
+2. `node scripts/gds/box.js provision <login> --apply`. Confirm in the Cloudflare dashboard: a tunnel `otb-term-<login>` exists with one healthy connector, and a proxied CNAME `term-<login>.example.com → <id>.cfargotunnel.com`.
+3. `GET /box/terminal` returns the stable `https://term-<login>.example.com`. Open it, basic-auth, `claude` — same as Leg A but on the stable host.
+4. Teardown: `node scripts/gds/box.js deprovision <login> --apply`. Confirm the tunnel **and** the CNAME are gone from the Cloudflare dashboard.
+
+### Gotchas surfaced by hardware verification ([#771](https://example.com/builders#/task/771))
+
+- **Packaged ttyd collision (fixed).** The Ubuntu `ttyd` apt package ships its OWN auto-started `ttyd.service` that binds `127.0.0.1:7681` with **no `--credential`** and runs `login`. It both blocks our `box-terminal.service` (the unit crash-loops on `EADDRINUSE`) and — because the tunnel fronts whatever sits on 7681 — would expose an **unauthenticated** terminal. Cloud-init and the `box-source-fetch` post-clone bring-up now `systemctl mask ttyd` so only our basic-auth ttyd binds the port. On a box provisioned before this fix: `sudo systemctl disable --now ttyd && sudo systemctl mask ttyd && sudo systemctl restart box-terminal`.
+- **IPv4-only Cloudflare calls (fixed).** The CF API token is IP-locked to the droplet's IPv4 (least privilege). The droplet also has IPv6 egress; `cf-tunnel.js` pins IPv4 (undici `Agent{connect:{family:4}}`) or every call 401s — mirroring `box-dns-cloudflare.sh`'s `curl -4`.
+- **⚠ Browser-only bootstrap gap (OPEN — follow-up).** The `ttyd`/`cloudflared` systemd units live in the repo (`/workspace/infra/*.service`), so they only start **after** `box-source-fetch` clones — which needs the **builder's GDS token on the box**. A pure Chromebook builder has no SSH and no terminal yet, so there is **no first-boot path to place that token** → the terminal never comes up for them. Today it only works if the token is seeded another way (SSH, or an operator). The fix (a follow-up) is to bring ttyd up at **first boot independent of the clone** — e.g. cloud-init writes the unit inline rather than copying it from the repo. Until then, the browser on-ramp is verified to work *given a token on the box*, not from a truly bare browser-only device.
+
+---
+
+## Part 3 — The box-heartbeat cron
+
+The cloud-init also wires `infra/box-heartbeat.sh` as a 5-minute cron job. This pings `/api/gds/box/heartbeat` while the box is in active use (interactive SSH session or CPU load > 0.2). Silence = the idle-suspend sweep ([#598](https://example.com/builders#/task/598)) is free to park the box.
+
+Without this cron the idle sweep measures **uptime** instead of **activity** and may park a box someone is actively using. Always verify it's wired:
+
+```bash
+crontab -l | grep box-heartbeat
+```
+
+If missing (pre-[#599](https://example.com/builders#/task/599) box):
+```bash
+(crontab -l 2>/dev/null; echo "*/5 * * * * /workspace/infra/box-heartbeat.sh >> /tmp/box-heartbeat.log 2>&1") | crontab -
+```
+
+---
+
+## Part 4 — Rank-gated source access ([#600](https://example.com/builders#/task/600), ADR 0031 §6 / §9.2)
+
+The box holds **no baked credential**. Instead it fetches a rank-scoped clone spec
+from the GDS at boot and on a 10-minute cron, via `infra/box-source-fetch.sh`
+(carried onto the box as base64 inside the cloud-init `write_files`). The flow:
+
+1. The box reads the builder's GDS session token (it appears after they authenticate
+   Claude Code / run `/builder-reauth` — so the **first clone lands after the builder
+   connects**, not at boot).
+2. It calls `GET /api/gds/box/source-access`. The GDS checks the builder's **live**
+   rank + status (ADR 0016 — unforgeable from the box):
+   - **403 `SOURCE_ACCESS_REVOKED`** — builder demoted below the floor or deactivated.
+     The script cuts the credentialed remote (box-side clawback); `box.js reconcile`
+     deprovisions the droplet.
+   - **200 `configured:false`** — the server's source credential isn't set yet (see
+     env below); the box logs guidance and no-ops.
+   - **200 `configured:true`** — returns the scope-appropriate spec + a short-lived
+     credential. The box clones/refreshes `/workspace`, **never persisting the token
+     in `.git/config`** (clones via a one-shot authenticated URL, then resets the
+     remote to the token-less URL).
+3. **Scope by rank** (`src/bongos/box-access.js`): a **Xenos → `starter`** sparse-checkout
+   of the safe surface (`.devcontainer`, `art`, `docs`, `modules/status-ui`,
+   `generative-ideas.md` — the §9.2 "first tasks on a scoped surface"); a **Metic+ →
+   `full`** clone. Graduation (`box.js reconcile`) rescopes a starter box to full; the
+   box re-clones on its next refresh.
+
+**Operator: configure the source credential** (in the **prod env**, never the repo —
+ADR 0022). The default provider (`src/bongos/box-credential.js`) is env-backed:
+
+```bash
+# In the prod environment (e.g. the droplet's systemd unit / .env):
+BOX_SOURCE_REPO_URL=https://github.com/<org>/example.git
+BOX_SOURCE_FULL_TOKEN=<redacted> PAT or installation token, contents:read>
+# Optional — make 'starter' a HARD boundary (a physically separate repo that cannot
+# read the crown jewels) instead of the default sparse-checkout defense-in-depth:
+BOX_SOURCE_STARTER_REPO_URL=https://github.com/<org>/example-starter.git
+BOX_SOURCE_STARTER_TOKEN=<token scoped to the starter repo>
+# Optional — advisory token TTL (how often the box re-fetches + re-checks live rank):
+BOX_SOURCE_TOKEN_TTL_SECONDS=3600
+```
+
+> **Honest limit (ADR 0031 §6).** Git cannot sub-path-scope a single private repo and
+> sparse-checkout is client-side, so the default `starter` surface is
+> **defense-in-depth + audit + revocation, not insider-proof DRM**. Set
+> `BOX_SOURCE_STARTER_REPO_URL` for a hard boundary. Every grant/denial is audited to
+> `box_events` (`source_grant` / `source_deny` / `source_revoke` / `source_rescope`).
+
+**The clawback (clean offboarding).** Revoking rank or deactivating a builder cuts
+source on two timelines: the **credential layer is instant** (their next
+`/box/source-access` fetch 403s), and the **droplet teardown** follows when the
+control plane runs reconcile:
+
+```bash
+node scripts/gds/box.js reconcile            # dry-run: show what would change
+node scripts/gds/box.js reconcile --apply    # deprovision below-floor/inactive boxes,
+                                             # rescope graduated ones (run on a schedule)
+```
+
+Verify the source-fetch cron + check progress on the box:
+
+```bash
+crontab -l | grep box-source-fetch
+tail /tmp/box-source-fetch.log
+WORKSPACE=/workspace /usr/local/bin/box-source-fetch.sh   # manual run
+```
+
+---
+
+## Quick reference: full operator flow
+
+```bash
+# 0. One-time: set BOX_SOURCE_REPO_URL + BOX_SOURCE_FULL_TOKEN in the prod env (Part 4)
+
+# 1. Provision the box
+node scripts/gds/box.js provision alice --apply
+
+# 2. Add the builder's SSH public key to the box
+ssh root@box-alice.dev.example.com "echo '...' >> ~/.ssh/authorized_keys"
+
+# 3. Generate and send managed-settings.json (Desktop / SSH path)
+bash infra/gen-managed-settings.sh alice > /tmp/alice-managed-settings.json
+# → email/DM to builder with placement instructions
+
+# 4. For thin-device (Chromebook) path: the web terminal is ready automatically
+#    (box-terminal.service + cloudflared.service start at boot; box-report-terminal
+#    publishes the URL + credential to the GDS within ~1 min)
+#    Builder self-serves: POST /api/gds/box/ensure, then poll GET /api/gds/box/terminal
+#    -> open the url, basic-auth as user 'otb' with the returned credential, run: claude
+#    -> then /remote-control to drive the box from the Claude app (ADR 0040)
+
+# 5. When the builder is done for the day, the idle sweep parks the box:
+node scripts/gds/box.js sweep-idle --apply   # control plane / prod cron
+
+# 6. Source materializes automatically once the builder authenticates on the box
+#    (the box-source-fetch cron clones the rank-scoped surface). Nothing to do here.
+
+# 7. Offboarding / demotion: the credential is cut on the builder's next fetch;
+#    reconcile then deprovisions the box (run on a schedule alongside the sweeps):
+node scripts/gds/box.js reconcile --apply
+```

package/docs/recipes/memory-backup.md

@@ -0,0 +1,71 @@
+# Recipe — Builder memory backup + restore drill
+
+> **Filed by:** task [#281](https://example.com/builders#/task/281) (V3.R62), GDS-V3 criterion C8 ("memory is local-first, server-canonical on ship"). Depends on the server-side memory store ([#273](https://example.com/builders#/task/273) R54) and the DB backup pipeline ([#412](https://example.com/builders#/task/412) local nightly + [#415](https://example.com/builders#/task/415) weekly offsite). Companion to [`gds-db-backup.md`](gds-db-backup.md); vendor choice in [ADR 0025](../adr/<redacted>.md).
+
+## Key decision — memory is backed up *as part of the full-DB backup*, not separately
+
+C8 makes each builder's local memory dir **server-canonical on ship**: `/builder-ship` upserts the builder's memory files into the `builder_memory` Postgres table ([#274](https://example.com/builders#/task/274) R55). Because that table lives in the same `example` database as everything else, **it is already captured by the full-DB backup pipeline** — there is deliberately **no separate `backup-memory.sh`**. A memory-only dump would duplicate what `pg_dump` already writes nightly and ship offsite weekly, and would be a second thing to keep in sync. One backup story, not two (same principle as [ADR 0025](../adr/<redacted>.md)'s retirement of the redundant [#7](https://example.com/builders#/task/7) cron).
+
+So builder memory inherits both layers:
+
+| Layer | Mechanism | Cadence | Retention |
+|---|---|---|---|
+| Local | `gds-db-backup.timer` → `pg_dump` → `/var/backups/gds/` ([#412](https://example.com/builders#/task/412)) | daily 04:00 UTC | 30 days |
+| Offsite | `gds-db-offsite.timer` → latest dump → DO Space `gds-offsite/` ([#415](https://example.com/builders#/task/415)) | weekly Sun 05:00 UTC | 4 weeks |
+
+The `builder_memory` table is in every dump. No extra moving parts.
+
+## The `builder_memory` table
+
+```
+builder_memory(builder_id, path, content, mtime, sha256, byte_len, created_at, updated_at)
+  PK (builder_id, path)
+```
+
+Each row is one memory file for one builder. The `sha256` + `byte_len` columns are the integrity fingerprint — which is what lets the restore drill below verify a restore **without ever reading the private `content`** (builder memory is per-builder private + isolated; see [#276](https://example.com/builders#/task/276) R57).
+
+## Restore drill — verify a builder's memory survives a restore
+
+Cadence: **quarterly** (alongside the general restore drill in [`gds-db-backup.md`](gds-db-backup.md)). The drill restores the latest dump into a throwaway sandbox DB and confirms a chosen builder's memory rows are byte-identical to production — by fingerprint, so no private content is printed.
+
+```bash
+# 1. Restore the latest local dump into a sandbox DB.
+DUMP=$(ls -1t /var/backups/gds/example-*.sql.gz | head -1)
+sudo -u postgres dropdb --if-exists example_mem_restore_test
+sudo -u postgres createdb example_mem_restore_test
+set -o pipefail
+gunzip -c "$DUMP" | sudo -u postgres psql -q -d example_mem_restore_test
+
+# 2. Pick a builder with memory (builder_id=1 here) and compare a
+#    privacy-preserving fingerprint: count + md5 over (path, sha256, byte_len).
+#    Never selects `content`.
+FP="SELECT builder_id, count(*),
+       md5(string_agg(path||':'||sha256||':'||byte_len, ',' ORDER BY path))
+    FROM builder_memory WHERE builder_id=1 GROUP BY builder_id;"
+psql -d example                         -tAc "$FP"   # production
+sudo -u postgres psql -d example_mem_restore_test -tAc "$FP"   # restored
+
+# 3. The two lines must be identical. Then clean up.
+sudo -u postgres dropdb example_mem_restore_test
+```
+
+### Last run
+
+- **2026-05-30** (task [#281](https://example.com/builders#/task/281), claim `#349`): builder 1, **3 files**, prod and restored fingerprints **MATCH** (`…<redacted>`). Restore completed with zero errors.
+
+## Caveat the drill caught — point-in-time vs. the data
+
+The first drill run **failed correctly**: the most recent nightly dump (04:02 UTC) predated migration 054 and the memory rows (which synced at 08:47 UTC that day), so the table wasn't in it. Two lessons baked in here:
+
+1. **A dump only contains what existed when it ran.** Memory synced since the last 04:00 UTC nightly is not in any backup until the next nightly; and not *offsite* until the next weekly run. Worst-case memory-loss window on whole-droplet loss is ~1 week (the offsite cadence). Tighten the offsite timer to daily if that ever matters.
+2. **After a migration that adds/relocates memory data, take a one-shot dump** rather than waiting for the nightly, if you want it backed up immediately:
+   ```bash
+   sudo systemctl start gds-db-backup.service   # fresh local dump now
+   sudo systemctl start gds-db-offsite.service  # push it offsite now
+   ```
+
+## See also
+
+- [`gds-db-backup.md`](gds-db-backup.md) — the full backup pipeline, install steps, and the general (non-memory-specific) restore drills.
+- [ADR 0025](../adr/<redacted>.md) — offsite vendor decision.
+- [ADR 0024](../adr/<redacted>.md) — the local-first / server-canonical memory model this backs up.

package/docs/recipes/memory-sync-conflicts.md

@@ -0,0 +1,98 @@
+# Recipe — Memory sync conflicts: policy + clobber recovery
+
+> **Filed by:** task [#277](https://example.com/builders#/task/277) (V3.R58), GDS-V3 criterion C8. Documents the conflict policy chosen by the R04 spike ([ADR 0024](../adr/<redacted>.md) §3) and implemented across [#274](https://example.com/builders#/task/274) (push), [#275](https://example.com/builders#/task/275) (pull), [#276](https://example.com/builders#/task/276) (isolation). Recovery leans on the backup pipeline from [#281](https://example.com/builders#/task/281) ([`memory-backup.md`](memory-backup.md)).
+
+## The policy in one line
+
+**Last-write-wins, per file, server clock authoritative.** No merge, no rejection, no warning.
+
+## How sync works (so the policy makes sense)
+
+Memory is **local-first, server-canonical on ship**. There are exactly two sync moments — no background polling, no mid-session sync:
+
+| Trigger | Direction | What happens |
+|---|---|---|
+| `/builder-ship` succeeds ([#274](https://example.com/builders#/task/274) R55) | **push** local → server | `POST /api/gds/memory/sync` upserts each local memory file into `builder_memory`. Every file is written unconditionally — the **last push to land wins**. |
+| `SessionStart` / `/builder-start` ([#275](https://example.com/builders#/task/275) R56) | **pull** server → local | client reports its newest local `mtime`; `GET /api/gds/memory/since?mtime=…` returns files the server holds that are **newer**; client writes them over local copies. |
+
+The stored `mtime` is the client's reported file mtime and is **advisory** — used only by the pull side's newer-than check. The server's own `updated_at` (trigger-maintained) is the authority for "when did this row actually change." A client cannot win a conflict by forging a future mtime: an unparseable mtime is normalized to `NULL`, and the server clock governs.
+
+`builder_id` is the GitHub user, not a session — so "two different builders racing on the same file" is impossible by construction. Every conflict below is **one builder, more than one machine or session**.
+
+## Edge cases
+
+### 1. Same file edited in two parallel sessions (two machines / two worktrees)
+
+Builder edits `MEMORY.md` on the desktop and, in a parallel session, on the laptop. Both ship.
+
+- Both sessions push. **Whichever push lands at the server second overwrites the first** — the server now holds exactly one version (the later push). No merge is attempted; the earlier push's content is gone from the server.
+- The "losing" machine still has its own local copy until its **next** `SessionStart` pull, which then overwrites it with the server's surviving version. After that pull, both machines agree.
+- **Net:** one version survives end-to-end; the other is lost. Lost-write window ≈ one session boundary. If the lost version mattered, recover it from backup (below).
+
+> Practical avoidance: don't run two live sessions editing the same memory file. Memory edits are cheap to redo; the system optimizes for "never half-merge a plausible-but-broken file" over "never lose a keystroke."
+
+### 2. Server has a newer version than local
+
+Builder shipped from the laptop yesterday (server now newer), then opens the desktop today.
+
+- `SessionStart` pull sees the server's `mtime` is newer than the desktop's local copy and **writes the server version over the local file**. Desktop is now current. This is the intended path — it's how memory "follows you between machines."
+- Risk only arises if the desktop had **unpushed local edits older** (by mtime) than the server's version — those would be overwritten on pull. In normal use local edits are the newest thing, so this is rare; when it bites, recover from backup.
+
+### 3. Local newer than server (the common, non-conflict case)
+
+Local edits since the last ship are newer; on next `/builder-ship` they push and become canonical. No conflict. This is the steady state.
+
+## Operator recovery — restore a clobbered file from backup
+
+When a builder confirms a real clobber, the lost version is recoverable from any `pg_dump` taken **before** the overwrite (nightly local, 30-day retention; weekly offsite, 4-week — see [`memory-backup.md`](memory-backup.md) / [`gds-db-backup.md`](gds-db-backup.md)). The dump must predate the bad push.
+
+```bash
+# 1. Restore a pre-clobber dump into a sandbox (NON-destructive to prod).
+DUMP=/var/backups/gds/example-<ts-before-the-clobber>.sql.gz
+sudo -u postgres dropdb --if-exists example_recover
+sudo -u postgres createdb example_recover
+gunzip -c "$DUMP" | sudo -u postgres psql -q -d example_recover
+
+# 2. Extract the old content to a FILE (never paste content inline — see the
+#    warning below). -tA = unaligned, no headers; one file's content out.
+#    (Archon-run, on the droplet; this is the one place reading content is OK —
+#     it's a deliberate recovery, logged by this recipe's existence.)
+sudo -u postgres psql -d example_recover -tAc \
+  "SELECT content FROM builder_memory WHERE builder_id=<id> AND path='<path>';" \
+  > /tmp/recovered.txt
+
+# 3. Hand the file back to the builder. Two ways:
+#    (a) PREFERRED — give them /tmp/recovered.txt; they drop it into their local
+#        memory dir and /builder-ship to re-push it. Keeps the builder as the
+#        author of their own memory, and the push path recomputes sha256/byte_len.
+#    (b) Archon writes it straight into prod IF the builder can't. Load the
+#        content from the FILE via a bound parameter — do NOT inline it into the
+#        SQL (a naive dollar-quote breaks if the content itself contains the
+#        quote tag, and string-pasting risks injection during a high-stress run):
+sudo -u postgres psql -d example \
+  -v content="$(cat /tmp/recovered.txt)" \
+  -c "INSERT INTO builder_memory (builder_id, path, content, sha256, byte_len)
+      VALUES (<id>, '<path>', :'content', encode(sha256(:'content'::bytea),'hex'),
+              octet_length(:'content'))
+      ON CONFLICT (builder_id, path) DO UPDATE
+        SET content=EXCLUDED.content, sha256=EXCLUDED.sha256, byte_len=EXCLUDED.byte_len;"
+#    (`:'content'` is psql variable interpolation with proper literal quoting —
+#     safe for arbitrary content, and sha256/byte_len are recomputed in-SQL so
+#     they can't drift from the restored bytes.)
+
+# 4. Clean up the sandbox and the temp file.
+sudo -u postgres dropdb example_recover && rm -f /tmp/recovered.txt
+```
+
+> **Privacy note:** another builder's memory is private ([#276](https://example.com/builders#/task/276) R57 isolation; Archon forensic reads are audit-logged). Recovery is the builder's own file, or an explicitly-authorized Archon action — never a casual cross-builder read. Prefer path (a) so the builder re-pushes their own content.
+
+## Why not something smarter? (rejected alternatives, from ADR 0024 §3)
+
+- **Three-way markdown merge** — rejected: a half-merged `MEMORY.md` that *looks* plausible but is structurally broken is worse than a clean lose-one. Revisit in V4 only if real collisions show up in V3.
+- **Optimistic concurrency** (reject stale-base pushes, force pull-first) — rejected: builders would just `--force` around it. Last-write-wins + a backup recovery path is simpler and handles the rare bad case.
+
+## See also
+
+- [ADR 0024](../adr/<redacted>.md) §3 — the decision + rejected alternatives.
+- [`memory-backup.md`](memory-backup.md) — backup cadence + the restore drill the recovery above relies on.
+- [`gds-db-backup.md`](gds-db-backup.md) — the underlying dump/offsite pipeline.