@stixxert/pi-docker-sandbox 1.0.1 → 1.1.1

package/README.md

@@ -32,6 +32,42 @@ daemon are a liability. With this extension:
   redirect it.
 - `docker_verify` runs a live isolation audit (PASS/FAIL per check).
 
+## Two modes
+
+**1. Deploy target (default).** `pi install` this package and the agent gets
+the `docker_*` tools: a private sbx microVM, with its own daemon, to deploy
+into. pi's own tools keep running wherever pi runs.
+
+**2. Execution backend.** Load `sandbox/` instead and pi runs *against* the
+sandbox: `bash`, `read`, `write`, `edit`, `grep`, `find`, `ls` and your `!`
+commands all execute inside the sbx microVM, while pi itself stays on the
+host with its own auth, config, sessions and model keys.
+
+```bash
+cd /path/to/project
+pi -e /path/to/pi-docker-sandbox/sandbox
+```
+
+This is pi's "route tool execution into an isolated environment" pattern —
+the same shape as pi's [Gondolin
+example](https://github.com/earendil-works/pi-mono), with an sbx microVM as
+the target rather than a local QEMU VM. See [sandbox/README.md](sandbox/README.md).
+
+It is also the **lighter alternative to running pi inside a sandbox**: no
+template with pi baked in, no bootstrap seeding state into the sandbox, no
+per-project auth/sessions/credential handling, no sandbox rebuild to pick up
+a new pi version. The tools are *overridden*, not added — so the model's
+tool list is byte-for-byte the built-in one and not one prompt token is
+spent on the sandbox.
+
+The sandbox it creates can itself be slim: [template/](template/) builds a
+**648 MB** baseline automatically (vs ~2.2 GB for the pi-bearing template),
+because pi no longer has to live inside it. See
+[template/README.md](template/README.md).
+
+Both modes can be active at once; the `docker_*` tools keep working when the
+backend is loaded.
+
 ## Requirements
 
 - [pi](https://pi.dev) (the extension runs in the host pi process)
@@ -148,6 +184,11 @@ persistent name (e.g. a shared sandbox reused across restarts), pin
 - `DOCKER_SANDBOX_TEMPLATE=<name>` — use a pre-baked `sbx template` for
   auto-created sandboxes (avoids re-pulling common images every session;
   create with `sbx template save <name>` from a prepared sandbox).
+- `DOCKER_SANDBOX_DEBUG=1` — emit the extension's lifecycle/GC diagnostics
+  (watchdog armed, teardown, startup sweep) to stderr. **Off by default**: the
+  extension runs inside the pi process, so stray console output would land on
+  the same terminal the TUI is drawing and corrupt the chat. Turn it on when
+  running `pi -p`, in a plain shell, or when diagnosing lifecycle issues.
 
 ## Ports (verified rules)
 

package/boundary.md

@@ -14,6 +14,16 @@ for the **agent** and for the **human operator**.
 > Gondolin is used throughout this document simply as the concrete reference
 > deployment.
 
+> **Two modes.** This document describes the default **deploy target** mode,
+> where the sandbox exists for `docker_*` and the agent keeps using its own
+> tools. The repo also ships an **execution backend** (`sandbox/`) that routes
+> pi's built-in tools into the same sandbox, for which the agent's own VM drops
+> out of the picture and workspace paths are used as-is (the workspace is
+> direct-mounted at its host absolute path inside the sandbox, so no
+> `/workspace` hop is involved). See
+> [sandbox/README.md](sandbox/README.md) and
+> [security.md](security.md#execution-backend-built-in-tools-routed-into-the-sandbox).
+
 ## Topology
 
 ```

package/index.ts

@@ -124,9 +124,9 @@ async function teardownSandbox(reason: string): Promise<void> {
 		].join("\n");
 		const child = spawn("/bin/sh", ["-c", script], { detached: true, stdio: "ignore", env: scrubbedEnv() });
 		child.unref();
-		console.error(`[docker-sandbox] session ${reason}: ${action} sandbox "${name}" (detached, retrying)`);
+		note(`session ${reason}: ${action} sandbox "${name}" (detached, retrying)`);
 	} catch (e) {
-		console.error(`[docker-sandbox] session ${reason}: teardown of "${name}" failed: ${(e as Error).message}`);
+		note(`session ${reason}: teardown of "${name}" failed: ${(e as Error).message}`);
 	}
 }
 
@@ -141,6 +141,22 @@ function findSbxCli(): string {
 	return "sbx";
 }
 
+/**
+ * Diagnostics are OFF by default on purpose: the extension runs inside the pi
+ * process, so a raw console write lands on the terminal the TUI is drawing and
+ * corrupts the chat transcript. Set DOCKER_SANDBOX_DEBUG=1 (or SBX_PI_DEBUG=1 /
+ * SBX_DEBUG=1) to get the lifecycle/GC lines on stderr — useful in `pi -p`, a
+ * plain shell or when diagnosing, harmless in the TUI because it is opt-in.
+ */
+function debugEnabled(): boolean {
+	return /^(1|true|yes|on)$/i.test((env.DOCKER_SANDBOX_DEBUG ?? env.SBX_PI_DEBUG ?? env.SBX_DEBUG ?? "").trim());
+}
+
+/** Diagnostic line — silent unless debug logging is enabled (see debugEnabled). */
+function note(message: string): void {
+	if (debugEnabled()) console.error(`[docker-sandbox] ${message}`);
+}
+
 /**
  * Non-secret vars always forwarded to children (the sbx CLI and shells need
  * them to function; they are not credentials).
@@ -1486,9 +1502,9 @@ function spawnWatchdog(): void {
 	try {
 		const child = spawn("/bin/sh", ["-c", script], { detached: true, stdio: "ignore", env: scrubbedEnv() });
 		child.unref();
-		console.error(`[docker-sandbox] watchdog armed for sandbox "${name}" (pid ${pid}, teardown=${mode}, keepalive=${keep})`);
+		note(`watchdog armed for sandbox "${name}" (pid ${pid}, teardown=${mode}, keepalive=${keep})`);
 	} catch (e) {
-		console.error(`[docker-sandbox] failed to arm watchdog: ${(e as Error).message}`);
+		note(`failed to arm watchdog: ${(e as Error).message}`);
 	}
 }
 
@@ -1644,6 +1660,36 @@ function textResult(text: string): AgentToolResult<undefined> {
 /* extension registration                                              */
 /* ------------------------------------------------------------------ */
 
+let lifecycleArmed = false;
+
+/**
+ * Arm everything that must outlive a session: the detached watchdog (teardown
+ * on pi exit + keepalive pokes while pi lives), the owner marker that stops GC
+ * reclaiming a live sandbox, and the startup sweep of stale sandboxes.
+ *
+ * Idempotent: the docker_* extension and the sandbox execution backend both
+ * call this from their own session_start, and loading both must not double-arm.
+ *
+ * Called separately from ensureSandbox() on purpose — ensureSandbox returns
+ * early when the sandbox ALREADY exists (a pinned DOCKER_SANDBOX, a resumed
+ * session), so arming only there would silently skip keepalive/teardown for
+ * every sandbox that was not created by this process.
+ */
+async function armSessionLifecycle(): Promise<void> {
+	if (lifecycleArmed) return;
+	lifecycleArmed = true;
+	spawnWatchdog();
+	writeOwnerMarker(sessionSandboxName());
+	const raw = Number(env.DOCKER_SANDBOX_GC_HOURS ?? "24");
+	if (!Number.isFinite(raw) || raw <= 0) return;
+	// Deliberately NOT awaited: the sweep is a background safety net for stale
+	// sandboxes from crashed sessions, and session start must not wait on
+	// `sbx ls` (and any sandboxd round trip) to get there.
+	void gcSweep(raw)
+		.then((summary) => note(summary))
+		.catch((e) => note(`gc at startup failed: ${(e as Error).message}`));
+}
+
 export default function (pi: ExtensionAPI) {
 	// Lifecycle: tear down this session's sandbox when the session ends
 	// (exit / Ctrl+C / Ctrl+D / SIGHUP / SIGTERM, /new, /resume, /fork).
@@ -1652,18 +1698,11 @@ export default function (pi: ExtensionAPI) {
 	});
 
 	// Crash safety net: sweep stale pi-sbx-* sandboxes at session start, and
-	// arm the watchdog + owner marker for the current sandbox name (covers
-	// /resume case).
+	// arm the watchdog + owner marker for the current sandbox name (covers the
+	// /resume case). Shared with the sandbox execution backend, which needs the
+	// same lifecycle when it is loaded on its own.
 	pi.on("session_start", async () => {
-		spawnWatchdog();
-		writeOwnerMarker(sessionSandboxName());
-		const raw = Number(env.DOCKER_SANDBOX_GC_HOURS ?? "24");
-		if (!Number.isFinite(raw) || raw <= 0) return;
-		try {
-			console.error(`[docker-sandbox] ${await gcSweep(raw)}`);
-		} catch (e) {
-			console.error(`[docker-sandbox] gc at startup failed: ${(e as Error).message}`);
-		}
+		await armSessionLifecycle();
 	});
 
 	pi.registerTool({
@@ -1932,4 +1971,23 @@ export default function (pi: ExtensionAPI) {
 }
 
 // Named exports for tests (pi's loader only calls the default factory).
-export { scrubbedEnv, envForwardMode, envAllowlist, envPassthrough, sessionSandboxName, mapHostPath, assertSafeArg };
+// The sbx kernel is also shared with the `sandbox/` execution-backend
+// extension (host pi + tools routed into the sandbox), so that the sandbox
+// lifecycle, env scrubbing and path confinement exist in exactly one place.
+export {
+	scrubbedEnv,
+	envForwardMode,
+	envAllowlist,
+	envPassthrough,
+	sessionSandboxName,
+	mapHostPath,
+	assertSafeArg,
+	hostRoot,
+	findSbxCli,
+	runSbxCli,
+	sandboxExists,
+	ensureSandbox,
+	teardownSandbox,
+	armSessionLifecycle,
+	debugEnabled,
+};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@stixxert/pi-docker-sandbox",
-  "version": "1.0.1",
+  "version": "1.1.1",
   "description": "pi extension: a private docker sandbox (sbx microVM with its own daemon) as the agent's deploy target — the host's docker is never exposed.",
   "license": "Apache-2.0",
   "publishConfig": {
@@ -12,6 +12,9 @@
   },
   "files": [
     "index.ts",
+    "sandbox/",
+    "template/",
+    "test-loader.mjs",
     "README.md",
     "boundary.md",
     "security.md",
@@ -24,6 +27,9 @@
   "scripts": {
     "typecheck": "tsc --noEmit",
     "test": "node smoke-test.mjs",
+    "e2e": "node sandbox/e2e.mjs",
+    "template": "bash template/build.sh",
+    "template:check": "bash template/build.sh --check",
     "prepublishOnly": "npm run typecheck && npm test",
     "release": "semantic-release"
   },

package/sandbox/README.md

@@ -0,0 +1,277 @@
+# sbx execution backend
+
+Run **pi on the host** with its built-in tools executed **inside a Docker
+Sandbox** (`sbx`) — the same shape as pi's
+[Gondolin example](https://github.com/earendil-works/pi-mono), except the
+sandbox is an sbx microVM instead of a local QEMU VM.
+
+```bash
+cd /path/to/project
+pi -e /path/to/pi-docker-sandbox/sandbox
+```
+
+That is the whole setup. There is no template to build, no pi to install
+inside the sandbox, and no state to seed.
+
+## What it routes
+
+`bash`, `read`, `write`, `edit`, `grep`, `find`, `ls`, plus the user's `!`
+commands. All of them are **overrides of the built-in tools** — same names,
+same schemas, same descriptions, same prompt snippets and guidelines. Only
+the execution changes.
+
+That is deliberate, and it is the whole answer to "does this cost context?":
+**overriding adds zero new tool schema.** Registering an `sbx_exec` tool
+instead would add tokens to the system prompt on *every turn, forever*.
+
+The system prompt's working-directory line is rewritten to say where commands
+actually run, so the model is never guessing.
+
+## Why not run pi inside the sandbox?
+
+Running pi inside an sbx sandbox is a bigger machine:
+
+| | pi inside the sandbox | this extension |
+|---|---|---|
+| Template with `pi` npm-installed | required | **not needed** |
+| Template with browser/pnpm baked | required for screenshots | optional, pi-free image is enough |
+| `bootstrap-pi.sh` seeding into sandbox `~/.pi` | required | **not needed** — host `~/.pi` is the real one |
+| Per-sandbox auth / sessions / model cache | required, per project | **none** — host config, host keys |
+| `sbx secret` to get a model key in | required | **not needed** |
+| `sbx create` + attach + type `pi` | required | one command from the project dir |
+| Pick up a new pi version | rebuild the sandbox | just restart pi |
+
+## Design notes
+
+**Paths are identical on both sides.** `ensureSandbox()` mounts the workspace
+at its host absolute path, so `/tmp/x/f.ts` is the same file inside the
+sandbox. No `/workspace` translation is involved (unlike the `docker_*` tools,
+which map through `mapHostPath`).
+
+**Bytes move as base64 in argv.** `sbx exec` stdin forwarding is not
+guaranteed, so file contents travel as base64 arguments (`base64 -d >
+"$2"`), chunked at 384 KB per call. The `read` path base64-decodes the file so
+images and any non-UTF-8 content survive intact.
+
+**Host secrets do not enter the sandbox.** pi's built-in `bash` tool builds
+the command environment from the *full* host environment, so exporting it
+verbatim would copy your API keys and tokens into the sandbox, where anything
+running there could read them. The backend forwards **only `PI_*` session
+metadata** (plus any names you opt into with `DOCKER_SANDBOX_ENV_ALLOWLIST`),
+and never `DOCKER_*`/`COMPOSE_*`. Covered by a test.
+
+**grep is reimplemented, not re-pointed.** pi's `grep` tool spawns host
+`ripgrep` for match discovery regardless of custom operations — so simply
+giving it sandbox operations would scan the *host* filesystem and require
+`rg` on the host. The tool is replaced wholesale by a walk-and-match over
+the transport, so matches come from sandbox content.
+
+**`.gitignore` is honoured by using git itself.** `grep`/`find` enumerate
+with `git ls-files --cached --others --exclude-standard`, which is exactly
+"tracked plus untracked-but-not-ignored" — so build output (`dist/`,
+`.next/`, coverage) and vendored trees stay out of results, matching what
+the built-in tool descriptions promise. It runs with `safe.directory=*`,
+because the workspace is a mount whose owner need not match the sandbox user
+(git otherwise refuses with "detected dubious ownership") — a read-only
+index query, so no repo-provided code is executed. When git is unavailable
+or the path is not a repo, a pruned walk is used and only `.git` and
+`node_modules` are skipped.
+
+**No shell injection surface.** Paths and file bodies are passed as
+*positional* arguments to `sh -c` (`"$1"`, `"$2"`), never spliced into the
+script text, so a path can never be read as shell syntax or as an option.
+`shQuote()` handles the environment exports.
+
+**Listings are one round-trip.** pi's `ls` tool calls `stat()` for every
+entry; over `sbx exec` that would be N+3 sandbox round-trips per listing. A
+single POSIX-sh pass returns `d`/`f` + name and is memoised for the duration
+of that one tool call (verified: 25 entries => ≤ 4 round-trips).
+
+**It degrades instead of breaking.** If `sbx` is missing or the sandbox
+cannot be provisioned, the tools fall back to local execution, the user is
+notified, and the system prompt says so explicitly — the agent is never led
+to believe it is sandboxed when it is not.
+
+## Trying it out (before publishing)
+
+One command, nothing installed, no settings touched:
+
+```bash
+bash sandbox/try.sh                 # auto: real sbx if usable, else a local container
+bash sandbox/try.sh --sbx           # force a real Docker Sandbox
+bash sandbox/try.sh --docker        # force a local container
+bash sandbox/try.sh -- --model x/y  # extra args are passed to pi
+```
+
+It loads the extension with `pi -e` — exactly how `pix` loads gondolin — so the
+blast radius is one pi process. Anything it creates (a container, or an sbx
+sandbox) is removed on exit.
+
+**The `--docker` path is not a toy.** It runs the *identical* ops layer with
+only the `exec <target> --` verb changed, which is what makes the backend
+testable on a machine that cannot run `sbx` at all (no KVM / nested virt).
+
+### Verifying it is actually routing
+
+Both tools are overridden, so check with something whose answer differs inside
+and outside:
+
+```bash
+# bash: the container image, not your host OS
+bash sandbox/try.sh --docker -- -p --tools bash "Run: cat /etc/os-release | head -1"
+```
+
+```bash
+# file tools: a path that exists ONLY inside the sandbox
+docker exec <container> sh -c 'echo hi > /opt/only-in-sandbox.txt'
+bash sandbox/try.sh --docker -- -p --tools read "Read /opt/only-in-sandbox.txt"
+```
+
+If the first reports the container's OS and the second returns the file, the
+routing works. If the extension failed to load you get a missing-tool error
+instead — never a silent fallback to the host (that case is reported in the
+system prompt and via `/sbx`).
+
+### On the host, with real sbx
+
+```bash
+brew install docker/tap/sbx && sbx login     # once
+cd /path/to/project
+bash /path/to/pi-docker-sandbox/sandbox/try.sh
+# or directly:
+pi -e /path/to/pi-docker-sandbox/sandbox
+```
+
+The extension auto-provisions the sandbox. `bash template/build.sh` first if you
+want the lightweight template; `try.sh` tells you when it is missing.
+
+### Notes on loading paths
+
+- `pi -e <file>` always works: `-e sandbox/index.ts`.
+- `pi -e <dir>` needs a `pi` manifest, so `sandbox/package.json` exists for that
+  (same shape as the gondolin extension).
+- **Do not symlink only `sandbox/` into `~/.pi/agent/extensions/`**: the
+  extension imports the shared sbx kernel from `../index.ts`, so it has to stay
+  inside the repository. Point `-e` at the repo instead (or install the repo as
+  a package and load the repo-relative subdirectory).
+
+## Configuration
+
+Reuses the `docker_*` extension's sandbox settings — the sandbox is the same
+kind of object, so `DOCKER_SANDBOX` (pin a name), `DOCKER_SANDBOX_CPUS`,
+`DOCKER_SANDBOX_MEMORY`, `DOCKER_SANDBOX_TEMPLATE`,
+`DOCKER_SANDBOX_WORKSPACE_RO`, `DOCKER_SANDBOX_KEEPALIVE` and
+`DOCKER_SANDBOX_TEARDOWN` all apply. `DOCKER_SANDBOX_WORKSPACE_RO=1` is the
+interesting one here: the sandbox then sees the project read-only while your
+edits go through pi's own tools.
+
+| Variable | Effect |
+|---|---|
+| `SBX_BACKEND=docker` + `SBX_DOCKER_CONTAINER=<id>` | route into a container instead (testing / non-sbx hosts) |
+| `DOCKER_SANDBOX_KEEPALIVE` | **default `1` here** — keeps the VM running for the life of the pi process; set `0` to allow idle-stop |
+| `DOCKER_SANDBOX_ENV_ALLOWLIST` | additionally export these host vars into the sandbox shell (default: `PI_*` only) |
+| `DOCKER_SANDBOX` | pin the sandbox name (also disables per-project derivation) |
+| `SBX_EPHEMERAL` | `1` = throwaway per-session sandbox, removed at exit |
+| `SBX_PI_DEBUG` | `1` = log per-phase startup timings to stderr |
+| `DOCKER_SANDBOX_TEARDOWN` | `remove` / `stop` / `none` (a per-project sandbox defaults to `none`) |
+
+Note: `DOCKER_SANDBOX_WORKSPACE_RO=1` is **not** compatible with this backend —
+the sandbox would mount the project read-only, so `write`/`edit` would fail.
+The extension warns at session start if it is set.
+
+Two variables are **exported** for sibling extensions:
+`PI_SBX_SANDBOX` (sandbox name) and `PI_SBX_BACKEND` (`sbx` / `docker`).
+An extension that shells out to a tool which only exists inside the sandbox
+(the webdev/`webshot` toolchain, for example) can use these to detect
+"host pi, but there is a sandbox" and route accordingly.
+
+## Lifecycle
+
+The sandbox is **one per project, reused across runs**, and stays warm for as
+long as pi is running:
+
+- **Per-project name.** The sandbox is `pi-sbx-<project>-<hash>`, derived from
+the nearest VCS root (the rule `sbxpi` uses), so running pi from a
+subdirectory lands in the same sandbox. A pinned name also makes teardown
+`none`, which is the point — see below.
+- **Keepalive is ON by default.** sandboxd stops an idle sandbox ~2–4 min after
+the last `sbx` call, which would make the first tool call after a pause pay a
+multi-second VM boot. A detached watchdog pokes the VM every ~60 s while pi
+runs, so it never goes cold mid-session. `DOCKER_SANDBOX_KEEPALIVE=0` opts out.
+- **It is not destroyed when pi exits.** The VM is idle-stopped by sandboxd (so
+  it costs no memory), but the sandbox and everything in it — pulled docker
+  images, installed packages — survives. The next run reuses it.
+- **...and pi being killed hard is still handled.** The watchdog is detached and
+  outlives pi, so keepalive/teardown bookkeeping never depends on a clean exit.
+- **`SBX_EPHEMERAL=1`** restores the old behaviour: a throwaway
+  `pi-sbx-<pid>-<rand>` sandbox removed at exit. Use it for one-off experiments.
+- A `docker` backend target is never managed — it is caller-supplied, and no sbx
+  lifecycle is armed for it.
+
+### Startup latency (why the name matters)
+
+A per-process name combined with teardown `remove` means **a fresh `sbx create`
+on every pi run** — 10–15 s, every time, because a brand-new VM has to be
+provisioned and its image layers prepared. That is the single biggest cost in
+this backend and it is entirely avoidable: with the per-project name the create
+happens once per project, and every later run just attaches (sub-second, plus a
+VM start if it has gone cold).
+
+To see where the time actually goes:
+
+```bash
+SBX_PI_DEBUG=1 pi -e /path/to/pi-docker-sandbox/sandbox
+
+[sbx] template: 210ms
+[sbx] ensure sandbox (create if missing): 13800ms     ← first run: it is creating
+[sbx] backend=sbx sandbox=pi-sbx-myapp-1a2b3c4d template=stock base keepalive=1 total=14100ms
+
+# second run, same project:
+[sbx] template: 190ms
+[sbx] ensure sandbox (create if missing): 640ms
+[sbx] backend=sbx sandbox=pi-sbx-myapp-1a2b3c4d template=stock base keepalive=1 total=830ms
+```
+
+Resolving the sandbox also happens **in the background**: `session_start` does
+not await it, so pi's prompt is usable immediately and the cost overlaps with
+you reading it instead of gating it. The first tool call awaits the same
+memoised promise.
+
+`/sbx` prints the active backend, target and whether the sandbox is
+project-scoped.
+
+## Known limitations
+
+- **Timeouts, not cancellation, for file tools.** pi's `*Operations`
+  interfaces (other than `BashOperations`) do not receive an `AbortSignal`,
+  so a wedged `sbx exec` is bounded by a 120 s timeout rather than cancelled
+  by Esc. `bash` gets a real signal and is killed on abort.
+- **Processes started inside the sandbox survive an abort.** Killing the
+  `sbx`/`docker` CLI's process group does not reach a process already running
+  inside the VM; a runaway dev server there lives until the sandbox is torn
+  down (or `SBX_BACKEND=docker`'s container is stopped).
+- **Names containing newlines** are not representable in the directory
+  listing format (the same trade-off the tools' text output already makes).
+- **`.gitignore` needs git.** Without git in the sandbox, only `.git` and
+  `node_modules` are skipped. sbx images ship git, so this is the exception.
+
+## Tests
+
+```bash
+npm run typecheck
+npm test          # docker_* extension: registration, guards, confinement
+npm run e2e       # this backend, end-to-end against a real container
+```
+
+`sbx` cannot run in CI or in a nested sandbox (it boots microVMs through a
+host hypervisor), which is exactly why the transport is pluggable: `npm run
+e2e` drives the identical ops layer against a real container via
+`docker exec`, and the only difference from the product path is which binary
+performs `exec <target> --`.
+
+## Relationship to the `docker_*` tools
+
+They stay, unchanged. But note the overlap: once `bash` runs *inside* the
+sandbox, `docker build` / `docker compose up` in a plain shell already hit the
+sandbox's own daemon, so the `docker_*` deploy surface becomes optional sugar
+rather than the only way to deploy.