self-bench 0.3.6 → 0.3.8

package/docs/operations.md

@@ -4,43 +4,73 @@ This document covers SelfBench configuration, persistence, authentication, the H
 
 ## Local stack
 
-`docker compose up` starts:
+`self-bench up` (or `bun run cli up --backend docker` from a checkout) starts:
 
 - Postgres for Temporal state;
+- Postgres for the site (`site-postgres`: users, connected repos, evaluation records);
 - Temporal;
-- the SelfBench API;
+- the SelfBench API, serving the `dist/review` bundle;
 - the SelfBench worker;
 - a persistent artifact volume.
 
-The worker mounts the host Docker socket for local sandboxes. The API never receives the Docker socket or model credentials. The local Postgres user and password are development defaults, both named `temporal`.
+The worker mounts the host Docker socket for local sandboxes. The API never receives the Docker socket or model credentials; the worker never receives the GitHub OAuth secret or session secret. The local Postgres users and passwords are development defaults (`temporal`/`temporal` and `selfbench`/`selfbench`).
 
-Temporal and artifact state live in the `selfbench_temporal-postgres` and `selfbench_artifacts` volumes. Back up those volumes before an upgrade when workflow history or generated artifacts must be retained. SelfBench has no application database migrations.
+### One stack per checkout
+
+Several checkouts (for example git worktrees) can run at once. `self-bench up` derives a Compose project from the checkout directory name and, from that, stable host ports and image tags, so two worktrees never share containers, volumes, ports, or images. Explicit environment always wins:
+
+| Value | Default for the canonical `self-bench` checkout and packaged installs | Default for any other checkout |
+| --- | --- | --- |
+| `COMPOSE_PROJECT_NAME` | `selfbench` | directory name, lowercased |
+| `SELFBENCH_SITE_PORT` | `8080` | stable hash of the project in 8100-8899 |
+| `SELFBENCH_TEMPORAL_PORT` | `7233` | stable hash of the project in 7300-8099 |
+| `SELFBENCH_IMAGE` | `selfbench:local` | `<project>-selfbench:local` |
+| `SELFBENCH_DOCKER_IMAGE` | `selfbench-sandbox:local` | `<project>-sandbox:local` |
+| `SELFBENCH_PUBLIC_URL` | `http://<SELFBENCH_SITE_HOSTNAME or 127.0.0.1>:<site port>` | same |
+| `SELFBENCH_SITE_BIND` | `127.0.0.1`, or `0.0.0.0` when the hostname is not loopback | same |
+
+`self-bench up` prints the resulting URL, project, and OAuth callback; `self-bench down` resolves the same project. Plain `docker compose up` uses the `selfbench` project on 8080 and 7233 unless those variables are exported, so it suits a single stack only.
+
+Compose and `self-bench up` both read `.env` from the checkout; copy `.env.example` to start. Keep `.env` free of per-stack values (project, ports, public URL) when it is shared between worktrees, and export those in the shell instead. With `GITHUB_OAUTH_CLIENT_ID`, `GITHUB_OAUTH_CLIENT_SECRET`, and `SELFBENCH_SESSION_SECRET` set, the API serves the signed-in selfbench.dev site (see [Site sign-in](#site-sign-in-selfbenchdev)); without them it serves the bearer-token Harbor Ledger. `SELFBENCH_SITE_HOSTNAME` is the name browsers open: leave it unset to stay on loopback, or set a tailnet or LAN name to reach the site from another machine.
+
+Manage development DNS and reverse proxies with your local tooling. Set `SELFBENCH_PUBLIC_URL` to the externally reachable origin and `SELFBENCH_SITE_BIND` / `SELFBENCH_SITE_PORT` to the address and port the proxy should reach.
+
+Temporal, site, and artifact state live in the `<project>_temporal-postgres`, `<project>_site-postgres`, and `<project>_artifacts` volumes. Back up those volumes before an upgrade when workflow history, users, or generated artifacts must be retained. Site database migrations (`drizzle/`) run automatically when the API or worker starts.
 
 ## Credentials
 
 self-bench requires GitHub and model credentials:
 
 - `gh auth login` supplies read access to merged pull requests. Export `GH_TOKEN="$(gh auth token)"` for the worker. Write access is not required.
-- `OPENAI_API_KEY` powers discovery, authoring, review, constrained repair, and model evaluation. This is the recommended model-authentication path.
+- `OPENAI_API_KEY` powers discovery, authoring rounds, and verification rounds. This is the recommended model-authentication path.
 
-Existing deployments may continue using ChatGPT subscription authentication by providing both `SELFBENCH_PI_AUTH_JSON` and `SELFBENCH_CODEX_AUTH_JSON`. API-key authentication takes precedence when `OPENAI_API_KEY` is set.
+For ChatGPT subscription authentication, provide `SELFBENCH_PI_AUTH_JSON` containing Pi's `openai-codex` OAuth credential. API-key authentication takes precedence when `OPENAI_API_KEY` is set. SelfBench does not install or invoke the Codex CLI; exported-task evaluation credentials belong to Harbor.
 
-Sandbox-provider credentials are separate. Modal accepts its mounted profile or token pair. For a local Vercel worker, `self-bench setup vercel` stores a project-scoped token in an owner-only local profile. Unattended workers use the equivalent `VERCEL_TOKEN`, `VERCEL_TEAM_ID`, and `VERCEL_PROJECT_ID` environment variables. Keep provider credentials on the worker; the API does not need them.
+Sandbox-provider credentials are separate. Modal accepts its mounted profile or token pair. For a local Vercel worker, `self-bench setup vercel` stores a project-scoped token in an owner-only local profile. Unattended Vercel workers use the equivalent `VERCEL_TOKEN`, `VERCEL_TEAM_ID`, and `VERCEL_PROJECT_ID` environment variables. E2B workers use `E2B_API_KEY` and optionally `E2B_DOMAIN`; E2B setup reads the same values but does not save them. Keep provider credentials on the worker. The API receives provider/template metadata for run manifests but never needs Vercel or E2B control credentials.
 
 ## Execution backends and Harbor
 
-SelfBench uses one provider for discovery, authoring, semantic review, and repair sandboxes. It separately invokes Harbor for nop/oracle validation. Docker and Modal default Harbor to the matching environment. Vercel has no Harbor environment, so `--harbor-environment docker|modal` is mandatory.
+SelfBench uses one provider for discovery, authoring-round, and verification-round sandboxes. It separately invokes Harbor for the build, smoke, nop, and oracle gates of every in-session `verify` and every submission. While an agent session runs, the worker polls the live sandbox's `/work/mailbox` through the provider's exec and file API (Docker `exec`/`cp`, Modal exec and filesystem, E2B commands and files, Vercel `runCommand` and file reads), so those APIs must stay reachable for the whole session. Every generation backend defaults Harbor to the matching environment; `--harbor-environment docker|modal|vercel|e2b|daytona` selects a different one. Daytona is a Harbor-only environment and reads `DAYTONA_API_KEY` from the worker. The pinned Harbor build is installed with its `e2b`, `daytona`, `modal`, and `vercel` extras; Harbor's Vercel environment boots a Vercel Sandbox from a cached snapshot and runs Docker inside it.
 
 ```bash
 self-bench up --backend docker                         # Docker + Docker
 self-bench up --backend modal                          # Modal + Modal
 self-bench up --backend vercel --harbor-environment docker
 self-bench up --backend vercel --harbor-environment modal
+self-bench up --backend vercel                         # Vercel + Vercel
+self-bench up --backend vercel --harbor-environment e2b
+self-bench up --backend vercel --harbor-environment daytona
+self-bench up --backend e2b                            # E2B + E2B
+self-bench up --backend e2b --harbor-environment docker
+self-bench up --backend e2b --harbor-environment modal
 self-bench up --backend docker --harbor-environment modal
 self-bench up --backend modal --harbor-environment docker
+self-bench up --backend modal --harbor-environment daytona
 ```
 
-Use `--modal-config` whenever either side uses Modal. A worker has one fixed pairing; do not run workers with different provider settings on the same Temporal task queue. Run and export metadata record both choices, plus the effective Vercel timeout cap when applicable.
+The hosted site offers only Modal, Vercel, and E2B generation with Modal, Vercel, E2B, or Daytona Harbor, each backed by an organization credential. Docker is not offered there because Docker generation and Docker Harbor both run on the shared worker. Hosted Harbor credentials travel as `SELFBENCH_HARBOR_E2B_API_KEY` and `SELFBENCH_HARBOR_VERCEL_TOKEN`, `SELFBENCH_HARBOR_VERCEL_TEAM_ID`, and `SELFBENCH_HARBOR_VERCEL_PROJECT_ID`, and take their provider names only inside Harbor's process, so generation and verification may use different accounts of the same provider.
+
+Use `--modal-config` whenever either side uses Modal. A worker has one fixed pairing; do not run workers with different provider settings on the same Temporal task queue. Run and export metadata record both choices, plus the configured hosted-provider timeout cap when applicable.
 
 ### Docker
 
@@ -67,11 +97,76 @@ self-bench up --backend modal --modal-config /absolute/path/to/.modal.toml
 
 When Modal is used for generation or Harbor, SelfBench mounts `~/.modal.toml` by default; `--modal-config` overrides that path. A secret manager may provide `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` instead. Empty token environment variables are removed at worker startup so they cannot override a valid mounted profile.
 
-Modal defaults to 20 concurrent worker activities. Discovery starts eight independently retryable shards, and candidate slots are continuously refilled. Discovery and authoring stop after eight minutes without process output; review stops after five. Discovery also has a 45-minute per-attempt deadline and up to three attempts per shard.
+Modal defaults to 20 concurrent worker activities. Discovery starts eight independently retryable shards, and candidate slots are continuously refilled. Discovery, authoring rounds, and verification rounds stop after eight minutes without process output. Discovery also has a 45-minute per-attempt deadline and up to three attempts per shard; authoring and verification rounds each request four hours because an in-session `verify` can take up to an hour and an agent has several.
+
+### E2B
+
+E2B generation defaults to E2B Harbor; choose Docker, Modal, Vercel, or Daytona instead with `--harbor-environment`. It requires a custom, prebuilt SelfBench template. Stock E2B templates do not contain the pinned Pi, GitHub CLI, system packages, or `/work` layout that SelfBench expects, so `SELFBENCH_E2B_TEMPLATE` has no default. SelfBench never installs those runtime dependencies while allocating a sandbox.
+
+#### Build the template
+
+Create an E2B API key, export it only in the shell that performs setup, and choose a versioned template name or tag:
+
+```bash
+export E2B_API_KEY=...
+# Optional only for an E2B-compatible private/control-plane domain:
+# export E2B_DOMAIN=e2b.example.com
+
+self-bench setup e2b --name selfbench-runtime:v1
+```
+
+This noninteractive command uses the pinned `e2b@2.46.0` SDK to parse the packaged `Dockerfile.sandbox` with the package root as its file context, then calls `Template.build`. It does not require the E2B CLI or a local Docker daemon. Setup accepts a lowercase `name[:tag]` (letters, digits, hyphens, and underscores, plus periods in a tag) and rejects malformed names before any SDK call. The Dockerfile pins its base image and tool versions, supplies an `amd64` default for E2B's `TARGETARCH` parser, and ends at `WORKDIR /work`. Build requests have a 60-second per-request control-plane timeout while the build itself may run longer. Build logs go to stderr; on success the command prints JSON containing the exact template reference returned by E2B, template ID, build ID, and a shell-safe `SELFBENCH_E2B_TEMPLATE` export. Preserve the build ID in deployment records and use a new versioned name/tag when rebuilding so run metadata can identify the intended runtime reference; a mutable tag is not an immutable build identifier.
+
+A standard SelfBench request expects 4 CPUs and 8,192 MiB. E2B 2.46 assigns CPU and memory when the template is built and exposes no per-sandbox resource override. Setup therefore uses those values by default:
+
+```bash
+self-bench setup e2b \
+  --name selfbench-runtime:v1 \
+  --cpus 4 \
+  --memory-mib 8192
+```
+
+The executor verifies E2B's allocated CPU and memory before uploading source files and fails with a resource-mismatch diagnostic if the configured template differs from the request. Change the setup resource flags only for a custom caller that also sets matching `SandboxRequest` resources; ordinary SelfBench activities use the standard values. Disk size is likewise template/platform controlled and cannot be mapped per create by this SDK version.
+
+E2B templates are durable account resources and are not removed by `self-bench down`. Build a new versioned template for runtime changes and retire old templates according to your E2B retention policy. Each workflow stage still gets a fresh sandbox from that template.
+
+#### Start a worker
+
+Configure the template reference printed by setup, keep the API key in the worker environment, and explicitly choose Harbor:
+
+```bash
+export E2B_API_KEY=...
+export SELFBENCH_E2B_TEMPLATE=selfbench-runtime:v1
+# Optional; defaults to the Hobby-compatible one-hour ceiling:
+export SELFBENCH_E2B_TIMEOUT_CAP=1h
+
+self-bench up --backend e2b --harbor-environment docker
+# Or, with Modal credentials/profile configured:
+# self-bench up --backend e2b --harbor-environment modal
+```
+
+Worker startup calls `Template.exists` with an explicit SDK client and a 30-second local/request timeout, and fails before polling Temporal if the credentials cannot access the template. The default activity concurrency is four. Reduce `SELFBENCH_ACTIVITY_CONCURRENCY`, often to `1`, when Docker Harbor or the E2B account cannot sustain four concurrent activities.
+
+E2B Hobby sandboxes have a one-hour maximum lifetime; paid plans can support up to 24 hours. SelfBench conservatively defaults `SELFBENCH_E2B_TIMEOUT_CAP` to `1h`. Set a larger cap only after verifying the account entitlement; values above `24h` are rejected. Discovery requests 45 minutes and each authoring or verification round requests four hours, so the configured cap centrally shortens only longer stages; under a one-hour cap an agent has room for at most one in-session verify. E2B also receives the effective stage timeout with lifecycle action `kill`, and SelfBench independently enforces the same hard deadline, returning exit 124 after a confirmed cleanup.
+
+Commands run under `/work`. Inputs and binary outputs are transferred with E2B's file API, and paths outside `/work` are rejected before allocation. Stdout/stderr stream progress while retaining only the latest 8 MiB per stream for diagnostics. Output inactivity cancels the command and sandbox. Workload environment and stage secrets are scoped to the command; SelfBench does not create durable account-level E2B Secrets.
+
+Every normal completion, command failure, cancellation, inactivity timeout, and hard timeout enters cleanup. On failure or termination, SelfBench requests command kill first, collects requested partial outputs in parallel under one bounded diagnostic deadline, and only then removes the sandbox. It first tries the sandbox handle, but never treats E2B's `kill(false)` alone as proof of absence: it falls back to static kill by sandbox ID and uses `getInfo`; only `SandboxNotFoundError` independently confirms that the sandbox is gone. Cleanup calls and retries share a bounded deadline, and a stuck provider promise cannot hold the activity open indefinitely.
+
+A create request carries unique SelfBench allocation metadata. If cancellation, timeout, or response loss leaves creation ambiguous before an ID is returned, cleanup searches all listed states (including paused sandboxes) by that metadata and also accepts a late-arriving create handle. If allocation absence or deletion cannot be confirmed within the cleanup window, the activity fails with the allocation context rather than silently reporting success. A provider could still allocate after that bounded recovery window; E2B's requested `onTimeout: kill` lifecycle is the final bound. A worker process crash can likewise bypass client cleanup.
+
+All `E2B_*` variables are treated as control-plane values. Compose passes the supported API key/domain settings only to the worker, not the API, and SelfBench strips the entire prefix from E2B workload commands and Harbor child environments. The API receives only `SELFBENCH_E2B_TEMPLATE` and `SELFBENCH_E2B_TIMEOUT_CAP` so it can stamp run/export metadata. The sandbox still receives repository content and the selected model/GitHub workload credentials, so use only trusted repositories and apply E2B account budget and network controls before unattended runs.
+
+Common failures:
+
+- `E2B template ... does not exist or is not accessible` at worker startup means the name/tag is wrong, the API key belongs to another account, or the optional domain is wrong. Re-export the exact `configure` value from setup or rebuild the template.
+- A resource-mismatch error means the template was built with CPU or memory that does not match the stage request. Rebuild the standard template with 4 CPUs and 8,192 MiB.
+- A timeout-limit error after raising the cap means the E2B plan does not support that lifetime. Restore `SELFBENCH_E2B_TIMEOUT_CAP=1h` or use a verified paid-plan value.
+- A cleanup error includes the sandbox ID or allocation context but redacts the API key. Inspect active sandboxes in E2B, kill any matching `selfbench_allocation` metadata, then resolve control-plane access before retrying.
 
 ### Vercel Sandbox
 
-Vercel is a generation backend only; choose Docker or Modal for Harbor. SelfBench supports both Vercel's 45-minute Hobby Sandbox ceiling and the longer paid-team ceiling. Discovery requests 45 minutes, review requests 15 minutes, and authoring and repair request two hours; setup detects the selected project's effective capability and caps every Vercel stage centrally when necessary. Sandbox use, VCR storage, memory, active CPU, and data transfer are metered by Vercel; configure Spend Management before unattended runs. Vercel Hobby use is intended for personal, non-commercial work.
+Vercel generation defaults to Vercel Harbor; choose Docker, Modal, E2B, or Daytona instead with `--harbor-environment`. SelfBench supports both Vercel's 45-minute Hobby Sandbox ceiling and the longer paid-team ceiling. Discovery requests 45 minutes and each authoring or verification round requests four hours; setup detects the selected project's effective capability and caps every Vercel stage centrally when necessary. Sandbox use, VCR storage, memory, active CPU, and data transfer are metered by Vercel; configure Spend Management before unattended runs. Vercel Hobby use is intended for personal, non-commercial work.
 
 #### Interactive local setup
 
@@ -122,7 +217,7 @@ self-bench up --backend vercel --harbor-environment modal
 
 Modal Harbor also needs the Modal profile or token pair. Vercel control credentials are removed from the Harbor child process for both Harbor environments. `self-bench up` resolves the profile, then validates the complete credential triple, digest-pinned image, and timeout cap before starting Compose.
 
-The setup probe records a two-hour effective SelfBench ceiling when the requested two-hour sandbox is accepted. If Vercel returns its exact 45-minute limit response, setup verifies a 45-minute sandbox, explains the impact, and asks before saving that cap. Longer authoring and repair requests then run for at most 45 minutes and return exit 124 on timeout, so only the affected candidate is rejected. Discovery and review retain their shorter requested limits. The effective cap is included in run and export metadata.
+The setup probe records a two-hour effective SelfBench ceiling when the requested two-hour sandbox is accepted. If Vercel returns its exact 45-minute limit response, setup verifies a 45-minute sandbox, explains the impact, and asks before saving that cap. Longer authoring and verification rounds then run for at most 45 minutes and return exit 124 on timeout, so only the affected round fails. Discovery retains its shorter requested limit. The effective cap is included in run and export metadata.
 
 #### Environment-only and unattended workers
 
@@ -209,32 +304,138 @@ self-bench run \
   --output ./self-bench-tasks.tar.gz
 ```
 
-The three tier counts total 1–10,000. Each is a fixed candidate authoring budget, not an accepted-task target: rejected candidates are not replaced. Discovery expands only until it fills each requested tier budget, then accepted tasks are exported.
+The three tier counts total 1–10,000. Each is an accepted-task target: discovery expands until it can fill every tier and over-fetches a small pool, and a rejected or infrastructure-failed candidate is replaced from that leftover pool until each tier is filled or the pool is exhausted. Accepted tasks are then exported.
+
+### Replaying known candidates
+
+`self-bench replay` starts a run from candidates of an earlier run instead of discovery, for example to re-run a few previously rejected candidates through the current pipeline:
+
+```bash
+self-bench replay \
+  --source-run sb-20260901-abcd1234 \
+  --candidate w0s2-uploader --candidate w1s0-legacy \
+  --output ./replayed-tasks.tar.gz
+```
+
+The worker rebuilds each candidate from the source run's artifacts: the retained human request from `runs/<source>/provenance/<candidate>.json`, the full candidate record from the discovery `report.json` located through the `w<wave>s<shard>-` ID prefix, or, failing that, the authored `definition.json` plus a `gh pr view` lookup of the completed commit. Candidate IDs are kept, artifacts are written under the new run ID, and authoring and verification start from fresh agent sessions. The same request shape is accepted by `POST /v1/runs` as a `replay` field (`{ "sourceRunId", "candidateIds" }`) in place of `repository`, `provenance`, and `candidateCounts`.
 
 `--output` implies `--wait`. It reports phase changes, requires a successful Temporal terminal state, downloads with create-only filesystem semantics, and verifies the API-provided SHA-256.
 
 A custom `--run-id` must contain 3–63 lowercase letters, digits, or hyphens and start with a letter or digit.
 
+#### Excluding pull requests earlier runs already processed
+
+Discovery only avoids pull requests found within the same run. A follow-up generation run over the same
+repository therefore proposes PRs an earlier run already authored, wasting agent time and producing duplicate
+tasks. Pass `--exclude-run` (repeatable) with every earlier run whose PRs must not be proposed again:
+
+```bash
+self-bench run --repo ~/code/posthog \
+  --easy-count 10 --medium-count 10 --hard-count 10 \
+  --exclude-run posthog-agent-pipeline-20-v1 \
+  --exclude-run posthog-agent-pipeline-replay-v1 \
+  --run-id posthog-agent-pipeline-30-v2 --output ./posthog-30-v2.tar.gz
+```
+
+Before the first discovery wave the worker runs `collectExcludedSourcePrs`, which reads each listed run's
+candidate records from the artifact store: the discovery checkpoints and reports under
+`runs/<runId>/discovery/`, a replay run's `runs/<runId>/provenance/replay.jsonl`, and, when present, an archived
+run status at `runs/<runId>/status.json` (a saved `GET /v1/runs/<runId>` body; each listed candidate resolves
+through its `provenance/<candidateId>.json` or authored `definition.json`). Every PR the run processed counts,
+whatever its outcome there: accepted, rejected, infrastructure-failed, or still pending. A rejected PR was
+judged on the same material, so retrying it belongs to `replay`, not to a fresh discovery. The set is sent with
+every wave and a candidate whose PR is excluded never enters the pool, even if a shard returns it. A listed run
+with no candidate records at all fails the run non-retryably (`ExcludedRunMissing`) so a mistyped ID cannot
+silently exclude nothing. `POST /v1/runs` accepts the same list as `excludeRuns`.
+
+Export dedupe rule: `buildExport` keeps the first accepted task per source pull request, in the order candidates
+were accepted, and drops later ones. The manifest's `acceptedCount` and `tasks` cover only kept tasks;
+`droppedDuplicates` lists each dropped task with its `sourcePr` and the `keptTaskId` it duplicates. Run status and
+`acceptedTaskIds` still report every accepted candidate; the export is the deduplicated deliverable.
+
+### Round artifacts
+
+Each authoring or verification round stores its decision directly under
+`runs/<runId>/<stage>/<candidateId>/round-<n>/` (`result.json`, `verdict.json`, `fix/`) and the pi session under
+`runs/<runId>/<stage>/<candidateId>/session/round-<n>.jsonl`. Everything a single attempt produces before the
+round is decided lives under `round-<n>/attempt-<m>/`: `prompt.md`, `coupling-evidence.json` (verifier round 1),
+`sandbox.log`, `sandbox-result.json` (the provider's exit code, the wrapper's own status, pi's exit code, and which
+declared outputs were collected), and the in-session `verify-<k>/` reports. A Temporal retry of the same round therefore never collides with the immutable
+artifacts of the attempt it replaces; its session is stored as `session/round-<n>-attempt-<m>.jsonl`.
+
+Large request files reach the sandbox by URL rather than upload: the verifier round asks the artifact store for a V4 signed read URL of the compiled task bundle (two-hour TTL) and the E2B executor has the sandbox `curl` it and verify the SHA-256 in place. Pushing hundreds of MB per sandbox through `files.write` hit the SDK's client-side request timeout once a couple of dozen rounds started together; E2B recommends the pull pattern. Providers without an in-sandbox download step fetch remote files on the worker and upload them inline.
+
+The round wrapper records its own exit status in `/work/wrapper-status` from its EXIT trap, and the worker trusts
+that file over the provider's reported exit code (a provider hard timeout stays authoritative). E2B has been
+observed to lose a long command's stream after the script finished and report a spurious exit code or a gRPC
+"terminated" error; with the status file collected, such a failure becomes a normal round result instead of a retry.
+
+## Harbor viewer
+
+The API serves a browser viewer at `/`. It has two sources, chosen by the server that serves it: a self-bench run (every candidate, including rejected and infrastructure-failed ones, with the artifacts and logs each stage wrote) when served by the API, and a local directory of Harbor tasks when served by `self-bench view`. Each task shows the compiled environment (task.toml, Dockerfiles, services, resources), the setup, smoke, and test commands with the selected tests, the instruction beside the gold and held-out test patches, and a file tree of everything in the bundle. In run mode the pipeline sheet lists each stage's artifacts with a one-line summary of what it concluded.
+
+The same viewer runs without Temporal or a token over any directory of Harbor tasks:
+
+```bash
+self-bench view ./self-bench-tasks --port 8090
+```
+
+Directories are recognized by a `task.toml` up to four levels deep, so an extracted self-bench export, a single task, or a Harbor tasks directory all work. Deep links carry `#run=`, `#task=`, and `#tab=`.
+
 ## HTTP API
 
-The CLI is the recommended client. The API exposes:
+The CLI is the recommended client for run workflows. Every site feature is also reachable over HTTP with a personal API key; see the [HTTP API reference](api.md) for the full route list and authentication. The run routes are:
 
 | Method | Path | Purpose |
 | --- | --- | --- |
 | `GET` | `/healthz` | Liveness check |
 | `POST` | `/v1/provenance?runId=...` | Store sanitized provenance JSONL |
-| `POST` | `/v1/runs` | Start a tiered candidate workflow |
+| `POST` | `/v1/runs` | Start a tiered candidate workflow (optionally with `excludeRuns`), or a replay of known candidates |
 | `GET` | `/v1/runs` | List workflows |
 | `GET` | `/v1/runs/:runId` | Read progress and rejection reasons |
 | `POST` | `/v1/runs/:runId/cancel` | Request Temporal cancellation |
 | `GET` | `/v1/runs/:runId/export` | Download a completed export |
+| `GET` | `/v1/viewer` | Viewer capabilities (`modes`) |
+| `GET` | `/v1/runs/:runId/candidates` | Every candidate with stage, reason, and definition summary |
+| `GET` | `/v1/runs/:runId/candidates/:taskId/artifacts` | Artifact keys grouped by pipeline stage, plus bundles |
+| `GET` | `/v1/runs/:runId/artifacts?key=...` | Stream one artifact under `runs/:runId/` |
+| `GET` | `/v1/runs/:runId/bundle?key=...` | Expand a Harbor task bundle into its text files |
 
 `/healthz` is unauthenticated. Every other route requires `Authorization: Bearer $SELFBENCH_API_TOKEN` when the token is configured. Startup fails if the API binds beyond loopback without a token.
 
-Run status includes its phase, accepted/rejected counts, per-candidate stage, discovery wave, completed/failed shard counts, and current candidates. Failed generation runs use a new run ID; the separate agent matrix is resumable through completed Harbor job reuse.
+### Site sign-in (selfbench.dev)
+
+Setting `GITHUB_OAUTH_CLIENT_ID` turns the same API into the selfbench.dev site: the bundle in `dist/review` renders the login page and signed-in shell instead of the Harbor Ledger, and these routes appear:
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/auth/github` | Redirect to GitHub with a state cookie (scopes `read:user read:org repo`) |
+| `GET` | `/auth/github/callback` | Exchange the code, record the user and their org memberships, set the session cookie |
+| `POST` | `/auth/logout` | Clear the session cookie |
+| `GET` | `/api/me` | The caller's login, name, avatar, organizations, and how they authenticated |
+| `*` | `/api/api-keys…` | Personal API keys: list, create (secret shown once), revoke |
+| `*` | `/api/orgs/:org/…` | Repositories, tasks, batches, evaluations, comparisons, and credentials; see the [API reference](api.md) |
+
+The session is a signed, HttpOnly, SameSite=Lax cookie valid for 30 days (Secure when `SELFBENCH_PUBLIC_URL` is https). Users live in the `users` table of `SELFBENCH_DATABASE_URL`; migrations run at startup. The user's GitHub token is stored encrypted under a key derived from `SELFBENCH_SESSION_SECRET` and is never sent to the browser. With sign-in enabled, `/v1/*` and `/api/*` answer 401 unless the request carries a valid session, a personal API key (`Authorization: Bearer sbk_…` or `X-API-Key`), or the operator bearer token; `/v1/viewer` stays public so the bundle can tell which host it is on. API keys are stored as SHA-256 hashes in the `api_keys` table and act as their owner; `read`-scoped keys may only send `GET` requests. `self-bench view <dir>` never requires sign-in.
+
+Compose: put the three sign-in variables in `.env` and set `SELFBENCH_PUBLIC_URL` to the browser-facing origin. Register that origin with `/auth/github/callback` as the GitHub OAuth callback.
+
+Hot-reload loop: `bun run dev:site` starts a Postgres container (`selfbench-site-postgres`, 127.0.0.1:5433), the API on 8087 with `SELFBENCH_TEMPORAL_CONNECT=lazy`, and Vite on 5173 proxying `/v1`, `/api`, and `/auth`. Register a GitHub OAuth app with callback `http://127.0.0.1/auth/github/callback` (GitHub lets loopback redirects use any port, so browse to `http://127.0.0.1:5173`) and put `GITHUB_OAUTH_CLIENT_ID`, `GITHUB_OAUTH_CLIENT_SECRET`, and `SELFBENCH_SESSION_SECRET` in `.env.site`.
+
+Run status includes its phase, accepted/rejected counts, per-candidate status with the current stage (`authoring` or `verification`) and round, discovery wave, completed/failed shard counts, and current candidates. Failed generation runs use a new run ID. Run exported tasks directly with Harbor; Harbor owns evaluation result persistence and retry behavior.
+
+`/v1/runs` also lists runs that exist only in the artifact store (status `ARCHIVED`) once Temporal retention has dropped their workflow. For those runs the candidate routes reconstruct each candidate from its artifacts: the stage is the furthest pipeline group that wrote anything, and a candidate counts as accepted when its latest verification round wrote an `accepted` `result.json` (a `rejected` round result in either loop marks the stage that ended it). Legacy runs without round results count as accepted when their latest coupling review is `clean`. Bundle expansion caches extracted bundles under the API host's temporary directory, keyed by artifact key, and never returns `repo.tar.gz` contents.
 
 SelfBench has no remote deletion route. Delete local artifact-volume data or GCS run prefixes through normal operator tooling.
 
+## Temporal workflow shape
+
+A run is the `selfBenchRunWorkflow` execution whose workflow ID is the run ID. It discovers candidates, then starts one `selfBenchCandidateWorkflow` child per candidate with the workflow ID `<runId>/candidate/<candidateId>` on the same task queue. The child runs that candidate's authoring and verification loops and returns its final progress plus the accepted task; it signals every progress change to the parent (`candidateProgress`), and the parent's `status` query and `GET /v1/runs/<runId>` merge those signals into the run status. The child's returned result is authoritative even if a signal is lost. Task-ID uniqueness across candidates is enforced by the parent when a child completes green.
+
+To inspect one candidate, open the child workflow in the Temporal UI (search for the run ID prefix, or the parent's "Child Workflows" list). Its history shows that candidate's activities, retries, and timeouts alone; the `candidateStatus` query returns its current progress. Cancelling the run cancels every child, and the parent close policy terminates any straggler. A child that fails outright (anything other than an exhausted or Harbor-infrastructure activity failure) is recorded as `infrastructure_failed` for that candidate; the run continues.
+
+Deployment note: this shape replaced a single workflow that drove every candidate through activities directly. Deploy a worker with the child-workflow shape only when no run is in flight. An in-flight run started under the old shape would replay against the new code and hit a non-determinism error; let running runs finish (or cancel and replay them) before restarting the worker on a build across that boundary.
+
 ## Configuration
 
 | Variable | Default | Used by |
@@ -246,16 +447,20 @@ SelfBench has no remote deletion route. Delete local artifact-volume data or GCS
 | `SELFBENCH_ARTIFACT_DIR` | `.selfbench/artifacts` | Local artifact store |
 | `SELFBENCH_GCS_BUCKET` | — | GCS artifact store |
 | `SELFBENCH_GCS_PREFIX` | `selfbench` | GCS artifact store |
-| `SELFBENCH_EXECUTION_BACKEND` | `docker` | Worker; `docker`, `modal`, or `vercel` |
-| `SELFBENCH_DOCKER_IMAGE` | `selfbench-sandbox:local` | Docker worker |
-| `SELFBENCH_HARBOR_ENVIRONMENT` | matching Docker/Modal backend | Worker; required as `docker` or `modal` for Vercel |
-| `SELFBENCH_ACTIVITY_CONCURRENCY` | `1` Docker, `20` Modal, `4` Vercel | Worker |
+| `SELFBENCH_EXECUTION_BACKEND` | `docker` | Worker; `docker`, `modal`, `vercel`, or `e2b` |
+| `SELFBENCH_DOCKER_IMAGE` | per checkout | Docker worker; sandbox image tag built by `self-bench up` |
+| `SELFBENCH_HARBOR_ENVIRONMENT` | matching Docker/Modal backend | Worker; required as `docker` or `modal` for Vercel/E2B |
+| `SELFBENCH_ACTIVITY_CONCURRENCY` | `1` Docker, `20` Modal, `4` Vercel/E2B | Worker |
 | `SELFBENCH_MODAL_APP` | `selfbench` | Modal worker |
 | `SELFBENCH_MODAL_ENVIRONMENT` | — | Modal worker |
 | `SELFBENCH_MODAL_IMAGE` | `node:22-bookworm` | Modal worker |
 | `SELFBENCH_MODAL_CONFIG_PATH` | `/dev/null` | Compose host mount; set by `self-bench up --modal-config` locally |
 | `SELFBENCH_VERCEL_IMAGE` | profile or — | Required digest-pinned VCR image for Vercel execution |
 | `SELFBENCH_VERCEL_TIMEOUT_CAP` | `2h` | Vercel worker and API; accepts integer milliseconds or `ms`, `s`, `m`, `h` units |
+| `SELFBENCH_E2B_TEMPLATE` | — | Required E2B template name/tag/ID for API and worker |
+| `SELFBENCH_E2B_TIMEOUT_CAP` | `1h` | E2B worker and API; accepts integer milliseconds or `ms`, `s`, `m`, `h` units, up to `24h` |
+| `E2B_API_KEY` | — | Worker-only E2B control credential; required for E2B workers |
+| `E2B_DOMAIN` | E2B default | Optional worker-only E2B control-plane domain |
 | `SELFBENCH_CONFIG_DIR` | `~/.selfbench` | Local CLI profile directory; not needed with a complete Vercel environment |
 | `VERCEL_TOKEN` | profile or unset | Vercel worker; explicit project-scoped access token |
 | `VERCEL_TEAM_ID` | profile or unset | Vercel worker |
@@ -263,14 +468,26 @@ SelfBench has no remote deletion route. Delete local artifact-volume data or GCS
 | `SELFBENCH_TEMPORAL_ADDRESS` | `127.0.0.1:7233` | API and worker |
 | `SELFBENCH_TEMPORAL_NAMESPACE` | `default` | API and worker |
 | `SELFBENCH_TASK_QUEUE` | `selfbench-dev` | API and worker |
-| `OPENAI_API_KEY` | — | Worker sandboxes and matrix harness |
-| `SELFBENCH_PI_AUTH_JSON` | — | Optional Pi subscription-auth fallback |
-| `SELFBENCH_CODEX_AUTH_JSON` | — | Optional Codex subscription-auth fallback |
+| `SELFBENCH_TEMPORAL_CONNECT` | `eager` | API; `lazy` defers the Temporal connection to first use |
+| `GITHUB_OAUTH_CLIENT_ID` | unset | API; enables site sign-in |
+| `GITHUB_OAUTH_CLIENT_SECRET` | — | API; required with the client id |
+| `SELFBENCH_SESSION_SECRET` | — | API; 32+ characters, signs session cookies and seals GitHub tokens |
+| `SELFBENCH_PUBLIC_URL` | `http://<hostname>:<site port>` | API; public origin, forms the OAuth callback URL |
+| `SELFBENCH_SITE_HOSTNAME` | `127.0.0.1` | `self-bench up`; hostname in the public URL, non-loopback binds `0.0.0.0` |
+| `SELFBENCH_DATABASE_URL` | compose: `site-postgres` | API and worker; Postgres holding users, connected repos, and evaluation records |
+| `SELFBENCH_EVAL_CREDENTIAL_KEY` | — | API and worker; 32-byte hex key encrypting saved evaluation credentials |
+| `SELFBENCH_SITE_PORT` / `SELFBENCH_SITE_BIND` | per checkout / `127.0.0.1` | Compose host port and bind address for the API |
+| `SELFBENCH_TEMPORAL_PORT` | per checkout | Compose host port for Temporal |
+| `COMPOSE_PROJECT_NAME` / `SELFBENCH_IMAGE` | per checkout | Compose project and API/worker image tag |
+| `OPENAI_API_KEY` | — | Worker sandboxes |
+| `SELFBENCH_PI_AUTH_JSON` | — | Optional Pi `openai-codex` subscription credential |
 | `GH_TOKEN` | — | Worker GitHub reads |
-| `CODEX_AUTH_JSON_PATH` | `~/.codex/auth.json` | Optional matrix subscription-auth file |
 
 ## Cloud topology
 
+For the reviewable GCP dev/prod Terraform foundation, bootstrap CLI, and deployment gates, see
+[`infra/README.md`](../infra/README.md). The infrastructure code does not provision or deploy itself.
+
 The API is a regular request-oriented HTTP service suitable for Cloud Run. Deploy `Dockerfile` with its default `node dist/api-main.js` command and port 8080:
 
 ```text
@@ -298,6 +515,17 @@ OPENAI_API_KEY=...
 
 The worker owns model, GitHub, Modal, and Harbor credentials. Use separate least-privilege service accounts and a secret manager. Grant GCS object access only to the configured prefix, use a TLS-enabled Temporal namespace, and keep API/worker image digests and `SELFBENCH_TASK_QUEUE` identical.
 
+For E2B generation on a long-running worker, replace the execution settings above with:
+
+```text
+SELFBENCH_EXECUTION_BACKEND=e2b
+SELFBENCH_HARBOR_ENVIRONMENT=docker  # or modal
+SELFBENCH_E2B_TEMPLATE=selfbench-runtime:v1
+SELFBENCH_E2B_TIMEOUT_CAP=1h         # raise only to a verified plan limit
+E2B_API_KEY=...
+# E2B_DOMAIN=...                     # only for a custom E2B domain
+```
+
 For Vercel generation, replace the execution settings above with:
 
 ```text
@@ -319,7 +547,7 @@ This repository defines the application boundary, not turnkey cloud infrastructu
 - Exports contain source snapshots, held-out tests, and reference solutions. They are sensitive and unencrypted.
 - Artifact references carry byte length and SHA-256; reads verify integrity.
 - Local artifact paths and GCS object names are confined to their configured roots. GCS IAM should enforce the same prefix independently.
-- Sandboxes receive only the selected model credential: `OPENAI_API_KEY` by default, or a stage-specific subscription credential for compatibility deployments.
+- Sandboxes receive only the selected model credential: `OPENAI_API_KEY` by default, or the isolated Pi `openai-codex` credential when subscription authentication is configured.
 - Sandboxes contain both a source checkout and a short-lived model credential. Use SelfBench only with repositories you trust to execute; it is not a malware-analysis service.
 - Docker uses disposable containers and volumes and removes them after normal completion. A host crash can leave resources for an operator to inspect and remove. Modal uses disposable Sandboxes. Vercel uses nonpersistent named sandboxes, attempts permanent deletion after each run, and fails the activity when deletion cannot be confirmed; inspect the project after worker crashes or cleanup failures.
-- Vercel control credentials authenticate only the worker's Sandbox control plane and are stripped before Harbor starts. They are never workload command secrets.
+- Vercel and E2B control credentials authenticate only the worker's sandbox control plane. Compose does not pass them to the API; `harborChildEnvironment` strips them before Harbor starts, and the E2B executor strips them from workload command environments. They are never workload secrets.

package/docs/solver-evaluations.md

@@ -0,0 +1,63 @@
+# Run tasks with Harbor
+
+Open a connected repository → **Review** to inspect and approve tasks. **Dataset** contains only human-approved tasks with accepted pipeline results. Select up to ten tasks and open the dedicated **Run** page. Its predefined model table is available even before credentials are configured. Select saved credentials, toggle compatible harnesses, choose a shared sandbox, then press **Run comparison**. Each model creates a separately tracked run on the same frozen tasks. **Results** shows runs and an accuracy/cost Pareto frontier using `@mupt-ai/dari-pareto`; select a run for its transcript, scores, and artifacts. Each task/harness pair gets one attempt, sequentially within its model run. Task generation, authoring, `nop`, and oracle verification remain separate workflows.
+
+## Settings → Credentials
+
+Add named credentials separately from Run. Supported providers are OpenAI, Anthropic, OpenRouter, and a custom OpenAI-compatible endpoint. Custom endpoints require HTTPS and an operator-approved hostname in `SELFBENCH_CUSTOM_MODEL_HOSTS`. Hosted sandboxes are E2B, Modal, and Daytona; E2B and Daytona use API keys, while Modal requires a token ID and secret. Docker is not offered in the hosted UI. Saving does not contact a model or allocate a sandbox. The explicit **Run comparison** action starts billable work.
+
+OpenAI credentials may instead use an explicitly uploaded Codex sign-in file. These credentials work only with the Codex harness, never fall back to an API key, and remain subject to subscription limits and sandbox charges. The worker never imports the host's sign-in automatically.
+
+Credentials are private to your user and reusable across your authorized repositories. PostgreSQL stores AES-256-GCM encrypted records; no separate vault service or Context Drop runtime is required. A separate deployment encryption key stays outside the database. Credentials are never returned by the API or placed in workflow inputs or browser storage. Only selected credentials enter the isolated Harbor process. Use HTTPS outside the encrypted local tailnet. Replacing a credential creates a new reference; deletion is blocked while a referencing run remains active. Database backups may retain deleted encrypted records under your backup retention policy.
+
+Comparisons are persisted before workflow submission. A retry uses the same frozen selection and child workflow IDs, so a partial submission can be resumed without starting duplicate work. Closing the page does not cancel runs.
+
+## Enable on the server
+
+Set `SELFBENCH_DATABASE_URL` and `SELFBENCH_EVAL_CREDENTIAL_KEY` identically on the API and evaluation worker. The key must be a securely generated 32-byte lowercase hex encryption key, separate from session/OAuth secrets. Back it up securely; losing it makes saved credentials unreadable. Startup applies the additive `evaluation_records` migration. Changing the key does not rotate existing records: retain the current key until an explicit decrypt/re-encrypt migration is available. Install `harbor[e2b,daytona,modal,vercel]==0.23.0` on the worker. Set `SELFBENCH_EVAL_TASK_QUEUE` identically on API and worker, then run `node dist/evaluation/worker-main.js`. This worker does not consume task-generation work. The normal combined worker remains supported when using its task queue.
+
+Alternatively, operators can preconfigure profiles with these non-secret settings on the API and Temporal worker:
+
+```sh
+SELFBENCH_EVAL_MODELS='[{"id":"my-openai-model","label":"My OpenAI model","model":"openai/YOUR_MODEL_ID","harnesses":["codex","pi"],"tenants":["YOUR_GITHUB_ORG"],"credentialEnv":"SELFBENCH_EVAL_SECRET_OPENAI"}]'
+SELFBENCH_EVAL_SANDBOXES='["docker"]'
+```
+
+The model name must be an actual model your provider account can use, not the example placeholder. OpenAI models support `codex` and `pi`; Anthropic models support `claude-code` and `pi`; the other supported providers use `pi`. Harbor's additional multi-field/cloud-identity authentication schemes (for example Bedrock and Vertex) are not exposed as fake single-key options. Provider access is not probed with paid calls. The worker fails closed if the selected configuration changed or its credential is missing.
+
+For legacy operator profiles, provision the corresponding `SELFBENCH_EVAL_SECRET_*` value **only on the worker**. Never place it in frontend variables or send an operator key to the browser. Other model keys, GitHub OAuth tokens, site/database credentials, unrelated sandbox credentials, and host subscription profiles are excluded. A separate temporary HOME prevents use of the operator's CLI login state. AWS identity and arbitrary harness imports are not exposed by the current runner.
+
+Docker requires a working worker-accessible Docker daemon and Compose. For operator profiles, Modal uses dedicated `SELFBENCH_EVAL_SECRET_MODAL_TOKEN_ID` and `SELFBENCH_EVAL_SECRET_MODAL_TOKEN_SECRET`; E2B and Daytona use `SELFBENCH_EVAL_SECRET_E2B_API_KEY` and `SELFBENCH_EVAL_SECRET_DAYTONA_API_KEY`. These are Harbor's built-in solver environments, not the separate generation executors. Generation credentials are not reused automatically.
+
+The worker verifies Harbor **0.23.0**, matching the existing Dockerfile pin. Harbor installs the selected solver CLI using its built-in harness. Installed agent version details remain in Harbor's trial result artifact. Restart/redeploy the API and worker through your normal deployment process after configuring them. This does not modify OAuth callback registration or existing preview services.
+
+## Results and safety
+
+E2B uses a packaged Harbor environment subclass that limits sandbox lifetime to one hour. The pinned upstream environment otherwise requests 24 hours, which Hobby accounts reject before the solver starts. All other E2B behavior, including network policy and cleanup, remains inherited from Harbor. The adapter does not retry sandbox creation automatically. Tasks exceeding the one-hour sandbox lifetime fail rather than silently extending usage.
+
+- Session authentication, tenant membership, connected repository ownership, task membership, and runnable bundle state are checked server-side. Run requests accept setup identifiers, not credentials, bundle paths or commands. Credentials are accepted only by the scoped setup endpoint. Tasks must have an accepted pipeline result and an explicit human approval, checked again when submitting.
+- Same-origin JSON is required to start a run. A request UUID is also its durable workflow identity. Retrying an unconfirmed submission with the same selection/UUID never starts a second Temporal workflow; a new explicit run gets a new UUID.
+- Temporal executes a separate `selfBenchEvaluationWorkflow`. Automatic activity and Harbor trial retries are disabled. Each Harbor invocation has a two-hour limit and requests sandbox deletion. A killed worker or forced timeout can leave sandbox resources; the failure view warns that an operator may need to verify cleanup. Local tests do not prove remote cleanup.
+- Runs survive page navigation and API restarts. Metadata and changing progress are append-only immutable snapshots under `evaluations/repos/<repo-id>/<uuid>/`; completed text artifacts live beside them. Unchanged poll results do not create snapshots. The worker and API must share the artifact store. Retention/garbage collection remains an operator responsibility.
+- Docker's mounted agent logs can update during execution. Modal's agent logs are downloaded after the solver finishes; the UI labels that limitation rather than presenting synthetic live transcript events. The browser polls every three seconds.
+- Completed views show each task/harness's verifier reward names and values, solver text, tool calls and outputs, raw logs, and sanitized text artifacts. A zero score is a legitimate completed result; missing scores and exceptions are failures, not zeroes. ATIF trajectories and Pi message events are normalized where present, with raw output as fallback.
+- Only selected text log/result filenames are exported, never whole sandbox directories, auth files, or standalone config files. Known worker secret values and common token patterns are redacted before persistence. Output is untrusted text, not rendered HTML. Redaction is not protection against a deliberately malicious solver encoding a credential; sandbox workloads must be trusted to receive the dedicated evaluation model key.
+- Text files are capped at 1 MiB each and 4 MiB per collection; the on-page log and transcript are bounded. Downloaded files are sanitized, potentially truncated exports, not byte-identical raw Harbor artifacts. No automatic public uploads occur.
+
+## Cost comparisons
+
+Estimated model cost is independent of authentication. Codex sign-in and API keys use the same token accounting: uncached input, cached input, cache writes, and output. For verified models, matching Harbor per-request token and cost records take precedence over reference rates, preserving per-request pricing tiers. Pi uses explicit reference rates: its unknown-model resolver can inherit another model's price metadata, so its dollar total alone is not trusted. A bare dollar total without matching usage is not accepted. Otherwise, dated reference rates are snapshotted with the run; aggregate usage beyond their conservative input bound is not estimated. These are token-equivalent estimates, not invoices; sandbox charges are excluded. Custom models can have a Harbor estimate even without reference pricing. Claude Code cost remains unavailable where its export cannot reliably separate cache-write usage. Missing estimates are never displayed as zero.
+
+The catalog was researched on September 5–6, 2026 against official provider documentation, OpenRouter's live model API, and Artificial Analysis coding-agent results. It includes GPT-6 Astra, GPT-5.6 Sol/Terra/Luna, Claude Fable 5.1/Opus 5/Sonnet 5, and current OpenRouter alternatives. Catalog availability is not proof of credential entitlement or a successful SelfBench run. No older-model fallback is performed. Existing runs retain their original model and pricing snapshots. OpenRouter routes with ambiguous cache-write units have no fixed reference estimate; measured Harbor usage can still supply a cost.
+
+The plot only compares completed runs with binary reward scores, verified model identities, and complete costs on the same frozen task/bundle snapshot. Separate dataset selections produce separate comparison groups. Missing costs are never treated as zero; those runs remain available in the results list. Failed/incomplete runs are not frontier points.
+
+## Local validation
+
+```sh
+bun test tests/evaluation-routes.test.ts tests/evaluation-runner.test.ts tests/evaluation-lifecycle.test.ts review/src/web/evaluation/evaluation.test.tsx
+bun run check
+bun run build
+```
+
+Tests use a local artifact store, PGlite, signed test sessions, and mocked Harbor commands. They do not run models or create Docker/Modal sandboxes. Real provider access, solver installation, live evaluation, and cloud cleanup require a separately authorized smoke run.

package/docs/task-construction.md

@@ -1,6 +1,6 @@
 # Task construction and validation
 
-SelfBench creates easy, medium, and hard Harbor evaluations from completed pull requests. This document defines the task boundary, validation gates, repair behavior, and export contents.
+SelfBench creates easy, medium, and hard Harbor evaluations from completed pull requests. This document defines the task boundary, the authoring and verification rounds, and export contents.
 
 ## Terms
 
@@ -31,9 +31,9 @@ Profiles are eligibility rules, not empirical claims about model success:
 | medium | at least 50 changed lines across 2 implementation paths | at least 1 | at least 1 |
 | hard | at least 100 changed lines across 3 implementation paths | at least 1 | at least 2 |
 
-Every accepted task also requires a held-out test patch with no file overlap with the reference patch, deterministic repository-native setup and tests, a passing nop/oracle split, and independent anti-coupling review.
+Every accepted task also requires a held-out test patch with no file overlap with the reference patch, deterministic repository-native setup and tests, a passing smoke/nop/oracle split, and acceptance by the independent verification agent.
 
-The size gate is mechanical. Generated or vendored code suitability remains a review judgment. Git LFS, submodules, generated changes, and service-heavy integration suites receive no special path and may be rejected during authoring or validation.
+The size gate is mechanical. Generated or vendored code suitability remains a verification judgment. Git LFS, submodules, generated changes, and service-heavy integration suites receive no special path and may be rejected during authoring or verification.
 
 ## Agent-visible boundary
 
@@ -45,23 +45,33 @@ An evaluated coding agent receives:
 
 It does not receive the held-out test patch or reference solution. Harbor mounts `solution/` only for the explicit oracle agent.
 
+The agent image commits the tree as it stands after `setup.sh` (the `selfbench-setup` commit) and snapshots that repository state to `/opt/selfbench/base.git`. The agent's diff (`agent.patch`) is therefore taken against the post-setup snapshot: files that setup creates and does not gitignore are never part of the agent's patch and cannot collide with the verifier image, which runs the same setup. The verifier image keeps the base commit as `HEAD`; when the gold patch changes dependency manifests it resets only those manifest paths after its extra setup pass.
+
 Held-out tests must exercise an existing public API, command, persistence boundary, or extension seam. They may not import gold-specific private helpers or prescribe exact internal SQL, query counts, private schemas, object identity, telemetry layout, incidental error wording, or UI composition unless the source request explicitly makes that artifact public.
 
-## Validation and repair
+## Authoring rounds and verification
+
+Every candidate passes through two agent-centred loops. Agents do the authoring and judging; the harness only renders, builds, and measures.
+
+**Authoring rounds.** One authoring agent session owns the complete Harbor task: instruction, definition (test command, fail-to-pass and pass-to-pass selection, timeouts, resources) and the environment contract (base image, root setup, setup, smoke command, environment variables, services, evidence). The agent writes its deliverable to `/work/task/` (`definition.json` with the environment contract, `instruction.md`, `test.patch`, `gold.patch`; `instruction.md` is authoritative for the prompt) and calls `verify`, which takes no arguments and reads that directory. The tool runs the static check inside the sandbox first (schema, environment policy, patch path safety, audit thresholds, and a dry render of the Harbor tree into `/work/rendered/`) and returns failures immediately. Otherwise it hands the payload to the worker through a sandbox mailbox and blocks while the worker's trusted compiler renders `task.toml`, both Dockerfiles, the scripts, and the repository snapshot, and Harbor builds the images and runs the smoke command, the `nop` split, and the `oracle` split. The structured report (compile, audit, build, smoke, nop, oracle) comes back as the tool result, so the agent iterates inside its own session; it has three `verify` calls per session. `submit_task` runs the static check again and records the task; when its payload equals the last green `verify`, the worker reuses that report instead of rebuilding. Otherwise the worker verifies the submission itself. If the sandbox dies or a submitted task is red, the same agent session is resumed in a fresh sandbox with the report as its next message (fallback loop). Three red rounds reject the candidate with the last report as the reason. A Harbor infrastructure failure counts as a red round with the build log as the report; three consecutive infrastructure rounds mark the candidate `infrastructure_failed` instead.
 
-Static audit runs before sandbox validation. Harbor then proves:
+The mailbox is a directory in the live sandbox: the tool writes `/work/mailbox/requests/<id>.json`, the worker's supervising activity polls it through the provider's live-sandbox exec and file API, archives each verify under `runs/<run>/<stage>/<candidate>/round-<n>/verify-<k>/`, and writes the response the tool is waiting on. Verifier commands run as the `verifier` user with `HOME=/home/verifier`.
 
-1. `nop`: selected new tests fail while selected regressions pass;
+The Harbor gates prove:
+
+1. `nop`: the smoke command succeeds in the built image, then selected new tests fail on the base snapshot while selected regressions pass;
 2. `oracle`: the reference patch applies and every selected test passes;
 3. determinism: the fail-to-pass selection passes a second time with the oracle.
 
-A fresh model session reviews the prompt and tests without inheriting the authoring conversation. When it finds repairable test-to-gold coupling, Temporal schedules one constrained repair in another sandbox. Repair may modify only paths already changed by the held-out test patch. It cannot change the request, base snapshot, or reference implementation.
+**Verification rounds.** A separate agent session that has not seen the authoring conversation receives the green task (instruction, held-out test patch, gold patch, environment contract, rendered files), the verification report, and the deterministic coupling evidence. It judges whether the task is a fair, self-contained benchmark: tests exercise public behaviour, no test-to-gold coupling, deterministic environment, instruction faithful to the human request. It either accepts or submits a fix. A fix may edit only the held-out tests and the environment contract (plus the test selection, timeouts, and resources that describe them), never the gold patch, the base commit, or the instruction. The verifier authors a fix as files too: held-out test edits in `/work/repo` plus an optional `/work/fix/definition.json` carrying only the changed fields (the tools regenerate `test.patch` from the working tree unless `/work/fix/test.patch` is present). It has the same in-session `verify` tool (two calls per session) for its fix; `submit_fix` runs the static check before it counts and reuses a matching green verify. Otherwise the worker re-runs compile, audit, and Harbor and resumes the same verifier session with the new report. A task is accepted only when the verifier accepts and the mechanical gates are green; otherwise the candidate is rejected with the verifier's reason after at most three verifier rounds.
+
+Every candidate in the discovered pool runs as its own child workflow in parallel; the requested counts size the pool (discovery targets 1.5 times the request per tier, dealt across its shards) rather than capping the result, so a run can export more accepted tasks than it asked for. Nothing is replaced or backfilled: a rejected candidate is simply a rejection in the run status.
 
-After repair, the task repeats static audit, nop, oracle, and independent review from the beginning. A second failure rejects the candidate.
+## Reproducible environments
 
-## Toolchains
+SelfBench does not infer a generic language toolchain. The authoring agent derives the environment contract from the exact pinned base commit: runtime, dependency installation, builds, fixtures, and local services come from the closest matching CI job, repository Dockerfile, devcontainer, lockfiles, and test scripts.
 
-The task compiler has setup layers for Node.js, Bun, Python, Go, and Rust. The author supplies fixed repository-native setup and test commands; compatibility is validated per candidate instead of promised for every repository.
+The resulting contract pins the base and service images by digest, records repository-file evidence for each choice, keeps service credentials local and non-secret, and separates verifier-only services from the agent environment. Secret-named variables are accepted only with fixed placeholder literals; values that look like real key material or interpolate host variables are rejected. The declared smoke command runs in the built verifier image before the `nop` split, so an environment defect is reported to the authoring agent as a red smoke or build gate rather than discovered later.
 
 When the reference patch changes a recognized dependency manifest or lockfile, the hidden verifier image repeats setup with the trusted reference state and then resets source files to the base snapshot. This prevents stale base dependencies from invalidating the oracle without exposing the reference patch to the coding-agent environment.
 
@@ -89,7 +99,7 @@ harbor-task/
 └── solution/
 ```
 
-The export includes the repository snapshot at each selected base commit, held-out tests, and reference solutions. It excludes Git history, local provenance/session records, and author/reviewer transcripts.
+The export includes the repository snapshot at each selected base commit, held-out tests, and reference solutions. It excludes Git history, local provenance/session records, and the authoring and verification agent sessions (those stay in the run's artifacts).
 
 The manifest digest detects accidental corruption but is not a signature because it sits inside the same archive. Extract only exports from a trusted SelfBench deployment and store them as private benchmark material.