self-bench 0.3.2 → 0.3.4

package/.dockerignore

@@ -5,6 +5,5 @@
 dist
 jobs
 node_modules
-review
 tests
 *.md

package/Dockerfile

@@ -6,8 +6,9 @@ COPY package.json bun.lock ./
 RUN bun install --frozen-lockfile
 COPY biome.json tsconfig.json tsconfig.build.json ./
 COPY src ./src
+COPY review ./review
 COPY scripts ./scripts
-RUN bun run build:server
+RUN bun run build
 
 FROM docker:29.4.0-cli AS docker-cli
 

package/Dockerfile.sandbox

@@ -1,4 +1,4 @@
-FROM node:22-bookworm
+FROM node:22-bookworm@sha256:0557ac14e0d45d02ed563067b82856ca5e7aa3437fa28d98d4350ea9c3d9494a
 
 ARG PI_VERSION=0.84.0
 ARG CODEX_VERSION=0.146.1

package/README.md

@@ -1,262 +1,206 @@
-# SelfBench
+# self-bench
 
-SelfBench turns completed GitHub changes into private [Harbor](https://harborframework.com/) evaluations for coding agents. It finds real feature requests, builds tasks from the repository's base commit, hides the tests and reference solution, and checks that each task fails without a solution and passes with one.
+[![npm version](https://img.shields.io/npm/v/self-bench?color=blue&label=npm)](https://www.npmjs.com/package/self-bench)
+[![CI](https://github.com/mupt-ai/self-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/mupt-ai/self-bench/actions/workflows/ci.yml)
+[![license](https://img.shields.io/github/license/mupt-ai/self-bench?color=green)](./LICENSE)
+[![Bun](https://img.shields.io/badge/runtime-Bun-f9f1e1?logo=bun&logoColor=000)](https://bun.sh/)
+[![TypeScript](https://img.shields.io/badge/lang-TypeScript-3178c6?logo=typescript&logoColor=fff)](https://www.typescriptlang.org/)
 
-SelfBench is distributed as the [`self-bench`](https://www.npmjs.com/package/self-bench) package and installed with **Bun**.
+**self-bench builds private coding-agent benchmarks from work already completed in your repository, so you can compare coding agents and models on tasks drawn from your own codebase.**
 
-## Choose your setup
+It finds completed requests from local coding sessions and merged GitHub pull requests, then reconstructs each task from the commit before the change. For every accepted task, self-bench creates hidden tests and a reference solution, proves that the task fails without a solution and passes with the original implementation, and exports a native task for [Harbor](https://harborframework.com/), a runner for coding-agent evaluations.
 
-| Setup | Best for | Execution | State |
-| --- | --- | --- | --- |
-| Local | Trying SelfBench or small runs | Docker on your machine | Local Docker volumes |
-| Temporal Cloud | Large repositories, long runs, and reliable unattended execution | Modal sandboxes | Temporal Cloud + object storage |
+The result is a private `.tar.gz` benchmark that you can run against multiple models:
 
-After a run is submitted, the workflow continues independently of the waiting CLI process. The local worker still depends on your machine and its Docker stack; Temporal Cloud is the recommended setup when a run may take hours or your laptop should not be responsible for the worker.
+```text
+Your repository history
+        ↓
+completed requests + implementations
+        ↓
+validated Harbor tasks with hidden tests
+        ↓
+gpt-5.6-luna vs gpt-5.6-terra vs gpt-5.6-sol
+```
 
-## Prerequisites
+## Quickstart
 
-Installing the package requires [Bun](https://bun.sh/) 1.3.14 or newer. Running evaluations additionally requires:
+This path runs the self-bench API, worker, and [Temporal](https://temporal.io/) workflow state locally while using [Modal](https://modal.com/) for disposable task-generation and validation sandboxes.
 
-- Docker with Compose
-- `gh`, authenticated with access to the source repository
-- Pi with an authenticated `openai-codex` account
-- Codex CLI with an authenticated account
+### Prerequisites
 
-Modal execution additionally requires a Modal account and token. Temporal Cloud execution additionally requires a Temporal Cloud namespace and a durable artifact store such as Google Cloud Storage.
-
-## Install
+- [Bun](https://bun.sh/) 1.3.14 or newer
+- Docker with Compose
+- [Modal](https://modal.com/) CLI, account, and token (`pip install modal`)
+- [`gh`](https://cli.github.com/), authenticated with read access to the repository
+- An OpenAI API key with access to `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna`
+- A Git checkout with a GitHub `origin` and completed work in its history
 
-Install the CLI from the npm registry with Bun:
+Install self-bench and authenticate Modal and GitHub:
 
 ```bash
 bun add --global self-bench
-self-bench --help
-```
-
-For one-off use without a global install:
-
-```bash
-bunx self-bench --help
-```
-
-The package exposes `dist/cli.js` as `self-bench`.
-
-To develop SelfBench itself, clone the repository and use Bun:
-
-```bash
-git clone https://github.com/mupt-ai/self-bench.git
-cd self-bench
-bun install --frozen-lockfile
-bun run build
-bun link
-self-bench --help
+modal token new
+gh auth login
 ```
 
-During development, run the TypeScript entrypoint without linking:
+Set the model and GitHub credentials used by the local worker, plus a random token that protects the local self-bench API:
 
 ```bash
-bun run cli -- --help
+export OPENAI_API_KEY=...
+export GH_TOKEN="$(gh auth token)"
+export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)"
 ```
 
-Authenticate the tools before starting a run:
+### 1. Start self-bench
 
 ```bash
-gh auth login
-# In Pi: /login -> OpenAI Codex
-codex login
+self-bench up --backend modal
+export SELFBENCH_API_URL=http://127.0.0.1:8080
 ```
 
-## Quick start: local Docker
+This starts Postgres, Temporal, the self-bench API, and a worker in Docker. The worker sends sandbox work to Modal; `SELFBENCH_API_URL` tells subsequent CLI commands where to reach the local API.
 
-This starts Postgres, Temporal, the SelfBench API, and a worker on your machine. The worker runs sandbox validation through Docker.
+### 2. Build a benchmark
 
 ```bash
-export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)"
-export GH_TOKEN="$(gh auth token)"
-
-self-bench up --backend docker
-export SELFBENCH_API_URL=http://127.0.0.1:8080
-
 self-bench run \
   --repo /absolute/path/to/your/repository \
-  --easy-count 30 \
-  --medium-count 30 \
+  --easy-count 10 \
+  --medium-count 10 \
   --hard-count 10 \
   --output ./self-bench-evals.tar.gz
 ```
 
-The `--repo` directory must be a Git checkout with a GitHub `origin`. SelfBench pins the current `HEAD`; uncommitted changes are ignored. It discovers candidate requests from sanitized local coding-session messages and merged, non-bot GitHub pull requests.
+Easy, medium, and hard candidates require at least 20, 50, and 100 changed implementation lines across 1, 2, and 3 paths respectively; the counts are generation budgets, not guarantees that every candidate will pass validation.
 
-This quick start authors 70 candidates: 30 easy, 30 medium, and 10 hard. Counts are authoring budgets, not accepted-task guarantees. Candidates can be rejected during authoring, validation, audit, or review, and rejected candidates are not replaced.
+The repository must be a Git checkout with a GitHub `origin`. self-bench pins its current `HEAD`, ignores uncommitted changes, and may take hours to author, validate, review, and export the accepted tasks. `--output` waits for completion and verifies the downloaded archive with SHA-256.
 
-A run may take hours. `--output` waits for the run to finish, then downloads and SHA-256-verifies the accepted tasks.
+### 3. Compare models with Harbor
 
-## Local commands
+Install Harbor and extract the generated tasks:
 
 ```bash
-self-bench status RUN_ID
-self-bench list
-self-bench cancel RUN_ID
-self-bench download RUN_ID ./self-bench-evals.tar.gz
-```
-
-If you are working from source rather than using `bun link`, use the same commands through Bun:
+uv tool install --python 3.12 'harbor[modal]==0.20.1.dev202608040148'
 
-```bash
-bun run cli -- status RUN_ID
+mkdir -p ./self-bench-export ./self-bench-tasks
+tar -xzf ./self-bench-evals.tar.gz -C ./self-bench-export
+for archive in ./self-bench-export/tasks/*.tar.gz; do
+  task_id="$(basename "$archive" .tar.gz)"
+  mkdir -p "./self-bench-tasks/$task_id"
+  tar -xzf "$archive" --strip-components=1 -C "./self-bench-tasks/$task_id"
+done
 ```
 
-Stop the local stack with Docker Compose:
+Run Harbor's Codex agent adapter once for all three models. Harbor evaluates every extracted task at high reasoning and keeps the model results in one job directory:
 
 ```bash
-docker compose down
-```
-
-Named Docker volumes retain Temporal history and generated artifacts. Back them up before removing them if you need to resume or inspect old runs.
-
-## Reliable setup for larger repositories
-
-For large repositories or unattended runs, use Temporal Cloud for workflow state and Modal for disposable execution sandboxes. Temporal Cloud does not host the SelfBench API or worker; deploy those separately.
-
-```text
-self-bench CLI
-    -> SelfBench API
-    -> Temporal Cloud namespace
-    -> persistent SelfBench worker
-    -> Modal sandboxes
-    -> GCS artifact storage
+harbor run \
+  --path ./self-bench-tasks \
+  --agent codex \
+  --model gpt-5.6-luna \
+  --model gpt-5.6-terra \
+  --model gpt-5.6-sol \
+  --ak version=0.146.1 \
+  --ak reasoning_effort=high \
+  --env modal \
+  --jobs-dir ./harbor-jobs \
+  --n-concurrent 20 \
+  --yes
 ```
 
-The API and worker must use the same image version, Temporal namespace, task queue, artifact configuration, and `SELFBENCH_BUILD_COMMIT`. Keep the worker running on a long-lived container service; do not run it on a scale-to-zero request service. The API can run as a normal HTTP service.
+The evaluated agent receives the base repository and task instruction, but not the hidden tests or reference solution.
 
-### 1. Prepare Temporal Cloud and GCS
+See [Running self-bench evaluations](docs/evaluations.md) for running one task and using the optional resumable matrix helper.
 
-Create a Temporal Cloud namespace and obtain its address, namespace, API key, and TLS settings. Create a GCS bucket for SelfBench artifacts and give the worker/API service accounts access only to the SelfBench prefix.
+## Run management
 
-Set these values in the API and worker environments:
+Closing the waiting CLI does not cancel a submitted workflow. If the local worker or Docker stack stops, work pauses until the worker is restarted.
 
 ```bash
-SELFBENCH_TEMPORAL_ADDRESS=your-namespace.tmprl.cloud:7233
-SELFBENCH_TEMPORAL_NAMESPACE=your-namespace
-SELFBENCH_TEMPORAL_API_KEY=...
-SELFBENCH_TEMPORAL_TLS=true
-SELFBENCH_TASK_QUEUE=selfbench-production
-SELFBENCH_ARTIFACT_BACKEND=gcs
-SELFBENCH_GCS_BUCKET=your-selfbench-artifacts
-SELFBENCH_GCS_PREFIX=selfbench
+self-bench list                    # find run IDs
+self-bench status RUN_ID
+self-bench cancel RUN_ID
+self-bench download RUN_ID ./self-bench-evals.tar.gz
 ```
 
-Do not use the local Compose defaults (`temporal` database credentials, loopback Temporal address, or local artifact volume) in a hosted deployment.
-
-### 2. Deploy the API
-
-Build and deploy `Dockerfile` as a service listening on port 8080. Configure:
+Stop the local stack with:
 
 ```bash
-SELFBENCH_API_HOST=0.0.0.0
-SELFBENCH_API_TOKEN=use-a-secret-value
-SELFBENCH_ARTIFACT_BACKEND=gcs
-SELFBENCH_GCS_BUCKET=your-selfbench-artifacts
-SELFBENCH_GCS_PREFIX=selfbench
-SELFBENCH_TEMPORAL_ADDRESS=...
-SELFBENCH_TEMPORAL_NAMESPACE=...
-SELFBENCH_TEMPORAL_API_KEY=...
-SELFBENCH_TEMPORAL_TLS=true
-SELFBENCH_TASK_QUEUE=selfbench-production
+self-bench down
 ```
 
-Then point the CLI at the API:
+Named Docker volumes retain Temporal history and generated artifacts.
 
-```bash
-export SELFBENCH_API_URL=https://selfbench-api.example.com
-export SELFBENCH_API_TOKEN=use-a-secret-value
-```
+## Other deployments
 
-### 3. Deploy the worker
+The quickstart is the recommended setup: a local stack with Modal sandboxes.
 
-Run the same image with this command:
+- **Local stack + Docker sandboxes:** use `self-bench up --backend docker` when you want all execution on your machine.
+- **Local stack + Vercel Sandbox:** run `self-bench setup vercel`, then explicitly choose Docker or Modal for Harbor validation.
+- **Temporal Cloud + Modal:** use this for persistent unattended workers and large repositories.
 
-```bash
-node dist/worker-main.js
-```
+See [Operations and deployment](docs/operations.md) for backend configuration, credentials, persistence, object storage, and the complete Temporal Cloud deployment.
 
-Give the worker the Temporal, GCS, and task-queue settings above, plus the credentials it needs to discover, author, validate, and review tasks:
+Generation sandboxes and Harbor validation are independent choices. Docker and Modal retain their matching defaults; Vercel must name a Harbor environment because Harbor does not currently provide a Vercel environment:
 
 ```bash
-SELFBENCH_EXECUTION_BACKEND=modal
-SELFBENCH_HARBOR_ENVIRONMENT=modal
-MODAL_TOKEN_ID=...
-MODAL_TOKEN_SECRET=...
-GH_TOKEN=...
-SELFBENCH_PI_AUTH_JSON=...
-```
-
-Provide the Codex auth JSON to the worker at `/home/node/.codex/auth.json` (the default location in the production image), for example through a read-only secret mount. Alternatively, mount it elsewhere and set `CODEX_AUTH_JSON_PATH` to that file. Keep model credentials in the worker only; the API does not need them.
+# Matching defaults
+self-bench up --backend docker
+self-bench up --backend modal
 
-### 4. Run against the hosted service
+# Vercel generation with either supported Harbor environment
+self-bench setup vercel
+self-bench up --backend vercel --harbor-environment docker
+self-bench up --backend vercel --harbor-environment modal
 
-```bash
-self-bench run \
-  --repo /absolute/path/to/your/repository \
-  --easy-count 2 \
-  --medium-count 2 \
-  --output ./self-bench-evals.tar.gz
+# Explicit cross-provider combinations are also supported
+self-bench up --backend docker --harbor-environment modal
+self-bench up --backend modal --harbor-environment docker
 ```
 
-The CLI uploads only repository metadata and sanitized provenance. The worker performs discovery, authoring, sandbox validation, review, audit, and export remotely.
+Provider selection belongs to the worker, so all runs on one task queue use the same pairing. See [Vercel Sandbox setup](docs/operations.md#vercel-sandbox) for project authentication, publishing the runtime image, limits, cleanup, and troubleshooting.
 
-### Large-repository guidance
+## How tasks are validated
 
-- Prefer Modal over local Docker so the run is not tied to your laptop's CPU, memory, or Docker daemon.
-- Start with one or two candidates per tier to confirm credentials, provenance, and repository compatibility before increasing the budget.
-- Treat counts as authoring attempts, not accepted-task guarantees.
-- Keep the worker alive for the entire run; discovery and authoring are retryable, but a missing worker stops progress.
-- Keep GCS and Temporal state persistent. Do not delete the artifact prefix or Temporal namespace while runs are active.
-- Use a dedicated task queue for each deployment and ensure the API and worker use exactly the same value.
-- Use `status`, `list`, and `download` from any machine that can reach the API.
+An accepted task must:
 
-## Backend options
+1. Preserve a real human request from repository history.
+2. Start from the repository state before the completed change.
+3. Include hidden tests that fail against the base snapshot.
+4. Pass after applying the original implementation.
+5. Survive deterministic reruns and an independent model review that rejects tests tied to private details of the reference solution.
 
-The local `up` command configures one execution backend for the stack:
-
-```bash
-# Local Docker sandboxes; simplest, usually one activity at a time
-self-bench up --backend docker
+Exports contain repository snapshots, hidden tests, and reference solutions. They are sensitive and unencrypted; keep them private.
 
-# Modal sandboxes; better for concurrency and larger runs
-modal token new
-self-bench up --backend modal
-```
+See [Task construction and validation](docs/task-construction.md) for the full acceptance rules and archive format.
 
-For Modal, `self-bench up` uses `~/.modal.toml` by default. Override it with:
+## Development
 
 ```bash
-self-bench up --backend modal --modal-config /absolute/path/to/.modal.toml
+git clone https://github.com/mupt-ai/self-bench.git
+cd self-bench
+bun install --frozen-lockfile
+bun run validate
 ```
 
-## Development
+Run the CLI directly from source:
 
 ```bash
-bun install --frozen-lockfile
-bun run check
-bun run test
-bun run build
-bun run validate
+bun run cli -- --help
 ```
 
 Useful development commands:
 
 ```bash
-bun run cli -- --help
 bun run dev:api
 bun run dev:worker
 bun run dev:review
 ```
 
-More detail is available in:
+## Documentation
 
-- [Task construction](docs/task-construction.md)
-- [Evaluation workflows](docs/evaluations.md)
+- [Task construction and validation](docs/task-construction.md)
+- [Running evaluations](docs/evaluations.md)
 - [Operations and deployment](docs/operations.md)
 
 ## License