self-bench 0.3.0 → 0.3.3

package/.dockerignore

@@ -0,0 +1,9 @@
+.git
+.github
+.selfbench
+.venv
+dist
+jobs
+node_modules
+tests
+*.md

package/Dockerfile

@@ -0,0 +1,38 @@
+FROM oven/bun:1.3.14-debian AS build
+
+WORKDIR /app
+ARG SELFBENCH_BUILD_COMMIT
+COPY package.json bun.lock ./
+RUN bun install --frozen-lockfile
+COPY biome.json tsconfig.json tsconfig.build.json ./
+COPY src ./src
+COPY review ./review
+COPY scripts ./scripts
+RUN bun run build
+
+FROM docker:29.4.0-cli AS docker-cli
+
+FROM node:22-bookworm
+
+ARG HARBOR_VERSION=0.20.1.dev202608040148
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends bash ca-certificates curl gh git jq ripgrep tar \
+  && rm -rf /var/lib/apt/lists/* \
+  && curl -LsSf https://astral.sh/uv/0.11.3/install.sh | env UV_INSTALL_DIR=/usr/local/bin UV_NO_MODIFY_PATH=1 sh \
+  && env UV_PYTHON_INSTALL_DIR=/opt/uv-python UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin uv tool install --python 3.12 "harbor[modal]==${HARBOR_VERSION}" \
+  && chmod -R a+rX /opt/uv-python /opt/uv-tools
+COPY --from=docker-cli /usr/local/bin/docker /usr/local/bin/docker
+COPY --from=docker-cli /usr/local/libexec/docker/cli-plugins /usr/local/libexec/docker/cli-plugins
+
+ENV NODE_ENV=production
+ENV PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin
+WORKDIR /app
+RUN mkdir -p /var/lib/selfbench/artifacts && chown -R node:node /var/lib/selfbench
+COPY --from=build /app/node_modules ./node_modules
+COPY --from=build /app/dist ./dist
+COPY src/extensions ./src/extensions
+COPY src/skills ./src/skills
+COPY package.json ./package.json
+
+USER node
+CMD ["node", "dist/api-main.js"]

package/Dockerfile.sandbox

@@ -0,0 +1,24 @@
+FROM node:22-bookworm
+
+ARG PI_VERSION=0.84.0
+ARG CODEX_VERSION=0.146.1
+ARG GH_VERSION=2.89.0
+ARG TARGETARCH
+
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends bash ca-certificates curl git jq ripgrep unzip xz-utils \
+  && rm -rf /var/lib/apt/lists/*
+RUN case "${TARGETARCH}" in \
+    amd64) GH_SHA256=d0422caade520530e76c1c558da47daebaa8e1203d6b7ff10ad7d6faba3490d8 ;; \
+    arm64) GH_SHA256=9e64a623dfc242990aa5d9b3f507111149c4282f66b68eaad1dc79eeb13b9ce5 ;; \
+    *) echo "unsupported architecture: ${TARGETARCH}" >&2; exit 1 ;; \
+  esac \
+  && curl -fsSL "https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_${TARGETARCH}.tar.gz" -o /tmp/gh.tar.gz \
+  && echo "${GH_SHA256}  /tmp/gh.tar.gz" | sha256sum -c - \
+  && tar -xzf /tmp/gh.tar.gz -C /tmp \
+  && mv "/tmp/gh_${GH_VERSION}_linux_${TARGETARCH}/bin/gh" /usr/local/bin/gh \
+  && rm -rf /tmp/gh.tar.gz "/tmp/gh_${GH_VERSION}_linux_${TARGETARCH}"
+RUN npm install --global --ignore-scripts "@earendil-works/pi-coding-agent@${PI_VERSION}"
+RUN npm install --global "@openai/codex@${CODEX_VERSION}"
+
+WORKDIR /work

package/README.md

@@ -1,254 +1,186 @@
-# SelfBench
+# self-bench
 
-SelfBench turns completed GitHub changes into private [Harbor](https://harborframework.com/) evaluations for coding agents. It finds real feature requests, builds tasks from the repository's base commit, hides the tests and reference solution, and checks that each task fails without a solution and passes with one.
+[![npm version](https://img.shields.io/npm/v/self-bench?color=blue&label=npm)](https://www.npmjs.com/package/self-bench)
+[![CI](https://github.com/mupt-ai/self-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/mupt-ai/self-bench/actions/workflows/ci.yml)
+[![license](https://img.shields.io/github/license/mupt-ai/self-bench?color=green)](./LICENSE)
+[![Bun](https://img.shields.io/badge/runtime-Bun-f9f1e1?logo=bun&logoColor=000)](https://bun.sh/)
+[![TypeScript](https://img.shields.io/badge/lang-TypeScript-3178c6?logo=typescript&logoColor=fff)](https://www.typescriptlang.org/)
 
-SelfBench is a TypeScript package. Use **Bun** to install dependencies, build the package, and run the development CLI.
+**self-bench builds private coding-agent benchmarks from work already completed in your repository, so you can compare coding agents and models on tasks drawn from your own codebase.**
 
-## Choose your setup
+It finds completed requests from local coding sessions and merged GitHub pull requests, then reconstructs each task from the commit before the change. For every accepted task, self-bench creates hidden tests and a reference solution, proves that the task fails without a solution and passes with the original implementation, and exports a native task for [Harbor](https://harborframework.com/), a runner for coding-agent evaluations.
 
-| Setup | Best for | Execution | State |
-| --- | --- | --- | --- |
-| Local | Trying SelfBench or small runs | Docker on your machine | Local Docker volumes |
-| Temporal Cloud | Large repositories, long runs, and reliable unattended execution | Modal sandboxes | Temporal Cloud + object storage |
+The result is a private `.tar.gz` benchmark that you can run against multiple models:
 
-After a run is submitted, the workflow continues independently of the waiting CLI process. The local worker still depends on your machine and its Docker stack; Temporal Cloud is the recommended setup when a run may take hours or your laptop should not be responsible for the worker.
+```text
+Your repository history
+        ↓
+completed requests + implementations
+        ↓
+validated Harbor tasks with hidden tests
+        ↓
+gpt-5.6-luna vs gpt-5.6-terra vs gpt-5.6-sol
+```
 
-## Prerequisites
+## Quickstart
 
-All setups require:
+This path runs the self-bench API, worker, and [Temporal](https://temporal.io/) workflow state locally while using [Modal](https://modal.com/) for disposable task-generation and validation sandboxes.
+
+### Prerequisites
 
 - [Bun](https://bun.sh/) 1.3.14 or newer
 - Docker with Compose
-- Node.js 22 or newer (the built service images use Node)
-- `gh`, authenticated with access to the source repository
-- Pi with an authenticated `openai-codex` account
-- Codex CLI with an authenticated account
-
-Modal execution additionally requires a Modal account and token. Temporal Cloud execution additionally requires a Temporal Cloud namespace and a durable artifact store such as Google Cloud Storage.
-
-## Install from source
-
-The package is not currently published to npm. Clone the repository and build it locally:
-
-```bash
-git clone https://github.com/mupt-ai/self-bench.git
-cd self-bench
-bun install --frozen-lockfile
-bun run build
-```
+- [Modal](https://modal.com/) CLI, account, and token (`pip install modal`)
+- [`gh`](https://cli.github.com/), authenticated with read access to the repository
+- An OpenAI API key with access to `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna`
+- A Git checkout with a GitHub `origin` and completed work in its history
 
-The compiled CLI is `dist/cli.js`, and the package exposes it as `selfbench`. To use `selfbench` directly from this checkout:
+Install self-bench and authenticate Modal and GitHub:
 
 ```bash
-bun link
-selfbench --help
+bun add --global self-bench
+modal token new
+gh auth login
 ```
 
-During development, run the TypeScript entrypoint without linking:
+Set the model and GitHub credentials used by the local worker, plus a random token that protects the local self-bench API:
 
 ```bash
-bun run cli -- --help
+export OPENAI_API_KEY=...
+export GH_TOKEN="$(gh auth token)"
+export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)"
 ```
 
-Authenticate the tools before starting a run:
+### 1. Start self-bench
 
 ```bash
-gh auth login
-# In Pi: /login -> OpenAI Codex
-codex login
+self-bench up --backend modal
+export SELFBENCH_API_URL=http://127.0.0.1:8080
 ```
 
-## Quick start: local Docker
+This starts Postgres, Temporal, the self-bench API, and a worker in Docker. The worker sends sandbox work to Modal; `SELFBENCH_API_URL` tells subsequent CLI commands where to reach the local API.
 
-This starts Postgres, Temporal, the SelfBench API, and a worker on your machine. The worker runs sandbox validation through Docker.
+### 2. Build a benchmark
 
 ```bash
-export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)"
-export GH_TOKEN="$(gh auth token)"
-
-selfbench up --backend docker
-export SELFBENCH_API_URL=http://127.0.0.1:8080
-
-selfbench run \
+self-bench run \
   --repo /absolute/path/to/your/repository \
-  --easy-count 30 \
-  --medium-count 30 \
+  --easy-count 10 \
+  --medium-count 10 \
   --hard-count 10 \
-  --output ./selfbench-evals.tar.gz
+  --output ./self-bench-evals.tar.gz
 ```
 
-The `--repo` directory must be a Git checkout with a GitHub `origin`. SelfBench pins the current `HEAD`; uncommitted changes are ignored. It discovers candidate requests from sanitized local coding-session messages and merged, non-bot GitHub pull requests.
+Easy, medium, and hard candidates require at least 20, 50, and 100 changed implementation lines across 1, 2, and 3 paths respectively; the counts are generation budgets, not guarantees that every candidate will pass validation.
 
-This quick start authors 70 candidates: 30 easy, 30 medium, and 10 hard. Counts are authoring budgets, not accepted-task guarantees. Candidates can be rejected during authoring, validation, audit, or review, and rejected candidates are not replaced.
+The repository must be a Git checkout with a GitHub `origin`. self-bench pins its current `HEAD`, ignores uncommitted changes, and may take hours to author, validate, review, and export the accepted tasks. `--output` waits for completion and verifies the downloaded archive with SHA-256.
 
-A run may take hours. `--output` waits for the run to finish, then downloads and SHA-256-verifies the accepted tasks.
-
-## Local commands
-
-```bash
-selfbench status RUN_ID
-selfbench list
-selfbench cancel RUN_ID
-selfbench download RUN_ID ./selfbench-evals.tar.gz
-```
-
-If you are working from source rather than using `bun link`, use the same commands through Bun:
-
-```bash
-bun run cli -- status RUN_ID
-```
+### 3. Compare models with Harbor
 
-Stop the local stack with Docker Compose:
+Install Harbor and extract the generated tasks:
 
 ```bash
-docker compose down
-```
-
-Named Docker volumes retain Temporal history and generated artifacts. Back them up before removing them if you need to resume or inspect old runs.
-
-## Reliable setup for larger repositories
-
-For large repositories or unattended runs, use Temporal Cloud for workflow state and Modal for disposable execution sandboxes. Temporal Cloud does not host the SelfBench API or worker; deploy those separately.
-
-```text
-selfbench CLI
-    -> SelfBench API
-    -> Temporal Cloud namespace
-    -> persistent SelfBench worker
-    -> Modal sandboxes
-    -> GCS artifact storage
-```
-
-The API and worker must use the same image version, Temporal namespace, task queue, artifact configuration, and `SELFBENCH_BUILD_COMMIT`. Keep the worker running on a long-lived container service; do not run it on a scale-to-zero request service. The API can run as a normal HTTP service.
+uv tool install --python 3.12 'harbor[modal]==0.20.1.dev202608040148'
 
-### 1. Prepare Temporal Cloud and GCS
-
-Create a Temporal Cloud namespace and obtain its address, namespace, API key, and TLS settings. Create a GCS bucket for SelfBench artifacts and give the worker/API service accounts access only to the SelfBench prefix.
-
-Set these values in the API and worker environments:
-
-```bash
-SELFBENCH_TEMPORAL_ADDRESS=your-namespace.tmprl.cloud:7233
-SELFBENCH_TEMPORAL_NAMESPACE=your-namespace
-SELFBENCH_TEMPORAL_API_KEY=...
-SELFBENCH_TEMPORAL_TLS=true
-SELFBENCH_TASK_QUEUE=selfbench-production
-SELFBENCH_ARTIFACT_BACKEND=gcs
-SELFBENCH_GCS_BUCKET=your-selfbench-artifacts
-SELFBENCH_GCS_PREFIX=selfbench
+mkdir -p ./self-bench-export ./self-bench-tasks
+tar -xzf ./self-bench-evals.tar.gz -C ./self-bench-export
+for archive in ./self-bench-export/tasks/*.tar.gz; do
+  task_id="$(basename "$archive" .tar.gz)"
+  mkdir -p "./self-bench-tasks/$task_id"
+  tar -xzf "$archive" --strip-components=1 -C "./self-bench-tasks/$task_id"
+done
 ```
 
-Do not use the local Compose defaults (`temporal` database credentials, loopback Temporal address, or local artifact volume) in a hosted deployment.
-
-### 2. Deploy the API
-
-Build and deploy `Dockerfile` as a service listening on port 8080. Configure:
+Run Harbor's Codex agent adapter once for all three models. Harbor evaluates every extracted task at high reasoning and keeps the model results in one job directory:
 
 ```bash
-SELFBENCH_API_HOST=0.0.0.0
-SELFBENCH_API_TOKEN=use-a-secret-value
-SELFBENCH_ARTIFACT_BACKEND=gcs
-SELFBENCH_GCS_BUCKET=your-selfbench-artifacts
-SELFBENCH_GCS_PREFIX=selfbench
-SELFBENCH_TEMPORAL_ADDRESS=...
-SELFBENCH_TEMPORAL_NAMESPACE=...
-SELFBENCH_TEMPORAL_API_KEY=...
-SELFBENCH_TEMPORAL_TLS=true
-SELFBENCH_TASK_QUEUE=selfbench-production
+harbor run \
+  --path ./self-bench-tasks \
+  --agent codex \
+  --model gpt-5.6-luna \
+  --model gpt-5.6-terra \
+  --model gpt-5.6-sol \
+  --ak version=0.146.1 \
+  --ak reasoning_effort=high \
+  --env modal \
+  --jobs-dir ./harbor-jobs \
+  --n-concurrent 20 \
+  --yes
 ```
 
-Then point the CLI at the API:
+The evaluated agent receives the base repository and task instruction, but not the hidden tests or reference solution.
 
-```bash
-export SELFBENCH_API_URL=https://selfbench-api.example.com
-export SELFBENCH_API_TOKEN=use-a-secret-value
-```
+See [Running self-bench evaluations](docs/evaluations.md) for running one task and using the optional resumable matrix helper.
 
-### 3. Deploy the worker
+## Run management
 
-Run the same image with this command:
+Closing the waiting CLI does not cancel a submitted workflow. If the local worker or Docker stack stops, work pauses until the worker is restarted.
 
 ```bash
-node dist/worker-main.js
+self-bench list                    # find run IDs
+self-bench status RUN_ID
+self-bench cancel RUN_ID
+self-bench download RUN_ID ./self-bench-evals.tar.gz
 ```
 
-Give the worker the Temporal, GCS, and task-queue settings above, plus the credentials it needs to discover, author, validate, and review tasks:
+Stop the local stack with:
 
 ```bash
-SELFBENCH_EXECUTION_BACKEND=modal
-SELFBENCH_HARBOR_ENVIRONMENT=modal
-MODAL_TOKEN_ID=...
-MODAL_TOKEN_SECRET=...
-GH_TOKEN=...
-SELFBENCH_PI_AUTH_JSON=...
+self-bench down
 ```
 
-Provide the Codex auth JSON to the worker at `/home/node/.codex/auth.json` (the default location in the production image), for example through a read-only secret mount. Alternatively, mount it elsewhere and set `CODEX_AUTH_JSON_PATH` to that file. Keep model credentials in the worker only; the API does not need them.
+Named Docker volumes retain Temporal history and generated artifacts.
 
-### 4. Run against the hosted service
+## Other deployments
 
-```bash
-selfbench run \
-  --repo /absolute/path/to/your/repository \
-  --easy-count 2 \
-  --medium-count 2 \
-  --output ./selfbench-evals.tar.gz
-```
+The quickstart is the recommended setup: a local stack with Modal sandboxes.
 
-The CLI uploads only repository metadata and sanitized provenance. The worker performs discovery, authoring, sandbox validation, review, audit, and export remotely.
+- **Local stack + Docker sandboxes:** use `self-bench up --backend docker` when you want all execution on your machine.
+- **Temporal Cloud + Modal:** use this for persistent unattended workers and large repositories.
 
-### Large-repository guidance
+See [Operations and deployment](docs/operations.md) for backend configuration, credentials, persistence, object storage, and the complete Temporal Cloud deployment.
 
-- Prefer Modal over local Docker so the run is not tied to your laptop's CPU, memory, or Docker daemon.
-- Start with one or two candidates per tier to confirm credentials, provenance, and repository compatibility before increasing the budget.
-- Treat counts as authoring attempts, not accepted-task guarantees.
-- Keep the worker alive for the entire run; discovery and authoring are retryable, but a missing worker stops progress.
-- Keep GCS and Temporal state persistent. Do not delete the artifact prefix or Temporal namespace while runs are active.
-- Use a dedicated task queue for each deployment and ensure the API and worker use exactly the same value.
-- Use `status`, `list`, and `download` from any machine that can reach the API.
+## How tasks are validated
 
-## Backend options
+An accepted task must:
 
-The local `up` command configures one execution backend for the stack:
+1. Preserve a real human request from repository history.
+2. Start from the repository state before the completed change.
+3. Include hidden tests that fail against the base snapshot.
+4. Pass after applying the original implementation.
+5. Survive deterministic reruns and an independent model review that rejects tests tied to private details of the reference solution.
 
-```bash
-# Local Docker sandboxes; simplest, usually one activity at a time
-selfbench up --backend docker
+Exports contain repository snapshots, hidden tests, and reference solutions. They are sensitive and unencrypted; keep them private.
 
-# Modal sandboxes; better for concurrency and larger runs
-modal token new
-selfbench up --backend modal
-```
+See [Task construction and validation](docs/task-construction.md) for the full acceptance rules and archive format.
 
-For Modal, `selfbench up` uses `~/.modal.toml` by default. Override it with:
+## Development
 
 ```bash
-selfbench up --backend modal --modal-config /absolute/path/to/.modal.toml
+git clone https://github.com/mupt-ai/self-bench.git
+cd self-bench
+bun install --frozen-lockfile
+bun run validate
 ```
 
-## Development
+Run the CLI directly from source:
 
 ```bash
-bun install --frozen-lockfile
-bun run check
-bun run test
-bun run build
-bun run validate
+bun run cli -- --help
 ```
 
 Useful development commands:
 
 ```bash
-bun run cli -- --help
 bun run dev:api
 bun run dev:worker
 bun run dev:review
 ```
 
-More detail is available in:
+## Documentation
 
-- [Task construction](docs/task-construction.md)
-- [Evaluation workflows](docs/evaluations.md)
+- [Task construction and validation](docs/task-construction.md)
+- [Running evaluations](docs/evaluations.md)
 - [Operations and deployment](docs/operations.md)
 
 ## License

package/biome.json

@@ -0,0 +1,18 @@
+{
+  "$schema": "https://biomejs.dev/schemas/2.5.7/schema.json",
+  "files": {
+    "includes": ["src/**/*.ts", "tests/**/*.ts", "review/src/**/*.{ts,tsx}"]
+  },
+  "formatter": {
+    "enabled": true,
+    "indentStyle": "space",
+    "indentWidth": 2,
+    "lineWidth": 100
+  },
+  "linter": {
+    "enabled": true,
+    "rules": {
+      "preset": "recommended"
+    }
+  }
+}