self-bench 0.3.0 → 0.3.2

package/.dockerignore

@@ -0,0 +1,10 @@
+.git
+.github
+.selfbench
+.venv
+dist
+jobs
+node_modules
+review
+tests
+*.md

package/Dockerfile

@@ -0,0 +1,37 @@
+FROM oven/bun:1.3.14-debian AS build
+
+WORKDIR /app
+ARG SELFBENCH_BUILD_COMMIT
+COPY package.json bun.lock ./
+RUN bun install --frozen-lockfile
+COPY biome.json tsconfig.json tsconfig.build.json ./
+COPY src ./src
+COPY scripts ./scripts
+RUN bun run build:server
+
+FROM docker:29.4.0-cli AS docker-cli
+
+FROM node:22-bookworm
+
+ARG HARBOR_VERSION=0.20.1.dev202608040148
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends bash ca-certificates curl gh git jq ripgrep tar \
+  && rm -rf /var/lib/apt/lists/* \
+  && curl -LsSf https://astral.sh/uv/0.11.3/install.sh | env UV_INSTALL_DIR=/usr/local/bin UV_NO_MODIFY_PATH=1 sh \
+  && env UV_PYTHON_INSTALL_DIR=/opt/uv-python UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin uv tool install --python 3.12 "harbor[modal]==${HARBOR_VERSION}" \
+  && chmod -R a+rX /opt/uv-python /opt/uv-tools
+COPY --from=docker-cli /usr/local/bin/docker /usr/local/bin/docker
+COPY --from=docker-cli /usr/local/libexec/docker/cli-plugins /usr/local/libexec/docker/cli-plugins
+
+ENV NODE_ENV=production
+ENV PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin
+WORKDIR /app
+RUN mkdir -p /var/lib/selfbench/artifacts && chown -R node:node /var/lib/selfbench
+COPY --from=build /app/node_modules ./node_modules
+COPY --from=build /app/dist ./dist
+COPY src/extensions ./src/extensions
+COPY src/skills ./src/skills
+COPY package.json ./package.json
+
+USER node
+CMD ["node", "dist/api-main.js"]

package/Dockerfile.sandbox

@@ -0,0 +1,24 @@
+FROM node:22-bookworm
+
+ARG PI_VERSION=0.84.0
+ARG CODEX_VERSION=0.146.1
+ARG GH_VERSION=2.89.0
+ARG TARGETARCH
+
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends bash ca-certificates curl git jq ripgrep unzip xz-utils \
+  && rm -rf /var/lib/apt/lists/*
+RUN case "${TARGETARCH}" in \
+    amd64) GH_SHA256=d0422caade520530e76c1c558da47daebaa8e1203d6b7ff10ad7d6faba3490d8 ;; \
+    arm64) GH_SHA256=9e64a623dfc242990aa5d9b3f507111149c4282f66b68eaad1dc79eeb13b9ce5 ;; \
+    *) echo "unsupported architecture: ${TARGETARCH}" >&2; exit 1 ;; \
+  esac \
+  && curl -fsSL "https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_${TARGETARCH}.tar.gz" -o /tmp/gh.tar.gz \
+  && echo "${GH_SHA256}  /tmp/gh.tar.gz" | sha256sum -c - \
+  && tar -xzf /tmp/gh.tar.gz -C /tmp \
+  && mv "/tmp/gh_${GH_VERSION}_linux_${TARGETARCH}/bin/gh" /usr/local/bin/gh \
+  && rm -rf /tmp/gh.tar.gz "/tmp/gh_${GH_VERSION}_linux_${TARGETARCH}"
+RUN npm install --global --ignore-scripts "@earendil-works/pi-coding-agent@${PI_VERSION}"
+RUN npm install --global "@openai/codex@${CODEX_VERSION}"
+
+WORKDIR /work

package/README.md

@@ -2,7 +2,7 @@
 
 SelfBench turns completed GitHub changes into private [Harbor](https://harborframework.com/) evaluations for coding agents. It finds real feature requests, builds tasks from the repository's base commit, hides the tests and reference solution, and checks that each task fails without a solution and passes with one.
 
-SelfBench is a TypeScript package. Use **Bun** to install dependencies, build the package, and run the development CLI.
+SelfBench is distributed as the [`self-bench`](https://www.npmjs.com/package/self-bench) package and installed with **Bun**.
 
 ## Choose your setup
 
@@ -15,33 +15,41 @@ After a run is submitted, the workflow continues independently of the waiting CL
 
 ## Prerequisites
 
-All setups require:
+Installing the package requires [Bun](https://bun.sh/) 1.3.14 or newer. Running evaluations additionally requires:
 
-- [Bun](https://bun.sh/) 1.3.14 or newer
 - Docker with Compose
-- Node.js 22 or newer (the built service images use Node)
 - `gh`, authenticated with access to the source repository
 - Pi with an authenticated `openai-codex` account
 - Codex CLI with an authenticated account
 
 Modal execution additionally requires a Modal account and token. Temporal Cloud execution additionally requires a Temporal Cloud namespace and a durable artifact store such as Google Cloud Storage.
 
-## Install from source
+## Install
 
-The package is not currently published to npm. Clone the repository and build it locally:
+Install the CLI from the npm registry with Bun:
 
 ```bash
-git clone https://github.com/mupt-ai/self-bench.git
-cd self-bench
-bun install --frozen-lockfile
-bun run build
+bun add --global self-bench
+self-bench --help
 ```
 
-The compiled CLI is `dist/cli.js`, and the package exposes it as `selfbench`. To use `selfbench` directly from this checkout:
+For one-off use without a global install:
 
 ```bash
+bunx self-bench --help
+```
+
+The package exposes `dist/cli.js` as `self-bench`.
+
+To develop SelfBench itself, clone the repository and use Bun:
+
+```bash
+git clone https://github.com/mupt-ai/self-bench.git
+cd self-bench
+bun install --frozen-lockfile
+bun run build
 bun link
-selfbench --help
+self-bench --help
 ```
 
 During development, run the TypeScript entrypoint without linking:
@@ -66,15 +74,15 @@ This starts Postgres, Temporal, the SelfBench API, and a worker on your machine.
 export SELFBENCH_API_TOKEN="$(openssl rand -hex 24)"
 export GH_TOKEN="$(gh auth token)"
 
-selfbench up --backend docker
+self-bench up --backend docker
 export SELFBENCH_API_URL=http://127.0.0.1:8080
 
-selfbench run \
+self-bench run \
   --repo /absolute/path/to/your/repository \
   --easy-count 30 \
   --medium-count 30 \
   --hard-count 10 \
-  --output ./selfbench-evals.tar.gz
+  --output ./self-bench-evals.tar.gz
 ```
 
 The `--repo` directory must be a Git checkout with a GitHub `origin`. SelfBench pins the current `HEAD`; uncommitted changes are ignored. It discovers candidate requests from sanitized local coding-session messages and merged, non-bot GitHub pull requests.
@@ -86,10 +94,10 @@ A run may take hours. `--output` waits for the run to finish, then downloads and
 ## Local commands
 
 ```bash
-selfbench status RUN_ID
-selfbench list
-selfbench cancel RUN_ID
-selfbench download RUN_ID ./selfbench-evals.tar.gz
+self-bench status RUN_ID
+self-bench list
+self-bench cancel RUN_ID
+self-bench download RUN_ID ./self-bench-evals.tar.gz
 ```
 
 If you are working from source rather than using `bun link`, use the same commands through Bun:
@@ -111,7 +119,7 @@ Named Docker volumes retain Temporal history and generated artifacts. Back them
 For large repositories or unattended runs, use Temporal Cloud for workflow state and Modal for disposable execution sandboxes. Temporal Cloud does not host the SelfBench API or worker; deploy those separately.
 
 ```text
-selfbench CLI
+self-bench CLI
     -> SelfBench API
     -> Temporal Cloud namespace
     -> persistent SelfBench worker
@@ -188,11 +196,11 @@ Provide the Codex auth JSON to the worker at `/home/node/.codex/auth.json` (the
 ### 4. Run against the hosted service
 
 ```bash
-selfbench run \
+self-bench run \
   --repo /absolute/path/to/your/repository \
   --easy-count 2 \
   --medium-count 2 \
-  --output ./selfbench-evals.tar.gz
+  --output ./self-bench-evals.tar.gz
 ```
 
 The CLI uploads only repository metadata and sanitized provenance. The worker performs discovery, authoring, sandbox validation, review, audit, and export remotely.
@@ -213,17 +221,17 @@ The local `up` command configures one execution backend for the stack:
 
 ```bash
 # Local Docker sandboxes; simplest, usually one activity at a time
-selfbench up --backend docker
+self-bench up --backend docker
 
 # Modal sandboxes; better for concurrency and larger runs
 modal token new
-selfbench up --backend modal
+self-bench up --backend modal
 ```
 
-For Modal, `selfbench up` uses `~/.modal.toml` by default. Override it with:
+For Modal, `self-bench up` uses `~/.modal.toml` by default. Override it with:
 
 ```bash
-selfbench up --backend modal --modal-config /absolute/path/to/.modal.toml
+self-bench up --backend modal --modal-config /absolute/path/to/.modal.toml
 ```
 
 ## Development

package/biome.json

@@ -0,0 +1,18 @@
+{
+  "$schema": "https://biomejs.dev/schemas/2.5.7/schema.json",
+  "files": {
+    "includes": ["src/**/*.ts", "tests/**/*.ts", "review/src/**/*.{ts,tsx}"]
+  },
+  "formatter": {
+    "enabled": true,
+    "indentStyle": "space",
+    "indentWidth": 2,
+    "lineWidth": 100
+  },
+  "linter": {
+    "enabled": true,
+    "rules": {
+      "preset": "recommended"
+    }
+  }
+}