self-bench 0.3.5 → 0.3.8

package/docs/api.md

@@ -0,0 +1,168 @@
+# HTTP API reference
+
+Everything the selfbench.dev site can do is available over HTTP. This document lists every route, how to authenticate, and what each call expects. The CLI-facing run routes under `/v1` are shared with the site and are also listed at the end.
+
+All responses are JSON unless noted. Errors carry `{ "error": "message" }` and sometimes a `code`. Paths that contain a repository take the connected repository's `owner/name` as two segments, so `/api/orgs/mupt-ai/repos/mupt-ai/self-bench/tasks` lists the tasks of `mupt-ai/self-bench` inside the `mupt-ai` organization. `:org` is a GitHub organization login, or your own login for the personal account; it is matched case-insensitively.
+
+## Authentication
+
+There are three ways to authenticate. Every `/api` and `/v1` route requires one of them; only `/healthz` and `/v1/viewer` are open.
+
+| Method | Header | Reaches |
+| --- | --- | --- |
+| API key | `Authorization: Bearer sbk_…` or `X-API-Key: sbk_…` | Every `/api` and `/v1` route, as the key's owner |
+| Browser session | `selfbench_session` cookie set by GitHub sign-in | Every `/api` and `/v1` route |
+| Operator token | `Authorization: Bearer $SELFBENCH_API_TOKEN` | `/v1` run routes only (no user identity) |
+
+### API keys
+
+Create keys under **Settings → API Keys** in the site. A key acts as the user who created it: it sees every organization that user belongs to, and organization roles still apply (only organization admins can change credentials, for example). The secret starts with `sbk_` and is shown exactly once; the server stores only a SHA-256 hash of it.
+
+Each key has a scope:
+
+- `write` (default) may call anything the user can.
+- `read` may only send `GET` requests. Any other method is refused with `403 {"error":"this API key is read-only"}`.
+
+Revoke a key from the same page, or with `DELETE /api/api-keys/:id`. A revoked or unknown key answers `401 {"error":"invalid API key","code":"invalid_api_key"}`. Creating and revoking keys requires a browser session; a key cannot mint further keys.
+
+Browser sessions are protected against cross-site requests: state-changing calls from a cookie session must be same-origin JSON requests. API-key requests are exempt, so scripts do not need to send an `Origin` header.
+
+```bash
+export SELFBENCH_API_KEY=sbk_…
+curl -H "Authorization: Bearer $SELFBENCH_API_KEY" https://selfbench.dev/api/me
+curl -H "X-API-Key: $SELFBENCH_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{"fullName":"mupt-ai/self-bench"}' \
+  https://selfbench.dev/api/orgs/mupt-ai/repos
+```
+
+## Account and keys
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/api/me` | The caller: `user`, `orgs` (with `kind` and `role`), `auth` (`session` or `api-key`), and `apiKey` (`name`, `scope`) when a key was used |
+| `GET` | `/api/api-keys` | Active keys of the caller: `id`, `name`, `prefix`, `scope`, `createdAt`, `lastUsedAt` |
+| `POST` | `/api/api-keys` | Body `{ "name": "CI", "scope": "read" \| "write" }`. Answers `201 { key, secret }`; the secret is never returned again. Browser session only |
+| `DELETE` | `/api/api-keys/:id` | Revokes a key. Browser session only |
+
+## GitHub discovery
+
+These call GitHub with the user's stored OAuth token.
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/api/orgs/:org/github-repos` | Repositories the user can see in that tenant, most recently pushed first (`repos`) |
+| `GET` | `/api/github-repos/:owner/:name` | One repository plus `mergedPullRequests` in the last year and the `since` date |
+
+## Connected repositories
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/api/orgs/:org/repos` | Connected repositories (`repos`), newest first |
+| `POST` | `/api/orgs/:org/repos` | Body `{ "fullName": "owner/name" }`. Connects a repository; `201` when new, `200` when already connected. Organizations may connect any public repository but only their own private ones |
+| `GET` | `/api/orgs/:org/repos/:owner/:name` | One connected repository (`repo`) |
+| `PATCH` | `/api/orgs/:org/repos/:owner/:name` | Body `{ "continuous": true \| false }`. Toggles building tasks as pull requests merge |
+| `DELETE` | `/api/orgs/:org/repos/:owner/:name` | Disconnects the repository |
+| `GET` | `/api/orgs/:org/task-counts` | Per-repository `total`, `accepted`, `needsReview`, `rejected`, and `lastPr`, keyed by full name |
+
+A repository object has `fullName`, `defaultBranch`, `private`, `continuous`, `connectedBy`, and `connectedAt`.
+
+## Pull requests and tasks
+
+All paths below are relative to `/api/orgs/:org/repos/:owner/:name`.
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/pull-requests?page=N` | Merged pull requests from GitHub, twenty per page: `pullRequests`, `nextPage`, `incomplete` |
+| `GET` | `/generation-options` | What a generation run may use: `models`, `sandboxes` (hosted providers only, never `docker`), `credentials`, and whether generation is `available` |
+| `POST` | `/tasks/from-pr` | Body `{ "pr": 123 \| "https://github.com/owner/name/pull/123", "generation"?: GenerationSettings }`. Builds one task from a merged pull request; `201 { task }` |
+| `GET` | `/tasks` | Every task of the repository (`tasks`), refreshed against running workflows |
+| `GET` | `/tasks/:runId/:taskId` | One task, by task id or candidate id (`task`) |
+| `DELETE` | `/tasks/:runId/:taskId` | Removes a finished task; `409` while it is still generating |
+| `PUT` | `/tasks/:runId/:taskId/review` | Body `{ "decision": "approve" \| "reject", "note": "…" }`. Records the human verdict (`task`) |
+| `DELETE` | `/tasks/:runId/:taskId/review` | Clears the human verdict (`task`) |
+| `GET` | `/tasks/:runId/:taskId/artifacts` | Pipeline artifact keys grouped by stage plus the task's bundles; read them with the `/v1/runs/:runId/artifacts` and `/bundle` routes |
+
+A task object carries `runId`, `taskId`, `candidateId`, `difficulty`, `stage`, `pipelineStatus`, the derived `state` (`needs_review`, `accepted`, `rejected`, `failed`, `in_progress`), `reason`, `sourcePr`, `sourceUrl`, `review`, `round`, `workflowId`, `startedBy`, `startedAt`, and `syncedAt`.
+
+`GenerationSettings` is `{ authorModel, verifierModel, reasoning: "low" | "medium" | "high", sandbox: "modal" | "vercel" | "e2b", modelCredentialId, sandboxCredentialId, sandboxImage?, harborEnvironment?: "modal" | "vercel" | "e2b" | "daytona", harborCredentialId? }`; the credential ids come from the organization credentials routes. `sandboxImage`, `harborEnvironment`, and `harborCredentialId` are required for `vercel` and `e2b` and rejected for `modal`. The full schema is `generationSettingsSchema` in `src/site/generation-settings.ts`.
+
+## Batches
+
+Relative to `/api/orgs/:org/repos/:owner/:name`.
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/batches` | Batch runs attached to the repository (`batches`: `runId`, `attachedBy`, `attachedAt`) |
+| `POST` | `/batches` | Body `{ "candidateCounts": { "easy": n, "medium": n, "hard": n }, "generation"?: GenerationSettings }`, 1–10000 candidates in total. Starts a batch; `202 { run, runId }` |
+| `GET` | `/batches/:batchId` | Live status of the batch: phase, counts, candidates, and task progress |
+| `POST` | `/batches/:batchId/cancel` | Requests cancellation; `202` |
+
+## Evaluations
+
+Relative to `/api/orgs/:org/repos/:owner/:name`. Single-model evaluation runs use the operator-configured model list; model comparisons (below) use organization credentials.
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/evaluations` | Past and running evaluations (`runs`), trial logs omitted |
+| `GET` | `/evaluations/options` | Configured `models` (each with its `harnesses`), `sandboxes`, `harborVersion`, whether personal setups are `configurable`, and the approved `tasks` that may be evaluated |
+| `POST` | `/evaluations` | Body `{ "id": uuid, "model", "harnesses": [...], "sandbox", "tasks": [{ "runId", "taskId" }] }`. Starts an evaluation; `202` with the run. Repeating the same `id` and selection resumes rather than duplicates; a different selection under a known `id` answers `409` |
+| `GET` | `/evaluations/:id` | One evaluation with every trial's log, steps, and artifact names |
+| `GET` | `/evaluations/:id/artifacts?name=…` | Downloads one trial artifact as text |
+| `POST` | `/evaluations/profiles` | Body per `setupSchema` in `src/evaluation/profiles.ts` (`provider`, `model`, `modelApiKey`, `sandbox`, sandbox credentials, optional `pricing`). Saves a personal model setup; `201` |
+
+## Model comparisons
+
+Relative to `/api/orgs/:org/repos/:owner/:name/evaluations`.
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/catalog` | The model catalog with reference pricing: `version`, `models`, `sandboxes`, `customHosts` |
+| `GET` | `/comparisons` | Comparisons of this repository with live status (`comparisons`) |
+| `POST` | `/comparisons` | Body per `comparisonSchema` in `src/evaluation/comparisons.ts`: `id` (uuid), `tasks`, `models` (`catalogId`, `credentialId`, `harnesses`, optional `customModel` and `thinking`), `sandbox`, and the sandbox credential. Creates and dispatches the comparison; `202`. A `submissionError` field means some runs were not confirmed and should be resumed |
+| `GET` | `/comparisons/:id` | One comparison with per-model run status |
+| `POST` | `/comparisons/:id/resume` | Re-dispatches unconfirmed runs with the same run ids; `202` |
+| `*` | `/credentials…` | Alias of the organization credential routes below |
+
+## Organization credentials
+
+Credentials are shared by everyone in the organization and encrypted at rest. Reads need membership; writes need the `admin` role.
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/api/orgs/:org/credentials` | Stored credentials without their secrets (`credentials`) and whether the caller `canManage` them |
+| `POST` | `/api/orgs/:org/credentials` | Body per `credentialSchema` in `src/evaluation/credentials.ts`: `name`, `kind` (`openai`, `anthropic`, `openrouter`, `custom`, `e2b`, `modal`, `daytona`, `vercel`), `value`, and kind-specific fields such as `endpoint`, `tokenId`, or `teamId`. `201` with the stored credential |
+| `POST` | `/api/orgs/:org/credentials/:id/delete` | Deletes a credential; `400` while a comparison still references it |
+| `POST` | `/api/orgs/:org/credentials/codex-login` | Body `{ "name": "Codex" }`. Starts a ChatGPT device sign-in for Codex; `202` with the session id and instructions |
+| `GET` | `/api/orgs/:org/credentials/codex-login/:id` | Sign-in status |
+| `POST` | `/api/orgs/:org/credentials/codex-login/:id/complete` | Stores the resulting credential once the browser side has approved |
+| `POST` | `/api/orgs/:org/credentials/codex-login/:id/cancel` | Abandons the sign-in |
+
+## Run artifacts and CLI routes
+
+The site's task pages read bundles and raw artifacts through the run routes that the CLI also uses. With a session or API key they are scoped to the caller like every other route; with `SELFBENCH_API_TOKEN` they are the operator's Harbor Ledger API. See [Operations](operations.md#http-api) for run submission details.
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/healthz` | Liveness check (unauthenticated) |
+| `GET` | `/v1/viewer` | Viewer capabilities (unauthenticated) |
+| `GET` | `/v1/runs` | Every run Temporal knows plus archived runs |
+| `POST` | `/v1/runs` | Start a candidate workflow or a replay |
+| `GET` | `/v1/runs/:runId` | Run status |
+| `POST` | `/v1/runs/:runId/cancel` | Request cancellation |
+| `GET` | `/v1/runs/:runId/export` | Download the export archive |
+| `POST` | `/v1/provenance?runId=…` | Store provenance JSONL for a run |
+| `GET` | `/v1/runs/:runId/candidates` | Every candidate of a run |
+| `GET` | `/v1/runs/:runId/candidates/:taskId/artifacts` | Artifact keys for one candidate |
+| `GET` | `/v1/runs/:runId/artifacts?key=…&start=…` | Stream one artifact, optionally from a byte offset |
+| `GET` | `/v1/runs/:runId/bundle?key=…` | Expand a Harbor task bundle into its text files |
+
+## Sign-in routes
+
+Used by the browser only; scripts should use API keys.
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/auth/github` | Redirect to GitHub |
+| `GET` | `/auth/github/callback` | Finish sign-in and set the session cookie |
+| `POST` | `/auth/logout` | Clear the session cookie |

package/docs/evaluations.md

@@ -1,5 +1,20 @@
 # Running self-bench evaluations
 
+## Signed-in workspace
+
+The web workspace keeps each repository's Dataset, Batches, Run, and Results pages together:
+
+- **Dataset** lists tasks by review status, with review and deletion actions. Accepted tasks are selected by default for evaluation.
+- **Batches** contains generation history and individual batch progress. Generate Batch separates candidate counts from model, sandbox, and credential configuration; Add PRs uses the same saved generation settings.
+- **Run** uses **Add Model / Harness** to configure independent model, credential, harness, and thinking selections. Available harnesses depend on the selected connection. Saved ChatGPT credentials require Codex for solver runs; OpenRouter connections support the workspace's cross-provider harness routes.
+- **Results** retains comparisons and individual solver results. A failed submission response is checked against its request ID before the workspace permits another submission, avoiding duplicate runs when the outcome is uncertain.
+
+Manage organization credentials in **Settings**. Generation accepts saved OpenAI API keys or ChatGPT sign-ins. Hosted generation sandboxes are Modal, Vercel, and E2B, each with its own credential; Docker is not offered because it would run sandboxes on the shared worker. E2B and Vercel generation still require their explicit runtime setting plus a Harbor verification environment (Modal, Vercel, E2B, or Daytona) and a matching credential. Vercel sandbox credentials are not Vercel AI Gateway connections.
+
+Model and sandbox execution is billed by the configured providers. The workspace does not start a run merely because a model or credential is selected.
+
+## Exported tasks
+
 A self-bench export contains a manifest and one archive per accepted Harbor task:
 
 ```text
@@ -16,7 +31,7 @@ tasks/
 
 Exports include repository snapshots, held-out tests, and reference solutions. They are sensitive and unencrypted; keep them private.
 
-The provider that generated an export does not constrain where Harbor runs it. Every task has the same provider-neutral Harbor layout, and `manifest.json` records the generation backend and the Harbor environment used for self-bench's nop/oracle gates. For later coding-agent trials, independently choose `--environment docker` or `--environment modal`. Harbor does not currently provide a Vercel environment; Vercel Sandbox is used only during benchmark generation.
+The provider that generated an export does not constrain where Harbor runs it. Every task has the same provider-neutral Harbor layout, and `manifest.json` records the generation backend and the Harbor environment used for self-bench's nop/oracle gates. For later coding-agent trials, independently choose `--environment docker`, `modal`, `vercel`, `e2b`, or `daytona`.
 
 ## Run one task
 
@@ -33,7 +48,7 @@ tar -xzf "./export/tasks/$TASK_ID.tar.gz" -C ./selected-task
 Install Harbor and run Codex against it with API-key authentication:
 
 ```bash
-uv tool install --python 3.12 'harbor[modal]==0.20.1.dev202608040148'
+uv tool install --python 3.12 'harbor[modal]==0.23.0'
 
 export OPENAI_API_KEY=...
 harbor run \
@@ -49,22 +64,34 @@ harbor run \
 
 The `solution/` directory is used only for explicit oracle validation and is never mounted for coding-agent trials.
 
-## Run the model matrix
-
-The included matrix runner evaluates every exported task with `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna`, all at high reasoning. It reuses completed Harbor jobs after a restart.
+## Run multiple tasks or models
 
-From the self-bench checkout:
+Use Harbor directly for batch and model-comparison runs. Extract each task into a directory containing its `harbor-task` contents:
 
 ```bash
-export OPENAI_API_KEY=...
-node dist/eval-main.js \
-  --export ./my-evals.tar.gz \
-  --jobs ./matrix-jobs \
-  --harbor harbor \
-  --environment modal \
-  --concurrency 20
+mkdir -p ./self-bench-tasks
+for archive in ./export/tasks/*.tar.gz; do
+  task_id="$(basename "$archive" .tar.gz)"
+  mkdir -p "./self-bench-tasks/$task_id"
+  tar -xzf "$archive" --strip-components=1 -C "./self-bench-tasks/$task_id"
+done
 ```
 
-The runner writes per-task Harbor jobs and a combined `./matrix-jobs/summary.json`.
+Then pass the directory and desired models to `harbor run`:
+
+```bash
+harbor run \
+  --path ./self-bench-tasks \
+  --agent codex \
+  --model gpt-5.6-luna \
+  --model gpt-5.6-terra \
+  --model gpt-5.6-sol \
+  --ak version=0.146.1 \
+  --ak reasoning_effort=high \
+  --env modal \
+  --jobs-dir ./harbor-jobs \
+  --n-concurrent 20 \
+  --yes
+```
 
-ChatGPT subscription authentication remains supported for existing deployments: omit `OPENAI_API_KEY` and pass `--auth /path/to/codex-auth.json`.
+Harbor owns agent authentication, concurrency, retries, result persistence, and reporting. Consult Harbor's agent documentation when using Codex subscription authentication or a different agent adapter.