self-bench 0.3.5 → 0.3.8

package/docs/task-construction.md

@@ -1,6 +1,6 @@
 # Task construction and validation
 
-SelfBench creates easy, medium, and hard Harbor evaluations from completed pull requests. This document defines the task boundary, validation gates, repair behavior, and export contents.
+SelfBench creates easy, medium, and hard Harbor evaluations from completed pull requests. This document defines the task boundary, the authoring and verification rounds, and export contents.
 
 ## Terms
 
@@ -17,7 +17,9 @@ SelfBench creates easy, medium, and hard Harbor evaluations from completed pull
 
 Each task is grounded in one retained human request. SelfBench prefers exact user messages from local Pi, Claude Code, or Codex sessions associated with the repository or its worktrees. For third-party repositories, an exact GitHub pull-request title and optional body from a non-bot author is also valid.
 
-GitHub records are stored with a distinct `github-pull-request` source type, canonical URL, and PR number. Discovery cannot pair the request from one PR with another PR. A model-authored benchmark instruction may restate the retained request but may not add behavior inferred only from the implementation or tests.
+GitHub records are stored with a distinct `github-pull-request` source type, canonical URL, and PR number. Discovery cannot pair the request from one PR with another PR. A local message may also carry an explicit PR binding supplied through `self-bench associate`; materialization must then reject every unbound local message for that PR, as well as omitting the GitHub title/body fallback. Association manifests contain only message identities and hashes of SelfBench's sanitized, whitespace-normalized content, while the existing provenance artifact retains that exact normalized text for the task. No model creates associations or request text.
+
+A model-authored benchmark instruction may restate the retained request but may not add behavior inferred only from the implementation or tests.
 
 ## Difficulty gates
 
@@ -29,9 +31,9 @@ Profiles are eligibility rules, not empirical claims about model success:
 | medium | at least 50 changed lines across 2 implementation paths | at least 1 | at least 1 |
 | hard | at least 100 changed lines across 3 implementation paths | at least 1 | at least 2 |
 
-Every accepted task also requires a held-out test patch with no file overlap with the reference patch, deterministic repository-native setup and tests, a passing nop/oracle split, and independent anti-coupling review.
+Every accepted task also requires a held-out test patch with no file overlap with the reference patch, deterministic repository-native setup and tests, a passing smoke/nop/oracle split, and acceptance by the independent verification agent.
 
-The size gate is mechanical. Generated or vendored code suitability remains a review judgment. Git LFS, submodules, generated changes, and service-heavy integration suites receive no special path and may be rejected during authoring or validation.
+The size gate is mechanical. Generated or vendored code suitability remains a verification judgment. Git LFS, submodules, generated changes, and service-heavy integration suites receive no special path and may be rejected during authoring or verification.
 
 ## Agent-visible boundary
 
@@ -43,23 +45,33 @@ An evaluated coding agent receives:
 
 It does not receive the held-out test patch or reference solution. Harbor mounts `solution/` only for the explicit oracle agent.
 
+The agent image commits the tree as it stands after `setup.sh` (the `selfbench-setup` commit) and snapshots that repository state to `/opt/selfbench/base.git`. The agent's diff (`agent.patch`) is therefore taken against the post-setup snapshot: files that setup creates and does not gitignore are never part of the agent's patch and cannot collide with the verifier image, which runs the same setup. The verifier image keeps the base commit as `HEAD`; when the gold patch changes dependency manifests it resets only those manifest paths after its extra setup pass.
+
 Held-out tests must exercise an existing public API, command, persistence boundary, or extension seam. They may not import gold-specific private helpers or prescribe exact internal SQL, query counts, private schemas, object identity, telemetry layout, incidental error wording, or UI composition unless the source request explicitly makes that artifact public.
 
-## Validation and repair
+## Authoring rounds and verification
+
+Every candidate passes through two agent-centred loops. Agents do the authoring and judging; the harness only renders, builds, and measures.
 
-Static audit runs before sandbox validation. Harbor then proves:
+**Authoring rounds.** One authoring agent session owns the complete Harbor task: instruction, definition (test command, fail-to-pass and pass-to-pass selection, timeouts, resources) and the environment contract (base image, root setup, setup, smoke command, environment variables, services, evidence). The agent writes its deliverable to `/work/task/` (`definition.json` with the environment contract, `instruction.md`, `test.patch`, `gold.patch`; `instruction.md` is authoritative for the prompt) and calls `verify`, which takes no arguments and reads that directory. The tool runs the static check inside the sandbox first (schema, environment policy, patch path safety, audit thresholds, and a dry render of the Harbor tree into `/work/rendered/`) and returns failures immediately. Otherwise it hands the payload to the worker through a sandbox mailbox and blocks while the worker's trusted compiler renders `task.toml`, both Dockerfiles, the scripts, and the repository snapshot, and Harbor builds the images and runs the smoke command, the `nop` split, and the `oracle` split. The structured report (compile, audit, build, smoke, nop, oracle) comes back as the tool result, so the agent iterates inside its own session; it has three `verify` calls per session. `submit_task` runs the static check again and records the task; when its payload equals the last green `verify`, the worker reuses that report instead of rebuilding. Otherwise the worker verifies the submission itself. If the sandbox dies or a submitted task is red, the same agent session is resumed in a fresh sandbox with the report as its next message (fallback loop). Three red rounds reject the candidate with the last report as the reason. A Harbor infrastructure failure counts as a red round with the build log as the report; three consecutive infrastructure rounds mark the candidate `infrastructure_failed` instead.
 
-1. `nop`: selected new tests fail while selected regressions pass;
+The mailbox is a directory in the live sandbox: the tool writes `/work/mailbox/requests/<id>.json`, the worker's supervising activity polls it through the provider's live-sandbox exec and file API, archives each verify under `runs/<run>/<stage>/<candidate>/round-<n>/verify-<k>/`, and writes the response the tool is waiting on. Verifier commands run as the `verifier` user with `HOME=/home/verifier`.
+
+The Harbor gates prove:
+
+1. `nop`: the smoke command succeeds in the built image, then selected new tests fail on the base snapshot while selected regressions pass;
 2. `oracle`: the reference patch applies and every selected test passes;
 3. determinism: the fail-to-pass selection passes a second time with the oracle.
 
-A fresh model session reviews the prompt and tests without inheriting the authoring conversation. When it finds repairable test-to-gold coupling, Temporal schedules one constrained repair in another sandbox. Repair may modify only paths already changed by the held-out test patch. It cannot change the request, base snapshot, or reference implementation.
+**Verification rounds.** A separate agent session that has not seen the authoring conversation receives the green task (instruction, held-out test patch, gold patch, environment contract, rendered files), the verification report, and the deterministic coupling evidence. It judges whether the task is a fair, self-contained benchmark: tests exercise public behaviour, no test-to-gold coupling, deterministic environment, instruction faithful to the human request. It either accepts or submits a fix. A fix may edit only the held-out tests and the environment contract (plus the test selection, timeouts, and resources that describe them), never the gold patch, the base commit, or the instruction. The verifier authors a fix as files too: held-out test edits in `/work/repo` plus an optional `/work/fix/definition.json` carrying only the changed fields (the tools regenerate `test.patch` from the working tree unless `/work/fix/test.patch` is present). It has the same in-session `verify` tool (two calls per session) for its fix; `submit_fix` runs the static check before it counts and reuses a matching green verify. Otherwise the worker re-runs compile, audit, and Harbor and resumes the same verifier session with the new report. A task is accepted only when the verifier accepts and the mechanical gates are green; otherwise the candidate is rejected with the verifier's reason after at most three verifier rounds.
+
+Every candidate in the discovered pool runs as its own child workflow in parallel; the requested counts size the pool (discovery targets 1.5 times the request per tier, dealt across its shards) rather than capping the result, so a run can export more accepted tasks than it asked for. Nothing is replaced or backfilled: a rejected candidate is simply a rejection in the run status.
 
-After repair, the task repeats static audit, nop, oracle, and independent review from the beginning. A second failure rejects the candidate.
+## Reproducible environments
 
-## Toolchains
+SelfBench does not infer a generic language toolchain. The authoring agent derives the environment contract from the exact pinned base commit: runtime, dependency installation, builds, fixtures, and local services come from the closest matching CI job, repository Dockerfile, devcontainer, lockfiles, and test scripts.
 
-The task compiler has setup layers for Node.js, Bun, Python, Go, and Rust. The author supplies fixed repository-native setup and test commands; compatibility is validated per candidate instead of promised for every repository.
+The resulting contract pins the base and service images by digest, records repository-file evidence for each choice, keeps service credentials local and non-secret, and separates verifier-only services from the agent environment. Secret-named variables are accepted only with fixed placeholder literals; values that look like real key material or interpolate host variables are rejected. The declared smoke command runs in the built verifier image before the `nop` split, so an environment defect is reported to the authoring agent as a red smoke or build gate rather than discovered later.
 
 When the reference patch changes a recognized dependency manifest or lockfile, the hidden verifier image repeats setup with the trusted reference state and then resets source files to the base snapshot. This prevents stale base dependencies from invalidating the oracle without exposing the reference patch to the coding-agent environment.
 
@@ -87,7 +99,7 @@ harbor-task/
 └── solution/
 ```
 
-The export includes the repository snapshot at each selected base commit, held-out tests, and reference solutions. It excludes Git history, local provenance/session records, and author/reviewer transcripts.
+The export includes the repository snapshot at each selected base commit, held-out tests, and reference solutions. It excludes Git history, local provenance/session records, and the authoring and verification agent sessions (those stay in the run's artifacts).
 
 The manifest digest detects accidental corruption but is not a signature because it sits inside the same archive. Extract only exports from a trusted SelfBench deployment and store them as private benchmark material.
 

package/drizzle/0000_glossy_johnny_blaze.sql

@@ -0,0 +1,97 @@
+CREATE TABLE "org_members" (
+	"org_id" bigint NOT NULL,
+	"user_id" bigint NOT NULL,
+	"role" text NOT NULL,
+	"synced_at" timestamp with time zone DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "orgs" (
+	"id" bigserial PRIMARY KEY NOT NULL,
+	"github_id" bigint NOT NULL,
+	"login" text NOT NULL,
+	"kind" text NOT NULL,
+	"name" text,
+	"avatar_url" text,
+	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
+	"updated_at" timestamp with time zone DEFAULT now() NOT NULL,
+	CONSTRAINT "orgs_github_id_unique" UNIQUE("github_id")
+);
+--> statement-breakpoint
+CREATE TABLE "repo_runs" (
+	"repo_id" bigint NOT NULL,
+	"run_id" text NOT NULL,
+	"attached_by" bigint NOT NULL,
+	"attached_at" timestamp with time zone DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "repos" (
+	"id" bigserial PRIMARY KEY NOT NULL,
+	"org_id" bigint NOT NULL,
+	"github_id" bigint NOT NULL,
+	"full_name" text NOT NULL,
+	"default_branch" text NOT NULL,
+	"private" boolean DEFAULT false NOT NULL,
+	"continuous" boolean DEFAULT false NOT NULL,
+	"connected_by" bigint NOT NULL,
+	"connected_at" timestamp with time zone DEFAULT now() NOT NULL,
+	CONSTRAINT "repos_github_id_unique" UNIQUE("github_id")
+);
+--> statement-breakpoint
+CREATE TABLE "tasks" (
+	"id" bigserial PRIMARY KEY NOT NULL,
+	"repo_id" bigint NOT NULL,
+	"run_id" text NOT NULL,
+	"candidate_id" text NOT NULL,
+	"task_id" text NOT NULL,
+	"source_pr" integer,
+	"source_url" text,
+	"difficulty" text NOT NULL,
+	"pipeline_status" text NOT NULL,
+	"stage" text NOT NULL,
+	"round" integer,
+	"reason" text,
+	"bundle_key" text,
+	"definition" jsonb,
+	"review_decision" text,
+	"review_note" text,
+	"reviewed_by" bigint,
+	"reviewed_at" timestamp with time zone,
+	"workflow_id" text,
+	"started_by" bigint,
+	"started_at" timestamp with time zone,
+	"synced_at" timestamp with time zone DEFAULT now() NOT NULL,
+	"created_at" timestamp with time zone DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "users" (
+	"id" bigserial PRIMARY KEY NOT NULL,
+	"github_id" bigint NOT NULL,
+	"login" text NOT NULL,
+	"name" text,
+	"avatar_url" text,
+	"github_token" text NOT NULL,
+	"github_scopes" text DEFAULT '' NOT NULL,
+	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
+	"last_seen_at" timestamp with time zone DEFAULT now() NOT NULL,
+	CONSTRAINT "users_github_id_unique" UNIQUE("github_id")
+);
+--> statement-breakpoint
+ALTER TABLE "org_members" ADD CONSTRAINT "org_members_org_id_orgs_id_fk" FOREIGN KEY ("org_id") REFERENCES "public"."orgs"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "org_members" ADD CONSTRAINT "org_members_user_id_users_id_fk" FOREIGN KEY ("user_id") REFERENCES "public"."users"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "repo_runs" ADD CONSTRAINT "repo_runs_repo_id_repos_id_fk" FOREIGN KEY ("repo_id") REFERENCES "public"."repos"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "repo_runs" ADD CONSTRAINT "repo_runs_attached_by_users_id_fk" FOREIGN KEY ("attached_by") REFERENCES "public"."users"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "repos" ADD CONSTRAINT "repos_org_id_orgs_id_fk" FOREIGN KEY ("org_id") REFERENCES "public"."orgs"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "repos" ADD CONSTRAINT "repos_connected_by_users_id_fk" FOREIGN KEY ("connected_by") REFERENCES "public"."users"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "tasks" ADD CONSTRAINT "tasks_repo_id_repos_id_fk" FOREIGN KEY ("repo_id") REFERENCES "public"."repos"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "tasks" ADD CONSTRAINT "tasks_reviewed_by_users_id_fk" FOREIGN KEY ("reviewed_by") REFERENCES "public"."users"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "tasks" ADD CONSTRAINT "tasks_started_by_users_id_fk" FOREIGN KEY ("started_by") REFERENCES "public"."users"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
+CREATE UNIQUE INDEX "org_members_pk" ON "org_members" USING btree ("org_id","user_id");--> statement-breakpoint
+CREATE INDEX "org_members_user_id" ON "org_members" USING btree ("user_id");--> statement-breakpoint
+CREATE INDEX "orgs_login" ON "orgs" USING btree ("login");--> statement-breakpoint
+CREATE UNIQUE INDEX "repo_runs_pk" ON "repo_runs" USING btree ("repo_id","run_id");--> statement-breakpoint
+CREATE INDEX "repos_org_id" ON "repos" USING btree ("org_id");--> statement-breakpoint
+CREATE UNIQUE INDEX "tasks_run_candidate" ON "tasks" USING btree ("run_id","candidate_id");--> statement-breakpoint
+CREATE UNIQUE INDEX "tasks_workflow_id" ON "tasks" USING btree ("workflow_id");--> statement-breakpoint
+CREATE INDEX "tasks_repo_id" ON "tasks" USING btree ("repo_id");--> statement-breakpoint
+CREATE INDEX "tasks_repo_pr" ON "tasks" USING btree ("repo_id","source_pr");--> statement-breakpoint
+CREATE INDEX "users_login" ON "users" USING btree ("login");

package/drizzle/0001_evaluation_records.sql

@@ -0,0 +1,5 @@
+CREATE TABLE "evaluation_records" (
+	"path" text PRIMARY KEY NOT NULL,
+	"version" integer NOT NULL,
+	"sealed" text NOT NULL
+);

package/drizzle/0002_task_deletion.sql

@@ -0,0 +1 @@
+ALTER TABLE "tasks" ADD COLUMN "deleted_at" timestamp with time zone;

package/drizzle/0003_drop_repo_runs.sql

@@ -0,0 +1,3 @@
+-- Batch generation also uses repo_runs for durable repository ownership.
+-- Keep this migration slot for databases that already applied the review-page draft.
+SELECT 1;

package/drizzle/0004_preserve_batch_runs.sql

@@ -0,0 +1,8 @@
+CREATE TABLE IF NOT EXISTS "repo_runs" (
+  "repo_id" bigint NOT NULL CONSTRAINT "repo_runs_repo_id_repos_id_fk" REFERENCES "repos"("id") ON DELETE cascade,
+  "run_id" text NOT NULL,
+  "attached_by" bigint NOT NULL CONSTRAINT "repo_runs_attached_by_users_id_fk" REFERENCES "users"("id"),
+  "attached_at" timestamp with time zone DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+CREATE UNIQUE INDEX IF NOT EXISTS "repo_runs_pk" ON "repo_runs" USING btree ("repo_id", "run_id");

package/drizzle/0005_api_keys.sql

@@ -0,0 +1,15 @@
+CREATE TABLE "api_keys" (
+	"id" bigserial PRIMARY KEY NOT NULL,
+	"user_id" bigint NOT NULL,
+	"name" text NOT NULL,
+	"prefix" text NOT NULL,
+	"key_hash" text NOT NULL,
+	"scope" text NOT NULL,
+	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
+	"last_used_at" timestamp with time zone,
+	"revoked_at" timestamp with time zone,
+	CONSTRAINT "api_keys_key_hash_unique" UNIQUE("key_hash")
+);
+--> statement-breakpoint
+ALTER TABLE "api_keys" ADD CONSTRAINT "api_keys_user_id_users_id_fk" FOREIGN KEY ("user_id") REFERENCES "public"."users"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+CREATE INDEX "api_keys_user_id" ON "api_keys" USING btree ("user_id");