bridgebench 3.1.0-alpha.1 → 3.1.0-alpha.2

package/README.md

@@ -13,11 +13,12 @@
 </p>
 
 <p align="center">
-  <a href="https://bridgebench.ai">bridgebench.ai</a> &nbsp;&bull;&nbsp;
+  <a href="https://bridgebench.ai">Leaderboard</a> &nbsp;&bull;&nbsp;
+  <a href="#review-the-benchmark">Review</a> &nbsp;&bull;&nbsp;
+  <a href="#how-a-match-becomes-a-ranking">Methodology</a> &nbsp;&bull;&nbsp;
   <a href="#the-arenas">Arenas</a> &nbsp;&bull;&nbsp;
-  <a href="#autonomous-match-lifecycle">Methodology</a> &nbsp;&bull;&nbsp;
-  <a href="#task-packs">Tasks</a> &nbsp;&bull;&nbsp;
-  <a href="#contributing">Contributing</a>
+  <a href="docs/README.md">Docs</a> &nbsp;&bull;&nbsp;
+  <a href="CONTRIBUTING.md">Contribute</a>
 </p>
 
 <p align="center">
@@ -29,27 +30,30 @@
 
 ## What is BridgeBench?
 
-BridgeBench measures how models perform as vibe coding partners. Every task is a software-engineering scenario — source code, diffs, CI logs, API specs, migrations, telemetry, agent sessions — and every deliverable is a question a coding agent would actually face.
+BridgeBench measures how models perform as vibe coding partners. Every task is
+a software-engineering scenario built from artifacts such as source code,
+diffs, CI logs, API specs, migrations, telemetry, and agent sessions.
 
-V3 is **arena-first**: models compete head-to-head on the same task, three independent model judges choose the stronger answer blind, and every majority decision awards one point and one Elo update. There is no weighted aggregate score and no opaque formula — a ranking is exactly the sum of the match record behind it, and the journal that produced it is replayable line by line.
+Models compete head-to-head on the same task. Three independent model judges
+compare their answers blind, a majority selects the stronger response, and the
+winner earns one point plus one Elo update. The append-only match journal is
+the source of truth; leaderboards and API responses are derived views.
 
-It's built by [BridgeMind](https://bridgemind.ai), the agentic organization behind BridgeSpace, BridgeVoice, and BridgeAgent. We benchmark models because we ship with them daily — the leaderboard is the same data we use to pick our own teammates.
+BridgeBench is built by [BridgeMind](https://bridgemind.ai), the agentic
+organization behind BridgeSpace, BridgeVoice, and BridgeAgent. We use the same
+benchmark data to choose models for our own work.
 
-## Architecture
+## Start with the path you need
 
-```text
-public task pack ─┐
-                  ├─> arena runner ─> blind judge panel ─> append-only journal
-private overlay ──┘                                          │
-                                                             ├─> verifier ─> reports
-                                                             └─> publisher ─> bridgebench.ai
-```
-
-The local journal is the execution record. Snapshots, leaderboards, and the bridgebench.ai API are derived views. `arena verify` validates every journal line and replays Elo before a report or publish operation trusts it.
-
-Public task prompts live in this repository. Active hidden references live in a separate private overlay. They are sent to the configured model judges during a match and may be synced to the private API store by maintainers. Withholding them reduces benchmark contamination risk; it does not make third-party processing impossible. The complete boundary is documented in [Private packs](docs/private-packs.md).
+| Goal | Start here |
+| --- | --- |
+| Understand and audit a result | [Reviewing BridgeBench](docs/reviewing-bridgebench.md) |
+| Understand the full arena contract | [Methodology](docs/methodology.md) |
+| Run paid matches or operate the dashboard | [Operator guide](docs/operator-guide.md) |
+| Author tasks or change the engine | [Contributing](CONTRIBUTING.md) |
+| Find a specific concept | [Documentation index](docs/README.md) or [glossary](docs/glossary.md) |
 
-## Public-clone quickstart
+## Review the benchmark
 
 Requirements: Node.js 20.19 or newer and npm 10 or newer.
 
@@ -57,184 +61,148 @@ Requirements: Node.js 20.19 or newer and npm 10 or newer.
 git clone https://github.com/bridge-mind/bridgebench.git
 cd bridgebench
 npm ci
-npx playwright install chromium
-
-npm run tasks -- validate
-npm run check
-npm run arena -- verify --category reasoning --journal test/fixtures/journals/valid.jsonl
+npm run review
 ```
 
-These commands need no API key and no private overlay. They validate the public packs, run offline checks, and replay a synthetic journal.
-
-## The arenas
-
-Two independent arenas ship today, each with its own task pack, journal, Elo ladder, and leaderboard:
-
-| Arena             | What it measures                                                                                                                                                                      | What a winning answer looks like                                                                                                                                                                       |
-| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| **Reasoning**     | Inference depth. Every task is fully determinable from its artifacts — interlocking specs, logs, code, and configs with planted decoy paths.                                          | Derives the one defensible resolution for every numbered deliverable, with the inference chain and artifact citations.                                                                                 |
-| **Hallucination** | Epistemic discipline. Tasks are seeded with false premises, missing evidence, fabrication bait (plausible entities that don't exist), conflicting sources, and near-duplicate values. | Answers the supported deliverables exactly, corrects false premises with the contradicting evidence, names precisely what is missing — and never invents entities, values, quotes, or blended figures. |
-
-The two arenas share the arena contract (pairing, blind three-judge panel, Elo) but never share ratings: a model's reasoning Elo says nothing about its hallucination Elo. Judges receive category-specific instructions — the reasoning panel punishes hedging on determinable questions, the hallucination panel weighs fabrication heaviest and treats blanket refusal as an error too.
-
-The same contract expands to security, debugging, and refactoring arenas next.
-
-## Match lifecycle
-
-1. A seeded scheduler selects one task and two distinct competitors while balancing exposure. The same seed always produces the same schedule.
-2. Both competitors receive identical task context and run concurrently. They don't know they're in a match.
-3. One exhausted competitor failure forfeits; two failures create a no-contest.
-4. Each judge independently receives the task, hidden rubric, and anonymous answers.
-5. Explicit model IDs, canonical slugs, provider names, and model-family names are redacted from responses; judges see only `Model A` and `Model B`.
-6. A/B order is independently permuted per judge to reduce position bias.
-7. Two valid votes decide the winner. Judges never see identities, ratings, costs, or other votes.
-8. The winner earns one point. Elo starts at 1000 and uses K=32.
-9. The complete result is appended to the journal before reports are rebuilt. Snapshots are derived, never authoritative.
-
-Candidate answers are treated as untrusted data. Judge prompts explicitly reject instructions embedded in answers; structured verdicts are schema validated; malformed verdicts get one retry and then abstain. No generated code or model-provided command is ever executed.
-
-The full protocol — scheduling math, anonymization rules, vote resolution, drift detection — is in [docs/methodology.md](docs/methodology.md).
-
-## Model roster and transport
-
-All requests go through [OpenRouter](https://openrouter.ai) using exact, pinned model slugs. `latest` aliases are prohibited. Before a paid run, the CLI verifies each ID and canonical slug against OpenRouter and confirms that judges still support structured output.
-
-The arena measures models _through one aggregator's routing_, not direct provider APIs. Every competitor and judge uses the same transport. OpenRouter's per-generation records (`arena generation <id>`) provide an independent accounting source for tokens, cost, and provider routing.
-
-The current roster and request policies are defined once in [`src/models.ts`](src/models.ts). Judges are never eligible competitors.
-
-## Maintainer/operator setup
-
-Paid runs require both:
-
-1. `OPENROUTER_API_KEY` in the process environment;
-2. `BRIDGEBENCH_PRIVATE_DIR` pointing to the private-pack checkout.
-
-Set an account-level spending limit before running matches. Never commit either value.
-
-## Run an arena
-
-```bash
-# Reasoning: 12 matches, reproducible seed, $25 stop boundary
-npm run arena -- run --category reasoning
-
-# The hallucination arena — same contract, its own tasks, journal, and Elo
-npm run arena -- run --category hallucination
-
-# Custom batch
-npm run arena -- run --category hallucination --matches 24 --seed july-calibration --max-cost-usd 40
+`npm run review` is the shortest credential-free audit path. It:
 
-# Sol/Fable pilot — repeat --competitor to set an explicit roster
-npm run arena -- run --category reasoning \
-  --competitor openai/gpt-5.6-sol \
-  --competitor anthropic/claude-fable-5
+1. checks documentation links, navigation, commands, and fixture references;
+2. validates all 36 public tasks and their pack invariants;
+3. verifies a bundled journal line against its run manifest;
+4. replays the majority outcome, point, and Elo update.
 
-# Resume the exact deterministic schedule after interruption or budget stop
-npm run arena -- run --category hallucination --matches 24 --seed july-calibration --max-cost-usd 40 --resume
+It needs no API key, private task overlay, network request, or paid model call.
+The fixture is synthetic and tests the audit mechanism, not model quality.
+Follow the [reviewer walkthrough](docs/reviewing-bridgebench.md) to connect
+that mechanism to a real public task and published results.
 
-# Rebuild reports for both arenas without API calls (or one: --category reasoning)
-npm run report
-
-# Verify before publishing
-npm run arena -- verify --category hallucination
-
-# Maintainers: an explicit API target and category are required
-# Set BRIDGEBENCH_API_URL and BRIDGEBENCH_ADMIN_KEY privately first.
-npm run tasks -- publish --category hallucination
-npm run arena -- publish --category reasoning
-```
-
-Category, seed, match count, model roster, task hashes, prompt policy, methodology, and engine version define the run manifest. `--resume` continues only the matching deterministic schedule.
-
-Without `--competitor`, runs use every enabled competitor. An explicit roster must contain at least two unique, enabled competitor models. Press Ctrl-C once to cancel: active model calls abort, completed matches stay journaled, and the same command resumes with `--resume`.
-
-Results are local, ignored by Git, and kept per arena:
+## How a match becomes a ranking
 
 ```text
-results/<category>/journal.jsonl      # append-only execution record
-results/<category>/runs/<run-id>.json # versioned run manifest
-results/<category>/snapshot.json      # verified, derived view
-results/<category>/leaderboard.md
+public task ─┐
+             ├─> two competitors ─> anonymous answers ─┐
+hidden rubric┘                                         ├─> three blind judges
+                                                      │
+                                                      └─> majority decision
+                                                               │
+                                                               v
+run manifest ─────────────────────────────────────> journal + Elo
+                                                               │
+                                                               v
+                                                   verified leaderboard
 ```
 
-A repeated schedule is rejected unless `--resume` is explicit. See [Replay Elo](docs/replay-elo.md) to audit a published ladder.
-
-## Local dashboard
-
-A BridgeMind-branded control surface split into three views — Arena, Leaderboard, and Matches — with a live competitor + anonymous-judge stage during runs, durable match history with raw responses and judge rationales, and an SSE activity feed.
-
-```bash
-npm run dashboard
-# Open http://127.0.0.1:4317
-```
-
-The control plane binds only to `127.0.0.1`. The API key stays in the server process and is never serialized to the browser. State-changing requests require a same-origin JSON request, only one run may be active at once, and the browser renders model output as escaped text.
-
-## Task packs
-
-Both packs hold 12 expert-difficulty tasks across six category-specific clusters (two tasks each). Tasks are deliberately heavyweight — five to eight interlocking artifacts (~9–18k characters) and four to ten numbered deliverables — so a match exercises real deliberation budget instead of a one-screen skim.
-
-**Domain invariant: every task is a coding / software-engineering scenario.** No generic business ops. Public prompts live under `tasks/<category>/public/`; hidden references live in the private overlay. Every journal line records the SHA-256 of both task halves, so task drift is externally detectable.
+1. A seeded scheduler selects a public task and two distinct competitors while
+   balancing exposure.
+2. Both competitors receive byte-identical task context and run concurrently.
+3. Each judge receives the task, hidden reference, and two anonymous answers.
+4. Model and provider identities are redacted, and answer order is independently
+   permuted for each judge.
+5. Two valid votes decide the winner. One exhausted competitor failure is a
+   forfeit; two failures or no majority produce a no-contest.
+6. A decided match awards one point and updates Elo from an initial rating of
+   1000 with K=32.
+7. The complete evidence record is appended before reports are rebuilt.
+
+Candidate answers are untrusted input. Judge prompts reject embedded
+instructions, verdicts are schema validated, malformed verdicts abstain after
+one retry, and model-provided code or commands are never executed.
+
+The [methodology](docs/methodology.md) specifies scheduling, anonymization,
+voting, failure outcomes, and replay rules in full.
 
-Cluster definitions, the public/private YAML schemas, and the authoring rules (including how decoys map to disqualifying errors) are in [docs/task-authoring.md](docs/task-authoring.md).
-
-## Debugging & the continuous improvement loop
+## The arenas
 
-Every arena and dashboard run writes a structured, key-redacted JSONL log to `results/<category>/logs/`. The triage command analyzes journals for anomalies — failed requests, suspiciously fast responses, truncation, judge abstentions — and a health stop halts runs that are mostly producing failures.
+Each arena has its own task pack, journal, Elo ladder, and leaderboard. Ratings
+never cross categories.
+
+| Arena | What it measures | A strong answer |
+| --- | --- | --- |
+| **Reasoning** | Inference across interlocking software artifacts with planted decoys. Every deliverable is determinable from the provided evidence. | Derives the defensible resolution, explains the inference chain, and cites the controlling artifacts. |
+| **Hallucination** | Epistemic discipline under false premises, missing evidence, fabrication bait, conflicts, and near-duplicate facts. | Answers supported items, corrects false premises, identifies missing evidence precisely, and does not invent. |
+| **Security** | Defensive analysis of fictional code that hides one real, reachable vulnerability among benign look-alikes, false positives, and shallow patches. No code is executed. | Proves the reachable source-to-sink or guard-bypass chain with cited evidence and calibrated severity, and declines to flag the benign traps. |
+| **BullShit** | Premise integrity under seeded nonsense — fabricated concepts, crossed domains, impossible quantities, reversed causality, pseudoscience, and loaded assumptions mixed with legitimate deliverables. | Names exactly what is nonsensical and why, corrects the premise to the nearest legitimate question and answers it, and still answers the sound deliverables instead of blanket-refusing. |
+| **Refactoring** | Behavior preservation under a transformation goal, with candidate rewrites that subtly change ordering, scope, contracts, or an edge case. | Traces equivalence across every affected path, cites the location and mechanism, and flags the rewrite that silently changes observable behavior. |
+| **Debugging** | Root-cause isolation from a failing system's evidence, among red-herring causes and shallow fixes. | Traces symptom to origin with cited evidence, names the one defensible root cause, and picks the fix that holds without regressing. |
+| **Generation** | Specification conformance across candidate implementations, where each near-miss violates one stated constraint or edge case. | Cites the exact spec clause and the distinguishing input, and identifies the implementation that meets every requirement. |
+| **Speed** | Raw latency: both models answer the same task and the faster completion wins. Decided by measured time-to-first-token and throughput — no judges. | Answers correctly and efficiently; the arena records TTFT and tokens-per-second and awards the win to the lower total completion time. |
+
+Each judged public pack contains 12 expert tasks across six category-specific
+clusters; the Speed pack contains 12 public-only workload tasks. See
+[Task authoring](docs/task-authoring.md) for their schemas, clusters, and
+enforced balance.
+
+## What evidence backs a result?
+
+| Evidence | Role | What a reviewer can check |
+| --- | --- | --- |
+| Public task | Exact prompt and artifacts sent to both competitors | Content, version, category, cluster, and `publicHash` |
+| Run manifest | Identity of the run | Seed, roster, task hashes, prompt policies, methodology, and engine version |
+| Match journal | Append-only execution record | Full responses, judge votes and rationales, outcome, cost, point, and Elo before/after |
+| Hidden reference | Expected resolution, evidence requirements, traps, and rubric | Its hash while active; its full contents after pack retirement |
+| Snapshot and leaderboard | Convenient derived views | Rebuild them only after the journal verifies |
+
+`arena verify` validates every journal line and replays Elo before reports,
+resume state, or publishing trust it. Read [Replay the Elo](docs/replay-elo.md)
+for the algorithm and exact audit command.
+
+### What verification does not prove
+
+Journal verification proves internal consistency and detects inconsistent
+edits, within-run reordering, task/manifest mismatch, and incorrect rating
+math. It does not authenticate the publisher or rule out a coordinated rewrite
+of the journal and manifest. It also cannot prove that a model judge made the
+best qualitative choice, that hidden references never reached a third-party
+provider, or that one aggregator represents direct provider behavior.
+
+All requests currently travel through OpenRouter using pinned model slugs.
+Active hidden references are sent to the configured judges and are withheld
+from the public repository until their pack retires. The
+[reviewer guide](docs/reviewing-bridgebench.md#what-the-evidence-proves) and
+[private-pack boundary](docs/private-packs.md) describe these limitations
+without claiming more than the evidence supports.
+
+## Repository map
 
-```bash
-npm run arena -- run --debug        # mirror every log entry to the console
-npm run triage                      # analyze the journal (auto-printed after every run)
-npm run arena -- generation gen-... # OpenRouter's ground-truth record for any journaled generation
+```text
+bridgebench/
+├── src/                         arena, judging, verification, reports, CLI
+├── tasks/<category>/public/     public task packs
+├── test/fixtures/               deterministic journals and run manifests
+├── ui/                          localhost dashboard
+├── docs/                        reviewer, protocol, authoring, and operator guides
+└── CONTRIBUTING.md              code, task, audit, and documentation workflow
 ```
 
-The loop: run → read the auto-printed health report → chase flags with the run log → fix the task, prompt, or model policy → rerun with a fresh seed → compare triage reports.
+Canonical executable sources:
 
-## Repository layout
+- Model roster and request policy: [`src/models.ts`](src/models.ts)
+- Category and methodology constants: [`src/contracts/categories.ts`](src/contracts/categories.ts)
+- Task loading and pack invariants: [`src/tasks.ts`](src/tasks.ts)
+- Journal verification and replay: [`src/verification.ts`](src/verification.ts)
 
-```
-bridgebench/
-├── src/
-│   ├── cli.ts             # thin executable entrypoint
-│   ├── commands.ts        # injectable CLI construction and command handlers
-│   ├── arena.ts           # match loop: forfeits, health stop, budget stop
-│   ├── scheduler.ts       # seeded, exposure-balanced deterministic scheduling
-│   ├── judges.ts          # anonymization, per-judge A/B permutation, majority vote
-│   ├── tasks.ts           # pack loader + public/private overlay resolution
-│   ├── verification.ts    # schema validation + deterministic journal replay
-│   ├── elo.ts / store.ts / report.ts / triage.ts
-│   └── dashboard/         # localhost-only control plane (SSE)
-├── tasks/
-│   ├── reasoning/public/       # 12 public task prompts
-│   └── hallucination/public/   # 12 public task prompts
-├── ui/                    # local dashboard SPA (React + Vite)
-├── test/                  # deterministic offline tests and fixtures
-└── docs/                  # indexed protocol and contributor references
-```
+## Develop and contribute
 
-## Development
+The full public-clone quality gate uses a mock OpenRouter gateway and requires
+no model API credentials:
 
 ```bash
+npm ci
+npx playwright install chromium
 npm run check
 ```
 
-`npm run build` also emits the versioned `bridgebench/contracts` subpath from `src/contracts/index.ts`. This is a local build entry point only; Phase 1 adds no package-publishing step.
-
-Tests use a mock OpenRouter gateway and never spend credits — they pass from a fresh public clone with no API key and no private overlay. Live catalog validation and arena runs are operator-invoked only.
-
-## Contributing
-
-BridgeBench is open source and open to builders:
-
-- **Propose tasks.** Validate one public task with `npm run tasks -- validate --file <path>`, then open a task proposal.
-- **Audit a ladder.** Run `arena verify` against the published journal. File an audit report if it fails.
-- **Extend the contract.** New arenas (security, debugging, refactoring) reuse the same match lifecycle; the season-engine branch holds a provider layer and browser harness earmarked for future suites.
-
-Start with [CONTRIBUTING.md](CONTRIBUTING.md) and the [docs index](docs/README.md).
+Paid arena commands are operator-invoked only and are not part of pull-request
+validation. BridgeBench accepts code, public task proposals, methodology
+audits, and documentation fixes. Start with [CONTRIBUTING.md](CONTRIBUTING.md).
 
 ## Project history
 
-This repository previously hosted the Season 1 _season-engine_ alpha: ten Three.js tasks scored in a browser and ranked by community A/B voting. That code remains on the [`season-engine-alpha`](https://github.com/bridge-mind/bridgebench/tree/season-engine-alpha) branch. Its results never mix with arena Elo.
+This repository previously hosted the Season 1 `season-engine` alpha: ten
+Three.js tasks scored in a browser and ranked by community A/B voting. That
+code remains on the
+[`season-engine-alpha` branch](https://github.com/bridge-mind/bridgebench/tree/season-engine-alpha).
+Its results never mix with arena Elo.
 
 ## License
 

package/dist/{chunk-UXFKTSQS.js → chunk-AY45YLYL.js}

@@ -1,7 +1,16 @@
 // src/contracts/categories.ts
 import { z } from "zod";
 var METHODOLOGY_VERSION = "arena-v0.3.0";
-var BenchmarkCategorySchema = z.enum(["reasoning", "hallucination"]);
+var BenchmarkCategorySchema = z.enum([
+  "reasoning",
+  "hallucination",
+  "security",
+  "bullshit",
+  "refactoring",
+  "debugging",
+  "generation",
+  "speed"
+]);
 var CATEGORIES = BenchmarkCategorySchema.options;
 var CATEGORY_CLUSTERS = {
   reasoning: [
@@ -19,6 +28,54 @@ var CATEGORY_CLUSTERS = {
     "knowledge-boundary",
     "conflicting-sources",
     "citation-fidelity"
+  ],
+  security: [
+    "vuln-discovery",
+    "taint-flow",
+    "authz-boundary",
+    "patch-verification",
+    "finding-triage",
+    "supply-chain"
+  ],
+  bullshit: [
+    "fabricated-concepts",
+    "crossed-domains",
+    "impossible-quantities",
+    "reversed-causality",
+    "plausible-pseudoscience",
+    "loaded-assumptions"
+  ],
+  refactoring: [
+    "behavior-preservation",
+    "extract-and-inline",
+    "dependency-decoupling",
+    "api-migration",
+    "dead-code-elimination",
+    "semantic-equivalence"
+  ],
+  debugging: [
+    "root-cause-isolation",
+    "regression-introduction",
+    "concurrency-defect",
+    "state-corruption",
+    "error-propagation",
+    "fix-adequacy"
+  ],
+  generation: [
+    "spec-conformance",
+    "edge-case-coverage",
+    "api-contract-adherence",
+    "algorithmic-correctness",
+    "constraint-satisfaction",
+    "interface-compatibility"
+  ],
+  speed: [
+    "short-completion",
+    "long-generation",
+    "structured-output",
+    "code-transformation",
+    "stepwise-reasoning",
+    "retrieval-synthesis"
   ]
 };
 var CATEGORY_META = {
@@ -29,6 +86,30 @@ var CATEGORY_META = {
   hallucination: {
     label: "Hallucination",
     tagline: "Tasks are seeded with false premises, missing evidence, and fabrication bait \u2014 the arena measures who stays grounded instead of inventing."
+  },
+  security: {
+    label: "Security",
+    tagline: "Tasks hide one defensible vulnerability among benign look-alikes, false positives, and shallow patches \u2014 the arena measures who proves reachable exploitability with evidence instead of crying wolf."
+  },
+  bullshit: {
+    label: "BullShit",
+    tagline: "Tasks seed confident nonsense \u2014 fabricated concepts, crossed domains, impossible quantities \u2014 among legitimate deliverables; the arena measures who corrects the user instead of fluently answering the unanswerable."
+  },
+  refactoring: {
+    label: "Refactoring",
+    tagline: "Tasks pair code with a transformation goal and candidate rewrites \u2014 the arena measures who preserves observable behavior and spots the rewrite that silently changes it."
+  },
+  debugging: {
+    label: "Debugging",
+    tagline: "Tasks supply a failing system and its evidence among red-herring causes and shallow fixes \u2014 the arena measures who isolates the one defensible root cause and the fix that actually holds."
+  },
+  generation: {
+    label: "Generation",
+    tagline: "Tasks pair a specification with candidate implementations \u2014 the arena measures who identifies the one that meets every constraint and edge case instead of the plausible near-miss."
+  },
+  speed: {
+    label: "Speed",
+    tagline: "Both competitors get the same task and race: no judges, no quality vote. The winner is decided deterministically by measured latency \u2014 time to first token and sustained output throughput settle who finishes the work faster."
   }
 };
 
@@ -47,7 +128,11 @@ var ModelCompletionSchema = z2.object({
   costUsd: z2.number().finite().nonnegative(),
   latencyMs: z2.number().finite().nonnegative(),
   finishReason: z2.string(),
-  attempts: z2.number().int().positive().optional()
+  attempts: z2.number().int().positive().optional(),
+  /** Time from request start to the first non-empty content delta (speed arena metric). */
+  ttftMs: z2.number().finite().nonnegative().optional(),
+  /** Request start to stream completion for the successful attempt (speed arena metric). */
+  totalMs: z2.number().finite().nonnegative().optional()
 });
 
 // src/contracts/journal.ts
@@ -121,6 +206,16 @@ var JUDGE_VERDICT_TRANSPORT_SCHEMA = {
     violations: { type: "array", items: { type: "string" } }
   }
 };
+var SpeedMetricSchema = z3.object({
+  ttftMs: z3.number().finite().nonnegative(),
+  tps: z3.number().finite().nonnegative(),
+  totalMs: z3.number().finite().nonnegative(),
+  outputTokens: z3.number().finite().nonnegative()
+});
+var SpeedMetricsSchema = z3.object({
+  a: SpeedMetricSchema,
+  b: SpeedMetricSchema
+});
 var NumericRecordSchema = z3.record(z3.number().finite());
 var JudgeVoteSchema = z3.object({
   judgeModelId: z3.string().min(1),
@@ -152,7 +247,7 @@ var MatchResultSchema = z3.object({
     category: BenchmarkCategorySchema.optional(),
     cluster: z3.string().min(1),
     publicHash: z3.string().min(1),
-    privateHash: z3.string().min(1)
+    privateHash: z3.string().min(1).nullable()
   }),
   competitors: z3.object({
     modelA: z3.string().min(1),
@@ -160,9 +255,10 @@ var MatchResultSchema = z3.object({
     responseA: CompetitorResponseSchema,
     responseB: CompetitorResponseSchema
   }),
-  outcome: z3.enum(["judged", "forfeit", "no-contest"]),
+  outcome: z3.enum(["judged", "forfeit", "no-contest", "speed-decided"]),
   winnerModelId: z3.string().min(1).nullable(),
   panel: PanelDecisionSchema.nullable(),
+  speedMetrics: SpeedMetricsSchema.nullable().optional(),
   eloBefore: NumericRecordSchema,
   eloAfter: NumericRecordSchema,
   pointAwarded: z3.boolean(),
@@ -214,6 +310,9 @@ var TaskPrivateSchema = z4.object({
     completeness: z4.string().min(1)
   }).strict()
 }).strict();
+function isCompleteArenaTask(task) {
+  return task.private !== null && task.privateHash !== null;
+}
 
 // src/contracts/index.ts
 var CONTRACTS_VERSION = "1.0.0";
@@ -229,6 +328,7 @@ export {
   CompetitorResponseSchema,
   JudgeVerdictSchema,
   JUDGE_VERDICT_TRANSPORT_SCHEMA,
+  SpeedMetricsSchema,
   JudgeVoteSchema,
   PanelDecisionSchema,
   MatchResultSchema,
@@ -239,5 +339,6 @@ export {
   TaskArtifactSchema,
   TaskPublicSchema,
   TaskPrivateSchema,
+  isCompleteArenaTask,
   CONTRACTS_VERSION
 };

package/dist/{chunk-MBC4C3VA.js → chunk-CJGHBY54.js}

@@ -6,17 +6,17 @@ import {
   createRunManifest,
   runIdFromManifest,
   runManifestHash
-} from "./chunk-KGZEQ6R5.js";
+} from "./chunk-IUPFMGUL.js";
 import {
   OpenRouterClient,
   TaskLoader,
   noopLogger,
   redactSecrets,
   resolveCompetitorRoster
-} from "./chunk-3CX4FIFZ.js";
+} from "./chunk-DVMGL3L7.js";
 import {
   METHODOLOGY_VERSION
-} from "./chunk-UXFKTSQS.js";
+} from "./chunk-AY45YLYL.js";
 
 // src/api-client.ts
 var ADMIN_KEY_HEADER = "x-bridgebench-admin-key";
@@ -138,9 +138,10 @@ async function postChunks(config, path2, batches, wrap, schema) {
 // src/publish.ts
 import { z } from "zod";
 async function publishTasks(category, config = resolveApiConfig()) {
-  const loaded = await new TaskLoader(category).loadAll({
-    requirePrivate: true
-  });
+  const requirePrivate = category !== "speed";
+  const loaded = await new TaskLoader(category).loadAll(
+    requirePrivate ? { requirePrivate: true } : {}
+  );
   const results = await postChunks(
     config,
     "/arena/tasks/import",