localarena 0.1.0 → 0.2.0

package/CHANGELOG.md

@@ -4,6 +4,28 @@ All notable changes to LocalArena are documented here.
 
 The project follows [Semantic Versioning](https://semver.org/).
 
+## [0.2.0] - 2026-07-26
+
+### Added
+
+- Live OpenAI-compatible providers for llama.cpp, Ollama, LM Studio,
+  OpenRouter, OpenAI, and custom base URLs.
+- Concurrent model-by-task evaluation with exact, containment, regular
+  expression, JSON, numeric, and live model-judge scoring.
+- A Python CLI for provider discovery, live runs, saved results, and
+  standalone HTML reports.
+- Shared evaluation configuration and run schemas, privacy-safe result
+  serialization, and judge-only model configuration.
+- Matching JavaScript APIs, TypeScript declarations, normalized usage data,
+  bounded transport behavior, and cancellation.
+
+### Changed
+
+- Prompt, answer, evaluator, and diagnostic content is excluded from saved
+  results by default and requires an explicit opt-in.
+- Generation retries are disabled by default to avoid duplicating ambiguous
+  provider requests or billing.
+
 ## [0.1.0] - 2026-07-25
 
 ### Added

package/README.md

@@ -1,210 +1,322 @@
-# localarena
+# LocalArena
 
-Local-first pairwise evaluation, deterministic scheduling, and Elo ranking for
-models, agents, prompts, algorithms, or anything else you want to compare.
+Run the models you choose against the tasks you choose, using live provider
+responses from your machine or an explicitly configured API endpoint.
 
 [![PyPI](https://img.shields.io/pypi/v/localarena)](https://pypi.org/project/localarena/)
 [![npm](https://img.shields.io/npm/v/localarena)](https://www.npmjs.com/package/localarena)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
 
-`localarena` is the small, dependency-free engine underneath a head-to-head
-arena. You generate two outputs and decide which side won; LocalArena keeps the
-match log, updates ratings, builds a fair next pairing, and gives you a portable
-leaderboard.
+LocalArena executes the complete model × task grid concurrently, applies
+deterministic scorers or a live judge model, records failures as data, derives
+Elo standings, and writes a standalone HTML report. It never substitutes
+recorded answers or copied leaderboard results for a live call.
 
-- Same package name and concepts in Python and JavaScript
-- No network calls, telemetry, model SDK, database, or implicit file writes
-- Deterministic round-robin and least-played scheduling
-- Standard Elo updates with wins, losses, and draws
-- Versioned JSON snapshots that can move between both runtimes
-- Detached metadata so callers cannot accidentally mutate arena state
+- llama.cpp, Ollama, and LM Studio on loopback or any reachable host
+- OpenRouter and OpenAI with separate runtime credentials
+- Any compatible Chat Completions endpoint through a custom base URL
+- Python CLI and library, plus matching JavaScript and TypeScript APIs
+- Exact, contains, regular-expression, JSON, numeric, and model-judge scoring
+- Privacy-safe result files by default: prompts and answers require opt-in
+- No runtime dependencies in either package
 
 ## Install
 
-Python:
+Python 3.10+:
 
 ```bash
 pip install localarena
 ```
 
-JavaScript or TypeScript (Node.js 18+):
+Node.js 18+:
 
 ```bash
 npm install localarena
 ```
 
-## Quick start
+## Run a live evaluation
 
-### Python
+Create `evaluation.json`:
 
-```python
-from localarena import Arena, Result
+```json
+{
+  "$schema": "./node_modules/localarena/schema/localarena-evaluation-config-v1.schema.json",
+  "name": "Local smoke test",
+  "concurrency": 2,
+  "models": [
+    {
+      "name": "local-model",
+      "provider": "ollama",
+      "model": "qwen3:0.6b",
+      "parameters": {
+        "max_tokens": 64,
+        "temperature": 0
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "id": "arithmetic",
+      "prompt": "Return only the result of 6 multiplied by 7.",
+      "evaluator": {
+        "type": "numeric",
+        "expected": 42,
+        "tolerance": 0
+      }
+    },
+    {
+      "id": "structured",
+      "prompt": "Return only JSON with an answer field equal to 42.",
+      "evaluator": {
+        "type": "json",
+        "compare": true,
+        "expected": {
+          "answer": 42
+        }
+      }
+    }
+  ]
+}
+```
 
-arena = Arena(["ollama/qwen3", "lm-studio/gemma"])
+Run every configured model against every configured task:
 
-# Generate both answers locally, ask a human or judge, then record the verdict.
-arena.record(
-    "ollama/qwen3",
-    "lm-studio/gemma",
-    Result.LEFT,
-    metadata={"prompt_id": "explain-rag"},
-)
+```bash
+localarena run evaluation.json \
+  --output results.json \
+  --report report.html
+```
 
-for standing in arena.standings():
-    print(standing.name, round(standing.rating, 2))
+The command prints progress as calls complete and returns a nonzero status if
+any generation or scoring row failed. A failure stays in `results.json`; it
+does not abort or silently shrink the matrix.
 
-print("Compare next:", arena.next_pair())
-```
+The complete multi-provider example is
+[examples/evaluation-config.json](examples/evaluation-config.json).
 
-### JavaScript
+## Providers
 
-```js
-import { Arena, Result } from "localarena";
+All built-in profiles use bounded, non-streaming Chat Completions requests and
+model discovery through `/v1/models`.
 
-const arena = new Arena(["ollama/qwen3", "lm-studio/gemma"]);
+| Profile | Default base URL | Default credential |
+| --- | --- | --- |
+| `llamacpp` | `http://127.0.0.1:8080/v1` | None |
+| `ollama` | `http://127.0.0.1:11434/v1` | None |
+| `lmstudio` | `http://127.0.0.1:1234/v1` | None |
+| `openrouter` | `https://openrouter.ai/api/v1` | `OPENROUTER_API_KEY` |
+| `openai` | `https://api.openai.com/v1` | `OPENAI_API_KEY` |
+| `custom` | Required in config | Optional `api_key_env` |
 
-arena.record("ollama/qwen3", "lm-studio/gemma", Result.LEFT, {
-  prompt_id: "explain-rag",
-});
+A model entry may override `base_url`, select any model ID, set generation
+parameters, and read credentials or sensitive headers from environment
+variables. Literal API keys are rejected by the CLI configuration loader.
+Hosted-provider keys are isolated: a standard cloud key is never forwarded
+automatically when that profile is pointed at a different base URL.
 
-for (const standing of arena.standings()) {
-  console.log(standing.name, standing.rating.toFixed(2));
-}
+Useful commands:
 
-console.log("Compare next:", arena.nextPair());
+```bash
+localarena providers
+localarena models ollama
+localarena models custom --base-url http://127.0.0.1:9000/v1
+localarena report results.json --output report.html
+```
+
+For an authenticated custom endpoint:
+
+```bash
+export LOCALARENA_CUSTOM_API_KEY="your-key"
+localarena models custom \
+  --base-url https://inference.example.com/v1 \
+  --api-key-env LOCALARENA_CUSTOM_API_KEY
 ```
 
-The library intentionally does not decide how outputs are generated or judged.
-That keeps private prompts and responses on your machine and lets the same
-engine work with Ollama, LM Studio, vLLM, llama.cpp, local functions, human
-votes, or any provider SDK you explicitly choose.
+Requests make one attempt by default because replaying an ambiguous generation
+can duplicate work or billing. Retries are explicit through
+`policy.max_attempts`.
+
+## Tasks and scoring
+
+Every native task is provider-independent JSON-safe chat messages plus one
+optional evaluator:
+
+| Evaluator | Use |
+| --- | --- |
+| `exact` | Complete answer equality with explicit whitespace/case rules |
+| `contains` | Require all or any configured strings |
+| `regex` | Search or full-output format validation |
+| `json` | Strict JSON validation or type-sensitive JSON comparison |
+| `numeric` | Numeric answer with an absolute tolerance |
+| `model_judge` | Live rubric scoring from another configured model |
+| `null` | Generate and retain run metadata without a score |
+
+Set `"judge_only": true` on a model entry to use that model for
+`model_judge` tasks without evaluating or ranking it as a contestant.
+
+`concurrency` bounds the number of in-flight rows. `repetitions` repeats the
+entire grid. Each model keeps its own base URL, credential, model ID, and
+generation settings, so local and hosted targets can run in the same
+evaluation.
+
+## Results and reports
+
+`results.json` follows the versioned
+[evaluation run schema](schema/localarena-evaluation-run-v1.schema.json). It
+contains:
+
+- one record for every model, task, and repetition;
+- normalized status, timing, finish reason, token use, and attempt count;
+- deterministic or live-judge scores;
+- per-model score, pass rate, error, latency, token, and judge aggregates;
+- Elo standings derived only from comparable successful task scores; and
+- an arena snapshot whose match history can move between Python and
+  JavaScript.
+
+JSON Schema validates the portable document shape. Load untrusted snapshots
+with `run_from_dict` or `runFromSnapshot` as well; the SDK loaders enforce the
+cross-field guarantee that every configured model × task × repetition has
+exactly one row.
+
+The HTML report is a single local file with no scripts, fonts, stylesheets, or
+network requests. It includes a leaderboard, score bars, a model-by-task heat
+map, and expandable row diagnostics.
+
+Prompt text, evaluator criteria and reference answers, generated text, score
+reasons, task metadata, and detailed errors are excluded from saved results
+and reports by default. Non-secret evaluator type and live-judge target
+provenance remain visible so scoring failures are attributable. Opt in only
+when the content is safe to retain:
+
+```bash
+localarena run evaluation.json \
+  --include-content \
+  --output results-with-content.json \
+  --report report-with-content.html
+```
 
-## Schedule a full arena
+Credentials, authorization headers, base URLs, and configured headers are
+never serialized into an evaluation run.
 
-`round_robin` / `roundRobin` returns each unordered pair once per round.
-Pair orientation flips every second round to reduce persistent left/right bias.
+## Python API
 
 ```python
-from localarena import Arena, Result, round_robin
+from localarena import (
+    EvaluationRunner,
+    ExactMatch,
+    ModelTarget,
+    PromptTask,
+    create_provider,
+    write_html_report,
+)
 
-models = ["a", "b", "c"]
-arena = Arena(models)
+provider = create_provider("ollama")
+target = ModelTarget(
+    name="local-model",
+    provider=provider,
+    model="qwen3:0.6b",
+    max_tokens=64,
+    temperature=0,
+)
+task = PromptTask.from_text(
+    "capital",
+    "Reply with exactly Paris.",
+    evaluator=ExactMatch("Paris"),
+)
 
-for left, right in round_robin(models, rounds=2):
-    # Replace this deterministic example with your own comparison.
-    winner = Result.LEFT if len(left) >= len(right) else Result.RIGHT
-    arena.record(left, right, winner)
+run = EvaluationRunner([target], [task], max_concurrency=2).run(
+    name="Python API example"
+)
+print(run.to_json())
+write_html_report(run, "report.html")
 ```
 
-```js
-import { Arena, Result, roundRobin } from "localarena";
+Provider construction does not perform network I/O. Calls occur only through
+`list_models()`, `generate()`, or an evaluation runner.
 
-const models = ["a", "b", "c"];
-const arena = new Arena(models);
+## JavaScript API
 
-for (const [left, right] of roundRobin(models, 2)) {
-  const winner = left.length >= right.length ? Result.LEFT : Result.RIGHT;
-  arena.record(left, right, winner);
-}
+```js
+import {
+  EvaluationRunner,
+  ExactMatch,
+  ModelTarget,
+  PromptTask,
+  createProvider,
+  writeHTMLReport,
+} from "localarena";
+
+const target = new ModelTarget({
+  name: "local-model",
+  provider: createProvider("ollama"),
+  model: "qwen3:0.6b",
+  requestOverrides: { maxTokens: 64, temperature: 0 },
+});
+const task = PromptTask.fromText(
+  "capital",
+  "Reply with exactly Paris.",
+  { evaluator: new ExactMatch("Paris") },
+);
+
+const run = await new EvaluationRunner([target], [task], {
+  maxConcurrency: 2,
+}).run({ name: "JavaScript API example" });
+
+console.log(run.toJSONString());
+await writeHTMLReport(run, "report.html");
 ```
 
-## Portable snapshots
+`AbortSignal` is supported for JavaScript provider calls and complete
+evaluation runs.
 
-Standings are derived from an append-only match history. A snapshot contains
-only the arena settings, contestant metadata, and that history:
+Python's `arun()` stops scheduling new rows when its coroutine is cancelled.
+A synchronous provider request already running in a worker thread continues
+only until that provider call completes or reaches its configured timeout.
 
-```python
-import json
-from localarena import Arena
+## Terminal and container benchmarks
 
-payload = json.dumps(arena.snapshot(), ensure_ascii=False, indent=2)
-restored = Arena.from_json(payload)
-```
+A prompt-only completion is not an official terminal benchmark run. Tasks that
+depend on a repository, shell, tools, container image, mutable state, time
+limits, or a benchmark-specific grader must stay in their authoritative
+harness. An adapter can import that harness result and reproducibility metadata
+into the rating layer; LocalArena does not pretend that sending the task prompt
+to a chat endpoint reproduces the environment.
 
-```js
-const payload = JSON.stringify(arena, null, 2);
-const restored = Arena.fromJSON(payload);
-```
+See [provider and task support](docs/provider-and-task-support.md) for the
+protocol rationale and external-adapter boundary.
 
-The schema-v1 keys are deliberately shared:
+## Elo core
 
-```json
-{
-  "schema_version": 1,
-  "initial_rating": 1000,
-  "k_factor": 32,
-  "contestants": [
-    {"name": "a", "metadata": {}},
-    {"name": "b", "metadata": {}}
-  ],
-  "matches": [
-    {
-      "id": 1,
-      "left": "a",
-      "right": "b",
-      "result": "left",
-      "metadata": {}
-    }
-  ]
-}
-```
-
-See [the JSON Schema](schema/localarena-v1.schema.json) for the serialized
-contract. Restoring a snapshot replays the history instead of trusting stored
-ratings or counters.
+The original dependency-free arena remains available for human or external
+verdicts:
 
-## Core API
+```python
+from localarena import Arena, Result
 
-| Python | JavaScript | Purpose |
-| --- | --- | --- |
-| `Arena(names, initial_rating=..., k_factor=...)` | `new Arena(names, { initialRating, kFactor })` | Create an arena |
-| `add(name, metadata)` | `add(name, metadata)` | Register a contestant |
-| `rating(name)` | `rating(name)` | Read one current rating |
-| `record(left, right, result, metadata)` | `record(left, right, result, metadata)` | Record one match |
-| `standings()` / `leaderboard()` | `standings()` / `leaderboard()` | Return rating-sorted rows |
-| `next_pair()` | `nextPair()` | Choose the least-played pair |
-| `snapshot()` / `to_json()` | `snapshot()` / `JSON.stringify(arena)` | Export schema v1 |
-| `Arena.from_snapshot()` / `from_json()` | `Arena.fromSnapshot()` / `fromJSON()` | Validate and replay |
-| `expected_score(a, b)` | `expectedScore(a, b)` | Compute Elo expectation |
-| `round_robin(names, rounds)` | `roundRobin(names, rounds)` | Build all-pairs schedule |
-
-Results are `"left"`, `"right"`, or `"draw"`. Python also exports the
-`Result` string enum; JavaScript exports a frozen `Result` object.
-
-## Rating behavior
-
-LocalArena uses the conventional Elo expectation:
-
-```text
-E(left) = 1 / (1 + 10 ^ ((right_rating - left_rating) / 400))
+arena = Arena(["model-a", "model-b"])
+arena.record("model-a", "model-b", Result.LEFT)
+print(arena.standings())
 ```
 
-A win scores `1`, a draw `0.5`, and a loss `0`. Both contestants receive equal
-and opposite rating adjustments, so total rating is conserved apart from
-normal floating-point precision.
-
-Elo is an online rating system: changing match order can change the final
-ratings. Keep and share the match log when reproducibility matters.
+The JavaScript package exposes the same `Arena`, `Result`, `roundRobin`, and
+schema-v1 snapshot contract.
 
-## Development
+## Development and release
 
 ```bash
-# Python
 python3 -m pip install -e python
 python3 -m unittest discover -s python/tests
-python3 -m build python
-python3 -m twine check python/dist/*
+python3 scripts/check_runtime_parity.py
 
-# JavaScript
 npm ci
+npm run check
 npm test
-npm audit --audit-level=low
 npm pack --dry-run
 ```
 
-CI tests all supported runtimes. Pushing an immutable `v*` tag runs the release
-workflow, verifies that the tag, Python package, and npm package versions agree,
-then publishes the exact tested artifacts to PyPI and npm.
+CI covers Python 3.10–3.13 and Node.js 18, 20, 22, and 24. A matching immutable
+`v*` tag builds and verifies both distributions, then publishes the exact
+artifacts to PyPI and npm.
 
 ## License
 

package/docs/provider-and-task-support.md

@@ -0,0 +1,165 @@
+# Provider and task support
+
+LocalArena evaluates outputs produced during the current run. It does not ship
+recorded model answers, copy scores from public leaderboards, or substitute
+cached responses when an endpoint is unavailable.
+
+The rating engine remains offline and dependency-free. Provider calls are an
+optional execution layer around that engine: generate two live answers, obtain
+a verdict, and record `left`, `right`, or `draw` with provenance in match
+metadata.
+
+## Provider architecture
+
+The first provider set uses one OpenAI-compatible Chat Completions transport
+with five named presets. A preset supplies safe defaults such as the base URL
+and expected authentication; the request, response, error, and model-listing
+logic stays shared.
+
+| Provider | Location | Default base URL | Chat and model endpoints | Authentication | Release behavior and notes | LocalArena adapter |
+| --- | --- | --- | --- | --- | --- | --- |
+| [llama.cpp server](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) | Local or self-hosted | `http://127.0.0.1:8080/v1` | `POST /chat/completions`; `GET /models` | None by default; Bearer token when the server is started with `--api-key` | Bounded non-streaming requests; use a server alias for a stable model ID, and configure a compatible chat template | Named preset over the shared transport |
+| [Ollama](https://docs.ollama.com/api/openai-compatibility) | Local | `http://127.0.0.1:11434/v1` | `POST /chat/completions`; `GET /models` | No server credential by default; some clients require a non-secret placeholder value, which Ollama ignores | Bounded non-streaming requests; models must be pulled before use | Named preset over the shared transport |
+| [LM Studio](https://lmstudio.ai/docs/developer/openai-compat) | Local | `http://127.0.0.1:1234/v1` | `POST /chat/completions`; `GET /models` | None when authentication is disabled; Bearer token when [authentication is enabled](https://lmstudio.ai/docs/developer/core/authentication) | Bounded non-streaming requests; the selected model must be available to the local server | Named preset over the shared transport |
+| [OpenRouter](https://openrouter.ai/docs/quickstart) | Hosted | `https://openrouter.ai/api/v1` | `POST /chat/completions`; [`GET /models`](https://openrouter.ai/docs/guides/overview/models) | Bearer API key | Bounded non-streaming requests; optional `HTTP-Referer` and `X-OpenRouter-Title` attribution headers | Named preset over the shared transport |
+| [OpenAI](https://developers.openai.com/api/reference/resources/chat) | Hosted | `https://api.openai.com/v1` | `POST /chat/completions`; [`GET /models`](https://developers.openai.com/api/reference/resources/models) | Bearer API key; optional organization and project headers | Bounded non-streaming requests | Reference preset for the shared transport |
+
+For all five presets, this release sends `stream: false`, reads the answer from
+`choices[0].message.content`, and reads model IDs from `data[].id`. Responses
+are buffered only up to the configured byte limit. The transport accepts an
+explicit base URL, so another compatible local or hosted server can be used
+without adding a new protocol implementation.
+
+### Why one transport
+
+All five servers deliberately expose the same small interoperability surface:
+Chat Completions, model listing, Bearer-style headers when authentication is
+needed, and the same core response envelope. Sharing that transport provides:
+
+- the same core request fields and normalized result contract in the Python
+  and JavaScript packages;
+- one place for timeouts, bounded response parsing, retries, response
+  validation, error normalization, and secret redaction;
+- contract tests that apply to every preset;
+- custom-base-URL support without a provider-specific dependency; and
+- a small portable parameter set: `model`, `messages`, `max_tokens`,
+  `temperature`, `seed`, and stop sequences.
+
+A compatibility layer is not a claim that every provider supports every
+vendor-specific feature. Provider-only request fields remain explicit
+pass-through options, and callers should use only fields supported by their
+selected endpoint. A genuinely different wire protocol belongs in a separate
+transport rather than in conditionals scattered through the shared one.
+
+## Live execution and network guarantees
+
+Provider-backed evaluation follows an explicit-network contract:
+
+1. Importing LocalArena, constructing an arena or provider, scheduling pairs,
+   reading standings, and loading or saving snapshots perform no network I/O.
+2. A network request occurs only when the caller explicitly asks to list
+   models, generate an answer, or run a provider-backed task.
+3. Each request goes only to the configured base URL. There is no silent
+   provider fallback, cloud fallback, endpoint discovery, model download, or
+   model pull.
+4. Local presets default to loopback addresses. Hosted presets must be chosen
+   explicitly and require their own credential.
+5. Generation is live. An unreachable endpoint, missing model, invalid
+   response, or timeout is reported as an error; it is never replaced with a
+   bundled or previously recorded answer.
+6. Generation makes one attempt by default. Retries are opt-in; when enabled,
+   they repeat the same configured request and remain visible in run metadata.
+   They do not switch models or providers.
+
+Custom HTTP base URLs are useful for a trusted machine on a private network.
+Hosted credentials should be sent only over HTTPS. The caller remains
+responsible for trusting a custom endpoint and for deciding whether prompts may
+leave the local machine.
+
+## Secret handling
+
+API keys are runtime configuration, not arena data.
+
+- Keys may be supplied directly for the current process or read from a
+  provider-specific environment variable.
+- A key is placed only in the authentication header for the configured
+  endpoint. It is not added to a URL, prompt, response, match metadata, or
+  snapshot.
+- Request diagnostics and exceptions redact authentication and caller-supplied
+  secret headers.
+- Result content retention defaults off. Prompts, evaluator criteria and
+  references, answer text, score reasons, and error details are retained only
+  when the caller explicitly opts in. Non-secret evaluator type and judge
+  target provenance remain available for auditability.
+- Unknown custom exceptions are recorded with generic details so an endpoint
+  or credential embedded in an exception cannot leak into a result. Structured
+  provider failures may add sanitized status, retryability, and attempt counts.
+- Local endpoints work without a real credential unless their operator has
+  enabled authentication.
+- OpenAI and OpenRouter use separate credentials. A missing key fails before a
+  hosted request is sent.
+- Snapshots can contain sanitized provenance such as provider name, model ID,
+  latency, token usage, finish reason, and a non-secret endpoint label. They
+  must never contain authorization headers or raw keys.
+
+## Task taxonomy
+
+The execution boundary is determined by what is needed to run and grade a task,
+not by the benchmark's name.
+
+| Task class | Examples | Execution path | Result recorded by LocalArena |
+| --- | --- | --- | --- |
+| Prompt preference | Two answers judged by a person or a live judge model | Native | `left`, `right`, or `draw`, plus prompt and run provenance |
+| Prompt with a deterministic scorer | Exact or normalized match, structured-output validation, classification, extraction, or caller-supplied scoring code | Native | The scorer's comparison converted to `left`, `right`, or `draw` |
+| Prompt with a rubric or reference | Free-form answer evaluated live against criteria or a reference answer | Native | The live evaluator's verdict and sanitized evidence |
+| Stateful tool or simulator task | A task that needs browser state, an external simulator, a service topology, or another controlled runtime | External adapter | A normalized verdict imported from the task's authoritative runner |
+| Containerized terminal benchmark | Repository repair, shell interaction, tests, filesystem mutation, environment reset, or image-based grading | Official environment and agent harness through an external adapter | The official harness outcome, task ID, and reproducibility metadata |
+| Historical or third-party result | A result created outside the current run | Explicit import only | Marked as imported; never presented as a live LocalArena execution |
+
+### Native prompt-scored tasks
+
+A task runs natively when it can be represented by JSON-safe messages and its
+score can be derived from returned content plus task-local criteria. LocalArena
+can schedule the pair, call each configured provider, invoke the scorer or live
+judge, and record the outcome without taking control of a shell or mutable
+environment. A target marked `judge_only` can score other targets without
+appearing as a contestant or receiving an arena rating.
+
+Native task runs should preserve enough provenance to reproduce the comparison:
+task and prompt identifiers or hashes, provider and model IDs, sampling
+parameters, judge configuration, timestamps, latency, token usage when
+reported, finish reason, and the scorer version. Secrets and sensitive prompt
+content remain opt-in metadata and are excluded by default.
+
+### External terminal and container tasks
+
+A terminal benchmark is more than a prompt. Its result depends on the exact
+repository state, container image, installed tools, shell policy, time limits,
+network policy, environment reset, agent loop, and grader. Reimplementing only
+its prompt through a chat endpoint would not reproduce the benchmark and must
+not be described as an official run.
+
+These benchmarks therefore stay under their published environment and agent
+harness. A narrow external adapter starts or observes that harness and returns
+a normalized run record to LocalArena. At minimum, that record should include:
+
+- benchmark and harness version;
+- task ID and attempt ID;
+- contestant and model configuration;
+- container image digest or equivalent environment identifier;
+- seed, start and finish timestamps, status, and timeout state;
+- authoritative score or pass/fail outcome;
+- references to logs or grader evidence; and
+- the mapping from that outcome to `left`, `right`, or `draw`.
+
+The adapter boundary keeps container lifecycle, tool execution, credentials,
+and benchmark-specific grading outside the dependency-free rating core. It also
+allows an official harness to evolve independently while LocalArena continues
+to schedule comparisons, retain provenance, and calculate standings.
+
+## Support rule
+
+Use the native path when the full task is “messages in, content out, score the
+content.” Use an external adapter when correctness depends on state or tools
+outside that exchange. In both paths, results come from an execution requested
+by the caller; no precomputed leaderboard data is treated as a live run.