eval-harness 0.2.18

package/README.md

@@ -0,0 +1,1046 @@
+# Eval Harness
+
+[中文文档](README.zh-CN.md)
+
+Release: `v0.2.18`
+
+Configurable evaluation harness for runner `work/` packages.
+
+Eval Harness is not bound to OpenCode. It is a runner-based evaluation platform
+for CodeX/Codex CLI, OpenCode, MiMo Code, Claude Code, and future runner
+adapters. See [Runner Adapters](docs/runners.md) for the
+runner model and extension points.
+
+The checked-in default config is a generic template. You can point it at any
+task repository and any submitted work repository through
+`config/eval.local.json`, Jenkins parameters, or a generated
+`eval.effective.json`.
+
+Jenkins setup links:
+
+- [Jenkins usage guide](docs/jenkins.md)
+- [Jenkins parameter comparison](docs/jenkins-parameter-comparison.md)
+- [Agent runner compatibility](docs/agent-runner-compatibility.md)
+
+This project turns a one-off runner submission into a reproducible,
+evidence-driven evolution loop. Instead of asking whether an agent "looked
+smart" in one console session, it asks whether the whole agent system can
+consistently understand the task, modify the work package, produce required
+artifacts, pass validation, and explain the next improvement.
+
+It is designed to let CodeX/Codex drive repeated agent runs, inspect deterministic
+evidence, compare models under the same task/work inputs, and feed the next
+smallest Skill, Agent, knowledge-pack, or Instruction improvement back into the
+work package.
+
+The intended loop is:
+
+```text
+prepare clean task workspace
+-> run selected runner with submitted work/INSTRUCTION.md
+-> collect work/result, reports, logs, validation, and code diff
+-> deterministic score
+-> deterministic accept/investigate/reject decision
+-> next-step recommendations and Codex/LLM judge report
+-> propose one small skill change
+-> rerun and compare
+```
+
+This repository starts with a deterministic runner/evaluator and Jenkins
+pipeline. It writes machine-readable advice for the next iteration, but it does
+not mutate skills automatically yet.
+
+## What You Configure
+
+The evaluator separates three concerns so it can be shared with other teams:
+
+- **Task repository**: the original project or problem package that the selected agent
+  must modify. Jenkins calls this `TASK_REPO_*`; config calls it
+  `source_repo`.
+- **Work repository**: the submitted `work` package that contains
+  `INSTRUCTION.md`, Skills, Agents, helper scripts, and result/report
+  contracts. Jenkins calls this `WORK_REPO_*`; config calls it
+  `submission_repo`.
+- **Execution matrix**: runner/model routes, concurrency, timeout, idle
+  timeout, provider failure grace, validation mode, and optional container
+  execution.
+
+For reproducibility, every Jenkins build writes the actual resolved values to
+`eval.effective.json` and exports it under `runs/eval.effective.json`.
+
+## Project Value
+
+`Eval Harness` is valuable because it makes agent improvement
+observable, comparable, and repeatable across configurable task/work inputs:
+
+- **From subjective logs to hard evidence**: every run is judged from concrete
+  artifacts such as `work/result/output.md`, `work/logs/trace/**`,
+  validation summaries, code diff, final result markers, and evaluator output.
+- **From single-run luck to convergence**: each run produces
+  `evaluation.json`, `judge-input.md`, `recommendations.md`,
+  `next-action.json`, and `skill-improvement-input.json`, so CodeX/Codex can turn
+  failures into the next targeted Skill, Agent, helper, or Instruction change.
+- **From model anecdotes to model comparison**: Jenkins can run any configured
+  runner/model matrix against the same task and work
+  package, then compare score, validation ratio, status, and recovery evidence.
+- **From shared workspace risk to isolated workers**: every agent worker gets
+  its own task copy, work copy, HOME, TEMP, cache, config, logs, and result
+  space, which makes parallel runs safe and failures attributable.
+- **From local demo to real judging shape**: the evaluator assumes the task
+  directory is dynamic, keeps paths platform-neutral, preflights external tools
+  on `PATH`, and separates task fetch, work fetch, prepare, execute, score,
+  aggregate, and export stages.
+- **From stuck jobs to fast feedback**: provider/model configuration errors,
+  unreachable providers, invalid credentials, rate limits, retry exhaustion, and
+  idle agent workers are detected early, recorded, cleaned up, and surfaced
+  in Jenkins.
+- **From vague failures to repair inputs**: validation logs are recovered into
+  evaluator-readable summaries when configured, while provider/runtime,
+  final-artifact, trace, protected-path, validation, semantic-memory,
+  agent-handoff, and judge-review failures are classified separately from
+  work-package quality.
+- **From prompt tweaking to Skill versioning**: the intended optimization unit
+  is small and reviewable: update one Skill, subagent, helper, or Instruction
+  concern, bump its version, rerun the harness, and keep the change only when
+  evidence improves.
+
+In short, this repository is the control plane for evolving a runner
+submission: Jenkins provides the execution surface, the selected runner performs
+the autonomous coding work, Agent Inspector provides model/proxy evidence, and
+the evaluator converts all of that into a repeatable score and next-step advice.
+
+## Current Defaults
+
+- Task repository: placeholder URL in `config/eval.example.json`.
+- Submitted work repository: placeholder URL in `config/eval.example.json`, `work/`.
+- Copy `config/eval.example.json` to `config/eval.local.json` or pass Jenkins `TASK_REPO_*` and `WORK_REPO_*` parameters for a real build.
+- Clone mode: prefer configured SSH URLs when present, fallback to HTTPS
+- Agent model route: no checked-in default; set it in config or Jenkins `MODEL_MATRIX`
+- Jenkins reasoning variant: `high`, applied only when the selected runner exposes a variant/agent flag
+- Agent Inspector proxy: optional, enabled only by config, `AGENT_INSPECTOR=true`, or `agent-inspector/*` model routes
+- Worker timeout: `90` minutes by default, configurable per run or Jenkins build
+- Worker execution contract: default runner commands expose `AGENT_EVAL_TIMEOUT_MINUTES`, reserve `AGENT_EVAL_FINALIZE_BEFORE_MINUTES=5`, and require `work/result/output.md`, `work/reports/FINAL_RESULT.json`, `work/reports/FINAL_RESULT.md`, and a real `work/reports/final-consistency-report.md` even for blocked or partial runs
+- Fetch timeout: `15` minutes by default for task/work clone
+- Idle timeout: `10` minutes without child agent output or watched artifact activity
+- Jenkins concurrency: default `1`, configurable from `1` to `5`
+- Jenkins runner matrix: comma/newline separated runner adapters, assigned by worker index; empty uses `agent.type` from config for every worker
+- Jenkins model matrix: comma/newline separated agent model routes, assigned by worker index
+- Jenkins tool paths: use Jenkins global `OEVAL_PYTHON_CMD`, `OEVAL_EXTRA_PATHS`, `Maven_Home`/`MAVEN_HOME`, or one-build override parameters; the pipeline does not commit host-specific tool paths
+- Jenkins config file: `EVAL_CONFIG` is optional; empty uses local `OEVAL_CONFIG`, then `config/eval.example.json`
+- Jenkins schedule: no checked-in cron trigger; add a Jenkins timer or branch-specific trigger in the deployment environment when scheduled runs are desired
+- Work version gate: `REQUIRE_WORK_VERSION=true` by default for traceable runs; set it to `false` for simple work packages without `versions.json`
+- Runtime isolation: enabled by default for HOME, TEMP, caches, and runner config
+
+## Input Contract
+
+To reuse this evaluator for another problem, provide two independent inputs:
+
+| Input | Required shape | Notes |
+| --- | --- | --- |
+| Task repository | A Git repository containing the project/problem to be repaired. | It is copied to each worker as `workspace/project`. Git metadata is preserved when available so the evaluator can produce diffs. |
+| Work repository | A Git repository containing the submitted work package. | `submission_repo.work_dir` or Jenkins `WORK_REPO_WORK_DIR` must point to the directory that contains `INSTRUCTION.md`. |
+| Work assets | Skills, Agents, helper scripts, and result/report conventions under the work directory. | The default installer supports nested `work/work/skills/*` and flat `work/skills/*` layouts, then installs them into the selected runner's local asset directories, such as `.opencode/*`, `.mimocode/*`, `.codex/*`, or `.claude/*`. |
+| Result artifacts | `work/result/output.md`, `work/reports/FINAL_RESULT.json`, `work/reports/FINAL_RESULT.md`, `work/reports/final-consistency-report.md`, and trace logs. | Missing artifacts are scored as evidence gaps instead of being hidden by console output. |
+
+Local directories can replace either Git clone through `PROJECT_SOURCE` and
+`WORK_SOURCE`. That is useful for debugging, while Git URL parameters are better
+for shared Jenkins jobs.
+
+## Quick Start
+
+The runner is platform-neutral. Use Python 3.10+ including Python 3.12, and keep external tools such as
+`git`, the selected agent CLI (`opencode`, `mimo`, `codex`, or `claude`), `agent-inspector`,
+`mvn`, and Java available on `PATH` or in the Jenkins `EXTRA_PATHS`/`OEVAL_EXTRA_PATHS` configuration.
+
+The expected real scoring baseline is Ubuntu 24.04.4 with Python 3.12, Node
+v24.13.0, OpenJDK 21, Maven 3.9.11, and Maven home at `/usr/local/maven3`
+when exposed as `Maven_Home`, `MAVEN_HOME`, or Jenkins `MAVEN_HOME_OVERRIDE`.
+
+Linux/macOS:
+
+```bash
+python3 -m venv .venv
+. .venv/bin/activate
+python -m pip install -e .
+eval-harness --help
+```
+
+Windows PowerShell:
+
+```powershell
+python -m venv .venv
+.\.venv\Scripts\python -m pip install -e .
+.\.venv\Scripts\eval-harness --help
+```
+
+NPM wrapper:
+
+```bash
+npx eval-harness --help
+npx eval-harness runner-presets
+```
+
+The npm package is a thin Node.js launcher around the bundled Python source. It
+still requires Python 3.10+ on `PATH`, or set `EVAL_HARNESS_PYTHON` to the
+Python executable to use. When run outside this repository, it falls back to the
+package's bundled `config/eval.example.json` and `schemas/` directory.
+
+Without installation:
+
+```bash
+PYTHONPATH=src python -m eval --help
+```
+
+```powershell
+$env:PYTHONPATH = "src"
+python -m eval --help
+```
+
+Copy the example config:
+
+```bash
+cp config/eval.example.json config/eval.local.json
+```
+
+```powershell
+Copy-Item config/eval.example.json config/eval.local.json
+```
+
+Customize the task and submitted work repositories in `config/eval.local.json`:
+
+```json
+{
+  "source_repo": {
+    "url": "https://example.com/course/task.git",
+    "ssh_url": "git@example.com:course/task.git",
+    "ref": "main",
+    "fallback_urls": []
+  },
+  "submission_repo": {
+    "url": "https://example.com/team/work.git",
+    "ssh_url": "git@example.com:team/work.git",
+    "ref": "main",
+    "work_dir": "work",
+    "fallback_urls": []
+  }
+}
+```
+
+For a one-off run without editing a checked-in file, generate an effective config:
+
+```bash
+PYTHONPATH=src python -m eval write-effective-config \
+  --config config/eval.example.json \
+  --output /tmp/oeval/eval.effective.json \
+  --task-repo-url https://example.com/course/task.git \
+  --task-repo-ref main \
+  --work-repo-url https://example.com/team/work.git \
+  --work-repo-ref main \
+  --work-repo-work-dir work
+```
+
+The command also accepts `--task-repo-ssh-url`, `--task-repo-fallback-urls`, `--work-repo-ssh-url`, and `--work-repo-fallback-urls`. Fallback URLs can be comma-separated or newline-separated.
+
+Then use the generated file exactly like a normal config:
+
+```bash
+PYTHONPATH=src python -m eval fetch-task --config /tmp/oeval/eval.effective.json --output /tmp/oeval/inputs/task --replace-existing
+PYTHONPATH=src python -m eval fetch-work --config /tmp/oeval/eval.effective.json --output /tmp/oeval/inputs/work --replace-existing
+PYTHONPATH=src python -m eval prepare --config /tmp/oeval/eval.effective.json --run-id demo-01 --project-source /tmp/oeval/inputs/task --work-source /tmp/oeval/inputs/work --replace-existing
+```
+
+For a real local run, make sure these tools are available on `PATH`:
+
+- `python3`, `python`, or another Python 3 command
+- `git`
+- `node`
+- the selected agent CLI: `opencode`, `mimo`, `codex`, or `claude`
+- `agent-inspector` when Agent Inspector is enabled
+- `mvn` and Java when validation is enabled
+
+When Agent Inspector is enabled, its configured provider/model must be healthy. The evaluator fails fast for unknown provider/model, unreachable provider, invalid credentials, rate limits, and retry exhaustion so Jenkins does not sit on a stuck worker.
+
+Prepare a run:
+
+```powershell
+python -m eval prepare --config config/eval.local.json
+```
+
+Execute the configured agent runner:
+
+```powershell
+python -m eval execute --run runs/<run-id>
+```
+
+Score a run:
+
+```powershell
+python -m eval score --run runs/<run-id>
+```
+
+Prepare, execute, and score:
+
+```powershell
+python -m eval run --config config/eval.local.json
+```
+
+Override the per-worker timeout when needed:
+
+```powershell
+python -m eval run --config config/eval.local.json --timeout-minutes 30
+python -m eval batch --config config/eval.local.json --count 5 --parallel 5 --timeout-minutes 30
+```
+
+If the selected agent CLI is not available yet, validate the harness with dry-run:
+
+```powershell
+python -m eval run --config config/eval.local.json --dry-run
+```
+
+Run five workers in parallel:
+
+```powershell
+python -m eval batch --config config/eval.local.json --count 5 --parallel 5
+```
+
+Run a Jenkins-like local pipeline once. This writes an effective config, fetches
+task/work inputs, runs workers in parallel, aggregates the batch, exports
+artifacts, and optionally validates the exported JSON:
+
+```powershell
+python -m eval run-once `
+  --config config/eval.local.json `
+  --batch-id local-three-model `
+  --count 3 `
+  --parallel 3 `
+  --prefer-ssh `
+  --model agent-inspector/MiniMax-M3 `
+  --model agent-inspector/deepseek-v4-pro `
+  --model agent-inspector/glm-5.2 `
+  --timeout-minutes 90 `
+  --idle-timeout-minutes deepseek-v4-pro=30,default=10 `
+  --variant high `
+  --replace-existing `
+  --validate-artifacts
+```
+
+Use `--task-repo-url`, `--task-repo-ref`, `--work-repo-url`,
+`--work-repo-ref`, and `--work-repo-work-dir` to override repositories for a
+single local run without committing private config. Add `--dry-run` to validate
+the flow without requiring a working agent CLI.
+
+Use `--smoke` for a short real dispatch check. It forces `DRY_RUN=false`,
+`timeout=5`, `idle_timeout=2`, disables validation/work-version/release gates,
+and is useful for proving the selected runner, model route, and Agent Inspector
+chain before a longer evaluation:
+
+```powershell
+python -m eval run-once `
+  --config config/eval.local.json `
+  --smoke `
+  --count 3 `
+  --parallel 3 `
+  --model agent-inspector/MiniMax-M3 `
+  --model agent-inspector/deepseek-v4-pro `
+  --model agent-inspector/glm-5.2
+```
+
+For `agent-inspector/*` model routes, make sure Agent Inspector is running
+before the local run:
+
+```powershell
+agent-inspector
+Invoke-RestMethod http://127.0.0.1:25947/api/health
+```
+
+The expected health response is `{"status":"ok"}`. The proxy URL
+`http://127.0.0.1:25947/proxy` is the API base URL injected into OpenCode/MiMo-like
+runners as `LLM_BASE_URL` and into Claude Code as `ANTHROPIC_BASE_URL`; it is
+not a browser UI page. Use `http://127.0.0.1:25947/` for the local UI. A
+successful dispatch can be confirmed in the selected runner log directory, such
+as `runs/<run-id>/<agent-log-dir>/opencode.log`.
+
+For troubleshooting, the same lifecycle is still available as separate commands.
+This runs one real 90-minute worker through the split stages:
+
+```bash
+export AGENT_EVAL_RUNS_DIR="${TMPDIR:-/tmp}/oeval/manual"
+python -m eval fetch-task --config config/eval.local.json --output "$AGENT_EVAL_RUNS_DIR/inputs/task" --replace-existing
+python -m eval fetch-work --config config/eval.local.json --output "$AGENT_EVAL_RUNS_DIR/inputs/work" --replace-existing
+python -m eval prepare --config config/eval.local.json --run-id manual-01 --project-source "$AGENT_EVAL_RUNS_DIR/inputs/task" --work-source "$AGENT_EVAL_RUNS_DIR/inputs/work" --replace-existing
+python -m eval execute --config config/eval.local.json --run "$AGENT_EVAL_RUNS_DIR/manual-01" --timeout-minutes 90 --idle-timeout-minutes 10
+python -m eval score --config config/eval.local.json --run "$AGENT_EVAL_RUNS_DIR/manual-01"
+```
+
+```powershell
+$env:AGENT_EVAL_RUNS_DIR = Join-Path $env:TEMP 'oeval/manual'
+python -m eval fetch-task --config config/eval.local.json --output "$env:AGENT_EVAL_RUNS_DIR/inputs/task" --replace-existing
+python -m eval fetch-work --config config/eval.local.json --output "$env:AGENT_EVAL_RUNS_DIR/inputs/work" --replace-existing
+python -m eval prepare --config config/eval.local.json --run-id manual-01 --project-source "$env:AGENT_EVAL_RUNS_DIR/inputs/task" --work-source "$env:AGENT_EVAL_RUNS_DIR/inputs/work" --replace-existing
+python -m eval execute --config config/eval.local.json --run "$env:AGENT_EVAL_RUNS_DIR/manual-01" --timeout-minutes 90 --idle-timeout-minutes 10
+python -m eval score --config config/eval.local.json --run "$env:AGENT_EVAL_RUNS_DIR/manual-01"
+```
+
+Dry-run the five-worker scheduler:
+
+```powershell
+python -m eval batch --config config/eval.local.json --count 5 --parallel 5 --dry-run
+```
+
+List and compare runs:
+
+```powershell
+python -m eval list
+python -m eval advise --run runs/<run-id>
+python -m eval compare --baseline runs/<old-run> --candidate runs/<new-run>
+```
+
+Useful platform maintenance helpers:
+
+```powershell
+python -m eval runner-presets
+python -m eval runner-version-check --config config/eval.local.json --markdown-output runs/preflight/runner-version-check.md
+python -m eval clean-runs --config config/eval.local.json --keep 10 --dry-run
+python -m eval release-notes --from-ref <previous-tag> --to-ref HEAD --output docs/release-notes-draft.md
+```
+
+Aggregate runs launched by an external scheduler:
+
+```bash
+export AGENT_EVAL_RUNS_DIR="${TMPDIR:-/tmp}/eval-harness-runs/batch-1"
+python -m eval run --config config/eval.local.json --run-id batch-1-01 --replace-existing
+python -m eval run --config config/eval.local.json --run-id batch-1-02 --replace-existing
+python -m eval aggregate --config config/eval.local.json --batch-id batch-1 --count 2 --parallel 2
+python -m eval export --config config/eval.local.json --batch-id batch-1 --count 2 --output runs --clean-output
+```
+
+```powershell
+$env:AGENT_EVAL_RUNS_DIR = Join-Path $env:TEMP 'eval-harness-runs/batch-1'
+python -m eval run --config config/eval.local.json --run-id batch-1-01 --replace-existing
+python -m eval run --config config/eval.local.json --run-id batch-1-02 --replace-existing
+python -m eval aggregate --config config/eval.local.json --batch-id batch-1 --count 2 --parallel 2
+python -m eval export --config config/eval.local.json --batch-id batch-1 --count 2 --output runs --clean-output
+```
+
+Run five workers through the configured container image:
+
+```powershell
+python -m eval batch --config config/eval.local.json --count 5 --parallel 5 --container
+```
+
+## Directory Model
+
+Each evaluation creates an isolated run directory:
+
+```text
+runs/<run-id>/
+  manifest.json
+  workspace/
+    project/        # clean clone of the original task repository
+    work/           # submitted work directory
+  evaluation/
+    evaluation.json
+    changed-files.md
+    judge-input.md
+    recommendations.md
+    next-action.json
+    skill-improvement-input.json
+    project.diff
+    project.diffstat
+    validation/
+  <agent-log-dir>/
+    opencode.log
+    sessions.json
+    sessions.md
+```
+
+The selected agent runs from `runs/<run-id>/workspace`. This matches the real grading shape: `work/INSTRUCTION.md` is stable, while the task project directory is dynamic.
+
+By default, `prepare` removes copied runtime artifacts such as `work/result/output.md`, `work/reports/FINAL_RESULT.*`, and `work/logs/trace/**`. This prevents stale logs from a previous run from influencing the current score.
+
+## Scoring
+
+The deterministic score checks hard evidence:
+
+- `work/result/output.md` exists.
+- `work/reports/FINAL_RESULT.json` and `FINAL_RESULT.md` exist.
+- The final result says `FINAL_RESULT_FINAL: true` or JSON `is_final: true`.
+- Configured protected paths were not changed.
+- Trace evidence exists under `work/logs/trace/`.
+- Validation summaries exist and pass.
+- Optional external validation commands can be run from the evaluator.
+
+Validation evidence can use either supported helper summary shape:
+command-oriented `results[]` entries or phase-oriented `phases[]` entries with
+a top-level `status`. A `phases[]` summary with all `PASS` phases is counted as
+machine-readable validation evidence. Output-only success claims in
+`work/result/output.md` are still treated as weak evidence until a
+`work/logs/trace/validation/**/summary.json` file exists.
+
+Final report quality is also checked. If
+`work/reports/final-consistency-report.md` is still the helper-created
+placeholder, scoring records a final-artifact finding and routes the next
+action toward finalization/report repair instead of treating a `PASS` marker as
+complete evidence.
+
+The 100-point score is also exposed as explicit dimensions so Jenkins, Codex,
+and downstream dashboards can diagnose convergence without reverse-engineering
+the total:
+
+- `artifacts` `/30`: required result files, trace files, final markers, and report artifacts.
+- `boundary` `/20`: protected task/design/test paths remain untouched.
+- `validation` `/30`: evaluator-readable validation summaries and pass evidence.
+- `final_status` `/20`: the declared final execution status and finality markers.
+
+Generated build outputs such as `target/`, `build/`, `.gradle/`, `__pycache__/`, and compiled archives/classes are recorded in `ignored_generated_changed_paths` but are not treated as protected-path violations. Real source changes under protected paths still fail the boundary check.
+
+The evaluator writes:
+
+```text
+runs/<run-id>/evaluation/evaluation.json
+runs/<run-id>/evaluation/changed-files.md
+runs/<run-id>/evaluation/judge-input.md
+runs/<run-id>/evaluation/recommendations.md
+runs/<run-id>/evaluation/next-action.json
+runs/<run-id>/evaluation/skill-improvement-input.json
+runs/batches/<batch-id>/summary.json
+runs/batches/<batch-id>/summary.md
+runs/batches/<batch-id>/model-report.json
+runs/batches/<batch-id>/model-report.md
+runs/batches/<batch-id>/experiment.json
+runs/batches/<batch-id>/comparison.json
+runs/batches/<batch-id>/comparison.md
+runs/history/batches.jsonl
+runs/history/index.json
+runs/history/index.md
+```
+
+`evaluation.json` contains the numeric score, `subscores`, `score_dimensions`, hard findings, final status, decision, structured recommendations, `failure_taxonomy`, `skill_improvement_input`, `changed_file_summary`, and `final_report_evidence`. `changed-files.md` is the Jenkins scoring-side changed-file report: it lists evaluator-detected project diffs, changed test files, ignored generated outputs, and the agent-reported changed files in `work/reports/FINAL_RESULT.json` when available. `recommendations.md` is the human-readable advice file, while `next-action.json` is the machine-readable next smallest change for Codex or another controller to apply. `skill-improvement-input.json` is the stronger controller input for the next Skill/Agent/Instruction edit: it names the primary failure category, target candidates, evidence paths, next action, and recommended change scope. The judge input is designed for Codex or another LLM reviewer to inspect evidence when the deterministic next action is too broad.
+
+Before scoring, the evaluator also checks for artifacts accidentally written at the workspace root, such as `reports/FINAL_RESULT.json`, `result/output.md`, or `logs/trace/**`. When those root-level artifacts are more complete than the required `work/...` artifacts, the evaluator copies them into `work/reports`, `work/result`, and `work/logs/trace`, records `artifact_recovery.root_workspace`, and leaves a trace under `work/logs/trace/evaluator-recovery/`. This preserves scoring correctness without changing the reusable work package; findings still note that root-level artifacts were recovered, and recommendations classify the issue as `artifact-path-contract`.
+
+The failure taxonomy deliberately separates environment/model/runtime defects from work-package defects. Current categories include provider/runtime, final artifacts, trace logging, artifact path contract, protected paths, validation failure, validation evidence, semantic memory, agent handoffs, changed-test review, coverage matrix, and judge-review fallback. This keeps a provider outage or missing final artifact from being mistaken for a design-repair weakness.
+
+Jenkins exports `runs/eval.effective.json` when an effective config exists. This file records the task repository, submitted work repository, refs, work directory, and fallback URLs used for that build, so shared evaluator runs remain reproducible without checking machine-specific config into Git.
+
+Batch summaries also expose convergence evidence:
+
+- `model`: the actual agent model route used by the worker.
+- `agent_runner` and `agent_log_dir`: the selected runner and exported log directory for the worker.
+- `opencode_primary_session` and `opencode_session_count`: legacy-compatible fields for the main agent session id and total agent/subagent session count extracted from the agent log.
+- `inspector_backfill_status` and `inspector_log_count`: best-effort Agent Inspector `/api/logs` evidence recovery when the agent log does not expose enough session data.
+- `failure_digest`: batch-level root-cause rows with run id, runner, model, outcome, reason, evidence, and next action.
+- `score_dimensions`, `subscores`, `best_subscores`, and `subscore_summary`: the score split used by the evaluator and the best run/model in the batch.
+- `validation_pass_ratio`: passed validation targets over scored validation targets after evaluator recovery.
+- `weak_validation_summary`: output-only validation claims found in `work/result/output.md` when the required machine-readable `work/logs/trace/validation/**/summary.json` evidence is missing. These claims are useful next-iteration input, but they do not replace formal validation summaries in the score.
+- `artifact_recovery`: recovered validation summaries or normalized final status markers, such as `status:SUCCESS->PASS`.
+- `best_by_model`: the best run for each model route in the same batch.
+- `model-report.md/json`: per-model average score, average dimensions, best run dimensions, decision distribution, validation ratio, and top next-action types.
+- `experiment.json`: batch-level comparability metadata such as models, source/work inputs, run ids, and best run.
+- `comparison.md/json`: regression comparison against the latest compatible historical batch, or a named `--baseline-batch-id`.
+- `history/index.md/json` and `history/batches.jsonl`: batch memory used to see score, decision, model, and recommendation trends across Jenkins builds.
+
+Use these fields to separate model quality from process quality. For example, a high score with artifact recovery suggests the model fixed code but ignored the finalization protocol, while `0/0` validation evidence plus a non-empty weak validation summary suggests the model probably ran checks but failed to write evaluator-readable summaries.
+
+Validate archived JSON contracts with:
+
+```powershell
+python -m eval validate-artifacts --root runs --batch-id <batch-id> --count <count>
+```
+
+The validator uses the checked-in `schemas/` directory and has no external Python package dependency.
+
+## Jenkins Parallel Mode
+
+Setup references:
+
+- [Jenkins usage guide](docs/jenkins.md)
+- [Jenkins parameter comparison](docs/jenkins-parameter-comparison.md)
+
+The included `Jenkinsfile` exposes a `CONCURRENCY` choice parameter for the number of concurrent agent workers. The default is `1`, and the allowed range is `1` through `5`. Jenkins first resolves a build-specific config, then materializes shared read-only inputs in separate nodes:
+
+```powershell
+python -m eval write-effective-config --config <EVAL_CONFIG> --output <external-runs>/eval.effective.json [repo overrides]
+python -m eval fetch-task --config <external-runs>/eval.effective.json --output <external-runs>/inputs/task --replace-existing
+python -m eval fetch-work --config <external-runs>/eval.effective.json --output <external-runs>/inputs/work --replace-existing
+```
+
+`EVAL_CONFIG` selects the config file to use for the build. Leave it empty to
+use a Jenkins-local `OEVAL_CONFIG` environment variable, then
+`config/eval.example.json` as the public fallback. Keep private files such as
+`config/eval.local.json` out of Git. `TASK_REPO_URL`, `TASK_REPO_SSH_URL`,
+`TASK_REPO_REF`, and
+`TASK_REPO_FALLBACK_URLS` override the task repository for one Jenkins build.
+`WORK_REPO_URL`, `WORK_REPO_SSH_URL`, `WORK_REPO_REF`,
+`WORK_REPO_WORK_DIR`, and `WORK_REPO_FALLBACK_URLS` do the same for the
+submitted work repository. Empty values keep the defaults from the selected
+config file.
+
+`SMOKE_MODE=true` is the fastest real-chain Jenkins check. It forces
+`DRY_RUN=false`, `TIMEOUT_MINUTES=5`, `IDLE_TIMEOUT_MINUTES=2`,
+`RUN_VALIDATION=false`, `REQUIRE_WORK_VERSION=false`, and disables release
+gates, while still executing the selected runner/model/Inspector path.
+
+`Resolve Effective Config` prints the exact task/work URLs, refs, work
+directory, and fallback URLs used by the build. `Export Artifacts` copies the
+same file to `runs/eval.effective.json`, so another person can inspect or rerun
+the batch with the same input definition.
+
+`PROJECT_SOURCE` can still point at a local task/project directory instead of cloning the task repository; `WORK_SOURCE` can point at a local submitted `work` directory instead of cloning the submitted work repository. Local source parameters take precedence at fetch time. Fetching the original task and fetching the submitted work are intentionally separate Jenkins phases, so a repository/network/configuration failure is visible before any agent worker starts.
+
+The Jenkins stage graph is grouped by the main lifecycle so long runs are easier
+to inspect:
+
+- `Environment Preparation`: tool path setup, Python/Git/evaluator checks, optional Agent Inspector health/model validation, and evaluator unit tests.
+- `Run Planning`: concurrency, timeout, provider-grace, regression, batch-id, run directory, optional Agent Inspector group declaration, and history seed resolution.
+- `Input Materialization`: input directory planning plus task/work source fetch and work-version reporting.
+- `Agent Execution`: the parallel worker group; each branch is labeled with worker index and model name, while prepare, optional Inspector run declaration, asset install, execute, optional Inspector session attach, score, and optional Inspector completion are printed as branch log sections.
+- `Validation And Reporting`: aggregate, optional Agent Inspector evidence export, artifact export, schema validation, run listing, and HTML report publishing.
+
+Each worker runs as a Jenkins `parallel` branch and no longer hides multiple actions behind one `python -m eval run` call:
+
+```powershell
+python -m eval prepare --run-id <run-id> --project-source <inputs/task> --work-source <inputs/work>
+python -m eval inspector-run --run <external-runs>/<run-id> --batch-id <batch-id> --phase create
+python -m eval install-agent-assets --run <external-runs>/<run-id>
+python -m eval execute --run <external-runs>/<run-id> --timeout-minutes <TIMEOUT_MINUTES>
+python -m eval inspector-run --run <external-runs>/<run-id> --batch-id <batch-id> --phase attach
+python -m eval score --run <external-runs>/<run-id>
+python -m eval inspector-run --run <external-runs>/<run-id> --batch-id <batch-id> --phase complete
+```
+
+The parallel branch name includes the worker index and short model name, such as `01 model-a`, so Jenkins Stage View shows which model each worker is running without a wall of per-step nodes.
+
+The external run root is passed through `AGENT_EVAL_RUNS_DIR`, so agent workspaces stay outside the evaluator checkout and cannot accidentally treat this repository as the task project root.
+
+Every branch gets its own isolated run space:
+
+```text
+<external-runs>/<run-id>/
+  inputs/task/         # shared or prefetched original task baseline, outside the worker
+  inputs/work/         # shared or prefetched submitted work baseline, outside the worker
+  workspace/project/   # independent copy of the prefetched task repository
+  workspace/work/      # independent copy of the submitted work directory
+  runtime/home/        # per-run HOME and USERPROFILE
+  runtime/tmp/         # per-run TEMP, TMP, and TMPDIR
+  runtime/cache/       # per-run XDG cache
+  runtime/config/      # per-run XDG config
+  runtime/npm-cache/   # per-run npm cache
+  control/timeout.json # live timeout control file watched during execution
+```
+
+Scoring treats `workspace/project/` as the task project modified by the selected agent. It does not diff the evaluator checkout or the whole Jenkins workspace. If `workspace/project/` remains a Git worktree, project changes are collected with Git. If the copy loses `.git`, the scorer falls back to a tree comparison between the separately fetched task input (`inputs/task`, recorded as `task_input.origin` or `project_source`) and `workspace/project/`.
+
+`ISOLATED_ENV` is enabled by default in Jenkins. The evaluator seeds known runner config locations into each run's runtime home before execution so that workers can stay isolated without sharing writable global state.
+
+After `Prepare`, every worker runs a separate `Install Agent Assets` stage. Default target directories depend on `agent.type`: OpenCode uses `.opencode/*`, MiMo Code uses `.mimocode/*`, Codex CLI uses `.codex/*`, and Claude Code uses `.claude/*`. A flat `work/skills` fallback is also checked for other package layouts. The install report is written under the selected runner log directory.
+
+Before workers start, Jenkins runs `Report Work Version` against the materialized submitted work directory. The console prints the resolved `versions.json`, `work package version`, all registered Skill/Agent versions, and any skill-pack versions discovered from `pack.json` files. With `REQUIRE_WORK_VERSION=true`, missing or unreadable version metadata fails before agent execution, which keeps real test logs tied to the exact work package under evaluation. Set it to `false` when onboarding a simple work package that has not adopted version metadata yet.
+
+Use the Jenkins `AGENT_ASSET_INSTALLS` text parameter to add more mappings after the default install, one per line:
+
+```text
+work/commands=>.opencode/command
+work/references=>.opencode/references
+```
+
+Recommended Jenkins parameters for a first real run:
+
+```text
+EVAL_CONFIG=
+CONCURRENCY=1
+TIMEOUT_MINUTES=90
+IDLE_TIMEOUT_MINUTES=10
+DRY_RUN=false
+SMOKE_MODE=false
+ISOLATED_ENV=true
+STREAM_OUTPUT=true
+CONSOLE_LOG_MODE=important
+MODEL_MATRIX=
+REASONING_VARIANT=high
+PROVIDER_API_FAILURE_GRACE=5
+REQUIRE_WORK_VERSION=true
+AGENT_INSPECTOR=false
+AGENT_INSPECTOR_CONFIG_DIR=<agent-inspector-data-dir-if-used>
+TASK_REPO_URL=<task repository https url>
+TASK_REPO_REF=main
+WORK_REPO_URL=<submitted work repository https url>
+WORK_REPO_REF=main
+WORK_REPO_WORK_DIR=work
+```
+
+For DeepSeek model routes that spend long periods in quiet subagent or subprocess phases, prefer a model-keyed idle timeout while keeping other workers at the default:
+
+```text
+IDLE_TIMEOUT_MINUTES=deepseek-v4-pro=30,default=10
+```
+
+For exploration, raise `CONCURRENCY` up to `5`. Each worker receives an independent task copy, work copy, HOME, TEMP, cache, config, optional Maven local repository, agent log, score output, and final work artifacts.
+
+`TIMEOUT_MINUTES` is the initial per-worker timeout. It accepts a single value (`90`), a worker-position list (`90,120,90`; the last value is reused), or model-keyed entries (`model-a=120,model-b=90,default=90`). Model keys can use the full route such as `agent-inspector/model-a` or the short model name. Jenkins prints the resolved timeout plan before workers start.
+
+`IDLE_TIMEOUT_MINUTES` follows the same single/list/model-keyed shape and also allows `0` to disable idle protection for the matching worker.
+
+Use an OS-local Agent Inspector data directory, for example `~/.agent-inspector` on Linux/macOS or a Jenkins service-account directory on Windows. Do not commit that machine path into this repository.
+
+Use Jenkins global `OEVAL_PYTHON_CMD` and `OEVAL_EXTRA_PATHS` to solidify service-account tools instead of repeating paths on every build. Use `PYTHON_CMD` only when auto-detection is not enough. Empty means Jenkins tries `OEVAL_PYTHON_CMD`, then `python3`/`python` on Linux/macOS and `python`/`python3`/`py -3` on Windows. Python 3.12 is supported; the evaluator avoids runtime dependencies and version-specific schema libraries. Use `MAVEN_HOME_OVERRIDE`, `Maven_Home`, and `MAVEN_HOME` only for configs whose validation commands need Maven. Use `EXTRA_PATHS` for one build or controller-level `OEVAL_EXTRA_PATHS` for scheduled builds when the Jenkins agent does not already expose required tools on `PATH`. The value is interpreted with the current node's path separator at runtime, so Windows, Linux, and macOS jobs can use the same Jenkinsfile without committed host paths. `PREFER_SSH_CLONE=true` tries configured SSH URLs first and falls back to HTTPS when SSH is unavailable; fetch stages also honor `FETCH_TIMEOUT_MINUTES`. For formal regression, enable `REQUIRE_ACCEPT=true` so the aggregate stage returns non-zero when every worker remains below `ACCEPT`; leave it disabled while exploring model behavior. Enable `REQUIRE_NO_REGRESSION=true` when a batch must not regress against `runs/history`; use `BASELINE_BATCH_ID` to compare against a specific batch and `REGRESSION_TOLERANCE` to allow a small best-score drop.
+
+When Agent Inspector is enabled, Jenkins runs `<python> -m eval check-inspector-models` against every entry in `MODEL_MATRIX`. This fails fast when Agent Inspector is unhealthy, a provider is unreachable, or a model name is not registered, instead of discovering the mistake in separate worker branches. Jenkins also runs `<python> -m eval check-inspector-mcp --batch-id <batch-id>` before declaring the Inspector group, so missing Streamable HTTP MCP tools are recorded as `runs/batches/<batch-id>/inspector-mcp-check.json` and `.md` with an `error_kind`.
+
+For non-dry real runs, Jenkins also runs `<python> -m eval check-tools` for executables listed in `WORKER_TOOLS`; leave it empty to use the selected runner default. Add `java,mvn` when the selected config uses Maven validation. `DRY_RUN=true` skips this worker-tool gate unless evaluator-side validation is enabled, so scheduler smoke tests can run on a lighter controller node.
+
+To compare two model routes in one real batch:
+
+```text
+CONCURRENCY=2
+MODEL_MATRIX=agent-inspector/model-a,agent-inspector/model-b
+DRY_RUN=false
+```
+
+The first worker uses `model-a` and the second worker uses `model-b`. If `CONCURRENCY` is larger than the number of entries, Jenkins reuses the last model entry for remaining workers.
+
+To compare multiple runner adapters in one Jenkins job, set `RUNNER_MATRIX` by worker index and keep `MODEL_MATRIX` aligned by worker index:
+
+```text
+CONCURRENCY=4
+RUNNER_MATRIX=codex,opencode,mimocode,claude
+MODEL_MATRIX=agent-inspector/MiniMax-M3,agent-inspector/deepseek-v4-pro,agent-inspector/glm-5.2,agent-inspector/MiniMax-M3
+DRY_RUN=false
+```
+
+Each worker writes a runner-specific effective config before `prepare`, but all workers reuse the same fetched task/work input snapshot. If `CONCURRENCY` is larger than the number of runner entries, Jenkins reuses the last runner entry for remaining workers.
+
+The Jenkinsfile ships without an active cron trigger. If you want scheduled
+runs, add a Jenkins timer in the job or add a branch-local trigger such as:
+
+```text
+trigger: H 22 * * *
+CONCURRENCY=3
+DRY_RUN=false
+MODEL_MATRIX=agent-inspector/model-a,agent-inspector/model-b,agent-inspector/model-c
+```
+
+`H 22 * * *` runs once during the Jenkins controller's local 22:00 hour and lets Jenkins choose a stable minute. Manual builds use the submitted parameters; timer-triggered builds, when you configure one in your environment, switch to the scheduled execution mode and reuse the configured `MODEL_MATRIX`. The nightly batch id uses the `nightly-<build-number>` prefix unless `BATCH_ID` is explicitly provided.
+
+`STREAM_OUTPUT` is enabled by default, but Jenkins uses `CONSOLE_LOG_MODE=important` by default. In this mode the controller console mirrors start, heartbeat, timeout, provider failure, error, build result, score, and session summary lines while the full stdout/stderr transcript is still written to `runs/<run-id>/<agent-log-dir>/opencode.log`. Use `CONSOLE_LOG_MODE=full` only for short single-worker debugging because high-volume agent JSON streams can put pressure on the Jenkins controller. Use `CONSOLE_LOG_MODE=off` when the archived agent log and HTML report are enough.
+
+Each isolated worker also receives its own Maven local repository under `<run>/runtime/m2-repository` via `MAVEN_OPTS=-Dmaven.repo.local=...`. This avoids concurrent workers corrupting or locking a shared service-account `.m2` repository while still letting project-declared Maven commands run normally.
+
+Runner commands are resolved from each worker's `agent.type`; `RUNNER_MATRIX` overrides that value per worker. OpenCode maps `REASONING_VARIANT` to `--variant`; Codex CLI and Claude Code ignore it. MiMo Code defaults to the harness-managed `harness-build` agent and ignores reasoning-style values such as `high`, `max`, `medium`, and `low` so Jenkins reasoning settings do not accidentally become MiMo agent names. Use `agent.variant` or a MiMo-specific command only when you intentionally want a MiMo agent such as `build`, `plan`, or `compose`. The default MiMo command passes `--dangerously-skip-permissions`, `--dir={workspace}`, and writes an isolated `harness-build` agent config with `external_directory=*` allowed for non-interactive Jenkins runs. Claude Code's own `--agent` flag selects a named Claude agent rather than a reasoning level, so configure it only in a Claude-specific command when needed. Codex and Claude strip the `agent-inspector/` route prefix before passing `--model`; Codex also normalizes display names such as `MiniMax M3` to `MiniMax-M3`. For OpenCode and MiMo Code, the same selected route is written into the isolated runtime config as `model`; MiMo Code also pins both `harness-build` and built-in `build` to that route so internal sessions do not fall back to `mimo-auto`. For Claude Code, the same resolved model is also pinned through `ANTHROPIC_MODEL`, `CLAUDE_CODE_SUBAGENT_MODEL`, alias model env vars, and isolated `.claude/settings.json`, so Claude subagents do not fall back to a default Haiku/Sonnet route during Inspector runs. Agent Inspector is optional: Jenkins prepares it only when `AGENT_INSPECTOR=true` or a model route uses `agent-inspector/*`. When enabled, the evaluator injects `LLM_BASE_URL=http://127.0.0.1:25947/proxy` before launching OpenCode/MiMo-compatible runners, can write isolated provider config for Codex/OpenCode/MiMo, and injects `ANTHROPIC_BASE_URL` plus `ANTHROPIC_API_KEY` before launching Claude Code. Runs that request `agent-inspector/*` also record an `inspector_route` check in `manifest.json` and `evaluation/evaluation.json`; runtime evidence such as `mimo/mimo-auto` during an Inspector run is reported as `inspector route mismatch`. Jenkins and local runs record runner version evidence through `runner-version-check` and each run's `manifest.json`. The shared runner prompt also asks workers to write full helper/finalization output to `work/logs/trace` before previewing it, instead of executing helpers through truncating pipes such as `Select-Object -First` or `head`.
+
+For Windows service deployments, do not leave the Jenkins controller at tiny defaults such as `-Xmx256m` when running three or more agent workers. A practical local baseline is `-Xmx2g`; use `-Xmx4g` when publishing full reports or retaining many active builds.
+
+After each agent worker exits, the evaluator parses `<agent-log-dir>/opencode.log` and writes `<agent-log-dir>/sessions.json` plus `sessions.md`. These files list the primary session, subagent sessions, parent-child relationships, agent names, model routes, token totals, error counts, and a best-effort Agent Inspector UI link such as `http://127.0.0.1:25947/?sessionId=<session-id>`. If the log does not expose enough session data, the evaluator attempts an Agent Inspector `/api/logs` backfill. Backfill now prefers the worker's `sessionId` or `clientPid` before model-wide queries, writes the full compact evidence set to `<agent-log-dir>/inspector-logs.json`, and keeps a preview table plus count/query metadata in `sessions.md`. The Jenkinsfile also uses Agent Inspector's Streamable HTTP MCP endpoint at `/api/mcp` to declare a batch group before workers start, declare each run after prepare, attach the discovered primary session after execute, update run status after score, and export group evidence before artifacts are copied back. MCP failures are classified as `mcp-unreachable`, `mcp-tool-missing`, `mcp-timeout`, `mcp-protocol`, or `mcp-error` and recorded as Inspector evidence under `<agent-log-dir>/inspector-*.json`, `runs/batches/<batch-id>/inspector-mcp-check.json`, or `runs/batches/<batch-id>/inspector-evidence.json`; they do not change the evaluator score because the agent log remains the source of truth.
+
+Heartbeat means the evaluator wrapper and child process are still alive; it does not prove the agent is making useful progress. `IDLE_TIMEOUT_MINUTES` is enabled by default at 10 minutes: if a worker produces no child output or watched artifact activity for that long, the evaluator writes `<agent-log-dir>/process-diagnostics-*.json` and `.md`, terminates it, records an `IDLE TIMEOUT` marker in the agent log, scores whatever partial artifacts exist, and lets the batch aggregate continue. The same process diagnostics are written before total timeout, provider fail-fast, and manual interrupt cleanup. They include the process tree, command lines, elapsed/idle timings, watched artifact paths, recent session/subagent activity, and recent log tail, so quiet subagent or subprocess hangs can be inspected after Jenkins archives `runs/**`. Watched artifact activity is checked by the Python runner, not by platform shell tools, and defaults to `workspace/work/logs/trace`, `workspace/work/reports`, and `workspace/work/result`. When those files change during a quiet subagent phase, Jenkins prints `[artifact-activity] ...` and the idle timer is reset. When Agent Inspector is enabled, the evaluator also polls `/api/logs` for the worker's known OpenCode session ids every `SESSION_ACTIVITY_CHECK_SECONDS` / `execution.session_activity_check_seconds` seconds, prints `[session-activity] ...`, and resets the idle timer only when that worker's own session has new Inspector evidence. `SUBAGENT_IDLE_GRACE_WINDOWS` and `execution.subagent_idle_grace_windows` default to `1`, granting one extra idle window when recent agent log activity shows a subagent was active; set it to `0` for strict fail-fast idle behavior.
+
+The total worker timeout can also be adjusted while a worker is already running. Each execute step writes and watches `<external-runs>/<run-id>/control/timeout.json`; the runner checks it every few seconds and records `TIMEOUT CONTROL UPDATED` in the agent log when the value changes. From another terminal or Jenkins helper job, update a running batch with:
+
+```bash
+python -m eval update-timeout --runs-dir <external-runs> --batch-id <batch-id> --count <workers> --timeout-minutes 120
+```
+
+Use `--workers 1,3` to update selected workers, or `--run <external-runs>/<run-id>` when you know the exact run directory. This changes the live timeout used by the evaluator wrapper; Jenkins build parameters remain immutable for the already-started build.
+
+If the agent reports a fatal model/provider setup problem, for example rate limiting, invalid API credentials, unknown model/provider, or provider connection failure, the evaluator writes a `PROVIDER FAILURE DETECTED` marker, terminates the agent process tree, and returns exit code `125`. Generic provider API stream failures are treated as transient by default: `PROVIDER_API_FAILURE_GRACE` and `execution.provider_api_failure_grace` default to `5`, so the evaluator writes `PROVIDER FAILURE WARNING` markers and lets the worker continue until the grace count is exceeded. Set the value to `0` for fail-fast behavior. This keeps Jenkins from waiting for the full idle timeout when the external model provider is clearly unavailable, while still giving models a chance to recover from temporary stream interruptions. Agent Inspector startup/health failures stop before the worker starts when Inspector is enabled. The pipeline still scores and exports partial artifacts so the failure reason remains visible in the agent log and the batch summary. If the worker is interrupted after producing validation logs but before writing final artifacts, scoring first tries helpers listed in `execution.final_result_recovery_helpers`, including a legacy helper argument set for older work packages. If helper recovery is unavailable or incompatible, the harness writes built-in `PARTIAL` final artifacts and records `final-result` in the recovery column; recovered artifacts do not hide failing validation.
+
+Provider failure detection is runner-aware. For structured JSON/JSONL streams, the harness scans only explicit error carriers such as top-level `error`, `request/response/session/turn failed`, Claude `result` events with `is_error=true` or an error subtype, `system` API error events, `level=ERROR/FATAL/CRITICAL`, and explicit runtime error payloads. It deliberately ignores assistant/model analysis, user tool results, tool-use events, Codex `command_execution` output, and agent messages, so business-domain text such as `401`, `403`, `FORBIDDEN`, `UNAUTHORIZED`, `/api/v1`, or `errorCode` in generated code, tests, validation logs, or reports is treated as task evidence rather than a provider outage. Plain stderr/stdout is still scanned for real provider tokens such as connection failures, invalid credentials, rate limits, unknown models, and retry exhaustion.
+
+After all branches finish, Jenkins runs these as separate nodes:
+
+```text
+<python> -m eval aggregate --batch-id <batch-id> --count <CONCURRENCY> --parallel <CONCURRENCY>
+<python> -m eval inspector-export-group --batch-id <batch-id>
+<python> -m eval export --batch-id <batch-id> --count <CONCURRENCY> --output runs --clean-output
+<python> -m eval validate-artifacts --root runs --batch-id <batch-id> --count <CONCURRENCY>
+<python> -m eval list
+```
+
+Before aggregation, Jenkins seeds the temporary external run directory from the previous workspace `runs/history` when it exists. If the workspace was wiped, the seed step is skipped and the current batch becomes a new baseline. `aggregate` writes `summary.json`, `summary.md`, `model-report.json`, `model-report.md`, `experiment.json`, `comparison.json`, `comparison.md`, and `history/**` under the external runs directory. `export` copies the consumable artifacts back into the Jenkins workspace `runs/` directory for archiving:
+
+- `runs/batches/<batch-id>/summary.json`
+- `runs/batches/<batch-id>/summary.md`
+- `runs/batches/<batch-id>/model-report.json`
+- `runs/batches/<batch-id>/model-report.md`
+- `runs/batches/<batch-id>/experiment.json`
+- `runs/batches/<batch-id>/comparison.json`
+- `runs/batches/<batch-id>/comparison.md`
+- `runs/batches/<batch-id>/inspector-mcp-check.json`
+- `runs/batches/<batch-id>/inspector-group.json`
+- `runs/batches/<batch-id>/inspector-evidence.json`
+- `runs/history/**`
+- `runs/<run-id>/manifest.json`
+- `runs/<run-id>/evaluation/**`
+- `runs/<run-id>/<agent-log-dir>/**`
+- `runs/<run-id>/workspace/work/result/**`
+- `runs/<run-id>/workspace/work/reports/**`
+- `runs/<run-id>/workspace/work/logs/trace/**`
+- `runs/index.md`
+- `runs/index.html`
+
+Input snapshots are exported with heavy generated folders filtered out (`.git`, `target`, `build`, `node_modules`, `.m2`, and similar cache/build directories). The copied `runs/inputs/task` and `runs/inputs/work` remain usable for reruns, but Jenkins artifacts stay small enough for the UI and API to load comfortably.
+
+Codex should read the archived `runs/batches/<batch-id>/summary.md` first, then inspect `comparison.md`, `history/index.md`, `model-report.md`, `experiment.json`, `inspector-mcp-check.md`, `inspector-evidence.md`, and the best run's `evaluation/skill-improvement-input.json`, `evaluation/next-action.json`, `evaluation/recommendations.md`, `evaluation/judge-input.md`, `<agent-log-dir>/sessions.md`, `<agent-log-dir>/inspector-*.md`, and agent log for the next skill, agent, or Instruction improvement. The summary and model report also expose the work package's semantic-memory score, which is derived from `work/logs/trace/memory/semantic-memory-events.jsonl` and helps identify runs that scored similarly but differed in rule/flow/route/final-checklist evidence.
+
+Jenkins does not natively render runner messages as a chat UI. The evaluator therefore exports `runs/index.html`, which renders the batch table, score dimensions and semantic-memory score for each run, agent session summaries, raw artifact links, log/output previews, evaluator recommendations, and next-action JSON. Full logs and result artifacts remain available as links instead of being fully embedded in the HTML. If the Jenkins HTML Publisher plugin is installed, the `Jenkinsfile` publishes this as `Eval Harness Report`; otherwise the same file is still archived under build artifacts.
+
+Recommended Jenkins plugins for this job:
+
+- Already required by the current pipeline shape: Pipeline, Git, Timestamper, Build Timeout, Workspace Cleanup, Pipeline Stage View.
+- Strongly recommended: HTML Publisher for the structured report page, AnsiColor for readable colored console output, Log Parser for highlighting timeout/error markers.
+- Optional: Pipeline Utility Steps for richer Jenkinsfile-side JSON summaries, Blue Ocean for a different parallel pipeline view.
+
+Default decisions:
+
+- `ACCEPT`: score is at or above `scoring.thresholds.accept_score`, final status is `PASS`, and no hard evidence is missing.
+- `INVESTIGATE`: enough evidence exists to inspect or compare, but the run is not ready to accept.
+- `REJECT`: final artifacts, trace logs, status, or protected-path boundaries are broken.
+
+## Agent Runners
+
+The harness currently supports these `agent.type` values:
+
+- `codex` or `codex-cli`: CodeX/Codex CLI runner.
+- `opencode`: OpenCode CLI runner.
+- `mimocode` or `mimo`: MiMo Code runner.
+- `claude` or `claude-code`: Claude Code runner.
+
+Each runner has its own default command, log directory, runtime profile seeding, and default work asset install targets. Use `execution.agent.command`, `execution.<runner>.command`, or `execution.<runner>_command` to override the default command without changing Python code.
+
+Default runner commands include the harness execution contract: respect `AGENT_EVAL_TIMEOUT_MINUTES`, leave time for finalization, write the required final artifacts, and avoid UTF-8 BOM output for Windows source edits. If you override a runner command, keep those requirements in the prompt.
+
+MiMo Code default runs are non-interactive by design. The harness writes an isolated `harness-build` MiMo agent that allows all tool permissions, including `external_directory=*`, and launches `mimo run` with `--dangerously-skip-permissions`, `--dir={workspace}`, and `--agent harness-build`. MiMo's built-in `build` agent leaves `external_directory=*` as `ask`, which becomes an auto-reject under Jenkins; the harness-managed agent avoids that approval prompt without modifying the user's global MiMo profile.
+
+CodeX/Codex CLI example:
+
+```json
+{
+  "agent": {
+    "type": "codex",
+    "executable": "C:\\Users\\you\\AppData\\Local\\OpenAI\\Codex\\bin\\<version>\\codex.exe"
+  },
+  "execution": {
+    "codex_command": [
+      "{agent_executable}",
+      "exec",
+      "--json",
+      "--skip-git-repo-check",
+      "--dangerously-bypass-approvals-and-sandbox",
+      "--cd",
+      "{workspace}",
+      "Read and execute work/INSTRUCTION.md. Work autonomously, do not ask for human input, and write all required result artifacts."
+    ]
+  }
+}
+```
+
+On Windows Jenkins service accounts, Codex CLI resolution first honors `agent.executable`, `codex.executable`, `execution.codex_executable`, `execution.codex.executable`, or `OEVAL_CODEX_EXECUTABLE`, then auto-discovers the current user's `%LOCALAPPDATA%\OpenAI\Codex\bin\*\codex.exe` before falling back to `PATH`. Prefer a runner-specific local override when the service account uses a nonstandard install; do not commit machine paths into shared config.
+
+The common `--model` override works for CodeX/Codex CLI as well:
+
+```powershell
+python -m eval execute --run runs/<run-id> --model gpt-5
+```
+
+Claude Code example:
+
+```json
+{
+  "agent": {
+    "type": "claude-code",
+    "executable": "claude"
+  },
+  "execution": {
+    "claude_command": [
+      "{agent_executable}",
+      "-p",
+      "--output-format",
+      "stream-json",
+      "--verbose",
+      "--dangerously-skip-permissions",
+      "Read and execute work/INSTRUCTION.md. Work autonomously, do not ask for human input, and write all required result artifacts."
+    ]
+  }
+}
+```
+
+For Agent Inspector routes, `agent-inspector/MiniMax-M3` is passed to Claude Code
+as `--model MiniMax-M3`, and the runner injects `ANTHROPIC_BASE_URL` plus
+`ANTHROPIC_API_KEY` into the isolated worker environment. The same model is
+also pinned for Claude subagents with `CLAUDE_CODE_SUBAGENT_MODEL`; for example
+`agent-inspector/glm-5.2` becomes `--model glm-5.2` and
+`CLAUDE_CODE_SUBAGENT_MODEL=glm-5.2`.
+
+Claude Code subagent model routing is intentionally pinned by the harness. Claude
+Code can spawn internal `Agent` / `Explore` subagents, and those subagents may
+have their own model defaults. During an Agent Inspector run, the harness writes
+the resolved worker model into the isolated worker's environment and
+`.claude/settings.json`:
+
+- `ANTHROPIC_MODEL=<resolved model>` for the main Claude Code session.
+- `CLAUDE_CODE_SUBAGENT_MODEL=<resolved model>` for spawned subagents.
+- Claude alias model environment variables, including the Haiku/Sonnet aliases,
+  so agent frontmatter defaults do not drift to another provider route.
+
+If a Claude worker fails with `PROVIDER FAILURE DETECTED: auth-config` and the
+agent log contains `Failed to authenticate. API Error: 403 Request not allowed`,
+check whether subagent log entries show a `resolvedModel` different from the
+requested Inspector model. For example, a run launched with
+`agent-inspector/glm-5.2` should not show subagents resolved as
+`claude-haiku-*`. After this pinning behavior is active, the same run should show
+the main worker and subagents using the same resolved model route.
+
+## Agent Command
+
+The selected runner command is normally supplied by the runner adapter.
+`config/eval.example.json` intentionally does not pin `opencode_command`,
+`codex_command`, `mimocode_command`, or `claude_command`. Override the command
+only when the default runner command is not suitable for your environment.
+
+Supported placeholders:
+
+- `{workspace}`: absolute workspace directory.
+- `{instruction}`: workspace-relative `work/INSTRUCTION.md`.
+- `{run_id}`: run identifier.
+- `{timeout_minutes}`: timeout minutes.
+
+Prefer these override locations:
+
+- `execution.agent.command` for a private single-runner config.
+- `execution.<runner>.command` or `execution.<runner>_command` for shared configs that may switch `agent.type`.
+
+Override shape:
+
+```json
+{
+  "agent": {
+    "type": "opencode"
+  },
+  "execution": {
+    "timeout_minutes": 90,
+    "agent": {
+      "command": [
+        "opencode",
+        "run",
+        "Read and execute the attached INSTRUCTION.md. Work autonomously, do not ask for human input, and write all required result artifacts.",
+        "--format",
+        "json",
+        "--print-logs",
+        "--log-level",
+        "INFO",
+        "--dangerously-skip-permissions",
+        "--file={instruction}"
+      ]
+    },
+    "agent_inspector": {
+      "enabled": false,
+      "port": 25947,
+      "base_url": "http://127.0.0.1:25947",
+      "proxy_url": "http://127.0.0.1:25947/proxy",
+      "health_url": "http://127.0.0.1:25947/api/health",
+      "providers_url": "http://127.0.0.1:25947/api/providers",
+      "mcp_url": "http://127.0.0.1:25947/api/mcp",
+      "mcp_enabled": false,
+      "mcp_required_tools": [],
+      "required_model": "",
+      "start_command": ["agent-inspector", "--background", "--no-open", "--port", "25947"],
+      "restart_command": ["agent-inspector", "--background", "--no-open", "--port", "25947", "--force-restart"],
+      "doctor_command": ["agent-inspector", "doctor", "--port", "25947"],
+      "startup_timeout_seconds": 15
+    }
+  }
+}
+```
+
+The command runs from `runs/<run-id>/workspace`. The evaluator resolves tools with `shutil.which()` before execution, so Windows command shims such as `opencode.cmd` work with Python `subprocess(shell=False)`.
+Legacy `execution.opencode_command` is still accepted for existing OpenCode configs, but new shared configs should use the generic or runner-specific keys above.
+
+The CLI can override the configured agent model without rewriting the whole command:
+
+```powershell
+python -m eval execute --run runs/<run-id> --model agent-inspector/model-a
+python -m eval batch --config config/eval.local.json --count 2 --parallel 2 --model agent-inspector/model-a --model agent-inspector/model-b
+```
+
+The CLI can also inject a runner-specific reasoning or agent variant without rewriting the command:
+
+```powershell
+python -m eval execute --run runs/<run-id> --variant high
+python -m eval batch --config config/eval.local.json --count 5 --parallel 5 --variant high
+```
+
+Provider API stream failure tolerance is also configurable without rewriting `config/eval.local.json`:
+
+```powershell
+python -m eval execute --run runs/<run-id> --provider-api-failure-grace 5
+python -m eval batch --config config/eval.local.json --count 3 --parallel 3 --provider-api-failure-grace 5
+```
+
+To route through Agent Inspector, configure an OpenCode provider named `agent-inspector`, set `execution.agent_inspector.enabled=true`, or pass Jenkins `AGENT_INSPECTOR=true` with `MODEL_MATRIX=agent-inspector/<model>`. Agent Inspector is expected on `PATH` as `agent-inspector`. Jenkins checks `http://127.0.0.1:25947/api/health` only when Inspector is enabled or inferred from a model route. Set the Jenkins `AGENT_INSPECTOR_CONFIG_DIR` parameter when the service account should use an existing provider directory, for example `~/.agent-inspector` on Linux/macOS or an equivalent service-account data directory on Windows. If container mode is enabled, update `execution.agent_inspector.proxy_url` to a host-reachable URL such as `http://host.docker.internal:25947/proxy`.
+
+## Container Mode
+
+Container mode keeps each agent execution isolated while the host evaluator still owns prepare and score.
+
+Enable it with CLI:
+
+```powershell
+python -m eval execute --run runs/<run-id> --container
+python -m eval batch --config config/eval.local.json --count 5 --parallel 5 --container
+```
+
+Or enable it in config:
+
+```json
+{
+  "container": {
+    "enabled": true,
+    "engine": "docker",
+    "image": "eval-harness-worker:latest",
+    "workspace_mount": "/workspace",
+    "extra_args": []
+  }
+}
+```
+
+The wrapper command is:
+
+```text
+docker run --rm -v <host-workspace>:/workspace -w /workspace <image> <opencode-command>
+```
+
+Use `--no-container` to force local execution even when the config enables containers.
+
+See `containers/opencode-worker.Dockerfile.example` for a starting point. The image must contain Python 3.12-compatible runtime, Git, the selected agent runner, and any extra validators or credentials needed by the selected config.
+
+## Tests
+
+Run unit tests without external services:
+
+```bash
+python -m unittest
+```
+
+```powershell
+python -m unittest
+```
+
+For the optional development tools:
+
+```bash
+python -m pip install -e ".[dev]"
+python -m pytest
+python -m coverage run -m unittest
+python -m ruff check .
+```
+
+## Evolution Policy
+
+Recommended later automation:
+
+- Change only one skill, subagent, or helper concern per iteration.
+- Use `evaluation/next-action.json`, `evaluation/recommendations.md`, `runs/batches/<batch-id>/summary.md`, and `runs/batches/<batch-id>/model-report.md` to choose the smallest next change.
+- Bump skill versions after every behavior change.
+- Sync `.opencode/skills` for directory skills and `.opencode/agents` for top-level subagents.
+- Rerun this evaluator.
+- Keep the change only when the score, decision, and hard-failure pattern improve.