@giamme/forge 0.0.0-stage → 0.8.0

package/SKILL.md

@@ -0,0 +1,122 @@
+---
+name: forge
+description: Dispatch implementation and independent diff review to user-selected models through local agent CLIs. Use for /forge or explicit requests to delegate coding or review; explanation-only questions do not authorize dispatch.
+argument-hint: '"<goal>" --dwarf <alias>[:effort[:harness]] [--qa <alias>] [--planner <alias>] [--yolo-dwarf] [--yolo-qa] [--decompose-level low|medium|high] [--fractal|--no-fractal|--fractal-auto-decompose] [--fractal-planner <spec>] [--jev|--no-jev] [--jev-shadow] [--no-memory] [--no-ripwire] [--ui|--no-ui] [--timeout <seconds>] [--retry-failed <n>] [--escalate-after <n>] [--qa-threshold P0|P1|P2|P3] [--shadow-qa <spec>[,<spec>]] [--shadow-rate <0..1>] [--explore <0..0.5>] [--fail-first]'
+allowed-tools: [Bash, Read]
+---
+
+# Forge
+
+Run an implementer (**dwarf**) followed by an independent **QA** review of its actual
+changes. Use the scripts for dispatch and capture; do not reconstruct harness commands.
+For explanation requests, explain the workflow without spending model quota.
+
+## Choices and boundaries
+
+- A dwarf model must be chosen by the user. Ask when it is missing; never invent one.
+- QA defaults to `opus`; effort defaults to `medium` for dwarf and `xhigh` for QA/planner.
+- Specs are `alias[:effort[:harness]]`, e.g. `sol:xhigh:openclaude`. Resolve using
+  [registry.tsv](registry.tsv); report any effort clamp. Literal model IDs require a harness.
+- Bypass flags apply only when explicitly requested for that role (`--yolo-dwarf`,
+  `--yolo-qa`). Bash access can mutate files despite disabled editing tools.
+- Ask **“Use Fractal for this run? [y/N]”** once before an interactive run, unless
+  the user supplied `--fractal`, `--no-fractal`, or `--fractal-auto-decompose`. Pass the answer explicitly to
+  the runner. Unattended runs default off. Installation never activates Fractal.
+  Resume and explicit retry retain the recorded choice. Read [Fractal](references/fractal.md)
+  when selected; Fractal completion never substitutes for Forge QA acceptance.
+- Ask **“Open forge-ui to watch this run? [y/N]”** once at the start of every new
+  interactive run, unless the user supplied `--ui` or `--no-ui`. Pass the answer to `plan`
+  or `forge-solo.sh` and relay the `forge-ui:` URL they print. Never ask on resume or retry;
+  unattended runs leave it off. A UI failure never changes the run. Read [forge-ui](references/ui.md).
+- A finished solo run, `plan` or `run` may end with one stderr line, `forge X is available
+  (you have Y): forge update` (checked at most once a day). Relay it once; run the update only
+  when the user asks (`forge update`, then restart the harness session). `FORGE_UPDATE_CHECK=off` silences it.
+- `--fractal-auto-decompose` enables automatic split-or-atomic decisions before each
+  eligible node implements. Include the effective planner and bounds in the approval
+  preview; child decisions need no additional user approval inside those bounds.
+  Forward `--fractal-planner <spec>` to the runner. When omitted, forward a selected
+  top-level `--planner` as `--fractal-planner`; otherwise each task's initial root model
+  plans all its descendants. Ordinary `--fractal` keeps worker-requested decomposition.
+- One implementer per tree. Solo edits remain uncommitted. Decomposed runs may commit
+  and merge on Forge task and integration branches. Updating the user's branch requires
+  separate authorization and `integrate --approved`. Never push automatically.
+- Show the decomposed task/model table and wait for approval before dispatch. Retry only
+  when requested; preserve failed task branches and worktrees. `run --retry-failed N`,
+  `--infra-retries M`, `--qa-threshold Pn` and `plan --shadow-qa` (a second review of a
+  sampled share of attempts, never read, to measure reviewer agreement) are opt-in: offer
+  them, never add them unprompted. So is `--fail-first` (solo or `plan`): the change's new
+  tests also run without its non-test changes, and the reviewer is asked to report any that
+  still pass as P2; it needs `.forge/verify-subset` and never changes a result by itself. `plan` explores by default (`--explore 0.2`: now and then
+  a row left to forge gets its neighbouring tier's model from the user's pools, odds
+  recorded for `forge dataset routing`); say so with the table and pass `--explore 0` when
+  the user declines. Retries follow a policy: after three failed attempts on one model the
+  next goes a tier up in the user's pools, or the task stops with a split suggestion
+  (`--escalate-after N`, `0` = off); an explicit `retry --dwarf` overrides it. `accept`
+  overrides a reviewer, so run it (with the user's `--reason` and
+  `--approved`) only on the user's explicit instruction; never self-accept.
+- Exit `8` / status `INFRA` is a quota, auth or network stop, not a task failure and no
+  attempt was spent: report it, say what to fix, and re-run only when told. For a multi-task
+  plan, offer the opt-in whole-run review (`review`, or `integrate --final-review <qa-spec>`);
+  it spends QA quota, so never run it unprompted.
+- Jev is optional, off unless configured and enabled, and always advisory: it never
+  changes a task's tier, never selects a model, and never substitutes for QA acceptance
+  or repository verification (`--jev-act` is still accepted and changes nothing). Read
+  [Jev](references/jev.md) when selected.
+
+## Common workflow
+
+1. Inspect the repository and existing work. State the goal, selected models and any bypass
+   flags. Keep complete requirements in `prompt.md`; short titles are insufficient for QA.
+2. Preflight every selected role before the first model call, including an optional planner:
+   `bash <skill_dir>/scripts/forge-dispatch.sh doctor --spec <spec> --role <role>`.
+   This checks resolution, executable and advertised flags without model quota. Authentication,
+   quota and live model availability remain unverified; a live probe must be explicitly requested.
+3. Use [solo runs](references/solo.md) for one implementation. For `--decompose-level
+   low|medium|high`, read [decomposition](references/decompose.md), prepare the task table,
+   obtain approval, then use `forge-parallel.sh plan/run`. Write the user's request verbatim
+   to `request.md` in the run or plan dir, and pass `--decompose-level` to `plan` with the
+   level you actually split at, even when the user named none. Keep each task small and end
+   its `prompt.md` with an `## Acceptance criteria` section of checks the reviewer can verify;
+   `plan` names tasks over 7K characters or 9 files, where first attempts mostly fail review.
+   Pass the recorded Fractal choice and requested limits to the execution runner; tier
+   pools go to solo directly or decomposed `plan`. Fractal does not replace planner dispatch.
+   `--timeout` applies to decomposed runs too (kept in the plan).
+4. Runners default to summary output. Read each `dwarf.last` and `qa.last` once; retrieve raw
+   logs only when needed. Report findings faithfully, and identify verification
+   limits. Source or review mutations invalidate acceptance. Missing verdicts are UNKNOWN.
+   An absent repository verification command means UNVERIFIED, even when QA passes.
+
+For a Fractal run, report its managed run ID with the Forge result. Use
+`<skill_dir>/forge fractal runs --repo <repo> --json` and `tree RUN_ID --json` to discover
+run/task/node IDs; use `status`, `logs` or `open` for inspection. Run-level `resume`
+recovers the recorded pipeline; scoped controls affect only the selected execution scope.
+Preserve failed work and state the distinction between execution completion, QA acceptance
+and verification. Read the Fractal reference before retrying, changing limits or exporting
+a portable report.
+
+Ripwire context is automatic when compatible; `--no-ripwire` or `FORGE_RIPWIRE=off`
+disables it. Pass opt-out to every role, including a separately dispatched planner.
+Runners offer installation once before agents; dispatch never prompts. See
+[Ripwire](references/ripwire.md) for installation, limits and fallback behavior.
+
+## Conditional references
+
+- [Planning](references/planning.md): read for `--planner` or an agreed approach.
+- [Solo](references/solo.md): command, snapshots, artifacts, exit codes and native-review limits.
+- [Decomposition](references/decompose.md): task schema, routing, waves, retries and integration.
+- [Harnesses](references/harnesses.md): invocation troubleshooting or adding a harness.
+- [Fractal](references/fractal.md): opt-in nested execution, installation, limits,
+  recovery, managed run commands, read-only dashboard and portable reports.
+- [Memory](references/memory.md): `.forge/` learning and spend records; `--no-memory` disables
+  them. Memory is written to the user's repository but excluded from implementation diffs.
+- [forge-ui](references/ui.md): the opt-in local run viewer; `forge ui open|status|stop`,
+  `FORGE_UI=off`, one shared server, privacy.
+- [Dataset](references/dataset.md): every run's inputs and outcomes, kept outside the repo
+  for training decision models; `forge dataset status|import|export|outcomes`, `FORGE_DATASET=off`.
+- [Jev](references/jev.md): optional TypeSafe System One judgments for routing, test
+  selection and memory curation; off by default. Chosen sites can go to Cloudflare's hosted
+  Clef instead (`forge jev setup --provider cloudflare`, `forge jev sites`, `FORGE_CLEF=off`).
+
+The common invocation is `/forge "<goal>" --dwarf <spec> [--qa <spec>]`. Advanced flags and
+commands live with their mode reference. Use [offline tests and evaluation cases](tests/README.md)
+when maintaining this skill.

package/VERSION

@@ -0,0 +1 @@
+0.8

package/forge

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+set -uo pipefail
+# npm runs this file through a symlink in its bin dir, usually a relative one. Follow the
+# links so ROOT is the package and not the bin dir; older macOS has no readlink -f.
+SELF="${BASH_SOURCE[0]}"
+while [ -L "$SELF" ]; do
+  LINK="$(readlink "$SELF")"
+  case "$LINK" in
+    /*) SELF="$LINK" ;;
+    *) SELF="$(dirname "$SELF")/$LINK" ;;
+  esac
+done
+ROOT="$(cd -P "$(dirname "$SELF")" && pwd)"
+
+usage() {
+  printf '%s\n' \
+    'usage: forge <command> [args]' \
+    '' \
+    'commands:' \
+    '  install   link forge into each agent harness on this machine; --dry-run previews' \
+    '  fractal   opt-in Fractal execution: install, doctor, runs, status, tree, logs, open, report, ...' \
+    '  jev       optional Jev advice: setup, status, doctor, enable, disable, backtest' \
+    '  dataset   every run'"'"'s inputs and outcomes: status, dump, import, export, outcomes, ...' \
+    '  ui        forge-ui, the run viewer: open [DIR], status, stop' \
+    '  update    fast-forward this checkout and refresh every harness installation' \
+    '  version   the installed version, and whether a newer release is out' \
+    '' \
+    'Run `forge <command> --help` for its options.' \
+    'Runs start from your agent harness (/forge "<goal>" --dwarf <model>); see README.md.'
+}
+
+case "${1:-}" in
+  install)
+    shift
+    exec /bin/bash "$ROOT/scripts/forge-install.sh" "$@"
+    ;;
+  fractal)
+    shift
+    exec python3 "$ROOT/scripts/forge-fractal.py" "$@"
+    ;;
+  jev)
+    shift
+    exec python3 "$ROOT/scripts/forge-jev.py" "$@"
+    ;;
+  dataset)
+    shift
+    exec python3 "$ROOT/scripts/forge-dataset.py" "$@"
+    ;;
+  ui)
+    shift
+    exec python3 "$ROOT/scripts/forge-ui.py" "$@"
+    ;;
+  update)
+    shift
+    exec /bin/bash "$ROOT/scripts/forge-update.sh" "$@"
+    ;;
+  version)
+    shift
+    exec python3 "$ROOT/scripts/forge-version.py" "$@"
+    ;;
+  -h|--help|help)
+    usage
+    exit 0
+    ;;
+  '')
+    usage >&2
+    exit 2
+    ;;
+  *)
+    printf 'forge: unknown command: %s\n\n' "$1" >&2
+    usage >&2
+    exit 2
+    ;;
+esac

package/package.json

@@ -1,6 +1,28 @@
 {
   "name": "@giamme/forge",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.8.0",
+  "description": "One model builds it. A different model reviews what it actually built.",
+  "keywords": ["claude-code", "codex", "opencode", "agent-skill", "code-review", "multi-model"],
+  "license": "MIT",
+  "author": "Giamme",
+  "repository": { "type": "git", "url": "git+https://github.com/Giamme/forge.git" },
+  "bin": { "forge": "forge" },
+  "os": ["darwin", "linux"],
+  "files": [
+    "forge",
+    "SKILL.md",
+    "VERSION",
+    "CHANGELOG.md",
+    "README.md",
+    "LICENSE",
+    "registry.tsv",
+    "references/",
+    "scripts/",
+    "ui/forge_ui/",
+    "ui/README.md",
+    "ui/THIRD_PARTY_NOTICES.md",
+    "ui/docs/api.md",
+    "!**/__pycache__"
+  ],
+  "publishConfig": { "access": "public" }
+}

package/references/dataset.md

@@ -0,0 +1,329 @@
+# Run dataset
+
+Every forge run records its inputs and outcomes in one global store outside every repo,
+so a decision model (Jev, Laya or any other) can later be trained or calibrated on what
+forge actually did. Run and plan dirs hold nearly all of this, but they live in `/tmp`
+and scratchpads and get deleted. `.forge/ledger.tsv` survives, but it keeps only spec
+strings and verdicts. It has no inputs, no resolved models, no attempt numbers and no
+tokens.
+
+```
+${FORGE_DATA_HOME:-${XDG_DATA_HOME:-~/.local/share}/forge}/dataset/
+  dataset.sqlite        events (rid-keyed, append-only) and repos
+  blobs/<aa>/<sha>.gz   prompts, diffs, messages, logs: capped, gzipped, deduplicated
+  errors.log            capture failures
+```
+
+Capture is on by default and costs a few milliseconds per execution. `FORGE_DATASET=off`
+turns it off for one invocation.
+
+## Raw events, not training examples
+
+The store holds **what happened, when it happened**: the state each decision saw and the
+outcomes that followed. It never holds (state, question, label) rows. Those are derived
+per decision by an exporter, so a question nobody has asked yet can still be answered
+from old runs without re-running anything. Labels come from folding a run's events in
+time order: an attempt that PASSed, then a merge that CONFLICTed, then a human `accept`.
+
+| kind | one per | what it holds |
+|---|---|---|
+| `start` | execution | a receipt written by `begin()`. A start with no matching end is an execution that was killed or is still running |
+| `dispatch` | model call | role, alias, harness, **resolved** model, effort after clamping, timeout and its source, delivery (custom or native review), phase timings, exit, timeout, infra class, tokens, parsed verdict and findings, `FORGE_LEARNING` lines. Blobs: the delivered prompt and the final message |
+| `attempt` | task attempt or solo run | status, attempt number (with refund and resume flags), trigger (`initial`, `auto`, `human`), dwarf and qa spec, declared vs changed files, numstat, drift, guard notes, gate line, patch key, the files this execution wrote and the ones it only carried. Blobs: the diff, retry findings, memory slices, capsule, Jev task files |
+| `verification` / `verify_run` | verify, and each command execution | command and its source (flag, `.forge/verify`, Jev), status, flaky, Jev triage. One `verify_run` per execution, each with its own log |
+| `plan` | `plan` | goal, original request, decompose level, exploratory routing decisions (`tasks[id].route`, with their odds), author (orchestrator or planner), per-task declared vs executed difficulty and who set it, pools, waves, Jev routing and coupling. Blobs: both task tables, prompts, approaches, excerpts of the declared files at plan time, the file inventory, `memory.md` |
+| `run_end` | `run` | results per task, the accepted integration commit, `memory.md` |
+| `invocation` | user-facing `forge-parallel.sh` subcommand | argv, exit, duration, plus final review, accept, integrate and split/combine payloads |
+| `event` | point event | `merge` (MERGED, CONFLICT with the files, INVALIDATED and why), `integrate_advanced`, `integrate_conflict`, `combine_conflict`, `sched_terminal`, `subset_run` (every outcome), `memory_record`, `memory_prune` |
+| `event` `escalation` | `retry --dwarf` | the model a retry escalated from and to, recorded at the rewrite, before the retried attempt begins |
+| `event` `shadow_review` | shadow review | how an opt-in second review ended (`done`, `invalid`, `overtime`, `orphaned`, ...), with the reviewed commit and both specs. Its dispatch is recorded with role `qa_shadow` |
+| `jev` | Jev request | site, answers, usage, latency, mode. Blob: the **full request** (state and questions). Its original sha is `jev.jsonl`'s `key` |
+| `fractal_step` / `fractal_node` | Fractal step, node state | decomposition decision, iteration, resolved model, cost, tokens, candidate diff |
+| `ledger` | ledger row | imported `.forge/ledger.tsv` rows |
+| `outcome` | integrated task, per horizon | written by `forge dataset outcomes`, never by a run: how many of the lines a task added survive N days after `integrate`, and what reverted it |
+
+Each blob reference carries `sha` (the stored bytes), `original_sha`, `n` (the original
+size), `kept`, and `cut`. A cut keeps the head and the tail, because a QA prompt ends with
+the diff and a log ends with the failure. Caps: prompt 1 MiB, diff 2 MiB, message 256 KiB,
+log 272 KiB. Transcripts (`*.log`) and `.cmd` (which some harnesses use to carry the
+prompt in argv) are never stored. A harness without structured output has its final
+message copied from the whole log, so for those harnesses only the tail is kept, flagged
+`final_source: log_copy`.
+
+## How it is captured
+
+All capture goes through a handful of hooks. Every one of them is fail-open: errors go
+to `errors.log`, nothing is printed, and nothing changes an exit code.
+
+- **`forge-runtime.py begin()/finish()`.** Every dispatch, task attempt, solo run,
+  verification and parallel run already passes through these. `finish()` captures from a
+  `finally:` block, so an extraction error cannot lose the record. This is the only
+  hook that sees every exit path: an empty diff, INVALIDATED, a timeout, an infra stop
+  or a `die`.
+- **Provenance travels with the execution.** `begin()` writes
+  `attempts/<role>-<uuid>/dataset.json` with the run id, task, starting attempt counter,
+  trigger and a signature (inode, mtime, size) of every task file. `finish()` credits a
+  file to this execution only when its signature changed. A dwarf that fails returns
+  before `changes.diff` is refreshed, and without this check it would inherit the
+  previous attempt's diff and verdict.
+- **Point events** (`forge_dataset_event`) record facts that disappear before any EXIT
+  trap could read them: a merge that gets aborted, a combine that rolls back, a user
+  branch that moves just before a cleanup that may still fail.
+- **An EXIT trap in `forge-parallel.sh`** records every user-facing subcommand, `die`
+  included.
+- **`forge_jev/client.py`** keeps the canonical request bytes of every live Jev call,
+  including calls made without a run dir. It skips fixture replays, the setup probe and
+  backtests.
+
+Run identity is a uuid in `<plan-or-run-dir>/dataset.uid`, created once (`O_EXCL`) by
+`plan` or by the first `begin()`. Split and combine copy attempt dirs unchanged, so a
+copied execution keeps its original identity and collapses to one event by rid.
+
+## Which forge produced an event
+
+Outcomes depend on the prompts, contracts and registry forge ran with, and the skill is
+often a live dev checkout that changes between releases. So every event carries four
+provenance columns:
+
+| column | what it identifies |
+|---|---|
+| `forge_version` | the release in `VERSION` |
+| `registry_sha` | the effective `registry.tsv` (or `FORGE_REGISTRY`) |
+| `code_sha` | the content of every file that shapes a run: `scripts/`, `references/`, `SKILL.md`, `VERSION`, `forge` and the effective registry. Two runs share it only if they ran the same code, uncommitted edits included |
+| `forge_rev` | the skill checkout's `HEAD` (`git rev-parse`, read-only), or null for a copied skill dir |
+
+An execution is stamped with what `begin()` saw, because its prompt was built then.
+`finish()` runs in another process; if the code changed in between, the event carries
+`code_changed: true` and can be left out of any per-regime analysis.
+
+An imported run was produced by whatever forge ran it, so the importer stamps nothing,
+unless a live `begin()` left its provenance in the attempt dir. Rows imported before
+schema 2 carry the importer's version by mistake, so readers treat `forge_version` and
+`registry_sha` on `src = 'import'` rows without a `code_sha` as unknown. The exporter
+gives those a `forge_rev_estimate`: the last forge commit before the run started, which is a
+guess and is labeled as one. Schema 2 added `code_sha` and `forge_rev`. The first
+capture after an upgrade adds them in one `BEGIN IMMEDIATE` transaction, and an older
+forge still running keeps writing.
+
+## Commands
+
+```sh
+./forge dataset status [-v] [--json]    # where, how much, and what looks incomplete
+./forge dataset dump [--kind K] [--run R] [--task T] [--since UNIX]   # events as JSON lines
+./forge dataset blob <sha>              # print one stored blob
+./forge dataset import <dir|archive|legacy-dir>... [--registry] [--fractal] [--ledger REPO] [--dry-run]
+./forge dataset export [--slice S]... [--limit N] [--seed N] [--out DIR]   # labeled decision cases
+./forge dataset outcomes [--horizon-days 14] [--repo PATH] [--dry-run]     # what integrated work became
+./forge dataset routing [--repo PATH] [--since DAYS] [--json]   # pass rate and runtime per tier and model
+```
+
+`status` runs a different completeness check for each mode, because each mode
+legitimately lacks different things. A solo run has no separate verification, and a
+final review's QA has no task attempt. The checks are:
+
+- executions that started but never ended
+- task QA dispatches with no task attempt
+- parallel PASSes with no merge outcome in a finished run
+- parallel runs with no end
+- blob references with no stored blob
+
+## Exporting labeled decisions
+
+`export` folds the events into cases in the System One wire format (one JSON object per
+line with `id`, `state`, `questions` and `expected`), the format decision-model
+benchmarks and fine-tuning recipes take. Each case is a decision Jev makes in forge,
+with its state built the way `forge_jev` builds it for that site and its label taken
+from what happened afterwards:
+
+| slice | question | label |
+|---|---|---|
+| `routing` | will the first attempt at this task pass review? (state adds the dwarf and reviewer models) | the final status of the first real attempt (see below) |
+| `routing` | which difficulty tier is this task? | the tier the orchestrator or planner declared. This is agreement, not an outcome |
+| `retry` | will this retry pass, given the review findings it was handed? (state adds the attempt number and the previous review) | the final status of that attempt (see below) |
+| `review` | will the independent reviewer pass this diff? (state names the reviewer's model, effort and harness) | the QA verdict. A diff whose reviews disagreed is dropped |
+| `review` | will most of the lines this diff adds stay in the code, unrewritten and unreverted? | from `outcomes`: at least 80% of the added lines still blamed to the task's commits at the horizon, and no revert. Only the review of the diff that landed gets it |
+| `review` | what is the most severe problem the reviewer will find? (`none`, `P3` to `P0`, the contract's scale) | the worst CONFIRMED finding, only when every CONFIRMED finding carries a severity and none is UNSTATED. A review whose findings cannot be fully read gets no label rather than a wrong `none` |
+| `drift` | is this undeclared file likely to need editing? | whether the passing attempt changed it. Negatives are undeclared files near the task, plus random ones |
+| `citation` | do the places this review cites appear in this diff? | the review paired with its own diff, or with another task's diff from the same repo that shares no file |
+| `memory` | is this recorded fact worth including for this task? | the fact's anchor is a file the task edited (true), or lies in an area it never touched (false). Anything in between is left out |
+
+The reviewer is part of the review state because strictness differs by reviewer: the
+same kind of diff passes far more often under one reviewer than another, so a label
+means little without knowing who gave it. Routing and retry states also carry a `size`
+block (`prompt_chars`, `declared_files`, `plan_tasks`, `deps`, `decompose_level`, from
+`forge_jev/features.py`), since task size moves the first-pass rate more than anything
+else in the history.
+
+An attempt is labeled by the last execution that carries its number, because a QA-only
+rerun keeps the number. Refunded and infra-stopped executions spent no attempt. PASS is
+true and FAIL is false. TIMEOUT is false only when the implementer's own dispatch ran out
+of time. Everything else says nothing about the implementer and is left out, counted
+under `censored` in `manifest.json`:
+- a reviewer's TIMEOUT or ERROR
+- a harness crash
+- forge's own failures between dispatches
+- UNKNOWN, INVALIDATED, INTERRUPTED
+
+A later success is never labeled as the first attempt's.
+
+Decisions with no outcome to label them are left out rather than labeled by opinion.
+These are QA effort sizing (recorded until 2026-10-17), the five routing rubrics, failure triage, and coupling, since
+no merge has conflicted yet.
+
+Routing cases of tasks routed with `plan --explore` add `route` to their metadata: the
+candidates, the pool's own pick, the choice, its propensity, and `followed`, which says
+whether attempt 1 really ran the chosen spec (an approval-gate edit or `retry --dwarf` can
+replace it). `assignments.jsonl` lists every such decision with its first attempt's outcome
+or censoring reason, including tasks no case was exported for. It is the randomized cohort,
+free of the export's own filters.
+
+Review cases whose attempt also had a shadow review (`plan --shadow-qa`) add `shadow` (its
+verdict, reviewer and worst finding) and `consensus` to their metadata. Only a shadow that
+finished on an untouched snapshot and reviewed the same commit counts. The primary
+labels never change, because `qa_shadow` dispatches are not reviews of record.
+`./forge dataset status` prints the agreement rate, overall and per model pair.
+
+`meta.jsonl` records each case's slice, run, repo and label source. It also holds hosted
+Jev's own answers where forge recorded them, as baselines that are never part of a case.
+Routing and retry cases add:
+- `decision_provenance` (the plan's) and `outcome_provenance` (the attempt's)
+- `code_changed`
+- the plan `options`
+- `started`
+
+Routing cases also add how the task went:
+- `attempts_observed`
+- `raw_verdict`
+- `gate_pass`, a reviewer FAIL the severity gate accepted
+- `first_raw_pass_attempt`
+- `human_accept`
+- `stop_reason`: `passed`, `accepted`, `noretry`, `interrupted` or `no_pass_observed`
+
+A final FAIL is not proof a task cannot pass: the retry budget ended the observation.
+The output defaults to `<data>/exports/forge-decisions/` (`0700`, files `0600`). It holds
+prompts, diffs and reviews from your repos, so keep it out of public repos and pass it
+only to local models. Builds are deterministic for a given store and `--seed`.
+
+## Routing
+
+`forge dataset routing` answers "which model should this tier use?" from the store, in two
+tables kept apart because they answer different questions.
+
+**Observational.** Every task's first attempt, by declared tier and the model and effort that
+ran it (resolved, so `sol` and `gpt-6.1-sol:medium:codex` share a row):
+- first-pass rate with a Wilson 95% interval
+- the share that passed review at some attempt, or that a human accepted
+- unfinished tasks
+- attempts per task
+- the implementer's median runtime
+
+Routing chose who ran what, so a model sent the hard tasks looks worse here than it is.
+Fractal runs are left out, since their dispatches have identities of their own.
+
+**Randomized.** The `plan --explore` draws, one row per (tier, pool pick, neighbour) pair.
+Round-robin pools make the pair differ between rows, and a model has no chance at all
+outside its own pairs, so pairs are never pooled. For each arm:
+- assigned tasks
+- first-pass as a Hajek (self-normalized inverse-propensity) mean
+- Kish's effective n
+- attrition: tasks not run as assigned, and tasks with no outcome
+- runtime
+
+Every task stays in the arm it was *assigned* (intent to treat), whatever the approval gate
+or a retry did afterwards. A task is bound to the last draw recorded before its first attempt
+started, so a later re-plan never relabels it. Draws from before the two-candidate rule
+(a pick among every spec of the plan) have no pair and are counted, not compared.
+
+**The decision rule** is fixed in advance and only advises. A pair is judged once, at the
+first of 30, 60 or 120 neighbour assignments it reaches, on exactly the assignments up to
+that point. The interval is a run-cluster bootstrap, shown only with at least 10 neighbour
+assignments across 3 runs.
+- For a **low** pair the neighbour is the stronger model: switching is advised only when the
+  interval is above 0.
+- For **medium** and **high** pairs it is the cheaper one: it is called non-inferior when the
+  interval's lower end is above -0.10 and its runtime is lower.
+
+At the default `--explore 0.2` a row changes model 10% of the time, so a pair needs weeks of
+normal use to reach its first window.
+
+## Outcomes after merge
+
+Every other label comes from a reviewer. `forge dataset outcomes` adds one that does not,
+read from the repo's history after the fact. `integrate` merges the run branch with
+`--no-ff` and fast-forwards your branch to that merge, never squashing, so each task's
+attempt commits land as they are and `git blame` still attributes a task's lines to them.
+For every integration, and every task it brought in, the harvester records:
+
+- **Survival:** of the text lines the task added (`base_ref..reviewed_commit`), how many
+  are still attributed to the task's commits at the horizon, per file. A deleted file
+  counts as none surviving. Binary files are skipped, and a task that only deleted lines
+  is `no_additions`.
+- **Reverts:** later commits whose message says `This reverts commit <sha>` for one of the
+  task's commits, its merge or the integration.
+- **The horizon:** the last commit on your branch's first-parent chain, after the
+  integration, dated before `integrate` + N days. The branch is resolved once, and the
+  chain is walked by position, so commit dates out of order cannot skip a commit.
+
+Two cases leave a task without a survival count:
+- **Pending:** the horizon has not been reached and no later commit has passed it yet.
+  Nothing is written, and the next harvest tries again.
+- **Lost:** a rebase, squash or reset took the integration off the branch. This means
+  unobservable, not failed.
+
+The rid names the integration, the reviewed commit and the horizon, so harvesting again
+adds nothing, and two horizons are two outcomes. Survival describes the diff that landed,
+with every retry in it, not the first model the task was routed to. That is why the
+exporter attaches it to that diff's review.
+
+Each verification event also records the commit it verified (`verification.commit`, named
+before any early return), so a FAIL or UNVERIFIED result can be joined to its code too.
+
+## Importing old runs
+
+`import` reads run dirs that predate live capture through the same readers. An execution
+seen both live and imported collapses to one event. Old dirs have no begin-time snapshot,
+so the importer will not guess which execution wrote a mutable file:
+
+- The **newest** task attempt gets the task dir's current files, flagged `legacy`.
+- **Earlier** attempts get only what is immutable:
+  - their own dispatch dirs
+  - the diff QA was shown (the last `` ```diff `` block of the QA prompt)
+  - the status that `attempts.tsv` logged, or failing that, the QA verdict, with
+    `status_source` saying which
+- Unfinished executions (still running, or killed) are skipped. The first write of a rid
+  wins, so importing one now would block the real outcome that live capture records when
+  it ends. Re-import the dir afterwards to fill anything live capture missed.
+- Repos under temp dirs (test runs) are skipped unless `--include-temp` is given.
+- Archives are extracted to a temp dir and mapped back to their original path. The legacy
+  run id hashes that original path, so the archive and the original dir agree on it.
+
+`~/.local/share/forge/legacy/` holds archives of run dirs taken before live capture, with a
+`MANIFEST.tsv`. Passing that dir to `import` reads every archive, plus the ledger copies
+listed in `ledgers/REPOS.tsv`.
+
+## Invariants
+
+- **Never inside the repo.** Capture refuses, silently and before any write, when the store
+  resolves inside the run's repo or a forge worktree. `integrate` refuses a dirty tree.
+- **Never read by a prompt builder.** Memory inject, the capsule and Ripwire must not read
+  the store. A reviewer that could see earlier runs' verdicts would not be independent.
+  `tests/test_dataset.py` enforces that outside its own package, the dataset is only
+  imported for capture.
+- **Read-only in your repos.** Capture and the harvester run only object-store git queries
+  in a repo: `rev-parse`, `rev-list`, `log`, `show`, `ls-tree`, `cat-file`, `diff` between
+  commits and `blame` at a commit. All run with `GIT_OPTIONAL_LOCKS=0`. None takes the
+  working tree or refreshes the index. `outcomes` reads the store through a read-only
+  connection and writes only what it found.
+- **Private.** Prompts, diffs and logs can contain secrets. The dir is `0700` and its files
+  `0600`. Do not sync it. Redaction belongs to whatever exports a training set.
+
+## Not captured
+
+- **Token counts for openclaude, opencode and antigravity.** Those harnesses expose none.
+  They are recorded as null.
+- **Dispatches refused before `begin()`** (an unknown alias, bad flags). No model ran.
+- **The orchestrator's own reasoning.** Only its outputs are recorded: the task table,
+  `request.md` and the approval edits visible between the planner's message and the
+  declared table.
+- **Counterfactuals.** Each task runs on one model, so nothing here says whether another
+  model would have done better. That needs deliberate head-to-head runs.