@giamme/forge 0.0.0-stage → 0.9.0

package/references/jev.md

@@ -0,0 +1,865 @@
+# Optional Jev advisory judgments
+
+Jev is TypeSafe's "System One" model: it answers narrow, typed questions (a
+yes/no probability, a single choice, an ordered score) over supplied state in
+about 100ms, with calibrated probabilities. Many questions per request evaluate
+in parallel, so asking a few extra costs little additional latency. Forge uses
+Jev to *advise*; Forge keeps every existing decision and gate. Jev never
+replaces model routing logic, test selection, QA acceptance or repository
+verification — it only offers a probability alongside them.
+
+## Privacy
+
+Off by default, and this is the main reason why. Each enabled capability
+transmits a narrow slice of state to an external API:
+
+- **routing** — task prompts and candidate model identifiers
+- **tests** — diff hunks and test names
+- **gates** — diff hunks and file excerpts relevant to the gate
+- **memory** — memory entry text being considered for curation
+
+`.forge/` and `.git` are excluded from every payload, in every capability, with
+no exception. The API key lives at
+`${XDG_CONFIG_HOME:-$HOME/.config}/forge/jev.json`, mode `0600`. It is never
+written to a run directory, a prompt, or a log.
+
+A site mapped to the Cloudflare provider (see [Cloudflare Clef](#cloudflare-clef-a-second-provider))
+sends the same slice to Cloudflare Workers AI instead of TypeSafe.
+
+## Setup
+
+```sh
+./forge jev setup                        # store an API key
+./forge jev status                       # resolved config, enabled capabilities
+./forge jev doctor [--live]              # config/key checks; --live also probes the API
+./forge jev enable [--capability NAME]   # persist enabled (optionally scoped)
+./forge jev disable [--capability NAME]
+```
+
+`TYPESAFE_API_KEY` in the environment overrides the stored key for that
+invocation only. Keys come from `console.typesafe.ai`.
+
+A second provider, Cloudflare's hosted Clef, is set up with
+`./forge jev setup --provider cloudflare --account-id <id>`; see
+[Cloudflare Clef](#cloudflare-clef-a-second-provider).
+
+## Commands
+
+User-facing:
+
+```
+forge jev setup                       # store a key, validate it, enable Jev
+forge jev status | doctor [--live]    # what is on; whether it works
+forge jev enable | disable [--capability routing|tests|gates|memory]
+forge jev backtest --repo <path> [--capability tests|drift] [--execute]
+```
+
+`score-plan`, `verify-discover` and `verify-triage` are also subcommands, but they are
+called by `forge-parallel.sh` rather than by hand. They are documented here only so that
+a line in a run log is traceable to the code that produced it.
+
+## Gating and precedence
+
+Four steps, implemented identically by `scripts/forge-jev-options.sh` (shell)
+and `scripts/forge_jev/` (Python) — keep them in sync:
+
+1. `FORGE_JEV=off` wins over everything: everything off, full stop. This is
+   the hard kill switch.
+2. Any `FORGE_JEV_<CAP>=on` acts as an allowlist — only those capabilities
+   run. `--jev-tests` alone works without `--jev`.
+3. Otherwise `FORGE_JEV=on` (from `--jev`) or the stored config's `enabled`
+   flag turns it on, and a capability explicitly set `off` is subtracted.
+4. Default: off.
+
+Flags: `--jev` / `--no-jev` (mutually exclusive), `--jev-act`, `--jev-shadow`,
+and `--jev-<cap>` / `--no-jev-<cap>` for `routing`, `tests`, `gates`, `memory`.
+A run directory freezes its resolved selection in `jev-selection.json`, the
+same way Fractal freezes `fractal-selection.json`, so resume and explicit
+retry make the same decisions as the original run.
+
+## Advisory, always
+
+Jev never selects a model, never changes a task's tier, never overturns a
+`FORGE_VERDICT`, and never marks anything verified. `--jev-shadow` records
+judgments and changes nothing at all.
+
+`--jev-act` still parses, so old commands and a frozen `jev-selection.json` keep
+working, and it still marks each judgment `acting` in `jev.jsonl` and the run
+dataset. Nothing reads it as permission: the one thing it unlocked, writing Jev's
+tier into the `difficulty` column, was removed on 2026-10-17 (see Routing).
+
+## Fail-open
+
+Any error, timeout, or missing key is logged as a skip reason and the run
+proceeds exactly as it would without Jev. A Jev network call can never fail a
+run.
+
+## Observability
+
+- `<run-dir>/jev.jsonl` — one row per request: site, question ids, answers,
+  confidences, latency, tokens, and whether the judgment was applied.
+- `<run-dir>/jev.skip` — present when a request was skipped, with the reason.
+
+## Backtest
+
+`forge jev backtest` measures whether Jev's `test_relevance` rubric (see
+`scripts/forge_jev/questions.py`) picks the right tests for a change, before
+that judgment is ever wired into a live decision.
+
+The label comes from git history, not from a model: a commit where a human
+changed source and test files together is treated as ground truth — "these
+are the tests a person judged relevant." Non-circular because nothing Jev (or
+any model) produced enters the label. `scripts/forge_jev/corpus.py` extracts
+these (change, tests-touched) pairs from `git log`.
+
+Candidates for a commit are the test files present in its **parent** tree, not
+at HEAD. Scoring against HEAD would ask about tests that did not exist yet:
+measured on one repo, a commit 300 back had 94 test files in tree against 416
+at HEAD, which would inflate the suite-reduction metric more than fourfold.
+
+```sh
+./forge jev backtest --repo PATH --limit 50            # dry estimate, no API call
+./forge jev backtest --repo PATH --execute --max-requests 200
+./forge jev backtest --repo PATH --execute --capability drift --json
+```
+
+`--execute` is required before backtest makes a single API call; without it,
+the command only estimates the run (commit count, request count) and says so.
+`--max-requests` caps spend by bounding how many Jev requests the run is
+allowed to make, regardless of `--limit`. Results land under `--out` (default:
+a directory under the run's own temp/output location) as raw judgments plus
+the printed summary — precision/recall of the rubric's "true" answers against
+the commit's actual test changes.
+
+The summary sweeps thresholds and reports micro/macro recall, precision,
+suite reduction, and `records_with_full_recall` — the fraction of commits where
+*every* labeled test was selected. That last one is the number that decides
+whether the judgment is safe to act on: a selector averaging 95% recall by
+missing one test in every commit is useless. For `--capability drift` the
+headline is instead precision at the warn threshold, because a warning that
+fires on files nobody edits trains people to ignore it.
+
+Three honesty caveats, printed with every run so they travel with the numbers:
+
+- Commit messages are written **after** the work and often name the very files
+  or behaviour that changed; a Forge task prompt is written before. Backtest
+  scores are therefore an **upper bound** on live performance.
+- A commit's test changes are a **lower bound** on the tests that were
+  actually relevant — humans miss tests too. High recall against this label
+  is necessary to trust the rubric, but not sufficient; it cannot prove the
+  rubric would catch tests no human thought to touch. (This one does not apply
+  to `--capability drift`, whose label is the commit's complete file set.)
+- Co-change encodes **one team's testing habits and repo layout**, not a
+  universal property of the change. A number measured on one repo does not
+  transfer to another's thresholds without re-measuring there. Measured here:
+  labeled tests are 1.6% of the candidate suite in one repo and 13.2% in
+  another, which are not the same problem.
+
+## Verification (the first live capability)
+
+Two judgments inside `verify_result`, the decomposed runner's verification step.
+Both are off unless the `tests` capability is enabled and a key is configured; with
+Jev inactive the function behaves exactly as it did before.
+
+**Finding a command.** Only when neither `--verify` nor `.forge/verify` exists — the
+case that reports `UNVERIFIED` today even when QA passed. Code enumerates commands
+that *demonstrably exist* in the repo (`package.json` scripts, Makefile targets,
+tox/nox, runner scripts under `tests/`, `scripts/`, `bin/`, a configured pytest,
+`go.mod`, `Cargo.toml`, `run:` steps in GitHub workflows, and facts recorded under the
+`## Verify` heading of `.forge/memory.md`); Jev picks one, or `none`. Each candidate is
+sent with its source and an excerpt of what it actually runs — a make target's recipe
+rather than the head of the Makefile — because the command name alone does not say
+whether it runs the suite. It selects, never generates, so it cannot propose `make test`
+for a repo with no Makefile. A non-executable runner is
+offered with its interpreter — forge's own `tests/check.sh` is mode 644 and its README
+says `bash tests/check.sh`.
+
+The chosen command is announced through `note` before it runs, recorded with its
+confidence and evidence in `<out>/verification.jev.json`, and then flows through the
+existing run/fingerprint/status logic unchanged. `UNVERIFIED` remains the outcome
+whenever nothing is chosen: Jev may find a command, never declare something verified.
+
+**Re-running a corroborated flake.** On a non-zero exit, Jev classifies the failure as
+`real_regression`, `known_flake`, `environment` or `generated_drift`. The command is
+re-run **once**, and only when all of these hold:
+
+- the exit was non-zero *and* the tree did not change — a tree change is never a flake;
+- the verdict is `known_flake` at or above the `flake_act` threshold (0.90 by default);
+- a `trap` already recorded in `.forge/memory.md` shares a distinctive token with this
+  failure's output.
+
+That last condition is deterministic and computed in code, not asked of the model. A
+model alone calling a failure a flake is how a real regression ships, and a trap noted
+months ago must not excuse an unrelated failure today. If the re-run also fails the
+status stays `FAIL` and the first log is preserved. `real_regression` is never re-run
+at any confidence.
+
+**Interaction with the deterministic rerun.** `forge-parallel.sh` now reruns *any* failed
+verify command once by default (`--verify-retries`, default `1`) when the tree did not
+change, with no model and no recorded trap needed: the rerun itself is the evidence, and a pass
+is reported loudly as `FLAKY` (`verification.flaky`) rather than as a clean pass. That rerun
+comes first, so Jev's triage above only runs when the deterministic one is disabled
+(`--verify-retries 0`) — in which case the corroboration requirements above still apply
+unchanged. A deterministic rerun that also fails ends in `FAIL` without consulting Jev.
+
+## Calibration (measured 2026-09-18)
+
+First measurements against the live API. Two repos for test selection, 18 for verify
+discovery. Every number below is from this machine's repos and does not transfer —
+re-measure before trusting a threshold elsewhere.
+
+### `tests_act` = 0.70 is supported by evidence, not a guess
+
+Discovery was run over every git repo on this machine (18 with candidates). The pick
+was compared against the command each repo's own README or `package.json` documents.
+
+| | |
+|---|---|
+| Fires at 0.70 | 10 of 18, **every pick correct** |
+| Lowest correct pick that fires | 0.73 (`ctrlrisk-tools-redigo`) |
+| Highest *incorrect* pick | 0.55 (`spankai` → `cargo test`) |
+| Margin around the threshold | **0.18** |
+
+`spankai` is the one confirmed wrong pick: a Rust + TypeScript monorepo whose README
+says `npm run check`, where Jev chose `cargo test` — a real test command that covers
+only the Rust half. The gate blocked it. Lowering the threshold to 0.58 would add four
+correct picks but leave only a 0.03 margin above that wrong one, so 0.70 stays.
+
+Confidence tracks how ambiguous the candidate set is, not how likely the pick is to be
+wrong: repos with 2 candidates land at 0.73–0.81, repos with 11–25 land at 0.58–0.61.
+This is why the gate was not `runs_tests`, which sat at 0.79–0.98 for every repo
+including the wrong `cargo test` at 0.97. `runs_tests` answers "is this a test command" —
+true of `cargo test` — and is nearly useless as a gate.
+
+Repeating the identical request three times gives a spread of 0.02–0.08 and never
+changes the pick, so these numbers are reproducible rather than one lucky sample.
+
+#### The gate is no longer Choice confidence, because that answers the wrong question
+
+Confidence measures *which* option wins. For a selection among several **acceptable**
+options that is the wrong quantity: two commands that both run the suite split the mass
+and depress it, and a tie is not doubt. Measured live on the jevscope repo, three times
+running — `bash scripts/check.sh` and `make test` both run its suite —
+
+| | |
+|---|---|
+| Choice confidence | **0.48–0.52**, against a 0.70 gate |
+| Mass on `none` | **0.01–0.02** |
+| Actual state of the repo | 48 tests, all passing |
+
+So forge reported `UNVERIFIED` for a verifiable repo while the model was 98% sure some
+listed command verified it. The gate now reads **1 − P(none)** — the probability that the
+repo is verifiable at all, which is what `tests_act` was always meant to bound. A split
+between correct commands leaves it high; genuine doubt piles onto `none` and drops it,
+and the winner is then just the tie-break. `client.probabilities` exposes the
+distribution the response was already returning and `choice()` was discarding. When a
+response omits it, the old confidence gate stands — a missing field is not certainty.
+
+#### …and 0.70 itself is now stale, for either gate
+
+`spankai` is the one confirmed wrong pick in the table above: `cargo test` covers only
+the Rust half of a Rust + TypeScript monorepo, and at 0.55 the 0.70 gate blocked it.
+**It no longer does.** Re-run today it scores 0.77–0.81, which clears 0.70 on the old
+gate as well as the new one.
+
+The cause is measurable rather than guessed. Suppressing `_evidence_excerpt` — restoring
+the state as it was when 0.55 was recorded — reproduces 0.52–0.59; restoring it gives
+0.77–0.81. Sending a `make` target's recipe raised confidence across the board, and it
+raised the wrong pick with everything else.
+
+So the whole table above was measured against a state production no longer produces, for
+the seventh time in this file. `tests_act` needs re-measuring across those 18 repos in
+the current state before 0.70 means anything, and **no number should be moved until it
+is** — fitting a new one to the single data point above is exactly the mistake being
+recorded here.
+
+### Test selection: no single threshold works across repos
+
+`backtest --capability tests` over 25 commits of `ctrlrisk-tools-enrich` (40 requests)
+and all 44 of `spankai` (44 requests), sweeping the per-file relevance probability:
+
+| repo | highest threshold holding recall ≥ 0.95 | suite reduction there |
+|---|---|---|
+| `ctrlrisk-tools-enrich` | 0.20 | 46% |
+| `spankai` | 0.45 | 36% |
+
+The safe threshold differs by 2.25× between two repos, which settles the question of
+whether one global default could serve: it cannot. Any future test-subsetting capability
+must calibrate per repo and store the result, not ship a constant. Note also that no
+answer ever exceeded 0.90 — the top of the probability range is unused by this rubric,
+so a threshold set there silently disables the capability.
+
+### A rejected change, recorded so it is not retried
+
+Feeding each repo's documented commands (README, CONTRIBUTING, AGENTS.md) into
+discovery state looked obviously right and **measured worse**. It fixed `spankai`'s pick
+but cut repos firing at 0.70 from 10 to 3, and it destabilised the judgment: against a
+baseline spread of 0.02–0.08, `ctrlrisk-intelligence` fell from a steady 0.81/0.81/0.83
+on the correct command to 0.30 on a narrow migration check. A README lists many
+commands and nothing marks which one is the gate, so the evidence is mostly noise.
+Half the damage was a prompt artifact — naming the state key when the list was empty
+cost 0.12–0.21 confidence on every repo that documents nothing, because "no candidate
+matches" reads as evidence against rather than absence of evidence. Fixing that
+recovered those repos exactly, and the doc-rich repos stayed erratic. Reverted.
+
+## Routing (shadow-first)
+
+Five `Score` rubrics per task, all in one request, evaluated in parallel. **This code,
+not the model, turns them into a tier** — `forge_jev/routing.py` holds the weights, so
+re-tuning them against accrued outcomes replays recorded scores instead of paying for
+inference again.
+
+| Rubric | Weight | Why |
+|---|---|---|
+| design judgment | 0.35 | `decompose.md` defines difficulty by this first |
+| state & concurrency subtlety | 0.25 | that table names it as what makes a ten-line diff `high` |
+| blast radius | 0.20 | how far a mistake reaches |
+| spec clarity | 0.15 | is the approach given, or must it be invented |
+| existing test coverage | **−0.05** | inverted: good coverage lowers the tier |
+
+Coverage is weighted lightly on purpose. It does not reduce the judgment a task demands,
+only what a mistake costs, and letting it dominate would send a genuinely hard change to
+a cheap model because the area happens to be well tested.
+
+The planner's own difficulty rating is **not** sent. Jev is a second opinion on the same
+evidence; showing it the answer first buys an agreement rate, not a judgment.
+
+**Escalation.** A composite within 0.05 of a tier boundary, or a confidence below 0.55,
+escalates one tier and records why (`boundary` / `low-confidence`). Paying for a stronger
+model on an ambiguous task is the cheap error.
+
+### Two modes, and why the tier is never written
+
+| Mode | What happens |
+|---|---|
+| `--jev-shadow` | Scores every task, writes `jev-routing.tsv` and `tasks/<id>/jev.json`, **changes nothing else** — not the table, not one dispatch |
+| `--jev` (default) | Adds an advisory line per task; the `difficulty` column is untouched |
+
+A third mode, `--jev-act`, could write the tier into `difficulty` once a repo held a
+calibration for that tier (`forge jev calibrate --write`: 30 tasks, a 95% Wilson lower
+bound over 0.80). No repo ever reached it, so it never acted, and on 2026-10-17 it was
+removed along with `calibrate` and the `routing_act` and `calibrate_floor` thresholds.
+
+The run dataset had settled the question by then. Jev's score predicts a task's first
+pass about as well as cheap features do (AUC 0.69 against 0.64–0.67 on older runs; on the
+newest runs every method, Jev included, sat near chance), and a model-swap test on Clef
+showed a decision model judging how hard a task is, not which model does it better:
+swapping the model explained 2% of the variance in its predictions. A tier is a choice of model, so it is measured instead: exploratory routing,
+on by default, and `forge dataset routing` compare models per tier from randomized
+assignments (see `decompose.md`, Exploratory routing).
+
+### What was measured
+
+Six real tasks from this repo, hand-labelled first, scored live:
+
+- **5 of 6 matched.** The sixth was a boundary escalation `medium`→`high`, the intended
+  conservative direction.
+- Confidence is high on easy work (0.89–0.93) and lower on hard work (0.67–0.76). That
+  asymmetry is the useful one: Jev is confident exactly where routing *down* saves money,
+  and defers to the human where getting it wrong is expensive.
+- End to end, a spelling fix declared `high` was routed to the cheap pool at 0.864 while a
+  concurrency redesign stayed on the expensive one.
+
+Composite confidence is the **weight-weighted** mean, not the minimum. The minimum was
+tried first and measured wrong: on a real task `blast_radius` returned 0.0 while
+`design_judgment` (weight 0.35) was 0.94, and the minimum reported 0.0 for a composite
+that was mostly settled. The weakest dimension is still reported separately as `weakest`.
+
+Six hand-labelled tasks is **not** a calibration. It is a smoke test that the rubrics
+point the right way.
+
+## Pre-dispatch gates
+
+Three plan-time warnings. All advisory, all fail-open, none can block a dispatch — a gate
+that can stop a dispatch is a gate that will eventually stop a good one.
+
+**Prompt adequacy** and **independent verifiability** ride inside the routing request.
+They need exactly the state routing already sends, and questions in one request evaluate
+in parallel, so they cost **no extra round trip**.
+
+**`files` drift prediction** needs a different universe — every tracked file, not the task
+text — so it is the one extra request Phase 4 adds, and only when the `gates` capability
+is on. `decompose.md` calls the `files` column "a promise, not a prediction"; Forge
+already detects a broken promise in `drift.txt`, but only after the wave has run. Asking
+at plan time turns a post-mortem into a warning while the plan can still change.
+
+### Each rubric has its own threshold, because they are not on one scale
+
+This is the same trap `tests_act` set earlier, and it was measured rather than guessed.
+
+| Gate | Threshold | Evidence |
+|---|---|---|
+| drift | `gate_warn` 0.60 | 50 commits × 2 repos: precision **1.000** and **0.934** |
+| prompt adequacy | `prompt_warn` 0.55 | unjudgeable prompts **0.06–0.27**, prompts stating checkable behaviour **0.85–0.96** |
+| independent verifiability | `verifiable_warn` 0.55 | fragments **0.26–0.65**, self-contained tasks **0.65–0.96** — precision gate, see below |
+
+Both text gates were first measured at a time when `prompt.md` did not reach the model:
+`score_task` sent only the task title. Those numbers (vague 0.04, specific 0.55–0.87)
+described data production never produced, and both thresholds were fitted to them.
+
+Re-measured with the real state — three runs over eight probe prompts against a real
+repo, plus the eight prompts of a real plan:
+
+**`prompt_adequacy` separates cleanly.** "Make the error messages better" scores 0.06;
+"harden the server — fix what you find" 0.15; "make it faster without breaking anything"
+0.27. A prompt that states checkable behaviour scores 0.85–0.96, and so did every one of
+the eight real task prompts. A 0.58-wide gap, so the threshold sits in the middle of it
+with 0.28 clear on each side. The old 0.30 classified every case correctly too, but sat
+**0.03** above the worst true positive — correct by luck rather than by margin, against a
+measured repeat-noise of 0.07.
+
+**`independently_verifiable` was unanswerable, and rewording it was never going to
+help.** The rubric asked whether confirming a task requires "work declared in a different
+task" — and `score_task` sent no other task. The model was asked about a plan it had
+never been shown, so it could only guess whether such a sibling existed. That is why the
+classes overlapped (0.34–0.56 against 0.43–0.90), and it is the same failure as
+`prompt.md`: a rubric scored against evidence that never reached it.
+
+The fix is state, not wording. `routing._siblings` now sends every other task in the plan
+— id, title, file scope, declared deps, capped at `MAX_SIBLINGS` — plus the task's own
+`deps`, and the rubric points at that list by name.
+
+Re-measured on the eight tasks of the real jevscope plan, eight repeats on every task the
+threshold depends on. Ground truth was taken from the **import graph and wave position**,
+not from the task titles — which corrected the labelling, not the model:
+
+| task | score | truth |
+|---|---|---|
+| `server` | 0.26–0.34 | fragment: wave 1, imports `discover`/`events`/`stats` and serves `page`'s `index.html`, **none declared** |
+| `page` | 0.43–0.53 | fragment: consumes `server`'s API; both defects that shipped were `page`↔`events` shape mismatches |
+| `readme` | 0.60–0.65 | fragment: documents the wave-2 CLI, and shipped wrong |
+| `cli` | 0.65–0.75 | verifiable: wave 2, so everything it needs has already landed |
+| `stats`, `events`, `discover`, `fixtures` | 0.67–0.96 | verifiable: own file set, own test file |
+
+**At the old 0.38 this gate caught nothing on a real plan** — every genuine fragment
+scored above it — exactly as wave coupling had never fired. 0.55 catches `server` with
+0.21 to spare and `page` with 0.02, and keeps **0.10** to the nearest true negative,
+which is the only margin here that beats the 0.07 noise floor. An advisory gate has to
+hold precision above all: one that cries wolf is one people switch off.
+
+The classes still touch at 0.65, where `readme` meets `cli`. So this is a precision gate
+that buys a clean left-hand side by giving up `readme` entirely, and **silence above 0.55
+still means "no evidence", not "verifiable"**.
+
+Using the single `gate_warn` of 0.60 for all three — which is what shipped first — warned
+on 5 of 9 and 5 of 11 perfectly good tasks in the hand-labelled set. It was caught by a
+live plan where a precisely specified task tripped both text gates. Precision is the only
+number that matters for a warning: one that fires on good work teaches people to ignore
+every future warning.
+
+Drift is the strongest of the three and the only one measured at corpus scale. Its
+threshold sits comfortably clear of the noise on real repos (non-edited files scored
+≤0.23 in a live check where the true file scored 0.65), but note that 0.65 is not a wide
+margin — a borderline file can fall either side between runs.
+
+The verifiability rubric was nearly discarded on a first reading that showed it failing
+to separate vague prompts from clear ones. That was a bad label, not a bad rubric:
+vagueness is not the same property as needing a sibling task to land first. Against the
+right labels it separates with a 0.27 margin.
+
+## Post-review annotation
+
+When a reviewer returns `FORGE_VERDICT: FAIL`, Jev reads the review against the diff and
+says whether the failure looks sound. **It cannot change the verdict, and no caller is
+given a way to.** The status file is written before this runs and nothing here rewrites
+it; a test asserts that structurally, by reading the shipped block and failing if a
+status write ever appears inside it.
+
+That restraint is the point. Forge's second invariant is reviewer independence, and a
+model that could talk a reviewer out of a FAIL would produce confident PASSes on code
+nobody checked — worse than having no reviewer. The asymmetry runs the other way too:
+nothing ever annotates a PASS into looking suspect, because a PASS is the outcome that
+gets merged and its reviewer's judgement is left alone.
+
+Two questions, one request:
+
+| | |
+|---|---|
+| **correctness** | does the review name a real bug, or only preferences about how the code is written? The QA prompt already says "style nits are not failures" |
+| **citation** | do the files, lines and quoted code the review cites actually appear in this diff? |
+
+Measured on three hand-written reviews against one diff:
+
+| review | correctness | citation | flagged |
+|---|---|---|---|
+| style nits only ("`i` should be `item`") | 0.21 | 0.80 | yes — style |
+| a real bug (discount below zero, with a triggering input) | 0.96 | 0.88 | no |
+| cites `src/payment.py`, absent from the diff | 0.45 | 0.02 | yes — citation |
+
+A low citation score suppresses the correctness reason rather than printing both. The
+hallucinated review scored 0.45 on correctness and would have been reported as "style
+preferences", which describes the wrong problem: nothing can be said about the
+correctness of code that is not there.
+
+## Wave coupling
+
+`compute_waves` bounds parallelism structurally: two tasks share a wave only when their
+declared file sets are disjoint. Disjointness cannot see the case where task A changes an
+interface task B consumes — both diffs are individually correct, both pass review against
+their own baseline, and the merge is broken. This asks about exactly that.
+
+**It never fired once, on any plan, until it was measured.** It borrowed `gate_warn`
+(0.60) while coupling probabilities live around 0.09–0.38. That was not a strict
+threshold; it was an unreachable one.
+
+Measured over 11 same-wave pairs from a real 8-task plan, five runs each, with ground
+truth taken from the run itself — two of those pairs produced a defect that actually
+shipped:
+
+| | median |
+|---|---|
+| `events`+`fixtures` — fixtures reads an event's answers object | **0.27** |
+| `events`+`page` — page renders normalised events | **0.25** |
+| `page`+`stats` — page rendered a repeats shape stats never emitted *(shipped bug)* | **0.23** |
+| `events`+`stats` — stats consumes `normalise()`'s output shape | **0.22** |
+| `discover`+`readme` — unrelated | 0.18 |
+| `discover`+`server`, `readme`+`fixtures` — unrelated | 0.13–0.14 |
+| `discover`+`fixtures`, `fixtures`+`page`, `discover`+`page` — unrelated | 0.09–0.10 |
+
+Two changes came out of it.
+
+**The requirements now reach the rubric.** It used to see `id (title) files: …` and
+nothing else, and in that state it had *no signal at all*: the pairs that produced real
+defects scored lowest of everything, below pairs that were entirely unrelated. An
+interface dependency between two tasks is written down in `prompt.md` and nowhere else —
+certainly not in a one-line title. File *contents* are still not sent; diffs nobody has
+written yet would be noise.
+
+**`coupling_warn` is 0.20, and at most three pairs are reported.** The threshold decides
+whether anything is worth saying; the ordering decides what gets said. Rank is the real
+signal — the coupled pairs took four of the top five places — while the absolute scores
+move with the plan: the same rubric sat at 0.09–0.27 in the measurement and 0.20–0.38 on
+the live plan, where a bare 0.20 warned about 12 of 21 pairs. A gate that fires on more
+than half of everything teaches people to ignore it.
+
+On the live plan it now names three pairs, all genuinely coupled, `page`+`server` highest
+— the page consumes the server's API contract. On a plan of eight unrelated tasks it says
+nothing.
+
+### The ordering half: `forward_dependency`
+
+The other shipped defect was `readme` documenting a CLI interface that `cli` had not
+built yet. Those two were in *different waves*, so they were never a candidate pair, and
+no same-wave check can see that shape. It is a different failure from coupling, too:
+not a broken merge between concurrent tasks, but a task reviewed, verified and merged
+**before the thing it was written against exists** — so its own review cannot catch it,
+and the error ships.
+
+`forward_pairs` forms ordered `(earlier, later)` pairs across waves, skipping any pair
+where LATER is named in EARLIER's declared deps — forge would not have placed EARLIER
+first in that case, so the risk cannot arise. Cross-wave pairs get their own
+`MAX_FORWARD_PAIRS` budget rather than sharing `MAX_PAIRS`, or one large first wave would
+spend the whole allowance inside itself and never ask the ordering question at all.
+
+Measured over that plan's seven cross-wave pairs, five runs each:
+
+| pair | score | |
+|---|---|---|
+| `readme` → `cli` | **0.88–0.90** | documents the CLI built in the next wave *(shipped bug)* |
+| `page` → `cli` | 0.13–0.15 | |
+| the other five | 0.05–0.07 | |
+
+A **0.73-wide gap**, the cleanest separation any rubric here has produced, so
+`forward_warn` sits in the middle of it at 0.50 — 0.35 clear below, 0.38 above, against a
+0.07 noise floor. Provisional in one specific way: the gap is enormous but rests on a
+single positive example. A second plan could move the positive class; it is unlikely to
+move it 0.38.
+
+Both halves run from one `forge jev coupling` invocation, so a plan gets both warnings or
+neither, and each row carries a trailing `kind` of `same-wave` or `forward`. On the real
+plan the gate now names all three of its genuine problems: `page`+`server` and
+`events`+`page` same-wave, and `readme`→`cli` at 0.88.
+
+Still advisory in every mode: it prints, and never reorders a wave or blocks a dispatch.
+
+## Memory curation
+
+Guarded so `--no-memory`, `FORGE_MEMORY=off` and a disabled capability each yield today's
+exact behaviour, verified against a pristine `HEAD` export.
+
+The asymmetry that shapes every default here: `memory.md` is injected into every dispatch
+of every future run, so a wrong fact is paid for forever, while `ledger.tsv` is
+append-only and never injected. **Nothing recorded is ever lost — only injection is
+gated.** A rejected line still lands in the ledger in full.
+
+| At `record` | |
+|---|---|
+| **quality gate** | a one-off observation is recorded but not injected |
+| **category correction** | refiles a fact under `verify`/`trap`/`finding`; `none` leaves it alone |
+| **semantic dedup** | merges a paraphrase onto the existing key |
+
+Dedup fixes a limitation `norm_key`'s own comment names: exact-match keys mean two runs
+learning the same thing in different words count as two facts, so neither reaches the
+two-distinct-runs promotion threshold for a `finding`. Live, four learnings produced:
+a rename note recorded but not injected (durable 0.1); a `trap` refiled to `verify`; a
+differently-worded restatement of it merged onto the same key; and a genuine new trap
+kept.
+
+**Category correction and dedup are two rubrics, and they have two thresholds.** They
+shared one `memory_dedup` value at 0.80 until both were measured. Splitting them was
+right; the first numbers put on them were not, and the way that went wrong is the useful
+part.
+
+The first pass fed existing entries as `verify | text` and read correct merges at
+0.99–1.00, which set the bar at 0.90. Running the real CLI against a real `memory.md`
+then returned **0.90 on a correct merge** — right on the threshold. Production strips
+entries to bare text under a `## section` heading, with no category prefix; the state
+was different, so the number was. Re-measured with the state
+`cmd_memory_curate` actually builds, six repeats per boundary case:
+
+| | wrong, must not act | right, must act | threshold |
+|---|---|---|---|
+| `memory_merge` | 0.56–0.72 | 0.91–1.00 | **0.85** |
+| `memory_recategorize` | 0.56–0.72 | 0.92–1.00 | **0.80** |
+
+Both rubrics separate in the same place. What sets the two numbers apart is therefore not
+the scale but the cost: a wrong merge loses a fact permanently, so it takes the top of the
+gap, while a wrong refile leaves the fact in place under the wrong heading and refusing a
+correction is the commoner harm, so it takes the bottom. **0.80 was a defensible value for
+one of the two all along** — the fault was never measuring it, and the first measurement
+was worse than the guess.
+
+The wrong merges are worth naming, because the threshold is the only thing that stops
+them: `config/prod.yaml` is ciphertext / `config/prod.yaml` must not hold plaintext
+secrets merged on 6 of 6 runs at 0.56–0.72. The rubric's `none` criterion already tells
+the model to keep both on a partial match and it picks one anyway — at a confidence that
+separates cleanly from a true paraphrase. That is the case for gating on confidence
+rather than wording the criterion harder.
+
+The refused corrections are facts that genuinely span two categories: a test that fails
+without `alembic upgrade head`, and `make test` regenerating fixtures so a dirty tree is
+expected. Both are verify and trap at once, which is exactly when the recorded category
+should stand.
+
+**A merged line is exempt from the quality gate.** The two judgments arrive in one
+request and originally did not consult each other, which broke dedup in the case dedup
+exists for. `rebuild` skips a non-injectable row *before* it counts anything, and a
+`finding` needs two distinct runs to be promoted — so a restatement judged not durable
+was dropped from the count of the fact it restates, and the two wordings still never
+reached two.
+
+Measured over paraphrases of facts already in memory: durability 0.44–0.83, against
+0.82–0.83 for the same facts stated fresh. A restatement reads as less durable than the
+thing it restates. That is the wrong question to ask of it: the entry it matched is
+already in `memory.md`, so durability was settled when *that* entry was written, and a
+line matching it is by definition not the one-off observation the gate is for. The
+exemption is the merge itself — a match refused by `memory_merge` grants nothing.
+
+The ledger gained an eleventh column for the injectable flag. `rebuild` treats a row
+without it as injectable, so an existing ledger keeps behaving as it did.
+
+| Elsewhere | |
+|---|---|
+| **semantic staleness** (`prune` only) | an entry whose anchor still exists but whose fact went false |
+| **injection slicing** (`inject`) | narrows the role slice to the ~8 facts bearing on this task |
+
+Staleness runs **only** on an explicit `forge-memory.sh prune`, never during `record`.
+`rebuild` runs after every dispatch, so a request per entry per dispatch would be the
+most expensive thing in a run.
+
+Its threshold is `stale_drop` at 0.25, far below `gate_warn`. Measured over six facts
+against real files: ones that had genuinely gone false scored 0.03–0.10, ones still true
+scored 0.47–0.96. `gate_warn`'s 0.60 sits inside the true range and would have deleted a
+fact the file still supports. Dropping a fact is irreversible and silent; keeping a stale
+one costs a line until someone notices.
+
+Slicing only removes lines — never adds, never reorders, and keeps the section heading
+above each kept line. The 40-line/4KB cap and the role slice remain the outer bound. Live
+on a 10-fact memory against a payments-retry task, it kept 8 and dropped the CSS build
+ordering and the parser cache — while keeping the idempotency trap.
+
+## Removed: effort sizing, spec parsing, Fractal re-rating
+
+All three were removed on 2026-10-17, and none had ever acted.
+
+- **Effort sizing** rode in the routing request and suggested a `low..ultra` level per
+  task, with QA effort sizing (`qa-effort`) doing the same for the reviewer. Over the run
+  dataset the dwarf suggestion separated first-pass outcomes at AUC 0.64 overall but
+  0.50–0.54 within a tier: all it knew was the tier. And 81% of its suggestions were
+  `xhigh` or higher. Effort is now measured like models are: pools that differ only in
+  effort (`--dwarf-low sol --dwarf-medium sol:high --dwarf-high sol:xhigh`) give
+  exploratory routing two arms per tier, and `forge dataset routing` reports whether the
+  higher effort passes more.
+- **`forge jev parse-spec`** mapped a sentence onto `--dwarf-<tier>` flags. It could only
+  map a sentence that already named the aliases, since `registry.tsv` says nothing about
+  cost or strength, which made it no shorter than typing the flags.
+- **Fractal child re-rating** could move a child between authorised pools, gated on the
+  same per-repo calibration as `--jev-act`, which no repo had. A child's difficulty is the
+  parent dwarf's own again.
+
+## Early test feedback (what became of per-task subsetting)
+
+Running tests inside a task worktree has no safe window: an artifact created before the
+diff is captured reaches QA as if the dwarf wrote it, and one created after trips the
+fingerprint re-checks in `do_task` and `merge_task`, which write `INVALIDATED` and block
+the merge. So this exports the reviewed commit to a throwaway directory and runs there.
+A test asserts the worktree's tree hash, file list and absence of `__pycache__` across a
+real run.
+
+**It is not a speed-up.** The full suite still runs at integration exactly as before, so
+this *adds* a run rather than replacing one. The saving the plan described required
+replacing verification, which the fingerprint rules forbid. What it buys instead is that
+a reviewer learns "your diff breaks `test_x`" while it can still act on that.
+
+### It is opt-in, and needs a template
+
+`.forge/verify-subset` must exist and contain `{files}`:
+
+```
+pytest -q {files}
+npm test -- {files}
+python3 -m unittest {files}
+```
+
+Deliberately a separate file from `.forge/verify`. There is no general way to restrict an
+arbitrary command to a file list, and the first version assumed there was — it appended
+paths to the verify command, which works for `pytest -q` and fails for `npm test`,
+`go test ./...`, `bash tests/check.sh` and `python -m unittest discover`, each for a
+different reason. Without the template this feature stays off.
+
+### A failure is only reported when the base passes
+
+The first run of this reported **"the likely-affected tests FAILED"** when the real
+problem was that `pytest` was not installed in the export. An environment failure
+presented to a reviewer as a broken diff is worse than silence, and pre-existing failures
+have the same shape.
+
+So on a failure the identical subset runs again against the commit the task started from,
+and the result is reported **only if the base passes**. That one extra run — paid only
+when something already failed — is what separates "this diff broke it" from "this was
+already broken, or cannot run here".
+
+## Cloudflare Clef (a second provider)
+
+Cloudflare Workers AI hosts Clef, a decision model that takes Jev's request shape
+(`{state, questions}`) and answers with a probability per option. Forge can send chosen
+**sites** (the `site=` name of each call: `jev-drift`, `jev-memory-slice`, ...) to it.
+Without a Cloudflare provider configured, every site asks TypeSafe exactly as before.
+
+```sh
+./forge jev setup --provider cloudflare --account-id <id> [--model clef|clef-flash] [--sites a,b]
+./forge jev sites                       # which provider answers each site, and whether it is ready
+./forge jev sites jev-drift=typesafe    # move a site back
+./forge jev doctor --live               # probes every provider a site uses
+```
+
+- **Token:** a Cloudflare API token with **Account > Workers AI > Read**, in
+  `cloudflare.env` beside `jev.json` (`CLOUDFLARE_API_TOKEN=...`, mode `0600`) or in
+  `CLOUDFLARE_API_TOKEN`; the account id in config or `CLOUDFLARE_ACCOUNT_ID`. Never argv,
+  never a run dir, a prompt or a log: reasons are redacted before they are cut.
+- **Privacy:** a mapped site sends Cloudflare what it would have sent TypeSafe: prompts,
+  diff hunks, file excerpts, memory entries. Cloudflare states it does not store or train on
+  these requests.
+- **Off switches:** `FORGE_CLEF=off` sends every site back to TypeSafe and stops the
+  predictions below; `FORGE_JEV_PROVIDER=typesafe|cloudflare` forces one provider for every
+  site; `FORGE_JEV_OUTCOMES=off` stops only the predictions; `FORGE_JEV=off` still stops all.
+- **Limits the client handles:** at most 64 questions per request (drift asks up to 120, so
+  it goes as two requests, concurrently, merged); a context of 64K tokens (`clef`) or 24K
+  (`clef-flash`) for state and questions together, beyond which Cloudflare cuts the state
+  silently, so the client refuses (`too-large`) rather than send it; `4006`/`3036` (allocation
+  used up) and `5035` (plan) are final, `3040`/`971` and 429 are retried within the deadline,
+  honoring `Retry-After`. Every requested answer must be present and in range, or the call
+  is `malformed`. A failure is "no judgment", as with TypeSafe.
+- **Thresholds are per model.** A cutoff measured on Jev's probabilities is not one on
+  Clef's: `model_thresholds.<model>.<name>` overrides a threshold for that model's answers.
+  Drift now reads `drift_warn`, which falls back to `gate_warn` until a model has its own.
+- **Cost and latency:** `clef` is $0.24 and `clef-flash` $0.038 per million input tokens,
+  output free. A drift request (two chunks) takes about 2.4 s at the median and 3.0 s at
+  p90, against Jev's 0.65 s and 10.8 s on the same live requests.
+
+### Which sites move: a paired replay, not an assumption
+
+`jev-drift` is the only site `setup` maps by default. Measured 2026-10-16 by replaying the
+stored live requests (the run dataset keeps each one whole) through Clef, labeled by the
+task's own outcome, older half of runs to calibrate, newer half to judge; a site moves only
+if Clef is non-inferior (paired run-bootstrap lower bound above -0.02) on at least 30
+labeled decisions from at least 8 held-out runs:
+
+| site | labeled | held-out | Clef AUC | Jev AUC | paired diff [95%] | moves |
+|---|---|---|---|---|---|---|
+| `jev-drift` | 4560 decisions, 16 positive, 17 runs | 9 runs | 0.985 | 0.972 | +0.013 [0.000, +0.028] | yes |
+| `jev-memory-slice` | 855 decisions, 144 positive, 10 runs | 5 runs | 0.951 | 0.953 | -0.003 [-0.064, +0.014] | no: too few runs |
+
+`jev-review-triage` was not replayed: its `correctness` half has no label, and the export's
+citation label (a review against its own diff or a substituted one) is not a live label.
+Clef keeps drift's shared 0.60 cutoff: on the calibration half neither model warned
+precisely (Jev flagged nothing at 0.60), so no Clef-specific value could be fitted there; at
+0.60 on the held-out half Clef flagged 1 file (right) and Jev 2 (1 right). The other sites
+stay on TypeSafe until a replay of their own justifies a move.
+
+### Silent outcome predictions
+
+With a Cloudflare provider configured, forge also asks Clef two questions forge has never
+asked, and records the answers without acting on them:
+
+- `clef-first-pass`, once per plan after the table rewrite that assigns every row its dwarf
+  and reviewer: will this task's first attempt pass review? Written to
+  `tasks/<id>/clef.first_pass` with the assignment it was about.
+- `clef-retry-pass`, when a dwarf attempt above 1 begins (not on a QA-only resume): will this
+  retry pass, given the findings it was handed? Written to `tasks/<id>/clef.retry_pass.<n>`.
+
+On forge's exported history Clef ranks these at AUC 0.73 (first pass, 253 cases) and 0.75
+(retry, 300), against hosted Jev's composite at 0.69 and cheap features at 0.67 and 0.61, but
+its raw probabilities are poorly calibrated (retry ECE 0.28) and the newest, largest tasks
+were too few to test. So nothing is printed and nothing acts: no prompt, verdict, tasks table,
+retry spec or findings ever sees a prediction. The export attaches them to the routing and
+retry cases (`first_pass_clef`, with whether attempt 1 ran the predicted dwarf, and
+`retry_pass_clef`), so their skill on new runs can be measured before anything relies on
+them. One bounded command per plan or retry: one deadline, at most 4 requests in flight,
+model resolution cached, an in-process watchdog; about $0.0025 a task with `clef`.
+
+## Status
+
+Every phase of the plan is implemented. What changed the picture since is that it was
+finally **run**, rather than tested: the first real `forge-parallel.sh` invocation with
+Jev enabled, on an 8-task plan, found twelve defects that 402 passing tests could not
+see. Most shared one shape — the code was correct and nothing reached it.
+
+- Every `--jev*` flag was unreachable: the option parser was sourced from inside
+  functions that had already walked their arguments, so `forge_jev_flag`,
+  `forge_jev_export`, `forge_jev_record` and `forge_jev_restore` had no callers at all.
+- `prompt.md` never reached routing or the gates, which judged tasks from a one-line
+  title. Both text gates had therefore been calibrated on data production never produced.
+- Both memory readers in `verify.py` parsed the *ledger* row format against the rendered
+  `memory.md`, so they were empty for every repo, always — which made the `known_flake`
+  re-run unreachable, since it requires a corroborating trap.
+- Wave coupling borrowed `gate_warn` and had never fired on any plan.
+- Every memory call was unlogged, and the review-triage flag printed where the default
+  output mode never shows it.
+
+**What acts today:** the drift, prompt-adequacy and verifiability gates, verification,
+early test feedback, memory curation, wave coupling (same-wave and cross-wave) and
+review-triage annotation. All are advisory —
+none blocks a dispatch, and nothing overturns a `FORGE_VERDICT`.
+
+**What was removed (2026-10-17):** the tier rewrite behind `--jev-act`, `calibrate`,
+effort and QA-effort sizing, `parse-spec` and Fractal re-rating. None had ever acted, and
+the run dataset showed decision models cannot choose models or efforts; statistics over
+randomized routing do that now. **Frozen:** wave coupling and forward-dependency scoring
+keep running as they are, and the decision-model fine-tuning pipelines get no further
+work. **Kept and maintained:** capture and replay, the provider client, drift and memory
+curation, where decision models measurably read well.
+
+**Known unsound, recorded rather than patched:**
+
+- `independently_verifiable` still cannot separate a task that documents later work
+  (`readme`, 0.60–0.65) from a real one (`cli`, 0.65–0.75). Putting the plan in state
+  fixed the rubric's worst end — it now scores the clearest fragment at 0.26–0.34, where
+  it used to score 0.63–0.65 — but the right-hand side remains a precision gate whose
+  silence is not evidence.
+- `tests_act` = 0.70 is stale. The semantics are fixed — the gate reads 1 − P(none)
+  rather than Choice confidence, so a tie between two correct commands no longer reads as
+  doubt — but the threshold behind it was calibrated before candidate evidence reached
+  the model, and the one confirmed wrong pick (`spankai` → `cargo test`) has risen from
+  0.55 to 0.77–0.81 and now clears the gate either way.
+- Wave coupling's same-wave half rests on one plan with roughly 0.02 of margin. Its
+  cross-wave half (`forward_dependency`, gap 0.73) is far better separated but rests on a
+  single positive example.
+
+The standing lesson, now learned seven separate times: a threshold named for one rubric
+and reused for another is wrong, and a number measured against state the model is not
+actually sent describes nothing. The sixth was `independently_verifiable`, which asked
+whether a task needed "work declared in a different task" while no different task had
+ever been in state — and the fix was to send the plan, not to reword the question. `jevscope` exists to make both visible — it reads the
+run logs back and reports each rubric's distribution and the spread across byte-identical
+repeated requests, which is the floor any threshold has to clear.