@kontourai/flow-agents 3.3.0 → 3.4.0

package/docs/specs/learning-review-proposals-contract.md

@@ -0,0 +1,340 @@
+# `kontour.learning-review-proposals` — kit/gate tuning proposal contract (v0.1)
+
+**Status:** ratified (flow-agents #352). **Kind:** `kontour.learning-review-proposals`.
+**Version:** `0.1`.
+
+## Purpose
+
+Turn already-accumulated `kontour.console.economics` records (fa#349) into **advisory,
+evidence-cited** kit/gate tuning proposals — the same "turn recorded outcomes into a proposal
+a human ratifies" shape as `routing-efficiency.sh` (#415), generalized from per-`(role,model)`
+to **per-kit** and **per-gate**. `scripts/telemetry/learning-review-proposals.sh` reads a
+window of `economics.jsonl`, joins each record's kit/gate identity from already-shipped
+sidecar shapes, computes aggregates, and emits proposals into a durable, idempotent ledger. A
+companion script, `scripts/telemetry/learning-review-decide.sh`, records the human
+ratify/reject/defer decision against a proposal id. Nothing here is ever auto-applied:
+`learning-review` step 2b (see `kits/builder/skills/learning-review/SKILL.md`) surfaces the
+output for review; a human decides.
+
+This is a pure **consumer**: it makes zero edits to `economics-record.sh`,
+`economics-record.schema.json`, or `gate-review`'s `InquiryRecord` output (ADR 0008/0010
+consume-never-fork). It never writes to `kits/**`, `.datum/config.json`, or any gate/flow
+config file.
+
+## Invocation
+
+```bash
+scripts/telemetry/learning-review-proposals.sh [economics.jsonl ...]      # defaults to the local economics log
+cat economics.jsonl | scripts/telemetry/learning-review-proposals.sh -    # records on stdin
+scripts/telemetry/learning-review-proposals.sh --since 2026-07-01 --until 2026-07-07 \
+  --sessions-root .kontourai/flow-agents --ledger .kontourai/telemetry/learning-review-proposals.jsonl
+```
+
+| Flag | Default | Meaning |
+| --- | --- | --- |
+| `--since VAL` | unbounded | date-only (`2026-07-01`), full ISO-8601 with `Z` (`2026-07-01T00:00:00Z`), full ISO-8601 with a `+00:00` UTC offset, or epoch-ms; lower bound on each record's `.at` |
+| `--until VAL` | unbounded | same accepted formats; upper bound on each record's `.at` |
+| `--sessions-root DIR` | `.kontourai/flow-agents` | where per-session `<task_slug>/trust.bundle` and `<task_slug>/gate-review.inquiries.json` are looked up for the kit/gate join |
+| `--ledger PATH` | `.kontourai/telemetry/learning-review-proposals.jsonl` (override via `LEARNING_REVIEW_PROPOSALS_LEDGER` env, mirrors `TELEMETRY_ECONOMICS_LOG_FILE`) | the durable, idempotent proposal ledger |
+| `-h`/`--help` | — | usage |
+
+Positional args and stdin follow the exact `routing-efficiency.sh` convention: explicit file
+args (a bare `-` means stdin), else stdin when piped, else the default log
+(`${TELEMETRY_ECONOMICS_LOG_FILE:-${TELEMETRY_DATA_DIR:-<repo>}/.kontourai/telemetry/economics.jsonl}`).
+
+**Accepted timestamp formats** (for `--since`/`--until` AND every record's own `.at`): an
+epoch-ms integer (as a JSON number or a digit-only string); a date-only string (`YYYY-MM-DD`,
+normalized to midnight UTC); a full ISO-8601 string with a literal `Z` UTC suffix; or a full
+ISO-8601 string with an explicit `+00:00` UTC offset (converted internally to `Z` before
+parsing — jq's `fromdateiso8601` only understands the `Z` form). Fractional seconds are
+accepted and stripped before parsing (e.g. `2026-07-01T00:00:00.123Z`, the default shape of a
+JS `Date.toISOString()` value) — precision finer than one second is not retained. Arbitrary
+non-UTC offsets are not supported. A `--since`/`--until` value in none of these forms is a
+CLI usage error: the analyzer exits non-zero with a stderr diagnostic and writes nothing,
+never a fabricated `"insufficient-data"` result (an unrecognized CLI argument is operator-
+correctable input, not a data gap). A record's own `.at` field, by contrast, degrades to an
+unbounded (`null`) timestamp on an unrecognized value — the same honest-degradation treatment
+as an absent `.at` — so one malformed record never aborts analysis of an otherwise-good window.
+
+Tunables (env, all optional): `LR_MIN_WINDOW_SAMPLE` (5), `LR_MIN_KIT_SAMPLE` (6),
+`LR_MIN_GATE_SAMPLE` (3), `LR_COST_RISE_PCT` (25), `LR_FLAT_FINDINGS_PCT` (10),
+`LR_GATE_FALSE_BLOCK_RATE` (0.5), `LR_GATE_WELL_CALIBRATED_RATE` (0.9).
+
+## Output document shape
+
+```json
+{
+  "schema": "kontour.learning-review-proposals",
+  "version": "0.1",
+  "window": { "since": "<epoch-ms|null>", "until": "<epoch-ms|null>" },
+  "records_considered": 0,
+  "outcome": "ok | insufficient-data",
+  "aggregates": {
+    "by_kit": [ { "kit_id": "...", "runs": 0, "...": "..." } ],
+    "by_gate": [ { "gate_id": "...", "fire_count": 0, "...": "..." } ],
+    "partial": false
+  },
+  "proposals": [
+    {
+      "proposal_id": "...",
+      "target": { "kind": "kit|gate", "id": "..." },
+      "pattern": "kit-review-cost-inflation | gate-false-block-review | gate-well-calibrated",
+      "proposed_change": "<human-readable recommendation>",
+      "severity": "advisory",
+      "evidence": { "cost": { "...": "..." }, "defect": { "...": "..." } },
+      "expected_effect": { "metric": "...", "direction": "decrease|increase|maintain", "description": "..." },
+      "decision": { "status": "proposed|ratified|rejected|deferred", "decided_by": null, "decided_at": null, "rationale": null },
+      "follow_on_ref": null,
+      "effect_observed": null,
+      "already_proposed": false
+    }
+  ],
+  "notes": [ "..." ]
+}
+```
+
+`window.since`/`window.until` in the OUTPUT document are the run's **effective** bounds:
+the `--since`/`--until` arg when given, else respectively the min/max `.at` (epoch-ms,
+normalized) actually observed among the records considered (or `null` when no records at
+all). This makes the effect-fill chronological comparison (below) well-defined even when a
+run is invoked with unbounded `--since`/`--until`.
+
+`already_proposed` is an OUTPUT-ONLY annotation (not stored on the ledger line itself): `true`
+when a proposal with the same `proposal_id` already exists in the ledger from a prior run (no
+duplicate line is appended); `false` when this run appended a brand-new ledger line for it.
+
+## Kit/gate identity join (honest, reuses already-shipped shapes — no new field on fa#349's record)
+
+For each economics record with a non-null `task_slug`, the analyzer looks for
+`<sessions-root>/<task_slug>/trust.bundle` and `<sessions-root>/<task_slug>/gate-review.inquiries.json`
+(both already-shipped, read-only inputs — never invoked, never modified):
+
+- **`kit_id`** = the mode (most frequent) of `[.claims[].claimType | split(".")[0]]` from that
+  session's `trust.bundle` (the real Kit Catalog namespace, e.g. `builder`,
+  `veritas-governance` — `kits/<name>/` on disk). If the `trust.bundle` is absent, empty, or
+  unreadable, `kit_id` degrades to `"unattributed"` — the same honest-degradation idiom
+  `economics-record.sh` already uses for `phases[].phase`.
+- **Per-gate rows** for that record = one row per `gate-review.inquiries.json` `InquiryRecord`:
+  `{ gate_id: .inquiry.target.fieldOrBehavior, outcome: .answer.value.calibration, fired: .answer.value.gateFired }`.
+  If the file is absent, empty, or unreadable, this record contributes zero gate rows (counted
+  separately in `notes` as "sessions without a gate-review join").
+- A record with a **null** `task_slug` (or a `task_slug` with no matching session directory)
+  gets `kit_id = "unattributed"` directly, with zero gate rows contributed — no lookup is
+  attempted.
+
+**Honest current-data gap (stated, not hidden):** real `economics.jsonl` records emitted today
+all carry `"task_slug":null` (fa#349 slice-1; `task_slug` plumbing was never wired end-to-end
+for the emitter's live invocation path). Until that separate, out-of-scope plumbing gap is
+fixed, every real run's `kit_id` degrades to `"unattributed"` and it contributes zero gate
+rows — per-kit/per-gate attribution is genuinely vacuous on live data today. This analyzer does
+not fabricate an identity to paper over that gap; fixtures (not live data) are what exercise
+the attributed path until `task_slug` is populated on real runs.
+
+**Trusted-as-is inputs (consume-never-fork):** `cost.estimated_cost_usd` and every other numeric
+field read from an `economics.jsonl` record is trusted exactly as fa#349 emitted it, including a
+negative or zero value — this analyzer never clamps, floors, or re-derives it; a negative/zero
+cost simply flows through the same average/percentage formulas as any other value (and can
+legitimately produce a negative `cost_trend_pct`, which is not an error).
+
+## Window and aggregates (exact formulas)
+
+All derived floating-point values (rates, percentages, dollar/second averages) are rounded to
+**4 decimal places** using `(<value> * 10000 | round) / 10000` — the same rounding idiom
+`routing-efficiency.sh` uses for `trouble_rate` (scaled to 4 places here instead of 3, for
+dollar-amount precision). Any ratio with a zero or otherwise undefined denominator is `null`,
+never a fabricated `0` or `Infinity`.
+
+### `by_kit[]`
+
+Records are grouped by `kit_id`. Within a group, records are sorted ascending by `.at` and
+split into a first half (oldest) and a second half (newest): `half_point = floor(runs / 2)`;
+the first `half_point` records are the "first half," the remaining `runs - half_point` records
+(the odd one out, if `runs` is odd, lands in the newer half) are the "second half."
+
+| Field | Formula |
+| --- | --- |
+| `kit_id` | group key |
+| `runs` | count of records in the group |
+| `first_half_avg_cost_usd` | mean of `.cost.estimated_cost_usd` over the first half (`null` if the first half is empty) |
+| `second_half_avg_cost_usd` | mean of `.cost.estimated_cost_usd` over the second half |
+| `cost_trend_pct` | `((second_half_avg_cost_usd - first_half_avg_cost_usd) / first_half_avg_cost_usd) * 100`; `null` if `first_half_avg_cost_usd` is `null` or `0` |
+| `first_half_findings_total` | sum of `.defects.findings_by_severity.{critical,high,medium,low}` over the first half |
+| `second_half_findings_total` | same, over the second half |
+| `findings_delta_pct` | `((second_half_findings_total - first_half_findings_total) / first_half_findings_total) * 100`; if `first_half_findings_total == 0`: `0` when `second_half_findings_total == 0` too, else `null` (a rise from a zero base has no defined percentage) |
+| `avg_wall_clock_s` | mean of `.time.wall_clock_s` over ALL records in the group (not half-split) |
+| `avg_human_wait_s` | mean of `.time.human_wait_s` over ALL records in the group |
+| `route_back_rate` | `sum(.iterations.route_backs) / sum(.iterations.count)` over ALL records in the group; `null` if the count sum is `0` |
+| `caught_false_completions_total` | sum of `.defects.caught_false_completions` over ALL records in the group |
+
+### `by_gate[]`
+
+Gate rows from every considered record's join (above) are flattened and grouped by `gate_id`.
+
+| Field | Formula |
+| --- | --- |
+| `gate_id` | group key (`inquiry.target.fieldOrBehavior`) |
+| `fire_count` | total gate rows for this `gate_id` (every calibration: `correct` + `false_block` + `missed_block`) |
+| `correct_count` | rows with `outcome == "correct"` |
+| `false_block_count` | rows with `outcome == "false_block"` |
+| `missed_block_count` | rows with `outcome == "missed_block"` |
+| `false_block_rate` | `false_block_count / fire_count` |
+| `avg_wall_clock_s_when_fired` | mean of the PARENT economics record's `.time.wall_clock_s`, over the distinct records that contributed at least one row for this `gate_id` with `fired == true` |
+| `avg_human_wait_s_when_fired` | same, for `.time.human_wait_s` |
+
+(`InquiryRecord`s themselves carry no wall-clock time; "when fired" averages borrow the
+parent record's own time fields for exactly the records whose gate-review join produced a
+`fired:true` row for that `gate_id`.)
+
+## Whole-window insufficient-data gate (AC4)
+
+`records_considered` = the count of `kontour.console.economics` records whose `.at` falls in
+`[since, until]` (both inclusive; unbounded when the flag is omitted). If
+`records_considered < LR_MIN_WINDOW_SAMPLE` (default `5`): top-level `outcome` is
+`"insufficient-data"`, `proposals` is `[]`, and `aggregates.partial` is `true` — but
+`aggregates.by_kit`/`by_gate` are STILL computed and returned, for transparency (never hidden,
+per `harness-capability-matrix.md`'s "degrade to an explicit unavailable, never fabricate").
+This whole-window gate is distinct from, and evaluated in addition to, the per-target sample
+gates (`LR_MIN_KIT_SAMPLE`/`LR_MIN_GATE_SAMPLE`) below, which just silently exclude a single
+under-sampled target from proposal consideration without flipping the top-level `outcome`.
+
+## Proposal rules (Goodhart-paired: `evidence.cost` + `evidence.defect` always co-required)
+
+Only evaluated when `outcome == "ok"` (i.e. the whole-window gate passed).
+
+1. **`kit-review-cost-inflation`** — fires when, for a `by_kit[]` entry: `runs >= LR_MIN_KIT_SAMPLE`
+   (default `6`) AND `cost_trend_pct >= LR_COST_RISE_PCT` (default `25`) AND
+   `findings_delta_pct <= LR_FLAT_FINDINGS_PCT` (default `10`) — cost is rising while defects
+   caught are flat or falling (the pattern a Goodhart-blind cost-only metric would miss).
+   - `evidence.cost = { first_half_avg_cost_usd, second_half_avg_cost_usd, cost_trend_pct }`
+   - `evidence.defect = { first_half_findings_total, second_half_findings_total, findings_delta_pct, caught_false_completions_total }`
+   - `expected_effect = { metric: "avg_cost_usd", direction: "decrease", description: "cost should come back down without a corresponding rise in escaped defects" }`
+2. **`gate-false-block-review`** — fires when, for a `by_gate[]` entry:
+   `fire_count >= LR_MIN_GATE_SAMPLE` (default `3`) AND `false_block_rate >= LR_GATE_FALSE_BLOCK_RATE`
+   (default `0.5`) — the gate is blocking on already-passing claims more often than not.
+   - `evidence.cost = { avg_wall_clock_s_when_fired, avg_human_wait_s_when_fired }`
+   - `evidence.defect = { fire_count, correct_count, false_block_count, missed_block_count }`
+   - `expected_effect = { metric: "false_block_rate", direction: "decrease", description: "false_block_rate should fall toward correctly-calibrated blocking" }`
+3. **`gate-well-calibrated`** (monitor-style, still `severity:"advisory"` — mirrors
+   `routing-efficiency.sh`'s `keep-tier`) — fires when `fire_count >= LR_MIN_GATE_SAMPLE` AND
+   `(correct_count / fire_count) >= LR_GATE_WELL_CALIBRATED_RATE` (default `0.9`) AND
+   `false_block_count == 0` AND `missed_block_count == 0`.
+   - `evidence.cost = { avg_wall_clock_s_when_fired, avg_human_wait_s_when_fired }`
+   - `evidence.defect = { fire_count, correct_count, false_block_count, missed_block_count }`
+   - `expected_effect = { metric: "false_block_rate", direction: "maintain", description: "gate is well-calibrated; no change expected or needed" }`
+
+A `by_kit`/`by_gate` entry under its rule's sample-size gate contributes **no proposal at
+all** for that rule (silently excluded from proposals, still visible in `aggregates`).
+
+Every proposal's `evidence` object structurally REQUIRES both `cost` and `defect`
+(`learning-review-proposals.schema.json`, mirroring fa#349's own `cost`+`defects`
+co-requirement, R7) — a proposal citing cost alone (or defect alone) is schema-invalid. This
+is the Goodhart guard: no proposal may claim "cheaper" without also showing "and here is what
+it caught or missed."
+
+## Idempotent ledger (`.kontourai/telemetry/learning-review-proposals.jsonl`)
+
+Durable, append-mostly JSONL — one proposal per line, in the exact shape above (minus the
+output-only `already_proposed` field). Default path
+`.kontourai/telemetry/learning-review-proposals.jsonl`, override via `--ledger PATH` or
+`LEARNING_REVIEW_PROPOSALS_LEDGER` (mirrors `TELEMETRY_ECONOMICS_LOG_FILE`'s local-first
+convention: the per-run snapshot lives in the session artifact dir; this ledger is the durable,
+cross-session trail).
+
+- **`proposal_id`** = `slugify("<since>_<until>_<target_kind>-<target_id>_<pattern>")`, where
+  `<since>`/`<until>` are the raw `--since`/`--until` argument strings (or the literal `all`
+  when the flag was omitted) and `slugify()` is the SAME idiom already used at
+  `evals/ci/run-baseline.sh:139-140`:
+  `tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9]+/-/g; s/^-//; s/-$//'` — no external hash
+  dependency.
+- On every run: proposals are computed fresh (stateless — the analyzer never reads its own
+  ledger to decide WHAT to propose, only to decide whether a computed proposal is new). For
+  each computed proposal, the ledger is checked for an existing line with the same
+  `proposal_id`:
+  - **Found** → mark `already_proposed:true` in this run's OUTPUT; no duplicate line is
+    appended.
+  - **Not found** → append a new ledger line with
+    `decision:{status:"proposed",decided_by:null,decided_at:null,rationale:null}`,
+    `follow_on_ref:null`, `effect_observed:null`, and mark `already_proposed:false` in the
+    output.
+- Running the analyzer twice over the identical window therefore produces zero duplicate
+  `proposal_id`s in the ledger (AC4) — the second run's output marks every proposal
+  `already_proposed:true`.
+
+## Decision recording (`learning-review-decide.sh`)
+
+```bash
+scripts/telemetry/learning-review-decide.sh <ledger-path> <proposal-id> \
+  --ratify|--reject|--defer --decided-by NAME [--rationale TEXT] [--follow-on-ref REF]
+```
+
+Finds the ledger line with the matching `proposal_id` and rewrites its `decision` object
+(`status`, `decided_by`, `decided_at: <now, ISO-8601>`, `rationale`) in place, atomically
+(temp file + `mv`). `follow_on_ref` is set ONLY when `--ratify` and `--follow-on-ref` are both
+present. Exits non-zero, writing nothing, when: the `proposal_id` is not found; more than one
+of `--ratify`/`--reject`/`--defer` is passed; or `--follow-on-ref` is passed without `--ratify`
+(the AC3 ratify-before-follow-on ordering guard — a follow-on work item can never be linked to
+an unratified proposal). All other ledger lines are left byte-identical.
+
+## Effect-fill pass (AC5)
+
+On every `learning-review-proposals.sh` invocation, AFTER computing and ledgering the current
+window's proposals: scan the ledger for entries where `decision.status == "ratified"` AND
+`effect_observed == null` AND the entry's `window.until` (epoch-ms) is chronologically before
+the CURRENT invocation's effective `window.since` (epoch-ms) — i.e. the ratified proposal's
+own window has fully elapsed before this new window begins. For each match, recompute the
+metric named in that entry's `expected_effect.metric` over the CURRENT window's aggregates,
+for the exact same `target`:
+
+- `target.kind == "kit"` → `after` = the current window's `by_kit[]` entry for that `kit_id`'s
+  overall average cost (`second_half_avg_cost_usd`, i.e. the newest-half average of the new
+  window — comparable apples-to-apples against the value that triggered the original
+  proposal); `before` = the ratified proposal's own `evidence.cost.second_half_avg_cost_usd`
+  (the elevated level that triggered it).
+- `target.kind == "gate"` → `after` = the current window's `by_gate[]` entry's
+  `false_block_rate` for that `gate_id`; `before` = the ratified proposal's own
+  `evidence.defect.false_block_count / evidence.defect.fire_count`.
+
+If no `by_kit`/`by_gate` entry exists for that target in the current window (e.g. the kit/gate
+had zero runs this window), `effect_observed` is left `null` this pass — an honest gap, not a
+fabricated measurement; it is retried on the next invocation that does have data.
+
+Writes:
+
+```json
+"effect_observed": {
+  "measured_at": "<ISO-8601, now>",
+  "metric": "avg_cost_usd | false_block_rate",
+  "before": 0.0,
+  "after": 0.0,
+  "moved": "improved | worsened | unchanged"
+}
+```
+
+`moved` is derived from `expected_effect.direction`: `"improved"` = the metric moved in the
+direction the proposal intended (`decrease` → `after < before`; `increase` → `after > before`);
+`"unchanged"` = `after == before` exactly; `"worsened"` = moved, but the wrong way (or any
+non-zero move when `direction == "maintain"`).
+
+## Guardrails (ADR 0008/0010, ADR 0002/0003 call 5)
+
+- **Advisory only.** Every proposal is `severity: "advisory"`. Neither
+  `learning-review-proposals.sh` nor `learning-review-decide.sh` EVER writes to `kits/**`,
+  `.datum/config.json`, or any gate/flow config file — they only read economics/session
+  artifacts and write to their own ledger.
+- **Human-ratified, decision-before-follow-on.** A recorded human decision
+  (`learning-review-decide.sh`) is required before any follow-on work item may cite a
+  proposal; `--follow-on-ref` is refused without `--ratify` (AC3).
+- **Consume-never-fork.** `economics-record.schema.json` (fa#349) and
+  `gate-review.inquiries.json` (#118-120) are read-only inputs, never modified, never
+  re-derived by inventing a parallel schema.
+- **No fabrication.** Ratios with an undefined denominator are `null`, never `0`; an
+  under-threshold window yields an explicit `"insufficient-data"` outcome, never a confident
+  but thin call; real records lacking `task_slug` degrade to `"unattributed"`, never an
+  invented kit identity.
+
+## Relationship to `routing-efficiency.sh` (#415)
+
+Same architecture, same input stream (`economics.jsonl`), same guardrails — generalized from
+per-`(role,model)` delegation efficiency to per-kit/per-gate review efficiency. Both are purely
+manual/skill-invoked (no scheduler/cron primitive exists in this repo for either); both are
+wired into `learning-review`'s closeout step (2a for routing, 2b for this contract).

package/docs/specs/routing-efficiency-review.md

@@ -0,0 +1,59 @@
+# Routing-efficiency review
+
+**Turn recorded delegation outcomes into advisory proposals about which model each role should route
+to — the #409 small-model value proof, pointed inward at our own agent fan-out (#415).**
+
+`scripts/telemetry/routing-efficiency.sh` reads the per-run economics records (`delegations[]` +
+`outcome`, from `economics-record.sh`) and, per `(role, model)`, asks: *is the model this role routes
+to actually efficient, or is a cheaper tier being re-worked so often it should be escalated (or an
+expensive tier so reliably accepted it is over-provisioned)?* `learning-review` runs it at closeout
+(step 2a) and a human ratifies any change.
+
+## Input / output
+
+```bash
+routing-efficiency.sh [economics.jsonl ...]     # defaults to .kontourai/telemetry/economics.jsonl
+cat economics.jsonl | routing-efficiency.sh -   # or records on stdin
+```
+
+Emits a `kontour.routing-efficiency` v0.1 document: `by_role_model[]` (per-group outcome counts +
+`trouble_rate`) and `proposals[]` (one advisory proposal per group), plus `notes`.
+
+## What it computes (honest by construction)
+
+For each `(role, bare-model)` group across all delegations:
+
+- `measurable = accepted + rework + diverged + failed` — **`unavailable` outcomes are excluded**. A
+  delegation whose outcome the harness could not expose is neither a success nor a failure, so it never
+  moves a rate (see `harness-capability-matrix.md`). It is counted separately as coverage.
+- `trouble_rate = (rework + diverged + failed) / measurable` — the share of *measurable* delegations
+  that did not cleanly accept.
+
+## Proposal kinds
+
+| Kind | Fires when | Meaning |
+| --- | --- | --- |
+| `escalate-minimum-tier` | `measurable ≥ ROUTING_MIN_SAMPLE` and `trouble_rate ≥ ROUTING_HIGH_TROUBLE` (0.5) | the cheap tier is under-routed for this role — consider raising its minimum tier |
+| `keep-tier` | `measurable ≥ min` and `trouble_rate ≤ ROUTING_LOW_TROUBLE` (0.1) | efficient — accepts cleanly; keep current routing (a candidate to try a *cheaper* tier lives in future work) |
+| `monitor` | `measurable ≥ min`, trouble in between | within tolerance; watch |
+| `insufficient-signal` | `measurable < min` | not enough measurable outcomes to judge — coverage reported, no confident call |
+
+Thresholds are tunable via env: `ROUTING_MIN_SAMPLE` (default 3), `ROUTING_HIGH_TROUBLE` (0.5),
+`ROUTING_LOW_TROUBLE` (0.1).
+
+## Guardrails (ADR 0002 / ADR 0003 call 5)
+
+- **Advisory only.** Every proposal is `severity: "advisory"`. The analyzer NEVER edits
+  `.datum/config.json` or any routing config — it only reads and reports.
+- **Human-ratified.** A person decides whether to act on a proposal; the resulting role→model change
+  travels the normal deliver loop, never an auto-apply.
+- **No fabrication.** `unavailable` outcomes are excluded from rates; a thin sample yields
+  `insufficient-signal`, not a confident verdict; empty input yields an empty `proposals[]`, never an
+  invented lesson.
+
+## Relationship to the value plane
+
+This is the same question the external value proof (`kontourai/evals`, #409) asks — "does routing the
+right model to each task pay off?" — asked of our *internal* delegation. The console renders the
+`(role, model)` cost + acceptance surface (#415 console panel); this analyzer produces the actionable
+proposal that closes the loop back to the routing ladder.

package/docs/verifiable-trust.md

@@ -1,31 +1,38 @@
-# Verifiable Trust — why "done" actually means done
+# Verifiable Trust — tamper-evident done
 
 > **The problem with autonomous coding agents: they grade their own homework.**
 > An agent writes the code, runs the tests, and reports "all green, shipped." If it's
 > wrong — or if it learns it can just *say* the tests passed — you find out in production.
-> Flow Agents is built so an agent **can't** mark work complete that isn't.
+> Flow Agents is built so "done" is a claim with evidence, not a status the model gets
+> to assert.
 
 ## The one-line pitch
 
 Flow Agents treats "the work is done" as a **claim that must be proven**, not a status the
-agent gets to assert. Completion is gated by **evidence the system re-derives itself**, and
-the authoritative check runs in **CI — an environment the agent can't disable or fake** —
-with **cryptographically signed provenance** of exactly what was verified.
+agent gets to assert. Completion is gated by evidence the system re-derives, with two
+different trust tiers:
 
-Most agent frameworks trust the model's self-report. Flow Agents doesn't trust the agent,
-its claims, *or* its environment.
+- **L1 local runtime gate and capture** — advisory, best-effort protection inside the agent
+  runtime. It catches native command evidence, stop-short behavior, and obvious divergence
+  before the work leaves the session.
+- **L2 controlled re-run anchor** — CI re-executes manifest commands and reconciles them
+  against git diff as ground truth. This is the authoritative boundary for CI-verified
+  status.
+
+Most agent frameworks trust the model's self-report. Flow Agents treats that self-report as
+input to reconcile, not as truth. The posture is **tamper-evident**, not tamper-proof.
 
 ## What that buys you
 
-- **"Done" you can rely on.** A finished task ends with real evidence — tests, build, lint,
-  review findings, captured command results — and the gate *re-derives* the verdict from
-  that evidence. A claimed pass that contradicts a captured failure is **blocked**, not shipped.
-- **Anti-gaming by design.** The gate independently captures real command results and
-  reconciles them against the agent's claims — namespace-agnostic, and independent of any
-  status the agent self-declares. Tricks like `npm test || true` (laundering the exit code)
-  are rejected.
-- **An external anchor that can't be switched off.** On every pull request, CI re-runs the
-  verification *fresh* in a clean environment and **fails the merge on any divergence**
+- **"Done" you can inspect.** A finished task ends with evidence — tests, build, lint,
+  review findings, captured command results, CI output, or an explicit `NOT_VERIFIED` gap.
+  A claimed pass that contradicts captured or reconciled evidence is blocked or downgraded,
+  not laundered into verified status.
+- **Anti-gaming by design.** Local capture raises the cost of direct native-command
+  self-tampering, and CI reconciles manifest commands against fresh results. Tricks like
+  `npm test || true` are rejected when they are represented as command evidence.
+- **An external anchor that can't be switched off locally.** On pull requests wired to the
+  trust anchor, CI re-runs verification fresh in a clean environment and fails on divergence
   between what the agent claimed and what CI actually observed. The agent can tamper with its
   own machine all it likes; it can't reach into CI.
 - **Signed provenance.** CI mints a Sigstore-signed attestation over its *own* results. The
@@ -46,7 +53,7 @@ its claims, *or* its environment.
   reproducible record of *what was verified and how* — provenance you can hand to an auditor,
   not a screenshot of a passing run.
 - **Multi-agent and at-scale delivery.** Every agent's output is held to the same external,
-  un-gameable bar — so you can fan out work without fanning out the risk that one agent
+  evidence-backed bar — so you can fan out work without fanning out the risk that one agent
   learns to game its gate.
 - **Platform teams adopting agents.** Add the trust anchor as a required check in your repos
   and get a consistent, enforced "agents must prove it" policy across every team.
@@ -55,20 +62,62 @@ its claims, *or* its environment.
 
 | | Typical agent setup | Flow Agents |
 |---|---|---|
-| "Is it done?" | The agent says so | Re-derived from independent evidence |
-| Failure hiding | Easy (claim pass, launder exit codes) | Caught — captured results reconcile against claims |
-| Where trust lives | In the agent's environment | **External** — CI re-runs fresh, agent can't disable it |
+| "Is it done?" | The agent says so | Re-derived from evidence; CI-verified only after the controlled anchor reconciles |
+| Failure hiding | Easy (claim pass, launder exit codes) | Native command capture locally; manifest-command reconciliation in CI |
+| Where trust lives | In the agent's environment | Local tier is advisory; authoritative tier is **external** CI + git ground truth |
 | Provenance | A log line | **Sigstore-signed** attestation of CI's own results |
 | Tampering with the gate | Possible | Required tests + code-owner review block it |
 
+## Trust tiers
+
+### L1: local runtime gate and capture
+
+The local tier runs inside the agent's runtime. It can steer the workflow, capture native
+command results, detect stop-short behavior, and warn or block depending on the host's hook
+surface. It is useful because it catches problems early, while the agent is still in the
+session.
+
+It is not the authoritative trust boundary. The agent and its tools can still affect the
+local machine. Local evidence is treated as session-local until it is either reconciled by
+the controlled anchor or explicitly carried as a lower-trust attestation.
+
+### L2: controlled CI re-run
+
+The controlled anchor runs outside the agent session. CI re-executes the manifest commands,
+checks the resulting status against the agent's claims, and uses git diff as the ground truth
+for what changed. A write that bypassed local capture still appears in the diff. A command
+claim that cannot be tied to a manifest command re-run does not become CI-verified just
+because the agent said it passed.
+
+The CI tier is where evidence can become independently reconciled. That is the line between
+"the session reported this" and "the controlled environment re-ran or inspected this."
+
+## Threat model note
+
+The local tier keys on native command tools. MCP or other non-native tool effects can bypass
+local capture, including file writes and test runs.
+
+That blind spot is disclosed, not hidden:
+
+- **File writes** still hit the CI/git boundary because the pull request diff is ground
+  truth for delivered changes.
+- **Non-native test runs** do not become CI-verified evidence unless the same check is
+  represented by a manifest command that CI re-runs. Otherwise they remain session-local or
+  `NOT_VERIFIED`.
+- **Bypassed local capture** should refuse, escalate, or degrade confidence. It must not be
+  laundered into a verified CI claim.
+
+Every trust claim should be traceable to public artifacts: the recorded evidence, the
+manifest command, the CI run, the git diff, or an explicit verification gap.
+
 ## The honest part
 
 This is a **defense-in-depth bar-raiser, not a magic wall** — and the docs say so plainly.
-The local gate raises the cost of casual or direct self-tampering; the *real* tamper-proof
-boundary is **external**: CI's fresh re-run, the CI-minted signatures, and human (owner)
-review — none of which the agent can reach. Known residuals (and their mitigations) are
-documented openly rather than hidden, because overstating security is its own risk. We'd
-rather you trust this *because* you can see where the lines are.
+The local gate raises the cost of casual or direct self-tampering; the real authoritative
+boundary is **external**: CI's fresh re-run, git diff reconciliation, CI-minted signatures,
+and human (owner) review. Known residuals are documented openly rather than hidden, because
+overstating security is its own risk. We'd rather you trust this *because* you can see where
+the lines are.
 
 > This posture wasn't designed on paper and declared safe — it was **earned by an adversarial
 > loop**: independent reviewers repeatedly tried to defeat the gate (and found real holes we

package/docs/workflow-usage-guide.md

@@ -201,6 +201,8 @@ Plans should preserve the work item's `R*` and `AC*` ids from `idea-to-backlog`.
 
 Keep an isolated worktree alive through review, verification, provider change creation, and provider checks so the branch remains inspectable and fixable without disturbing the main checkout. Remove the worktree only after the change is merged or accepted, the branch is abandoned, or the user explicitly asks to collapse the isolation. Do not copy files back into the original checkout by hand; Git is the merge surface.
 
+After creating or entering a fresh worktree, run `npm ci` there before pre-push checks or evals; each worktree needs its own `node_modules`.
+
 When `pull-work` chooses a worktree, record `worktree_lifecycle`: path, branch, retain-until condition, cleanup owner, cleanup command, and cleanup blockers. Publish-change retains the worktree for review/CI fixes; final acceptance or explicit abandonment owns cleanup.
 
 Example prompt:
@@ -581,6 +583,14 @@ Use learning-review. Capture facts, decisions, gaps, follow-ups, and durable kno
 
 ## Resumable sessions
 
+Builder build sessions also project their canonical Flow run into the resume block.
+Start a newly selected session with `flow-agents builder-run start --session-dir
+.kontourai/flow-agents/<slug>`. Thereafter, follow `next_action.skills` or
+`next_action.operations`, record the current gate evidence, and use the exact
+`next_action.command` if automatic synchronization was interrupted. See
+[`docs/spec/builder-flow-runtime.md`](spec/builder-flow-runtime.md) for the ownership,
+trust-binding, route-back, and artifact-root contract.
+
 When a session resumes (after context compaction, an agent restart, or a cross-session
 handoff), the workflow-steering hook emits a `RESUME:` block on `SessionStart` that
 gives the resuming agent immediate situational awareness without blocking or auto-deciding.