@nurix/etna 0.15.7 → 0.16.0

package/data/harnesses/discover/rules/self-correction.md

@@ -0,0 +1,177 @@
+# Self-Correction — Defect Inspection · Run Measurement · Iterate-or-Propose
+
+The reference for **Mode 4** (`discover` harness). After a run finishes (Discovery → Ingestion →
+Evaluation), the self-correcting step does THREE things, in order:
+
+- **A. Inspect** the finished run against a built-in **defect checklist** → a concrete list of
+  *behavioral defects* in the prior steps (a list, never a vibe).
+- **B. Measure** the run against a fixed **rubric** → a score + vibe band. This runs **every
+  iteration, including the final one** — the self-correcting step ALWAYS evaluates the current run.
+- **C. Iterate-or-propose.** Each defect is classified by ONE test — *does fixing it require editing a
+  harness file (a rule or an agent)?* — into **within-run** (auto-iterate on THIS target, no
+  approval) vs **harness-changing** (the approval-gated proposal). The within-run loop improves the
+  current run; the harness-changing list improves the NEXT, different target.
+
+> **The two gates that bind this step (both unchanged):**
+> 1. **Harness changes are approval-gated.** The main session presents `04-self-correction-proposal.md`
+>    and STOPS; nothing is applied until the user accepts per-item. Structural edits are flagged and
+>    never auto-applied.
+> 2. **Cost/state/credential actions are confirmation-gated.** The within-run iterate loop is
+>    **read-only / no-cost only.** Any fix needing a Pass-2 / credit-spend / state-change / credential
+>    action routes through the explicit-confirmation gate (ingestion §7 rule 4, §9.2) — never the
+>    auto-loop. The loop can never silently spend money or mutate an app to chase a score.
+
+---
+
+## A. The defect checklist (what to inspect — by the mode that produced it)
+
+Inspect the run and emit **every** defect that fired, each as:
+`{mode, defect, severity (high|med|low), evidence anchor (research/<target>/…), within-run | harness-changing}`.
+This list is the load-bearing output of stage A — never skip a category silently; "none in this
+category" is itself recorded.
+
+### Discovery defects
+- **Access axis mis-graded** — esp. `auth:have-login` asserted on a *claimed/expected* login that was
+  never live-probed (should be `auth:claimed` until confirmed); any `source`/`runtime`/`presence` value
+  later contradicted by what Ingestion actually found. *(high — it mis-routes the whole run)*
+- **Method graded from a label, not a retrieved artifact** — `openapi-verbatim` on a page merely
+  *titled* OpenAPI with no served, parseable spec. *(med)*
+- **A predicted-available dimension came back blocked/absent** — Discovery over-promised. *(med)*
+- **A sibling-declared count/host-list treated as a ceiling**, not a floor. *(low)*
+- **Scope verdict wrong** — a suite/primitive/ambiguous/ethics call that should have fired (or wrongly did). *(high)*
+
+### Ingestion defects
+- **Redaction miss** — a secret-shaped value reached `raw/`. *(HIGH — also hard-zeros the safety axis)*
+- **Method ≠ planned** — the capture used a weaker method than Discovery predicted, with no recorded reason. *(med)*
+- **Missing contract artifact** — a `raw/` file the dimension's contract requires was not produced. *(med)*
+- **Low completeness with no recorded gap/reason.** *(med)*
+- **Blocked-where-predicted-available and not recovered** — no retry / second method / fold. *(med)*
+- **Seam forked** — `api-path-catalog`/`feature-flags` written under a dimension's `raw/` instead of
+  appended to `dimensions/_shared/` (ingestion §6). *(med)*
+- **Same artifact double-mined** and the two counts read as corroboration rather than a range. *(med)*
+- **Silent absence** — a dimension dropped with no `status: absent/folded` record. *(med)*
+- **Went silent / looked hung** — a long fan-out with no summary-first partial (ingestion §8). *(low)*
+- **An absence asserted that contradicts a sibling**, without a second-method re-check (ingestion §7 rule 10). *(med)*
+
+### Evaluation defects
+- **A conflict left flagged-but-unresolved** where the direct-observation tiebreaker applies. *(med)*
+- **Over-promotion** — a confidence band raised without ≥2 *independent* dimensions (or on two views of one artifact). *(med)*
+- **Inference overrode a direct observation.** *(high)*
+- **A single-source / low-confidence claim stated as fact.** *(med)*
+- **Marketed write/execution behavior promoted to fact** on a `write_side_observed:false` session. *(med)*
+- **A gap not surfaced**, or a count-divergence rendered as a conflict instead of a range. *(low)*
+- **The published-vs-app-own API diff omitted** where a published API exists. *(med)*
+
+### Cross-cutting defects
+- **Provenance frontmatter incomplete or dishonest** — `status`/`confidence` not matching reality. *(high)*
+- **An absent dimension not recorded as a finding.** *(low)*
+
+> The checklist is the floor, not the ceiling — a run may exhibit a behavioral defect not listed here.
+> Record a novel defect with the same shape; if it would recur on a different target, route it
+> harness-changing (and it becomes a checklist addition in its own right).
+
+---
+
+## B. The measurement rubric (the score — always computed)
+
+Five axes, 0–20 each → **/100**, so iterations are comparable and "deteriorating" is a real drop, not
+noise. The score is a **heuristic to catch regression and give the loop direction** — not a precise grade.
+
+| # | Axis (0–20) | How to score |
+|---|-------------|--------------|
+| 1 | **Coverage** | `20 × (dimensions captured 'complete' ÷ dimensions reachable per the access vector)`; **−4** per dimension blocked where Discovery predicted available. |
+| 2 | **Provenance integrity** | `10 × (captures with complete honest frontmatter ÷ all captures)` + `10 × (evaluation claims anchored+provenance-tagged ÷ all claims)`. |
+| 3 | **Reconciliation correctness** | start 20; **−3** per weighting violation (over-promotion, tiebreaker skipped, single-source-as-fact, count-divergence-as-conflict, published-vs-app diff missing where a public API exists). |
+| 4 | **Calibration & safety** | gaps/absences recorded honestly + confidence bands honest, scaled to 20. **A redaction miss hard-zeros this axis** and flags the run `unsafe` regardless of the others. |
+| 5 | **Defect load** | `20 − Σ(severity weight)` over stage-A defects (high=3, med=2, low=1), floored at 0. |
+
+**Rollup:** sum the five → `score/100`. **Vibe band:** ≥85 strong · 70–84 solid · 50–69 thin · <50 weak.
+Plus a one-line gestalt (e.g. "solid coverage, but evaluation over-promoted two single-source claims").
+
+**The iteration ledger** (written into `04-self-correction-proposal.md`):
+
+| iter | score | Δ | cover | prov | recon | calib | defects | within-run fixes applied | remaining |
+|------|-------|---|-------|------|-------|-------|---------|--------------------------|-----------|
+| 0 | … | — | … | … | … | … | … | (baseline) | … |
+| 1 | … | +n | … | … | … | … | … | re-mined entitlements; appended seam | … |
+
+---
+
+## C. Iterate-or-propose (the loop)
+
+**Classify every stage-A defect by one test:** *does fixing it require editing a rule or an
+agent file?*
+
+- **No → within-run.** Fixable by re-running a Mode 1–3 step on THIS target, better (re-collect a
+  dimension deeper, re-run a reconciliation pass, do a missed entitlements mine, run the §7-rule-10
+  second probe, append a forked catalog into `_shared/`, recover a blocked dimension after a trivial
+  fix). **The main session auto-iterates these — no approval** (within the read-only/no-cost boundary).
+- **Yes → harness-changing.** A generic methodology fix the NEXT different target would also hit →
+  accumulate into `04-self-correction-proposal.md`, **approval-gated** (present + STOP).
+- A single defect may be **both**: fix it within-run now AND propose the rule so it never recurs.
+
+### The loop (driven by the main session, not a sub-agent)
+```
+score[0] = measure(run); best = run; iter = 0
+loop:
+  defects   = inspect(run)                       # stage A
+  within    = defects.where(within-run AND read-only/no-cost)
+  harness   = defects.where(harness-changing)
+  if within is empty:                  break     # CONVERGED
+  apply within-run fixes → run'                  # re-run the cheapest affected Mode 1–3 steps
+  iter += 1; score[iter] = measure(run')         # stage B — ALWAYS measure
+  if score[iter] < score[iter-1] - TOL:          # REGRESSION (clear drop, not noise)
+      run = best; break                          #   revert to the best prior run, stop
+  best = run = run'
+  if iter >= CAP:                      break      # CAP ≥ 5
+present(harness) as 04-self-correction-proposal.md + the iteration ledger    # approval gate
+```
+
+**Stop conditions** (any one ends the loop):
+1. **Converged** — no within-run defects remain.
+2. **Regression** — `score[iter] < score[iter-1] − TOL` → **revert to the best prior run and stop**
+   (never let the loop make the run worse chasing a defect). `TOL` is a small tolerance (≈2 points) so a
+   within-noise wobble doesn't false-stop — stop only on a *clear* drop; always keep the highest-scoring
+   run as `best`.
+3. **Cap** — `iter` hits the cap. **The cap is ≥ 5**: the loop must be *willing* to run at least five
+   iterations on a still-improving run before giving up; raise it while the score keeps climbing.
+
+---
+
+## Output — `04-self-correction-proposal.md`
+
+```markdown
+---
+target: <slug>
+proposed_at: <YYYY-MM-DD>
+status: PROPOSED            # → APPLIED only after per-item approval (harness-change rows)
+awaiting_approval: true
+run_score: <final /100>     # the score of the run AS SHIPPED (post-iteration)
+vibe: strong | solid | thin | weak
+iterations: <n>
+stopped_on: converged | regression | cap
+---
+# <Target> — Self-Correction (Mode 4)
+## Run measurement          (the final score + per-axis breakdown + the one-line vibe gestalt)
+## Iteration ledger          (the per-iteration table — score trajectory + fixes applied each round)
+## Behavioral defects found  (stage-A list: mode · defect · severity · evidence anchor · within-run|harness)
+## Within-run fixes applied  (what the auto-loop re-ran on this target, and the score delta each bought)
+## Harness-change proposals  (APPROVAL-GATED table: # · sev · additive|structural · finding+anchor · fix · target file)
+## Approval gate             (present + STOP; main session applies accepted items, flips each to APPLIED, logs)
+```
+
+A **thin harness-change table with a high `run_score` and `stopped_on: converged`** is the success
+signal — the run self-corrected to a good state and left little for the harness to learn. A **batch**
+of runs additionally rolls up into a cross-target synthesis that promotes only patterns ≥2 *independent*
+targets hit (guard against a homogeneous batch: 6/6 of one category is one vote, not six).
+
+---
+
+## How Mode 4 uses this reference
+Stage A — inspect against the checklist, emit the defect list. Stage B — score the run (always).
+Stage C — split within-run vs harness-changing; **auto-iterate** the within-run set (read-only/no-cost,
+≥5 iterations capable, stop on regression/convergence/cap), re-measuring each round; **accumulate** the
+harness-changing set into the approval-gated proposal. Present the proposal + ledger and STOP. The main
+session applies accepted harness rows, flips them to APPLIED, logs the change, and (per ingestion /
+the orchestrator) mirrors to the harness home. Promote only what the next, different target would hit;
+prefer additive edits; structural edits are flagged, never auto-applied.