@vibgrate/haile 2026.916.2 → 2026.921.1

package/README.md

@@ -64,3 +64,219 @@ VG_HAILE_RELEASE=1 pnpm --filter @vibgrate/haile build  # refuses without wasm
 pnpm --filter @vibgrate/haile test                  # loader ↔ kernel contract (kernel cases need the wasm)
 (cd crate && cargo test)                            # kernel unit tests, native
 ```
+
+## Decision Engine (Stage 0 — not yet wired into any production code path)
+
+> **Status: Stage 0.** This is package/ABI/provider scaffolding only. No
+> production code path calls `createDecisionProvider()` yet — see
+> [`docs/DECISION-ENGINE.md`](../../docs/DECISION-ENGINE.md) for the
+> "as built" reference (architecture diagram, worked examples with rendered
+> screenshots, real benchmark numbers) and
+> [`docs/DECISION-ENGINE-WASM-SPEC.md`](../../docs/DECISION-ENGINE-WASM-SPEC.md)
+> for the full original design (that spec originally proposed a standalone
+> `packages/vibgrate-decision/` package; the maintainer decided afterward
+> that this capability should instead live inside the existing
+> `@vibgrate/haile` package, reusing its Rust crate / build / package
+> infrastructure rather than standing up a parallel one — everything else in
+> the spec, including the ABI shape, the six decision families, and every
+> non-negotiable safety rule, applies unchanged). The seed-pack scoring
+> weights in `crate/src/decision/pack_data.rs` are honest, hand-written
+> placeholders — **not trained on a real corpus** — pending a later stacked
+> PR's benchmark harness and a real, labelled `decision-pack.json`.
+
+The Vibgrate Decision Engine is a **separate, additive, probabilistic**
+capability living alongside Haile's deterministic architecture classification
+above, in the same compiled WASM artifact. Given evidence Vibgrate already
+knows (impact analysis, test results, verification history, Haile's own
+abstention, …), it answers one of six small bounded questions and returns a
+calibrated probability, never a fact. It never establishes facts, never
+overrides deterministic evidence, and can never be mistaken for Haile's
+`classify()` result — every `DecisionResult` carries its own
+`"schema":"vg.decision.result.v1"` and `"source":"decision"`, fields Haile's
+`HaileClassification` never has.
+
+```ts
+import { createDecisionProvider } from "@vibgrate/haile";
+
+const decisions = createDecisionProvider();
+// null when dist/engine/haile.wasm lacks the decision/ export family —
+// never a stub provider. decide() itself never throws either: a
+// missing/broken engine or a rejected request is always `null`.
+decisions?.decide({
+  schema: "vg.decision.request.v1",
+  decision: "failure_related",
+  context: {},
+  evidence: [
+    { id: "impact:AuthService", kind: "impact", feature: "stack_symbol_in_impact", value: true },
+    { id: "test:AuthServiceTest", kind: "test", feature: "failing_test_covers_changed_symbol", value: true },
+  ],
+});
+```
+
+### The six decision families, one worked example each
+
+**`task_complexity`** — recommend an initial Code Mode (advisory only; explicit `--mode`/`--model`/`--provider`, project config, and an active pin all outrank it, and `resolveMode()` still performs the hardware fit).
+
+```json
+// in:  {"schema":"vg.decision.request.v1","decision":"task_complexity","context":{},
+//       "evidence":[{"id":"e1","kind":"change","feature":"migration_indicator","value":true},
+//                    {"id":"e2","kind":"architecture","feature":"architecture_boundaries_crossed","value":true}]}
+// out: {"schema":"vg.decision.result.v1","decision":"task_complexity","primitive":"choice",
+//       "selected":"forge","probabilities":[{"id":"spark","probability":0.0119},
+//       {"id":"flow","probability":0.1078},{"id":"forge","probability":0.8803}],
+//       "confidence":0.8803,"band":"high","abstain":false,
+//       "reasonCodes":["architecture_boundaries_crossed","migration_indicator"],
+//       "evidenceRefs":["e2","e1"],"calibrationId":"task-complexity-seed-1","source":"decision"}
+```
+
+**`failure_related`** — is a new test/compiler/runtime failure probably caused by the current edit? May route toward the bounded repair loop; may never suppress the failure.
+
+```json
+// in:  {"schema":"vg.decision.request.v1","decision":"failure_related","context":{},
+//       "evidence":[{"id":"impact:AuthService","kind":"impact","feature":"stack_symbol_in_impact","value":true},
+//                    {"id":"test:AuthServiceTest","kind":"test","feature":"failing_test_covers_changed_symbol","value":true}]}
+// out: {"schema":"vg.decision.result.v1","decision":"failure_related","primitive":"binary",
+//       "selected":"related","probabilities":[{"id":"related","probability":0.9837},
+//       {"id":"unrelated","probability":0.0163}],"confidence":0.9837,"band":"high","abstain":false,
+//       "reasonCodes":["stack_symbol_in_impact","failing_test_covers_changed_symbol"],
+//       "evidenceRefs":["impact:AuthService","test:AuthServiceTest"],
+//       "calibrationId":"failure-related-seed-1","source":"decision"}
+```
+
+**`repair_action`** — after a failed verification attempt, which bounded VG Code mechanism runs next (respecting `maxRepairRounds`)? It never writes code itself.
+
+```json
+// in:  {"schema":"vg.decision.request.v1","decision":"repair_action","context":{},
+//       "evidence":[{"id":"e1","kind":"verification","feature":"capsule_stale","value":true}]}
+// out: {"schema":"vg.decision.result.v1","decision":"repair_action","primitive":"choice",
+//       "selected":"refresh_capsule","probabilities":[{"id":"retry_local_repair","probability":0.2123},
+//       {"id":"refresh_capsule","probability":0.577},{"id":"expand_context","probability":0.0707},
+//       {"id":"run_additional_verification","probability":0.0639},
+//       {"id":"escalate_reasoning","probability":0.0474},{"id":"ask_human","probability":0.0287}],
+//       "confidence":0.577,"band":"low","abstain":false,"reasonCodes":["capsule_stale"],
+//       "evidenceRefs":["e1"],"calibrationId":"repair-action-seed-1","source":"decision"}
+```
+
+**`completion_confidence`** — does the implementation appear to satisfy the task? Asymmetric by design: may say "not done yet"; may never bypass a failed test or a protected finding.
+
+```json
+// in:  {"schema":"vg.decision.request.v1","decision":"completion_confidence","context":{},
+//       "evidence":[{"id":"e1","kind":"test","feature":"tests_passed_ratio","value":1},
+//                    {"id":"e2","kind":"diagnostic","feature":"protected_findings_present","value":true}]}
+// out: {"schema":"vg.decision.result.v1","decision":"completion_confidence","primitive":"binary",
+//       "selected":"incomplete","probabilities":[{"id":"complete","probability":0.0037},
+//       {"id":"incomplete","probability":0.9963}],"confidence":0.9963,"band":"high","abstain":false,
+//       "reasonCodes":["tests_passed_ratio","protected_findings_present"],
+//       "evidenceRefs":["e1","e2"],"calibrationId":"completion-confidence-seed-1","source":"decision"}
+```
+
+**`finding_attention`** — queue-prioritisation only. Never modifies severity, CVSS, EPSS, KEV status, reachability, RiskScore, DriftScore, `protected_finding`, or the review decision.
+
+```json
+// in:  {"schema":"vg.decision.request.v1","decision":"finding_attention","context":{},
+//       "evidence":[{"id":"e1","kind":"diagnostic","feature":"severity_score","value":1},
+//                    {"id":"e2","kind":"diagnostic","feature":"exploit_maturity_score","value":1}]}
+// out: {"schema":"vg.decision.result.v1","decision":"finding_attention","primitive":"choice",
+//       "selected":"immediate","probabilities":[{"id":"immediate","probability":0.7516},
+//       {"id":"review","probability":0.1518},{"id":"normal","probability":0.092},
+//       {"id":"defer","probability":0.0046}],"confidence":0.7516,"band":"medium","abstain":false,
+//       "reasonCodes":["severity_score","exploit_maturity_score"],"evidenceRefs":["e1","e2"],
+//       "calibrationId":"finding-attention-seed-1","source":"decision"}
+```
+
+**`architecture_fallback`** — only when Haile's own classifier has explicitly abstained. Uses its own small role vocabulary (never Haile's `taxonomy::ROLES`), reads Haile's abstention only as one input feature, and — like every other decision above — always carries `"source":"decision"` so it can never be mistaken for Haile's deterministic classification.
+
+```json
+// in:  {"schema":"vg.decision.request.v1","decision":"architecture_fallback","context":{},
+//       "evidence":[{"id":"e1","kind":"architecture","feature":"role_hint_authenticate","value":1},
+//                    {"id":"e2","kind":"architecture","feature":"haile_abstained","value":true}]}
+// out: {"schema":"vg.decision.result.v1","decision":"architecture_fallback","primitive":"choice",
+//       "selected":"authenticate","probabilities":[{"id":"authenticate","probability":0.8007},
+//       {"id":"persist","probability":0.0399},{"id":"respond","probability":0.0399},
+//       {"id":"validate","probability":0.0399},{"id":"orchestrate","probability":0.0399},
+//       {"id":"render","probability":0.0399}],"confidence":0.8007,"band":"high","abstain":false,
+//       "reasonCodes":["haile_abstained_fallback_engaged","role_hint_authenticate"],
+//       "evidenceRefs":["e2","e1"],"calibrationId":"architecture-fallback-seed-1","source":"decision"}
+```
+
+### Determinism, calibration, and abstention
+
+Same input JSON always yields byte-identical output for a given engine/pack
+version (no randomness, clock, or unordered iteration). Confidence bands
+(`high`/`medium`/`low`/`abstain`) are per-decision, versioned thresholds
+(`crate/src/decision/calibration.rs`), not one global cutoff. An honest
+`abstain` (e.g. no evidence supplied) is a normal, fully-shaped result, not
+an error — the host falls back to existing behaviour exactly as it does when
+the engine itself is absent.
+
+### Runnable examples (`examples/decision/` — one worked example each)
+
+```bash
+pnpm --filter @vibgrate/haile run examples:decision   # all six, in sequence, real WASM output
+pnpm --filter @vibgrate/haile exec tsx examples/decision/failure-related.example.ts   # one at a time
+```
+
+Six small standalone scripts, one per decision family, each building a
+realistic `DecisionRequest` with an inline comment on every evidence field,
+calling the real `createDecisionProvider()`, and pretty-printing the
+`DecisionResult`. `docs/generate-screenshots.mjs`
+(`pnpm --filter @vibgrate/haile run docs:screenshots`) runs these same six
+scripts to regenerate [`docs/DECISION-ENGINE.md`](../../docs/DECISION-ENGINE.md)'s
+worked-example JSON and screenshots — see that doc for the full architecture
+reference plus rendered examples.
+
+### Tests
+
+```bash
+pnpm --filter @vibgrate/haile test   # includes tests/decision-*.test.ts and tests/bench-harness.test.ts
+(cd crate && cargo test)             # includes crate/src/decision/**/*'s own unit tests
+```
+
+### Benchmarking (`bench/` — spec §17/§18/§32)
+
+```bash
+pnpm --filter @vibgrate/haile run bench:decision                      # all six decisions
+pnpm --filter @vibgrate/haile run bench:decision:failure-related      # one decision
+pnpm --filter @vibgrate/haile run bench:decision:generate-corpus      # regenerate bench/corpus/*.jsonl
+```
+
+`bench/evaluate.ts` loads each labelled corpus under `bench/corpus/*.jsonl`,
+splits it by `repo_id` (spec §32 — never by random row), runs the real
+`decision-wasm` backend (`bench/backends/decision-wasm.ts`, wrapping
+`createDecisionProvider()` above) over the held-out test split, and prints
+accuracy, precision, recall, F1, Brier score, negative log likelihood,
+expected calibration error, coverage, accuracy-by-confidence-band, and
+abstention rate — plus a naive-heuristic baseline for comparison
+(`bench/backends/full-model.ts`; Jev is stubbed unavailable per spec §24,
+`bench/backends/jev.ts`).
+
+> **Every corpus row is SYNTHETIC** (`bench/corpus/README.md`) and the
+> seed-pack weights (`crate/src/decision/pack_data.rs`) are untrained
+> placeholders — so the numbers below are genuinely mediocre. That is the
+> correct, honest Stage 0 result, not a bug in the harness: this corpus is
+> for validating the benchmark machinery itself, not for judging production
+> readiness. It is **not** the real 500–1,000-example corpus spec §31/§32
+> requires before any production hook.
+
+Real output from an actual run (`pnpm --filter @vibgrate/haile run bench:decision`), unedited:
+
+```
+=== Summary — vibgrate-decision-wasm across all six decisions ===
+decision               |  n(test)  |  accuracy  |  F1(macro)  |  Brier   |  NLL     |  ECE     |  coverage  |  abstain%
+task_complexity        |  15       |  46.7%     |  24.6%      |  0.5782  |  0.8517  |  0.3216  |  73.3%     |  26.7%
+failure_related        |  20       |  35.0%     |  25.9%      |  1.2023  |  2.6754  |  0.6201  |  100.0%    |  0.0%
+repair_action          |  18       |  100.0%    |  100.0%     |  0.2879  |  0.6553  |  0.4757  |  66.7%     |  33.3%
+completion_confidence  |  20       |  55.0%     |  52.0%      |  0.7553  |  1.3406  |  0.3805  |  85.0%     |  15.0%
+finding_attention      |  16       |  37.5%     |  34.4%      |  0.5932  |  1.0679  |  0.1527  |  31.3%     |  68.8%
+architecture_fallback  |  21       |  100.0%    |  100.0%     |  0.2486  |  0.5423  |  0.3416  |  76.2%     |  23.8%
+```
+
+Per-decision output also prints a naive-heuristic-baseline row for
+comparison and accuracy broken out by confidence band; run the command
+above for the full table. `repair_action` and `architecture_fallback`
+scoring 100% here is a property of their small synthetic test splits (18
+and 21 rows respectively, after the repo-level split), not a general claim —
+`task_complexity` and `failure_related`, evaluated against the same kind of
+corpus, land far lower. Treat every number as "the harness works and these
+are the real, unflattering seed-pack results," not as a production
+readiness signal.