@clear-capabilities/agentic-security-scanner 0.128.1 → 0.132.0

package/src/posture/CLAUDE.md

@@ -9,6 +9,8 @@ Annotators that run **after** every detector has emitted, plus state stores read
 
 **Calibration + held-out evaluation** — `calibration.js`, `calibration-drift.js`, `validator-metrics.js`, `holdout-eval.js`. The seed corpus lives at `calibration-seed.json`; held-out labels are taken via `loadLabeledJsonl`. Brier and ECE both live in `holdout-eval.js`; never reintroduce a "fit-on-the-table" version.
 
+**Published accuracy scorecard (R3)** — `accuracy-scorecard.js`. Pure aggregation + markdown/JSON rendering for `docs/SCORECARD.md`; the impure driver that performs the corpus and self-scan runs is `scripts/scorecard.mjs` (`npm run scorecard`). Every rate is carried as `{n, d}` and rendered through `formatRate()` so a percentage can never appear without its denominator; entries a run could not score are excluded from every denominator *and* disclosed by name. No F1 is emitted — see the module header for why, and don't add one without a labelled real-world population to measure precision over.
+
 **Cross-language taint** — `cross-lang-{openapi,grpc,graphql,orm,queues,meta}.js`. Each parses a contract artifact (`openapi.json`, `*.proto`, `*.graphql`, queue config) and emits a chain finding when the same data crosses a language boundary into another module's finding.
 
 **Risk amplification** — `epss.js`, `kev` (in `version.js`), `blast-radius.js`, `crown-jewels.js`, `exploitability.js`, `bounty-prediction.js`, `risk-in-dollars` (lives in `scripts/`, not here).
@@ -19,15 +21,61 @@ Annotators that run **after** every detector has emitted, plus state stores read
 
 **Fix lifecycle** — `fix-history.js` (apply + backup + recover), `fix-verify.js` (closed-loop re-scan + lint), `fix-plan.js` (oversized-patch fallback), `regression-test-gen.js`, `deterministic-fix.js` (safe context-independent literal-swap patch synthesis — md5/sha1→sha256, TLS verify-off→on — materialized on demand by `mcp/synthesize_fix`; every patch still passes through `apply_fix`'s inline verify before it lands).
 
+**Measured fix loop (R5)** — `fix-metrics.js`. `verifyFix` times each stage
+(`rescan`/`lint`/`tests`/`honesty`) and appends one record per attempt to
+`.agentic-security/fix-metrics.jsonl`; `summarizeFixDurations` turns those into
+the reported distribution, surfaced on `scan.fixMetrics` and as a stderr line
+on human formats. Distinguish it from `time-to-fix.js`, which *estimates*
+engineering hours from family and patch shape before anything runs — this
+module reports only what was observed.
+
+Three bucketing rules are load-bearing and each has a test that fails if
+relaxed: **failed attempts never enter the validated distribution** (a failed
+verification short-circuits, so blending them makes a worse pipeline look
+faster); **"tests skipped" is bucketed apart from "tests passed"**
+(`validatedWithoutTests`, since a project with no detectable suite reaches
+`ok:true` on a weaker and much cheaper check); and **per-stage timings come
+from validated runs only** (a failed run truncates every stage after the
+failure point). Buckets partition the attempts, so the counts always sum.
+Percentiles are nearest-rank — every figure reported is a duration some run
+actually took — and are flagged `reliable:false` below n=10 rather than hidden
+or quoted as settled. Recording goes through `isSafeStateDir`, so it declines
+rather than creating a stray state dir outside a project.
+
 **Agentic verification** — `verifier.js`, `verifier-target.js`, `verifier-ephemeral.js`, `harness-discovery.js`, `adversary-agent.js`, `defender-agent.js`, `auditor-agent.js`, `three-agent-pipeline.js`.
 
 **Methodology additions (`docs/AGENTIC_METHODOLOGY_PRD.md`)** — default-on annotators/artifacts that layer the agentic-hunter methodology on the deterministic engine:
 - `falsification.js` — default falsification pass. For each taint-style finding, tries to DISPROVE it (locate a context-matched control on the path, reusing `dataflow/sanitizer-proof.js`'s shape rules read-only); a blocked finding is demoted + `quarantined`, never removed and never severity-touched (recall-preserving, like `proof-gate`). Wired after `annotateProofGate`. Opt out: `AGENTIC_SECURITY_NO_FALSIFICATION=1`. Optional LLM tier over survivors when an endpoint is configured.
 - `entrypoint-inventory.js` — attack-surface completeness ledger. Enumerates every entry point (HTTP/queue/cron/CLI/env/upload/webhook) with a disposition each; on `scan.entrypointInventory`.
-- `root-cause-sweep.js` — from confirmed findings, finds sibling instances detectors missed with total-count accounting (`found === candidates + mitigated`); on `scan.rootCauseSweep`.
+- `root-cause-sweep.js` — from confirmed findings, finds sibling instances detectors missed with total-count accounting (`found === candidates + mitigated`); on `scan.rootCauseSweep`. Searches the corpus **once per distinct sink pattern**, not once per finding — findings deriving the same pattern share one walk and one set of (read-only) match records. The counts are always exact; the materialised `instances` list is a bounded sample (`INSTANCE_SAMPLE_LIMIT`, 100) and says so via `instancesTruncated`. Both properties are load-bearing on large corpora: the per-finding walk was O(findings × corpus-bytes) and the instance records were O(findings × matches), which together exhausted a 6 GB heap on a 40k-file suite. If you touch this module, keep the own-site exclusion **per pattern group** — resolving it globally makes a group subtract an exclusion it never matched and drives counts negative.
 - `model-routing.js` — capability-based CWE/severity→model policy; stamps `finding.dispatchModel` (strongest for crypto/auth/critical, mid for injection, cheapest for low-sev hardening) for cost-sensitive subagent dispatch.
 - `fix-honesty-gate.js` — deterministic honesty gates on fix output: a residual-risk hand-wave guard, a cited-file:line requirement for any FP/safe verdict, and FULL/MITIGATION/WORKAROUND completeness tiers. Consumed by `fix-verify.js` when the caller supplies fix metadata; the closed-loop test leg (`fix-verify-loop.js`) is wired into `mcp/apply_fix` behind `AGENTIC_SECURITY_FIX_RUN_TESTS=1`.
 
+**Relevance scoping (R6 + R9)** — `relevance.js`. Turns the two existing *inventories* into *inputs*: `entrypoint-inventory.js` supplies the attack surface, `threat-model.js` supplies assets/boundaries/STRIDE, and `annotateRelevance(findings, ctx)` scores each finding by how reachable and how threat-modelled it is. Sets `entrypointReachable: true|false|null`, `relevance` (0..1), `relevanceTier: 'direct'|'indirect'|'unreachable'|'unknown'`, `relevanceFactors[]`, and re-ranks `exploitability` (ordinal priority, ×1.15 direct / ×0.6 unreachable, floored at 0.05, tier label recomputed on the same thresholds `annotateExploitability` uses). Reachability is a forward BFS over a literal-specifier import graph (JS/TS relative + Python dotted + Java FQCN) starting at every entry-point file.
+
+Its contract is **recall-preserving, same precedent as `falsification.js` / `dataflow/proof-gate.js`**: it never removes a finding, never touches `severity`, and never asserts `unreachable` without positive evidence — a negative verdict additionally requires the intra-repo import graph to be hole-free *along the reachable set* (an unresolved relative import or a non-literal `require(x)` in a reachable file hides a possible edge, so every would-be `unreachable` degrades to `unknown`). `null`/`'unknown'` is a first-class state and is **not** the same as `false`. Wired in `engine.js` after the entry-point inventory is built and after every finding has been appended (multi-sink and cross-language chains included) so nothing escapes annotation; this is the one annotator that deliberately runs after `why-fired`.
+
+**Enforced verification separation (R7)** — `verification-separation.js`. The falsification pass could already try to *disprove* a finding; what it could not do was prove the checker was not the producer. This module supplies that structural guarantee:
+
+- `recordProducer(finding, producerId)` stamps provenance **write-once** — a later party cannot re-stamp itself as producer to manufacture separation.
+- `assertSeparation(finding, verifierId)` refuses when verifier === producer, and **fails closed** when no producer was recorded (unestablishable separation is not separation).
+- `recordVerdict(finding, {verifierId, lens, verdict, reason})` runs that check itself, so there is no path to a recorded verdict that skips it. `lens` is the perspective (`'control-flow'`, `'reachability'`, `'data-shape'`, `'llm-review'`); `verdict ∈ 'upheld'|'refuted'|'undecided'`. One verifier gets one vote per lens — a re-vote replaces rather than stuffs.
+- `consensusOf(finding) -> {verdict, upheld, refuted, undecided, lenses[]}` — majority, `'undecided'` on a tie or on no verdicts.
+
+Producer ids are namespaced `detector:<parser>`, verifier ids `verifier:<name>`, so the two spaces cannot collide. **Nothing throws** (posture convention): every entry point returns `{ok:false, refused:true, reason}`.
+
+Contract is **recall-preserving, same precedent as `falsification.js` / `proof-gate.js`**: a `refuted` verdict never removes a finding and never touches `severity`. It is a triage signal, not a deletion.
+
+Wired in `falsification.js`: the detector is stamped as producer, the falsification pass records under `VERIFIER_FALSIFICATION` on the `control-flow` lens, and the optional LLM tier records separately under `VERIFIER_LLM_REVIEW` on the `llm-review` lens — which is what makes a contested finding legible *as contested* (upheld vs refuted → consensus `undecided`) instead of resolved by whoever spoke last. Result lands on `finding.verification = {producer, verdicts[], consensus}`.
+
+**Run attestation (R4)** — `attestation.js`. Turns determinism from an implementation property into something a third party can check. `computeRunAttestation({findings, engineVersion, rulesetVersion, bundleSha, root, sign})` returns `{digest, algorithm, findingCount, engineVersion, rulesetVersion, bundleSha, canonicalisation, proves, doesNotProve, signature?}`; `verifyRunAttestation(attestation, {findings, …})` re-derives and returns `{ok, reason}`.
+
+Canonicalisation is an **allowlist, not a denylist** — each finding reduces to `id ⇥ severity ⇥ file ⇥ line ⇥ cwe ⇥ vuln`, rows sorted, multiplicity preserved. That is what makes the digest independent of emission order, run ids, timestamps, durations, separator style, and the absolute prefix (when `root` is given), while a changed severity/file/line/rule id/cwe or a finding appearing or disappearing all change it. A new volatile field cannot leak in without being added to the allowlist deliberately. `parser` and `family` are deliberately **excluded**: `parser` records which analysis engine fired, which is environment-sensitive (the Python AST path vs. its regex fallback), so including it would report an environment difference as a findings difference.
+
+**What it proves / does not prove** — the attestation carries both statements inline, and both are asserted by a test so they cannot be quietly dropped. It proves two finding sets with the same digest under the same canonicalisation are the same findings from the same engine/ruleset/bundle. It does **not** prove cross-machine reproducibility — no second machine was compared, and this repo makes no such claim. Signing reuses `integrity.js`'s per-install HMAC key handling verbatim (`signLastScan`); no second key mechanism was introduced. `verifyLastScan` is *not* reused because it verifies against a sibling `.sig` file whereas an attestation carries its signature inline, so verification re-signs and compares in constant time. Being symmetric, the signature is tamper-evidence for the operator, not third-party non-repudiation.
+
+Wired in `bin/agentic-security.js` after every filter and after `makeDeterministic`, over `normalizeFindings(scan)` — i.e. it attests the set that actually ships — and surfaced as `attestation` in `toJSON`. `bundleSha` is read from the sidecar next to the *running* bundle and is `'unavailable'` when running from source, rather than reporting a dist hash that may not correspond to this run.
+
 **Integrity + signing** — `integrity.js` (per-install HMAC for `last-scan.json`), `rule-pack-signing.js`. The HMAC key lives at `$XDG_CONFIG_HOME/agentic-security/scan-key`; override via `$AGENTIC_SECURITY_HMAC_KEY`. Premortem-derived; do not regress to hostname-derived.
 
 **Rule lifecycle** — `custom-rules.js` (YAML pattern DSL), `rule-overrides.js` (`disable:` gated on signature), `rule-packs.js`, `rule-synthesis.js` (proposes suppressions from triage feedback), `ruleset-version.js`.
@@ -44,6 +92,150 @@ Annotators that run **after** every detector has emitted, plus state stores read
 - **No throwing.** Every annotation in `engine.js` is wrapped `try { … } catch (_) {}`. Your annotator must degrade gracefully — set `null` on the field and continue.
 - **Dead-module test.** `npm run test:lifecycle` fails the build if you export a public symbol from a posture module that no other source file imports. Wire it in `engine.js` (or allowlist it with a written reason in `test/no-dead-modules.test.js`).
 
+## Execution-proof tiers (R2)
+
+`proof-tier.js` + `execution-proof.js` add a fourth axis to a finding's
+credibility, orthogonal to `confidence`/`exploitability`: whether the bug was
+*run*, not just reasoned about.
+
+**The four tiers** (`PROOF_TIERS`, most-proven first):
+
+- `execution-proven` — a generated PoC ran inside the sandbox and the sandbox
+  observed the predicted effect (a marker file the PoC's payload should have
+  written showed up). The strongest claim the pipeline can make.
+- `proof-failed` — a PoC ran and the marker did **not** appear. This is a
+  **triage signal, not a false-positive verdict**. Absence of proof is not
+  proof of absence: the PoC may be wrong, the param key may be misinferred,
+  or the vulnerable path may need state the single-shot PoC didn't set up.
+  Never auto-close or downgrade severity off `proof-failed` alone.
+- `taint-proven` — the analyser's static reasoning (`IR-TAINT`/`MULTI-SINK`)
+  found it; nothing executed. This is `proofTierOf()`'s default when no
+  execution evidence has been attached.
+- `unproven` — no analyser backing recorded at all (e.g. `REGEX` parser).
+
+**Why a marker file, not an exit code.** The sandbox cannot reliably
+distinguish "the payload was denied by confinement" from "the payload ran
+and happened to exit 0" — both look like a clean exit from the parent
+process's point of view. A marker file the PoC only writes *if its exploit
+path actually executes* turns that ambiguity into a directly observable
+fact: the file exists, or it doesn't. Exit code alone is used only for
+timeout/crash detection, never as the proof signal itself.
+
+**The backend is recorded in every evidence object** (`proofEvidence.backend`,
+e.g. `'userspace'`) because not all confinement backends carry the same
+guarantee. Two backends now carry a verified-by-execution escape contract:
+`userspace` (verified on the macOS development host) and `namespace`, which
+implements write confinement as well as network isolation (read-only rebind of
+the mount tree, read-write rebind of the sandbox root, capability drop before
+exec) and whose escape suite has now **RUN and passed on a Linux host in CI** —
+see `sandbox/CLAUDE.md` for the host and the eight cases. `execution-proven`
+evidence from either of those backends stands on an executed escape contract.
+The `disabled` backend never produces evidence at all: it refuses to run,
+which is why the tier stays static rather than becoming `proof-failed`.
+Keep reading `proofEvidence.backend` anyway — it is what makes a tier
+re-auditable when a backend's contract changes, and a backend added later
+starts out unverified by default. `attachProofTier()` also enforces the demotion
+guard: `ran:false` can never yield `execution-proven` or `proof-failed`,
+regardless of what tier was requested — it falls back to the finding's
+static standing (`proofTierOf`).
+
+**`ran` means the PoC executed, not that `runConfined` returned.** A sandbox
+that could not start (`status:'error'` — confinement binary missing, namespaces
+denied) and a refusal (`status:'disabled'`) both leave the PoC unexecuted, so
+`execution-proof.js` records `ran:false` and a reason naming the sandbox
+failure, leaving the finding at its static tier. Calling that `proof-failed`
+would report a broken sandbox as a failed exploit attempt — a claim about the
+*finding* derived from evidence that only concerns the *host*.
+
+`proofTier`/`proofEvidence` are copied through `report/index.js`'s
+`normalizeFindings()` only when the annotator actually attached them —
+never synthesised at the report layer.
+
+## Corpus auto-enrolment (R2's differentiator)
+
+`corpus-enroll.js` + `corpus-match.js` turn an execution-proven finding into a
+permanent CVE-replay corpus entry, so every exploit the pipeline proves once is
+defended by the baseline gate forever.
+
+**`corpus-match.js` is shared with the gate on purpose.** `bench/cve-replay/runner.mjs`
+imports `matcherFor`/`preHit`/`postHit` from it. If enrolment verified a
+candidate with a different matcher than the gate scores with, it would commit
+entries that fail CI. The pre/post asymmetry (pre matches `vuln` OR `family`
+and regex-tests `cwe`; post is strict on `vuln` and exact on `cwe`) is
+reproduced verbatim from the runner — `bench/cve-replay/CONTRIBUTING.md` records
+it as known imprecision, and changing it would re-verdict the whole committed
+baseline, which is a corpus migration rather than a refactor.
+
+**Nothing is written that has not been scored.** `enrollProvenFinding` builds
+the entry in a temp dir, scans `pre/` and `post/`, and moves it into the corpus
+only on `pre:TP post:TN`. There is no force flag, and `scoreCandidate` is
+deliberately unexported so no caller can score by one route and write by
+another — that unscored-write path is the v0.106.0 mistake this module would
+otherwise automate. Refusals also cover: a tier that disagrees with its own
+`proofEvidence`, `ran !== true`, an `execution-proven` tier with nothing
+observed, a missing `post/` (never synthesised by deleting the vulnerable
+line — it would pass for the wrong reason), a `post/` identical to `pre/`, a
+path escaping the entry dir, a `pre/` not containing the finding's file, and a
+duplicate id. The manifest's `vuln_match` is regex-escaped so an unrelated
+detector cannot satisfy the entry.
+
+**New entries land in `capability/`, never `regression/`.** `regression/` is
+the CI-gated tier and graduation into it is a human decision with a stated
+policy; an automated writer must not decide what blocks everyone's build.
+
+**Operator entry point:** `scripts/enroll-proven-finding.mjs <project>`
+(`--dry-run` scores without writing). It proves findings itself — the scan
+pipeline does **not** attach a `poc` to findings or promote proof tiers, so
+`last-scan.json` never contains an `execution-proven` finding on its own. PoCs
+come from the PoC-generator. Enrolment additionally needs fixed content
+(`finding.fix.patch`) for `post/`; a proven finding with no fix is reported as
+skipped, not dropped. After enrolling, refresh the baseline
+(`npm run bench:cve-replay:update-baseline`) and commit it.
+
+## Scan checkpointing / resume (R8)
+
+`scan-checkpoint.js` lets an interrupted scan resume instead of restarting, which
+is what caps usable repository size today. **Opt-in only**: `AGENTIC_SECURITY_RESUME=1`
+(or `runScan(root, {resume:true})`). Default behaviour is byte-for-byte unchanged
+and nothing is written.
+
+**What is checkpointed.** Only the per-file loop in `engine.js#runFullScan` — the
+one place per-file work happens. Each completed file's *entire* contribution is
+persisted (routes, findings, taint sources/sinks/sanitizers, logic vulns, secrets,
+at-rest/in-transit ciphers, the suppression-log delta, and the per-file taint
+result the cross-file pass reads), not just its findings. Everything after the
+loop — cross-file taint, gadget detection, the whole annotation pipeline — re-runs
+from scratch, so nothing that depends on the global picture can be stale by
+construction. On replay, `pfr[p]`'s arrays are rebuilt as slices of the aggregate
+arrays, so object identity between the two matches an uninterrupted run exactly.
+
+**The property.** A resumed scan must produce the same finding set as an
+uninterrupted one; a checkpoint that silently drops findings turns a slow scan
+into a quietly incomplete one, which is worse than no checkpoint. `test/scan-checkpoint.test.js`
+asserts this end-to-end: a child process is hard-exited (`process.exit`, no
+unwinding) partway through a real scan, the resumed scan is compared against a
+genuinely uninterrupted one, and the fixture is asserted to exercise every
+channel inside the replayed prefix so a dropped channel cannot go unnoticed.
+
+**Invalidation is deliberately blunt.** The run key covers engine version,
+ruleset version, bundle SHA, a content hash of every scanned and dependency file
+(which subsumes mtime), and every `AGENTIC_SECURITY_*` env switch. If any of it
+moved, the checkpoint is discarded and the scan starts clean. Redoing work is
+slow; resuming stale work is a correctness bug.
+
+**Crash safety: append-and-fsync.** JSONL — one header line pinning the run key,
+then one record per file carrying a SHA-256 of its own payload, each written with
+a single `writeSync` and `fsyncSync`'d before the next file is analysed. Recovery
+reads forward while records verify and truncates at the last byte that did, so a
+torn or tampered tail is dropped rather than resumed into. Nothing is rewritten
+in place. Values JSON cannot round-trip (Date/RegExp/Map/function/…) are refused
+rather than recorded lossily — that file just gets rescanned. On clean completion
+the checkpoint is removed, so the next run cannot resume consumed state.
+
+State lives at `<scanRoot>/.agentic-security/scan-checkpoint.jsonl`; like every
+other module here, nothing throws — a failure to open, read or append degrades to
+"no checkpoint", i.e. a normal full scan.
+
 ## Gotchas
 
 - The seed `calibration-seed.json` is small (n < 30 for several families). Don't treat it as a held-out set — that's `holdout-eval.js`'s job, against an externally-supplied JSONL.

package/src/posture/accuracy-scorecard.js

@@ -0,0 +1,317 @@
+// Accuracy scorecard (roadmap R3) — aggregation + rendering.
+//
+// WHY THIS LIVES IN src/posture/ RATHER THAN ENTIRELY IN scripts/:
+// every figure this project publishes has to be re-derivable and testable.
+// The aggregation and the rendering are pure functions of their inputs, so
+// they belong where the unit-test suite already runs (tests execute against
+// `src/` directly). The thin driver at `scripts/scorecard.mjs` does the
+// impure half — running the corpus, running the self-scan, reading the
+// bundle hash and the git commit — and hands the results here. That split is
+// what makes the "hand-computed fixture" test in
+// `test/accuracy-scorecard.test.js` possible at all.
+//
+// INTEGRITY CONTRACT (the reason the item exists — read before editing):
+//   · Nothing here invents a rate. Every rate is carried as {n, d} and
+//     rendered through formatRate(), which always prints the numerator and
+//     the denominator alongside the percentage. A denominator of 0 renders
+//     "0/0 (n/a)" — never 0% and never 100%.
+//   · Entries the run could not score (parser unavailable, scan error) are
+//     removed from every denominator AND disclosed by name. Scoring them
+//     either way would manufacture a number out of an environment fault.
+//   · No F1 is emitted. F1 needs a precision measured over a labelled
+//     real-world population; this corpus is curated known-vulnerable
+//     fixtures and their fixed counterparts, so its "precision" would be
+//     computed over a denominator that does not mean what the word implies.
+//     See renderScorecardMarkdown()'s methodology section.
+//   · Output is a deterministic function of the inputs apart from one
+//     timestamp line (TIMESTAMP_MARKER), so the document regenerates
+//     byte-identically on an unchanged tree.
+
+// The single line of the rendered document that is permitted to vary between
+// two runs on an unchanged tree. Tests and the regeneration check key on it.
+export const TIMESTAMP_MARKER = 'Generated (UTC)';
+
+/**
+ * Render a rate with its raw counts. This is the ONLY sanctioned way to put a
+ * percentage into the scorecard: a bare percentage with no visible
+ * denominator is exactly the kind of unfalsifiable claim this document exists
+ * to avoid.
+ */
+export function formatRate(n, d) {
+  if (!d) return `${n}/${d} (n/a)`;
+  return `${n}/${d} (${((n / d) * 100).toFixed(1)}%)`;
+}
+
+function emptyBucket(key) {
+  return { key, entries: 0, detection: { n: 0, d: 0 }, silence: { n: 0, d: 0 } };
+}
+
+function accumulate(bucket, status) {
+  bucket.entries++;
+  if (status.includes('pre:TP')) { bucket.detection.n++; bucket.detection.d++; }
+  else if (status.includes('pre:FN')) { bucket.detection.d++; }
+  if (status.includes('post:TN')) { bucket.silence.n++; bucket.silence.d++; }
+  else if (status.includes('post:FP')) { bucket.silence.d++; }
+}
+
+function sliceBy(scored, key) {
+  const map = new Map();
+  for (const d of scored) {
+    const k = d[key] || 'unknown';
+    if (!map.has(k)) map.set(k, emptyBucket(k));
+    accumulate(map.get(k), d.status);
+  }
+  // Sorted by key so the document is stable across runs.
+  return [...map.values()].sort((a, b) => a.key.localeCompare(b.key));
+}
+
+/**
+ * Aggregate the per-entry detail records emitted by the corpus runner.
+ *
+ * Input records: { cve, tier, cwe, language, status, error? } where status is
+ * the runner's own string form, e.g. "pre:TP post:TN", "pre:FN", "env-error".
+ */
+export function aggregateCorpus(detail) {
+  const all = [...(detail || [])];
+  const notScored = all
+    .filter(d => !d.status || !d.status.startsWith('pre:'))
+    .map(d => ({ cve: d.cve, tier: d.tier || null, language: d.language || null, status: d.status || 'unknown', error: d.error || null }))
+    .sort((a, b) => String(a.cve).localeCompare(String(b.cve)));
+  const scored = all.filter(d => d.status && d.status.startsWith('pre:'));
+
+  const overall = emptyBucket('overall');
+  for (const d of scored) accumulate(overall, d.status);
+
+  return {
+    totalEntries: all.length,
+    scoredEntries: scored.length,
+    notScored,
+    overall: { detection: overall.detection, silence: overall.silence },
+    byLanguage: sliceBy(scored, 'language'),
+    byCwe: sliceBy(scored, 'cwe'),
+    byTier: sliceBy(scored, 'tier'),
+  };
+}
+
+/**
+ * Build the machine-readable scorecard model. `inputs`:
+ *   provenance   { engineVersion, bundleSha256, commit, nodeVersion, generatedAt }
+ *   corpusDetail per-entry records from a corpus run performed THIS run
+ *   selfScan     { targets: {name:{total,byFile}}, polyglot: {total,byLanguage} }
+ *                — measured THIS run
+ *   committed    { corpusBaseline, proofCorpus } — read from committed files,
+ *                labelled as such in the output, never used to derive a rate
+ */
+export function buildScorecard(inputs) {
+  const corpus = aggregateCorpus(inputs.corpusDetail);
+  const selfScan = inputs.selfScan || {};
+  const targets = {};
+  for (const k of Object.keys(selfScan.targets || {}).sort()) {
+    targets[k] = { total: selfScan.targets[k].total, byFile: selfScan.targets[k].byFile || {} };
+  }
+  const committed = inputs.committed || {};
+  const proof = committed.proofCorpus || null;
+  return {
+    schema: 'agentic-security/accuracy-scorecard@1',
+    provenance: { ...inputs.provenance },
+    methodology: {
+      population: 'curated known-vulnerable fixtures (pre/) and their fixed counterparts (post/)',
+      detectionRateMeans: 'share of known-vulnerable fixtures where the expected finding fired',
+      silenceRateMeans: 'share of fixed counterparts where the same finding correctly did not fire',
+      notGeneralRecall: true,
+      notGeneralFalsePositiveRate: true,
+      f1Emitted: false,
+      f1OmissionReason: 'no labelled real-world population is available to measure precision over, so any F1 would combine denominators that do not describe the same population',
+    },
+    corpus: {
+      measuredThisRun: true,
+      totalEntries: corpus.totalEntries,
+      scoredEntries: corpus.scoredEntries,
+      notScored: corpus.notScored,
+      overall: corpus.overall,
+      byLanguage: corpus.byLanguage,
+      byCwe: corpus.byCwe,
+      byTier: corpus.byTier,
+    },
+    selfScan: { measuredThisRun: true, targets, polyglot: selfScan.polyglot || { total: 0, byLanguage: {} } },
+    committedInputs: {
+      corpusBaseline: committed.corpusBaseline
+        ? { source: 'bench/cve-replay/corpus-baseline.json', generatedAt: committed.corpusBaseline.generatedAt, total: committed.corpusBaseline.total, passing: committed.corpusBaseline.passing }
+        : null,
+      proofCorpus: proof
+        ? {
+          source: 'bench/proof-corpus/results/summary.json',
+          bundleSha: proof.bundleSha || null,
+          targetCount: proof.targetCount ?? null,
+          ok: proof.ok ?? null,
+          failed: proof.failed ?? null,
+          targets: (proof.targets || []).map(t => ({
+            id: t.id, commit: t.commit, status: t.status,
+            filesInScope: t.coverage?.totals?.inScope ?? null,
+            filesParsed: t.coverage?.totals?.parsed ?? null,
+            determinismChecked: t.determinism?.checked ?? null,
+            determinismIdentical: t.determinism?.identical ?? null,
+            resultsEmitted: t.determinism?.results ?? null,
+          })).sort((a, b) => String(a.id).localeCompare(String(b.id))),
+        }
+        : null,
+    },
+  };
+}
+
+function rateRow(r) {
+  return `| ${r.key} | ${r.entries} | ${formatRate(r.detection.n, r.detection.d)} | ${formatRate(r.silence.n, r.silence.d)} |`;
+}
+
+/** Render the human-readable scorecard. Deterministic apart from one line. */
+export function renderScorecardMarkdown(m) {
+  const p = m.provenance || {};
+  const c = m.corpus;
+  const L = [];
+  L.push('# Accuracy scorecard');
+  L.push('');
+  L.push('Generated by `npm run scorecard`. Every figure below comes from a run');
+  L.push('performed by that command, except where a line is explicitly labelled');
+  L.push('*committed artifact* — those carry the timestamp and commit of the run');
+  L.push('that produced them.');
+  L.push('');
+  L.push('## Provenance');
+  L.push('');
+  L.push('| Field | Value |');
+  L.push('| --- | --- |');
+  L.push(`| Engine version | ${p.engineVersion || 'unknown'} |`);
+  L.push(`| Bundle SHA-256 | \`${p.bundleSha256 || 'unknown'}\` |`);
+  L.push(`| Commit | \`${p.commit || 'unknown'}\` |`);
+  if (p.worktreeClean !== undefined) {
+    L.push(`| Worktree at measurement time | ${p.worktreeClean ? 'clean' : 'DIRTY — the commit above does not fully describe what was measured'} |`);
+  }
+  L.push(`| Node | ${p.nodeVersion || 'unknown'} |`);
+  L.push(`| Corpus entries | ${c.totalEntries} (${c.scoredEntries} scored) |`);
+  L.push(`| ${TIMESTAMP_MARKER} | ${p.generatedAt || 'unknown'} |`);
+  L.push('');
+  L.push('## What these numbers are, and what they are not');
+  L.push('');
+  L.push('The population is a curated corpus of known-vulnerable code fixtures');
+  L.push('(`pre/`) each paired with its fixed counterpart (`post/`). Two rates are');
+  L.push('reported, always with their raw numerator and denominator:');
+  L.push('');
+  L.push('- **Detection rate** — of the vulnerable fixtures, the share where the');
+  L.push('  expected finding fired. This is **not** general-purpose recall against');
+  L.push('  real-world code: the corpus is curated, each entry was added because it');
+  L.push('  represents a class worth covering, and nothing here samples the');
+  L.push('  distribution of vulnerabilities in arbitrary repositories.');
+  L.push('- **Correct-silence rate** — of the fixed counterparts, the share where');
+  L.push('  the same finding correctly did not fire. This is **not** a general');
+  L.push('  false-positive rate: the denominator is one narrowly-scoped fixed file');
+  L.push('  per entry, not a population of real code in which any rule could');
+  L.push('  misfire.');
+  L.push('');
+  L.push('The corpus is also **gated**: an entry is admitted only once it scores');
+  L.push('detected-on-`pre` and silent-on-`post`, and a committed baseline fails the');
+  L.push('build on any drift. A detection rate at the ceiling is therefore expected');
+  L.push('by construction — it is evidence that nothing regressed, not evidence that');
+  L.push('there is no headroom. What moves it is adding entries for classes not yet');
+  L.push('covered, which is why the corpus size is reported alongside the rate.');
+  L.push('');
+  L.push('**No F1 is reported, deliberately.** An F1 requires a precision measured');
+  L.push('over a labelled real-world population. This project does not have one, so');
+  L.push('a precision computed from the corpus alone would divide by a denominator');
+  L.push('that does not describe the population the word implies. The precision-side');
+  L.push('signal reported instead is the self-scan section below: exact finding');
+  L.push('counts on this repository\'s own non-fixture source, where every count is');
+  L.push('reviewed by hand and any movement is a real change. Publishing a number we');
+  L.push('cannot defend would cost more credibility than the number is worth.');
+  L.push('');
+  L.push('## Corpus results (measured this run)');
+  L.push('');
+  L.push('| Population | Detected / correctly silent |');
+  L.push('| --- | --- |');
+  L.push(`| Vulnerable fixtures (\`pre/\`) — detection | ${formatRate(c.overall.detection.n, c.overall.detection.d)} |`);
+  L.push(`| Fixed counterparts (\`post/\`) — correct silence | ${formatRate(c.overall.silence.n, c.overall.silence.d)} |`);
+  L.push('');
+  if (c.notScored.length) {
+    const word = c.notScored.length === 1 ? 'entry' : 'entries';
+    L.push('**' + c.notScored.length + ' corpus ' + word + ' could not be scored.** Excluded from every');
+    L.push('denominator above rather than counted as a miss:');
+    L.push('');
+    for (const n of c.notScored) {
+      const why = n.error ? ' — ' + n.error : '';
+      L.push('- `' + n.cve + '` (' + (n.language || 'unknown') + '): ' + n.status + why);
+    }
+    L.push('');
+  } else {
+    L.push('All corpus entries scored; no entry was excluded.');
+    L.push('');
+  }
+  L.push('### By language');
+  L.push('');
+  L.push('| Language | Entries | Detection (`pre/`) | Correct silence (`post/`) |');
+  L.push('| --- | --- | --- | --- |');
+  for (const r of c.byLanguage) L.push(rateRow(r));
+  L.push('');
+  L.push('### By CWE');
+  L.push('');
+  L.push('| CWE | Entries | Detection (`pre/`) | Correct silence (`post/`) |');
+  L.push('| --- | --- | --- | --- |');
+  for (const r of c.byCwe) L.push(rateRow(r));
+  L.push('');
+  L.push('### By corpus tier');
+  L.push('');
+  L.push('| Tier | Entries | Detection (`pre/`) | Correct silence (`post/`) |');
+  L.push('| --- | --- | --- | --- |');
+  for (const r of c.byTier) L.push(rateRow(r));
+  L.push('');
+  L.push('## Precision-side signal: self-scan (measured this run)');
+  L.push('');
+  L.push('The engine scanned this repository\'s own hand-reviewed source. These are');
+  L.push('absolute finding counts, not a rate — there is no labelled ground truth');
+  L.push('over this code, so no precision figure is derived from it. What it');
+  L.push('supports is a movement claim: any change in these counts between');
+  L.push('releases is a real change in what the engine reports on unchanged code.');
+  L.push('');
+  L.push('| Target | Findings |');
+  L.push('| --- | --- |');
+  for (const [k, v] of Object.entries(m.selfScan.targets)) L.push(`| \`${k}\` | ${v.total} |`);
+  L.push(`| \`polyglot\` fixture (expected 0) | ${m.selfScan.polyglot.total} |`);
+  L.push('');
+  L.push('Per-file counts are in `docs/scorecard.json`.');
+  L.push('');
+  L.push('## Committed artifacts referenced (not re-run by this command)');
+  L.push('');
+  const cb = m.committedInputs.corpusBaseline;
+  if (cb) {
+    L.push(`- **Corpus baseline** (*committed artifact*, \`${cb.source}\`, generated ${cb.generatedAt}):`);
+    L.push(`  ${cb.passing}/${cb.total} entries recorded as passing. The gate`);
+    L.push('  `npm run bench:cve-replay:check` fails the build on any drift from it.');
+    L.push('  The rates above are computed from this run, not from this file.');
+  }
+  const pc = m.committedInputs.proofCorpus;
+  if (pc) {
+    L.push(`- **Third-party repository run** (*committed artifact*, \`${pc.source}\`,`);
+    L.push(`  bundle \`${pc.bundleSha}\`): ${pc.ok}/${pc.targetCount} targets completed.`);
+    L.push('  Reported for scale and parse coverage only — these repositories have no');
+    L.push('  vulnerability ground truth, so no accuracy rate is derived from them.');
+    L.push('');
+    L.push('| Repository | Commit | Status | Files parsed / in scope | Results emitted | Deterministic re-run |');
+    L.push('| --- | --- | --- | --- | --- | --- |');
+    for (const t of pc.targets) {
+      const det = t.determinismChecked ? (t.determinismIdentical ? 'identical' : 'checked, not byte-identical') : 'not checked';
+      L.push(`| ${t.id} | \`${String(t.commit || '').slice(0, 12)}\` | ${t.status} | ${formatRate(t.filesParsed ?? 0, t.filesInScope ?? 0)} | ${t.resultsEmitted ?? 'n/a'} | ${det} |`);
+    }
+  }
+  L.push('');
+  L.push('## Reproducing any figure here');
+  L.push('');
+  L.push('| Figure | Command |');
+  L.push('| --- | --- |');
+  L.push('| Detection / correct-silence, and every slice | `node bench/cve-replay/runner.mjs --json` |');
+  L.push('| Corpus drift gate | `npm run bench:cve-replay:check` |');
+  L.push('| Self-scan counts | `node bench/self-scan/measure.mjs --json` |');
+  L.push('| Self-scan drift gate | `npm run bench:self-scan:check` |');
+  L.push('| This whole document | `npm run scorecard` |');
+  L.push('');
+  L.push('Running `npm run scorecard` twice on an unchanged tree produces an');
+  L.push('identical document apart from the generated-timestamp row above.');
+  L.push('');
+  return L.join('\n');
+}

package/src/posture/api-contract.js

@@ -19,7 +19,7 @@
 
 import * as fs from 'node:fs';
 import * as path from 'node:path';
-import * as yaml from 'js-yaml';
+import * as yaml from '../util/yaml.js';
 
 const CONTRACT_FILE_PATTERNS = [
   { glob: /(?:openapi|swagger)\.(?:ya?ml|json)$/i, kind: 'openapi' },