@clear-capabilities/agentic-security-scanner 0.130.0 → 0.133.0

package/CHANGELOG.md

@@ -1,5 +1,252 @@
 # Changelog
 
+## 0.133.0 — two ways findings could be silently deleted, both closed
+
+Four rounds of adversarial premortem against this repository's own artifacts.
+Two rounds found working exploits in shipped code; both are fixed and both are
+pinned by tests that run the original attack. The rest is measurement honesty —
+several gates turned out to prove less than they claimed, including one defect
+this effort introduced and then caught.
+
+### Security
+
+- **A signature anyone could forge could disable any detector.**
+  `verifyLastScan` accepted a second key derived as
+  `sha256(<constant salt> + ':' + hostname)`. The salt is a constant in
+  published, npm-shipped source and a hostname is not a secret. Because the
+  `disable:` list in `rules.yml` is gated on that verification, a signature
+  forged from public information alone switched off arbitrary detectors and the
+  scan reported clean. Demonstrated end to end: a command-injection finding went
+  from 1 reported to 0, and back to 1 after the fix.
+
+  This had been known and fixed once already. The 0.62.0 entry below introduced
+  the per-install key precisely because the old one was "hostname-derived and
+  publicly forgeable in CI / containers", and kept legacy verification "for one
+  release to migrate existing signed scans". It was still accepted **seventy
+  minor releases later**. A migration window nobody closes is not a migration
+  window; it is the vulnerability, kept. Verification now accepts exactly one
+  key. Signatures made under the legacy key stop verifying — intended, and it
+  fails closed.
+
+- **The LLM validator cache was a finding-deletion primitive.** A cache hit
+  assigned a verdict directly, and a `reject` verdict drops a finding. The cache
+  was read with a bare `JSON.parse(readFileSync(...))`, so planting one file
+  under `.agentic-security/llm-cache/` deleted a critical finding with no model
+  call and no network. The key is derivable by anyone with repo access, and CI
+  restoring a cache directory between runs delivers it without a repo write at
+  all. Cache entries are now HMAC-signed with the same mechanism `last-scan.json`
+  already used; an unsigned, tampered or foreign-keyed entry is a MISS, never a
+  verdict.
+
+- **A `reject` can no longer delete a strongly-provenanced finding.** The code
+  asserted that prompt-injecting the validator was harmless because "the worst
+  an attacker can produce is escalate". That was false: the challenge/nonce
+  cross-check defends against forged and replayed responses, not against a model
+  persuaded by source it legitimately read. Findings from real analysis
+  (taint-proven, multi-sink, execution-proven) are now demoted to `escalate`
+  rather than dropped, so the guarantee is structural instead of asserted.
+
+- **Coverage reduction is now visible in the artifact.** A `disable:` that took
+  effect produced findings that were simply absent — indistinguishable from
+  clean code to whoever reads the report. `suppressedRules` now carries the
+  count, per-rule severity breakdown, example locations and the AUTHORITY the
+  suppression ran under, so a signed suppression reads differently from an
+  env-var opt-out. Authorised suppressions are reported too: a signature proves
+  who asked, not that the hidden findings stopped existing.
+
+- **Signatures carry key provenance** (`env` / `per-install` / `ephemeral`).
+  `env` means whoever set the environment could have signed the run; `ephemeral`
+  means the key could not be persisted and the signature will never verify
+  again. Neither was inferable from the digest.
+
+- **The suppression quorum has a floor of 2.** `AGENTIC_SECURITY_LEARN_QUORUM=1`
+  was honoured, so a single triage verdict could suppress a finding — and with
+  family+path matching, a whole family across a path. The root guidance warned
+  about exactly this; the code did not enforce it.
+
+### Measurement honesty
+
+- **The corpus is fitted to the detectors it measures, and now says so.** 98% of
+  entries are self-authored synthetic fixtures and none come from the
+  disclosed-PoC tier. `npm run corpus:provenance` prints the composition on every
+  run and fails a commit that lands a detector together with the corpus entries
+  exercising it — caught against real history, including one such commit in this
+  very effort. This stops the loop tightening; it does not make the corpus
+  independent, and the docs no longer imply otherwise.
+
+- **The precision gate covered 6% of the source.** It ran over `hooks/` and
+  `scripts/` — 22 files — while `scanner/src` (383 files, the entire product) sat
+  outside it. Now 240 files. The `scanner/src` count is published as a DRIFT
+  TRIPWIRE, explicitly not hand-reviewed and explicitly not a precision figure,
+  because a scanner's own source contains sink patterns as data.
+
+- **The determinism gate exercised only the layer that cannot vary.** The
+  original fixture produced findings from regex and structural detectors alone. A
+  second fixture now drives the interprocedural taint engine and the Python
+  parser, digests are compared per fixture so a divergence names the layer, and
+  the comparator fails if the deep fixture degraded to the syntactic layer on any
+  machine.
+
+- **The scorecard gate could not detect a stale scorecard.** It compared only the
+  engine version, so a document measured over 200 corpus entries passed while the
+  corpus held 210 — every published rate computed over a population that no
+  longer existed. It now compares the population too.
+
+- **The independent-evaluation gate passed on a 4-sample smoke fixture** its own
+  README says must never be cited, because its thresholds had been calibrated to
+  pass. It now uses the README's own figures, treats exceeded calibration targets
+  as violations rather than notes, and refuses to emit a pass over the built-in
+  fixture however the thresholds are set. It fails today, correctly.
+
+- **Claims re-scoped to what is measured.** The roadmap carries a "What the gates
+  do not prove" section, states outright that the false-positive-rate goal is not
+  met, and points at the two harnesses built for that gap — both of which need
+  data, not code. R13 is downgraded from "landed" to "mechanism landed, no
+  observation pipeline": nothing constructs its ledger, so it can never downgrade
+  anything.
+
+### Fixed along the way
+
+- **The validator cache had never persisted a single entry.** `safeWriteState`
+  refused every directory nested under `.agentic-security/`, so `llm-cache/`,
+  `fix-history/` and `sbom-history/` were all unwritable while a
+  `validator-cache stats|gc` subcommand managed a cache that was always empty.
+  Found by a positive-control test asserting a legitimately written entry
+  round-trips.
+
+- **Sandbox timeouts now use SIGKILL.** The kernel does not deliver
+  default-action signals to a PID namespace's pid 1 from outside it, so SIGTERM
+  was dropped and a payload ran to completion against a 1200 ms budget — measured
+  in CI at 30057 ms. Proof execution also gained its own aggregate wall-clock
+  budget, because a count cap bounds nothing in time.
+
+- **Cost reporting stopped presenting an estimate as spend.** The endpoint's
+  usage report was discarded, so the ledger always booked the pre-call worst
+  case. Usage is now plumbed through, and any estimated component renders as
+  "at most $X" with the reason.
+
+## 0.132.0 — a proven exploit becomes a permanent regression test
+
+The corpus stops being only a regression net and starts being fed by the engine
+itself. Every claim below was verified by a command in the session that made it.
+
+- **Execution-proven findings auto-enrol as corpus entries.** This was R2's
+  differentiator and the last missing piece of it. A finding whose
+  proof-of-concept RAN in the sandbox and produced the predicted effect can now
+  be turned into a permanent `pre:TP post:TN` corpus entry, so every exploit
+  proved once is defended against its own regression forever. Demonstrated end
+  to end rather than in a unit test: a real command-injection finding was proved
+  in the sandbox, its fix supplied the `post/` tree, and the entry was enrolled
+  and scored by the real gate — **the baseline moved 199 → 200**. Proven in the
+  other direction too: with the entry's `pre/` neutered the gate reports
+  `REGRESSED (1)` and exits non-zero.
+- **Nothing reaches the corpus unscored.** The entry is built in a temporary
+  directory, `pre/` and `post/` are scanned, and it is moved into the corpus
+  only on `pre:TP post:TN`. There is no force flag, and the scoring function is
+  unexported so no caller can score by one route and write by another — the
+  v0.106.0 mistake (fixtures committed without verifying they score) is the one
+  thing an automated writer must never industrialise. Scoring itself now lives
+  in one module shared by the enroller and the corpus runner, so the two cannot
+  drift apart; the refactor was proven behaviour-preserving at 199/199, no drift.
+  New entries land in `capability/`, never the CI-gated `regression/` tier — a
+  machine must not decide what blocks everyone's build.
+- **`verify_fix` now runs the proof-of-concept against the candidate patch.** A
+  re-scan only proves the DETECTOR stopped firing, which a cosmetic edit
+  achieves; re-running the exploit proves the hole is shut. Still-exploitable
+  after the patch is a hard failure. Deliberately asymmetric: a PoC that could
+  not run is recorded `inconclusive` and excluded from the verdict, because
+  reading "could not prove it" as "fixed" is exactly the false confidence this
+  leg exists to prevent. Both directions were executed against the real sandbox.
+- **Time-to-validated-fix is now measured and reported**, closing R5. Every
+  verification stage is timed and appended to a per-project log, and the
+  reported distribution is deliberately hard to flatter: failed attempts never
+  enter the validated median (they short-circuit, so blending them makes a worse
+  pipeline look faster), "no test suite to run" is bucketed apart from "tests
+  passed", and per-stage timings come from validated runs only. Percentiles are
+  nearest-rank — every figure shown is a duration some run actually took — and
+  are flagged unreliable below n=10 rather than quoted as settled.
+- **Cross-machine determinism now has a gate behind it.** A dependency-free
+  fixture is scanned on two operating systems and the run-attestation digests
+  must match. The comparator refuses every route to a meaningless pass: fewer
+  than two attestations, two runs from the same platform, a zero-finding digest,
+  mismatched canonicalisations, unparseable input. Each refusal was fired
+  deliberately and confirmed to exit non-zero.
+
+Honest limits, stated rather than implied:
+
+- The enrolment loop is **not yet automatic end to end**. Nothing in the scan
+  pipeline attaches a proof-of-concept to a finding or promotes proof tiers, so
+  a scan never produces an `execution-proven` finding on its own; PoCs come from
+  the generator and are proved at enrol time. Automatic attachment during a scan
+  is the remaining work.
+- The cross-machine determinism jobs have **not yet run in CI**, so no second
+  machine has actually been compared. Same-machine repeatability is verified.
+  The attestation's own "does not prove" statement is unchanged and stays
+  correct either way: one attestation is one run on one machine.
+- The sandbox wall-clock timeout stops the **direct child, not the process
+  tree**, on both backends. No test asserts a backgrounded grandchild dies.
+
+Also in this release:
+
+- **Documentation corrected against evidence.** The roadmap and the sandbox and
+  posture guides still described the kernel-namespace backend as "implemented,
+  unverified" after CI had already proved otherwise; they now cite the run
+  (Ubuntu 24.04, kernel 6.17.0-1020-azure, 41 assertions, 0 failures, all eight
+  escape cases). The corpus entry count was three tiers and 14 entries out of
+  date.
+- **Dependency advisories cleared** in the extension tree and `@types/node`
+  brought current in both trees.
+
+## 0.131.0 — decorated files stop vanishing; the Linux sandbox is verified
+
+Two correctness fixes, both found by pushing measurement further than the last
+release did.
+
+- **Decorator syntax is now accepted by the IR frontend.** The parser was told
+  about TypeScript and JSX but not decorators, so it rejected the ENTIRE file on
+  the first `@tracked` it met — no error, no warning, just findings that never
+  existed. Measured on a live third-party target: **4,023 of 4,271 JavaScript
+  files parsed before, 4,262 after** — 239 files, roughly 6% of that project,
+  were invisible to the scanner and are now analysed. `decorators-legacy` plus
+  `decoratorAutoAccessors` was chosen over the modern `decorators` variant
+  because the modern one cannot parse TypeScript parameter decorators; it would
+  have swapped one blind spot for another. No new dependency. Guarded by a
+  regression test proven in both directions — 5/5 pass with the fix, 4 fail
+  without it.
+- **The kernel-namespace sandbox now confines writes, and is verified.** It
+  previously confined network only and had never been executed. It now enters a
+  private mount namespace, rebinds every mount read-only except the sandbox
+  root, and drops the entire capability set before exec so a payload cannot
+  rebind the tree writable. A CI job relaxes the host restriction on
+  unprivileged user namespaces and runs the real escape suite: a write outside
+  the root is blocked with no file created, and outbound egress is blocked. The
+  verifier exits non-zero unless that suite actually RAN, so a skip can never be
+  read as a pass. Execution-proof on Linux no longer rests on an unexercised
+  backend.
+
+Also in this release:
+
+- **A quadratic blowup in the root-cause sweep**, which ran on every full scan:
+  it re-split the whole corpus into lines once per finding and kept a record per
+  finding x matching line. The largest benchmark corpus went from dying after
+  55+ minutes to **exit 0, 933 MB peak, 330s** — smaller and roughly ten times
+  faster, with metrics identical before and after.
+- **Benchmark corpora are fetched by pinned commit** instead of guessing a clone
+  depth; six corpora had never been scoreable because their pin sat more than
+  100 commits back.
+- **A pre-push gate** runs bundle integrity, the full suite, the corpus baseline
+  and the precision baseline before anything leaves the machine, plus branch
+  protection requiring green checks to merge.
+- **A dependency-currency release gate**: any advisory at moderate or above
+  fails with no opt-out; anything behind latest fails unless explicitly held
+  with a stated reason and a review date that expires.
+- **The glob dependency was replaced by the platform built-in** — 92 to 73
+  production packages — after a differential over 320 trees and 1,419,229 paths
+  showed zero differences.
+
+`npm test` 2072/0; cve-replay 199/199; self-scan no drift; proof corpus
+ghost/superset/godot all 100% parse coverage.
+
 ## 0.130.0 — the roadmap's first ten: provable security over orchestration parity
 
 A capability roadmap (`docs/ROADMAP.md`) plus its first ten items, derived from a

package/bin/agentic-security.js

@@ -17,7 +17,7 @@ import { recordScan, formatStreakLine, formatGradeDelta } from '../src/posture/s
 import { ingestAndMerge } from '../src/sca/sarif-ingest.js';
 import { loadProfile, saveProfile, detectProfile, renderAttributionLine, ATTRIBUTION, ATTRIBUTION_URL } from '../src/posture/profile.js';
 import { applySuppressions, addSoftAcceptance, expiredSoftAcceptances } from '../src/posture/suppressions.js';
-import { applyOverrides, validateOverrides } from '../src/posture/rule-overrides.js';
+import { applyOverrides, validateOverrides, suppressionReport, renderSuppressionSummary } from '../src/posture/rule-overrides.js';
 import { listPacks, loadPack, applyPacks } from '../src/posture/rule-packs.js';
 import { writeLockfile, verifyLockfile, makeDeterministic, isDeterministic } from '../src/posture/deterministic.js';
 import { enrichWithEPSS } from '../src/posture/epss.js';
@@ -28,7 +28,7 @@ import { syncTickets } from '../src/integrations/tickets.js';
 import { decide as decideNextAction, explain as explainDecision } from '../src/posture/router.js';
 import * as triage from '../src/posture/triage.js';
 import { buildSlackDigest, buildDiscordDigest, postWebhook, buildJiraIssue, buildPrComment, buildSiemEvent, loadIntegrationConfig } from '../src/integrations/index.js';
-import fg from 'fast-glob';
+import { globFiles } from '../src/util/glob.js';
 
 // last-scan.json integrity helpers — implementation in posture/integrity.js
 // so the MCP server tools can share verification.
@@ -483,7 +483,7 @@ async function cmdScan(args) {
   // fingerprint and tracking provenance via sources[].
   if (args.flags['ingest-sarif']) {
     const glob = args.flags['ingest-sarif'];
-    const paths = await fg(glob, { dot: false, onlyFiles: true });
+    const paths = await globFiles(glob);
     if (paths.length) {
       const r = ingestAndMerge(scan, paths);
       if (process.stderr.isTTY) process.stderr.write(`[ingest] merged ${r.merged} / added ${r.added} findings from ${paths.length} SARIF file(s)\n`);
@@ -512,6 +512,19 @@ async function cmdScan(args) {
   scan.findings    = applyOverrides(scan.findings    || [], targetAbs);
   scan.secrets     = applyOverrides(scan.secrets     || [], targetAbs);
   scan.logicVulns  = applyOverrides(scan.logicVulns  || [], targetAbs);
+  // Coverage reduction belongs in the ARTIFACT, not only in a log line. A
+  // `disable:` that takes effect otherwise produces findings that are simply
+  // absent, which is indistinguishable from clean code to whoever reads the
+  // report. Recorded whether the suppression was authorised or not — an
+  // authorised one still hides results.
+  try {
+    const _sup = suppressionReport(targetAbs);
+    if (_sup) {
+      scan.suppressedRules = _sup;
+      const _line = renderSuppressionSummary(_sup);
+      if (_line) process.stderr.write(`⚠️  agentic-security: ${_line}\n`);
+    }
+  } catch { /* reporting must never fail a scan */ }
 
   // Curated rule packs: --pack <name> (repeatable). Narrows findings to the
   // CWEs covered by the requested pack(s).
@@ -575,6 +588,7 @@ async function cmdScan(args) {
   // only — a failure here must never fail a scan.
   try {
     const { computeRunAttestation } = await import('../src/posture/attestation.js');
+    const { keyProvenance } = await import('../src/posture/integrity.js');
     const { effectiveVersion } = await import('../src/posture/ruleset-version.js');
     scan.attestation = computeRunAttestation({
       findings: normalizeFindings(scan),
@@ -584,6 +598,11 @@ async function cmdScan(args) {
       root: targetAbs,
       sign: true,
     });
+    // P1-3 — a signature is only as meaningful as the key behind it.
+    // `env` means whoever set the environment could have signed this;
+    // `ephemeral` means the key could not be persisted, so this signature will
+    // never verify on any later run. Neither is inferable from the digest.
+    if (scan.attestation) scan.attestation.keyProvenance = keyProvenance();
   } catch { /* attestation is metadata; never fail a scan over it */ }
 
   // R2: Always emit machine-readable artifacts to .agentic-security/.
@@ -655,6 +674,23 @@ async function cmdScan(args) {
           if (sla) process.stderr.write(`⏰ agentic-security: ${sla}\n`);
         }
       } catch { /* MTTR is best-effort — never block a scan write */ }
+
+      // R5 — report the observed time-to-validated-fix distribution from the
+      // fix attempts recorded by `verifyFix`. This is measurement, not
+      // estimation: it says nothing until fixes have actually been verified in
+      // this project, and it prints nothing when there is nothing measured.
+      // Skipped under --deterministic for the same reason MTTR is: the
+      // durations are wall-clock and would break byte-identical state.
+      try {
+        const { fixDurationReport, renderFixDurationSummary } = await import('../src/posture/fix-metrics.js');
+        const fixMetrics = fixDurationReport(path.resolve(target));
+        if (fixMetrics.attempts > 0) {
+          persistedScan.fixMetrics = fixMetrics;
+          const isJsonFmt = format === 'json' || format === 'sarif' || format === 'cyclonedx' || format === 'sbom' || format === 'spdx' || format === 'vex' || format === 'openvex' || format === 'pbom' || format === 'aibom';
+          const line = renderFixDurationSummary(fixMetrics);
+          if (!isJsonFmt && line) process.stderr.write(`🔧 agentic-security: ${line}\n`);
+        }
+      } catch { /* fix metrics are best-effort — never block a scan write */ }
     }
     // #22 — live-secret validation (opt-in, offline-degrading). Label each
     // detected secret live | dead | unknown via a read-only provider "whoami".