@clear-capabilities/agentic-security-scanner 0.134.0 → 0.136.9

package/CHANGELOG.md

@@ -1,5 +1,437 @@
 # Changelog
 
+> **A note on `docs/*_PRD.md` references below.** Product requirement documents in
+> this project are removed once their workstreams land — the durable parts move
+> into the code and tests that enforce them, where they cannot drift out of date.
+> Entries below cite PRD files that were live at the time of that release and no
+> longer exist. They are left as written: a changelog records what was true when
+> it was written, and rewriting past entries to hide a since-deleted file would
+> make the history less accurate, not more.
+
+
+## 0.136.9 — the real bug: the single-file bundle was never actually self-contained
+
+0.136.8's diagnostic logging answered the question immediately:
+
+```
+Error [ERR_MODULE_NOT_FOUND]: Cannot find module
+'.../scanner/dist/449.index.js' imported from
+'.../scanner/dist/agentic-security.mjs'
+```
+
+`dist/agentic-security.mjs` is documented and committed as a self-contained
+single-file bundle — `.gitignore`'s own comment says the reusable
+`scan.yml` workflow fetches *only that one file* from
+raw.githubusercontent.com so downstream users need no install step. That
+was never quite true. `bin/agentic-security.js` lazily loads each
+subcommand's implementation with `await import('../src/...')` — about 58
+call sites, deliberately, so running `agentic-security scan` doesn't pay
+the cost of loading every posture/discovery/compliance module `agentic-security
+compliance` or `agentic-security hunt` would need. `ncc` code-splits every
+one of those into its own `dist/NNN.index.js` chunk rather than inlining
+it, and the chunk is loaded at runtime via a path relative to the bundle's
+own location on disk — but `.gitignore` only ever allowlisted the main
+`.mjs`, its sha256 sidecar, and the compliance-frameworks data. All 38
+chunk files were silently gitignored. `.claude/settings.local.json`-style
+invisible: they exist on any machine that has ever run `npm run build`
+locally (which is every contributor's, permanently, from the first
+`npm install`), so nobody — human or gate — had a checkout that lacked
+them until this release's hosted CI runs did.
+
+This is the third occurrence of the identical bug class in this file's own
+history — `.gitignore` already carries a comment about the same thing
+happening to `compliance-frameworks/` data before ("the shipped CLI
+silently listed ZERO frameworks and exited 0"). Same shape both times: a
+single tracked entry point quietly depends on sibling files nobody
+allowlisted.
+
+Fixed by allowlisting `scanner/dist/*.index.js` and committing all 38
+current chunks (744 KB), and by adding a permanent regression guard
+(`test/dist-chunks-tracked.test.js`, wired into `test:lifecycle`) that
+diffs `dist/*.index.js` on disk against `git ls-files dist` and fails
+loud, by name, the moment a future build produces a chunk nobody
+allowlisted — rather than waiting for a lazy code path to hit it in
+production. Verified RED against the pre-fix git state (all 38 chunks
+correctly reported untracked) and GREEN after staging them.
+
+## 0.136.8 — instrumenting a real, still-unexplained hosted-CI-only failure
+
+0.136.7's hosted release run got past every other check and failed on
+exactly one: `test/mttr.test.js`'s CLI-wiring test, which asserts a real
+`agentic-security scan` reports an `mttr` field in `last-scan.json`. That
+field comes from a deliberately best-effort code path in `cmdScan`
+(`bin/agentic-security.js`) wrapped in a bare `catch { /* MTTR is
+best-effort — never block a scan write */ }` — correct as a product
+decision (a scan must never fail because a secondary metric couldn't be
+computed), but it means whatever throws inside that block has never been
+visible to anyone, including this investigation.
+
+This is now confirmed NOT the shared-runner flakiness 0.136.5/0.136.6
+blamed it as: reran the identical failed job against the identical commit
+(`gh run rerun --failed`, no code change) and got the exact same failure,
+same file, same line, same message, a fourth consecutive time. Extensive
+local reproduction attempts — simulating `CI=true`/`GITHUB_ACTIONS=true`,
+running the full 1328-test `test:posture` scope concurrently to match CI's
+exact invocation shape, invoking the real bundled CLI directly under the
+same env vars — all passed cleanly. Whatever this is, it is deterministic
+on GitHub's hosted runner and has not reproduced anywhere else tried so far.
+
+Rather than keep guessing, the silent catch now logs the actual error to
+stderr whenever `CI`/`GITHUB_ACTIONS` is set (or `AGENTIC_SECURITY_MTTR_DEBUG=1`
+locally), and the test's own assertion message now includes the scan
+subprocess's stderr. Scan behavior is unchanged — this is instrumentation
+only, shipped specifically to get a real answer out of the next hosted run
+instead of another guess.
+
+## 0.136.7 — the doc-drift checker was resolving paths outside the repo
+
+0.136.6's hosted release run failed on a check this same audit added a few
+releases ago: `check-doc-drift.mjs`'s own regression test, "the real
+repository currently has zero mechanically-checkable drift," flagged root
+CLAUDE.md's documented (and correctly gitignored) `.claude/settings.local.json`
+reference as a dangling path.
+
+The actual defect was in `resolveCandidate`'s search bases, not in the
+reference: for a CLAUDE.md near the repo root, one of the fallback bases is
+"two directories up from the CLAUDE.md's own directory" — meant for nested
+CLAUDE.md files reaching back toward the repo root, but for the ROOT
+CLAUDE.md itself that lands OUTSIDE the checkout entirely, in whatever
+happens to be the parent of wherever the repo was cloned. On the
+maintainer's laptop that's their home directory, which happens to contain
+an unrelated, machine-global `~/.claude/settings.local.json` from ordinary
+Claude Code usage — so the checker "resolved" the reference against a file
+that has nothing to do with this project, passed locally, and failed on a
+clean CI checkout where no such coincidence exists. `resolveCandidate` now
+rejects any candidate base or resolved path outside the repo root, and
+`settings.local.json` — genuinely optional, gitignored, user-created — joins
+the checker's existing known-example-basename allowlist as a second,
+independent fix.
+
+Reproduced without touching CI: moved the local override file aside,
+confirmed the existing regression test still passed (proving the checker was
+resolving against something else entirely), traced it to the home-directory
+escape, fixed both the escape and the allowlist gap, verified clean with the
+file present and absent, then restored it.
+
+## 0.136.6 — correction: the "flaky" dataflow tests were a real, deterministic gap
+
+0.136.5's entry below called a cluster of `test:dataflow` failures on the
+hosted runner resource-contention flakiness, because a second, simultaneous
+workflow run of the identical commit showed a completely different failure
+and neither reproduced locally. That diagnosis was wrong, and the actual
+cause is more interesting: 13 test files pass `runScan(dir, { deep: true })`
+to exercise the interprocedural taint engine directly, but never opt into
+`AGENTIC_SECURITY_DEEP_IN_CI` — and `engine.js` deliberately auto-disables
+deep mode under any detected CI environment unless that second flag is also
+set, precisely so a pathological file can't hang a CI pipeline. Every one of
+those 13 files has therefore been silently CI-broken (never actually
+exercising the deep engine there, only recording the "skipped in CI"
+placeholder finding) since the day it was written — invisible until this
+release's tag push put 62 accumulated commits through hosted CI for the
+first time. Reproduced deterministically with `CI=true node --test
+<file>.test.js` locally (no CI service required), fixed by threading
+`deepInCi: true` alongside `deep: true` at all 21 call sites across those 13
+files, and confirmed both directions: `npm test` is clean with `CI` unset
+and with `CI=true` set.
+
+The earlier flakiness diagnosis wasn't entirely wrong — the *other* workflow
+run's single MTTR-wiring failure genuinely didn't reproduce on a third run
+and remains unexplained — but it was wrong about *this* failure cluster,
+which was 100% reproducible once isolated with the right environment
+variable rather than blamed on shared-runner load.
+
+## 0.136.5 — the release workflow gets its own missing dependency
+
+0.136.4 was tagged and its release workflow ran — for the first time ever,
+since the 60-odd commits it carried had accumulated across many local
+sessions without a single hosted-CI push. The gate caught a real gap in
+itself: `nist-catalog-freshness` (a check this same release adds, see below)
+shells out to `scripts/nist-compliance/build-catalog.py`, which needs
+`openpyxl` to read the source workbook. That's present on the maintainer's
+machine via a system Python install, but nothing installs it on the hosted
+runner — so the check could never have passed there. `release.yml` now
+installs it explicitly before the gate runs.
+
+The same CI run also showed roughly a dozen dataflow tests fail — a
+completely different set than the *other* workflow (`ci.yml`) failed on the
+same commit at the same time (one flaky MTTR-wiring test, no overlap with the
+dataflow set). Neither set reproduces locally, isolated or otherwise. That
+non-overlap is the signature of resource-contention flakiness on a shared
+runner, not a deterministic regression, so it isn't chased further here — but
+it's worth knowing about if a future release gate flakes on `npm test` again.
+
+Per the project's own precedent (see 0.136.1 below): a tag that failed to
+publish stays where it is rather than being moved. 0.136.4 is that tag. This
+ships as 0.136.5 instead, otherwise identical.
+
+## 0.136.4 — dominance-correct implicit-flow taint, and wiring three dead-reachable tools
+
+A capability-PRD follow-up audit found several places where a real, tested
+mechanism existed in the code but nothing in the product could actually reach
+it. This release closes those gaps rather than documenting them further.
+
+**`implicit-flow.js` now uses real dominance, not a depth counter.** The
+branch-scoping check for implicit taint (does a constant-argument sink sit
+*inside* a tainted branch, or after it closes?) was a path-dependent DFS
+depth-counter, which cannot distinguish a join point from a nested branch and
+both over- and under-attributes depending on CFG shape. It's replaced with a
+proper dominance check (`ssa.js`'s `computeDominators`, already used for φ-node
+placement, now exported and reused here) plus a predecessor-count "sole
+parent" test — needed because an else-less `if`'s CFG lowering links the
+condition directly to the join node, which a naive dominance check alone would
+still misread as a branch root. `dataflow/engine.js`'s implicit-flow post-pass
+is split into two correctly-scoped passes: a sink-call-must-be-inside-the-
+branch check for constant-argument leaks, and an ungated check for
+already-tainted-variable-as-argument leaks, which don't need the sink itself
+to be branch-local.
+
+**Three previously-dead mechanisms are now reachable.** `verify_fix`'s PoC-
+recheck leg always reported `not-requested` — the caller never had the PoC to
+pass it, so it's now looked up server-side from the finding's own
+`last-scan.json` entry. `fix-honesty-gate.js`'s deterministic honesty checks
+(vague-assurance residual prose, unbacked false-positive verdicts) were fully
+built and consulted by the verifier, but `fixMeta` was never in `apply_fix` or
+`verify_fix`'s MCP schema, so no caller could ever supply one; a dishonest
+`fixMeta` now blocks the write itself, not just the report. `verifyRunAttestation`
+had no CLI caller at all — `verify-attestation` now auto-detects a
+run-attestation shape and re-scans the target to check it reproduces the
+attested digest, backed by two new release-gate checks
+(`attestation-self-check`, `nist-catalog-freshness`).
+
+**Stale docs, fixed instead of flagged.** `docs/compliance/{nist-ai-600-1,
+owasp-asvs}-coverage.md` carried static control tables that drifted from the
+live evaluator; both now point at the `/compliance` walkthrough/report instead
+of duplicating data that can go stale. The ASVS doc also had a genuine
+version mismatch (4.0.3 vs. the 5.0 catalog actually in use). A new
+`scripts/check-doc-drift.mjs` catches this class of staleness mechanically —
+it resolves every backtick-quoted path/export reference in a CLAUDE.md file
+against the real filesystem — after this audit found several by hand.
+
+**CVE-replay corpus:** four new capability entries (IaC open-ingress, LLM
+system-prompt injection, MCP untrusted-install, API missing-auth/BFLA),
+closing four of six previously-flagged zero-coverage categories; each verified
+`pre:TP post:TN` against the real runner before joining the baseline. SBOM and
+SCA-reachability are documented in `bench/cve-replay/CONTRIBUTING.md` as
+structurally unable to fit this corpus's binary presence/absence schema —
+they're covered by their own test suites instead.
+
+CMP-1's family-alias table also closed three more gaps
+(`k8s-pod-security-privileged`, `mcp-audit.js`'s `agent-tool-exec` backfill,
+dependency-confusion family tagging).
+
+**No functional change from 0.136.2.** This version exists for one reason, and
+it is worth stating plainly rather than dressing up: 0.136.2 reached npm from a
+maintainer's laptop, not from CI, so it carries **no provenance attestation** —
+nothing ties that tarball to this repository or this commit beyond trust in the
+publisher.
+
+npm will not accept a re-publish of an existing version, so obtaining provenance
+requires a new one. 0.136.3 is that, and nothing else.
+
+The release path itself was already proven end to end on 0.136.2: the gate passed
+on a clean runner, npm signed a provenance statement and recorded it in the
+Sigstore transparency log, and only the final registry upload was rejected —
+because the token in CI was not authorized to publish. With a valid automation
+token that last step completes, and the attestation that was already being
+generated actually lands.
+
+Verify it yourself once published:
+
+```
+npm view @clear-capabilities/agentic-security-scanner@0.136.3 --json | jq .dist.attestations
+```
+
+`null` means it went out unattested again. `dist.signatures` is NOT the same
+thing — the registry signs every package it serves; provenance is the separate
+Sigstore statement binding the artifact to its source.
+
+## 0.136.2 — authenticate the gate on the path that actually runs it
+
+0.136.1 removed the self-deadlock and the release workflow got further: every
+gate check passed, then `npm publish` failed with *"the forge CLI is not
+authenticated"*.
+
+The gate runs **twice** in that job. Once as an explicit `Release gate` step,
+which sets `GH_TOKEN` and passed. Then again inside `npm publish`, which
+triggers `prepublishOnly` — and that step set only `NODE_AUTH_TOKEN`. The second
+run had no token, could not read hosted CI, and refused.
+
+The gate was right to refuse: unverifiable is not green, and the remedy it
+suggests — `--allow-unverified-ci` — would have published without proving the
+commit was green at all. So the fix is to authenticate it, not to relax it.
+`checks: read` is now declared explicitly too; the gate passed without it on the
+default token, but depending on an undeclared default is how a tightened default
+becomes a mystery failure a year later.
+
+The same shape as the deadlock it follows: a control that works on the path
+that was tested, and fails on the path that actually ships.
+
+## 0.136.1 — the release gate stops deadlocking on itself
+
+`npm publish` had been impossible since 0.135.0, and the cause was not the one I
+reported. Not a missing npm token: **the release gate was waiting for itself.**
+
+The release workflow's job is named `publish`. The gate runs inside that job,
+queries hosted CI for HEAD, sees a check run named `publish` that is
+`in_progress` — itself — and requires it to finish before allowing the release.
+It never can. Every tag push failed this way with all nine real checks green, and
+the resulting `publish: failure` then blocked local publishes too, which is the
+error that finally surfaced it.
+
+`.github/required-checks.json` gains a third category, **self**, beside blocking
+and informational. A self check is EXCLUDED, not trusted: it cannot report a
+conclusion until the gate it contains has already passed, so requiring it is a
+deadlock and believing it would be believing a check that has not run. The file
+already insisted every check be classified deliberately — this is the category
+that was missing, and its absence meant `publish` fell through to the safe
+default of blocking, which was exactly wrong here.
+
+Proven in three directions, because excluding a check must not weaken a gate:
+a pending self check no longer blocks, a FAILED self check no longer blocks, and
+a genuinely red blocking check still does. Verified against live CI state:
+`PASS Hosted CI is green for HEAD`.
+
+v0.135.0 and v0.136.0 were tagged but never reached npm for this reason. The
+tags stay where they are — a public tag is not moved — so this ships as 0.136.1.
+
+## 0.136.0 — NIST Privacy Framework 1.1, and the frameworks that never shipped
+
+### A privacy compliance scan that says what it did not check
+
+All 104 PF 1.1 controls, assessed on every scan, artifacts at
+`.agentic-security/privacy-framework.{json,md}`. Each gap is emitted as an
+ordinary finding (`family: privacy-compliance`, `CWE-359`) carrying an actionable
+remediation, so `/fix` handles it like anything else.
+
+The design turns on one column in NIST's own workbook. PF 1.1 rates each control
+for code-testability — **23 yes, 33 partial, 48 no** — and that rating says a
+control *could* be assessed from source, not that this engine assesses it.
+Collapsing the two is how a privacy report marks "the organizational mission is
+communicated" as PASSED because no rule fired against it, and someone hands that
+to an auditor. So every control lands in exactly one stated bucket:
+
+| Bucket | Meaning |
+|---|---|
+| gap | mapped to an engine signal, and that signal is failing — the only bucket that emits a finding |
+| not assessed | NIST rates it code-testable, this engine has no signal — named, never a pass |
+| manual | NIST rates it not code-testable — governance, outside any scanner's reach |
+| satisfied | mapped, and the signal is clean |
+
+Measured on a live fixture: 9 gaps, 20 satisfied, 27 not assessed, 48 manual.
+The satisfied rate is reported over the **29 assessed** controls, never over 104.
+
+A **vacuous-satisfaction guard** was added after the module's own test caught it:
+a `family:`-mapped control clears when no findings of that family are open, which
+is equally true of a scan that read zero files. Pointing the tool at an empty
+directory was reporting privacy controls as satisfied on the strength of having
+looked at nothing. Now every mapped control degrades to *not assessed* and the
+summary says so.
+
+Findings are opt-in (`AGENTIC_SECURITY_PRIVACY_FRAMEWORK=1`). The assessment
+always runs and persists; appending findings by default would change every
+severity count and gate verdict downstream, and a compliance opinion should not
+silently become someone's build failure.
+
+### A real `compliance` subcommand
+
+```
+agentic-security compliance [--gap] [--list] [--walkthrough <id>]
+                            [--format cli|json|md] [--fail-on gap]
+```
+
+`/compliance --privacy` was documented as a mode with no binary behind it. It
+reads `last-scan.json` rather than re-scanning — a compliance answer is a
+statement about a scan that happened. With no scan to read it exits **2** rather
+than assessing an empty project. Exit codes: 0 report produced, 1 only with
+`--fail-on gap` and a failing control, 2 nothing to assess.
+
+### Every bundled framework had been invisible from the published artifact
+
+Running the new subcommand from the shipped bundle printed nothing and exited 0.
+`auditor-walkthrough` resolves its data directory from `import.meta.url` — inside
+the bundle that is `dist/`, and `dist/compliance-frameworks/` never existed
+because the build only emitted the `.mjs`. `listFrameworks` catches the readdir
+failure and returns `[]`.
+
+So GDPR, ASVS, NIST AI 600-1 — all nine — worked perfectly from source and were
+**silently absent from the npm package**. Nothing caught it because every test
+ran against `src/`. The build now copies the data next to the bundle, the data is
+tracked in git for the same reason the bundle is, and a test drives the BUNDLE so
+it cannot regress.
+
+Twice in this release, testing the shipped artifact rather than the source found
+something the whole suite was blind to.
+
+### Also
+
+Removed all remaining PRD documents, repairing 18 files that referenced them.
+`bench/proof-corpus`'s section-level citations (parse-coverage rule, acceptance
+criterion 2, criterion 4, the disclosure boundary) are now stated inline and
+owned by the files that depend on them — the rationale outlived the document.
+
+## 0.135.0 — a scan stops modifying what it scans, and the benchmark stops flattering
+
+Two findings, one of which reverses something this project previously reported.
+
+### `--no-state`: a scan is an observation again
+
+Pointing the engine at a directory used to write **10 files** into it. For a user
+that means CI asserting a clean tree fails after a scan, and scanning a
+dependency or a customer's code leaves artifacts in a tree they own. Worse, our
+own output contains CWE identifiers, so a second scan could read the first
+scan's conclusions as source.
+
+`AGENTIC_SECURITY_NO_STATE=1` (and `--no-state`) now adds **zero paths** while
+reporting byte-identical findings. Both halves are asserted, and the test was
+proven to FAIL with the switch off — the switch must change what is written,
+never what is found. The engine also skips `.agentic-security/` when walking, so
+our output can never become our input.
+
+Three defects were found in the guard meant to enforce this, each worth more
+than the line that fixed it:
+
+- **`git status` is not sufficient evidence.** Git does not track empty
+  directories, so an earlier revision reported a CLEAN tree while still creating
+  `sbom-history/` and `fix-history/`. Directory creation is mutation: it fails on
+  a read-only mount and is litter in someone else's repo. The acceptance test
+  compares full path listings.
+- **The guard was blind to `bin/`** — where the three largest artifacts
+  (`findings.json`, `last-scan.json`, `.sig`) are written. A seam guard that
+  cannot see the CLI entry point misses the primary writer.
+- **The detector counted documentation as a violation**, then over-corrected: a
+  glob inside `// .agentic-security/rules/*.yml` opened a block comment that
+  consumed 12,198 characters and hid a real violation. Comment-strip order is
+  now load-bearing and asserted in both directions.
+
+Stated plainly: 55 modules still build state paths by hand and remain on a
+migration ledger. What changed permanently is that a 56th cannot be added.
+
+### The independent recall figure was wrong, and the correction is downward
+
+Benchmark trees had been contaminated by the engine's own state files (220
+polluted trees, 544 carrying `CWE-` strings). Fixing that coincided with a second
+change — restricting matches to the files the advisory's fix commit touched — and
+recall fell from a previously reported 33.6% to 12.7%.
+
+Attributing that fall to the wrong cause would have been the same reasoning error
+as the contamination, pointed the other way, so the runner now scores both ways
+in one pass:
+
+| | advisory-local (**the claim**) | wide (diagnostic) |
+|---|---|---|
+| recall | **12.7%** (14/110) | 33.6% (37/110) |
+| precision | **50.0%** (14/28) | 50.0% (37/74) |
+| F1 | **0.203** | 0.402 |
+
+The wide figure is identical to the pre-purge number. **The contamination was
+real and had to be fixed, but it was not inflating the measurement** — every one
+of the 20.9 points comes from the benchmark becoming honest about *where* a
+finding has to be. 12.7% is the true recall, and it always was.
+
+It is published as a low number rather than quietly requalified, because the
+point of owning the instrument is to be able to trust it when it disagrees.
+
+
 ## 0.134.0 — the loop closes, and the logic tier learns how to be wrong
 
 The remaining PRD epics. Two of them are new capability; the other two are the

package/bin/agentic-security-audit.js

@@ -23,6 +23,7 @@ import * as fs from 'node:fs';
 import * as path from 'node:path';
 import { verifyAuditLog } from '../src/mcp/audit.js';
 
+import { statePath } from '../src/posture/state-dir.js';
 function args() {
   const a = process.argv.slice(2);
   const sub = a[0] || 'review';
@@ -48,7 +49,7 @@ function _parseDuration(s) {
   return n * u;
 }
 
-function _logPath(root) { return path.join(root, '.agentic-security', 'mcp-audit.log'); }
+function _logPath(root) { return statePath(root, 'mcp-audit.log'); }
 
 function _readEntries(root) {
   const fp = _logPath(root);

package/bin/agentic-security-consistency.js

@@ -26,6 +26,7 @@ import * as fs from 'node:fs';
 import * as path from 'node:path';
 import { measureConsistency, summarize } from '../src/llm-validator/consistency.js';
 
+import { statePath } from '../src/posture/state-dir.js';
 function args() {
   const a = process.argv.slice(2);
   const out = { trials: 5, top: 5, json: false, root: process.cwd() };
@@ -40,7 +41,7 @@ function args() {
 
 async function main() {
   const opts = args();
-  const scanFile = path.join(opts.root, '.agentic-security', 'last-scan.json');
+  const scanFile = statePath(opts.root, 'last-scan.json');
   if (!fs.existsSync(scanFile)) {
     console.error(`no last-scan.json at ${scanFile} — run a scan first`);
     process.exit(2);