@clear-capabilities/agentic-security-scanner 0.134.0 → 0.136.2

package/CHANGELOG.md

@@ -1,5 +1,201 @@
 # Changelog
 
+> **A note on `docs/*_PRD.md` references below.** Product requirement documents in
+> this project are removed once their workstreams land — the durable parts move
+> into the code and tests that enforce them, where they cannot drift out of date.
+> Entries below cite PRD files that were live at the time of that release and no
+> longer exist. They are left as written: a changelog records what was true when
+> it was written, and rewriting past entries to hide a since-deleted file would
+> make the history less accurate, not more.
+
+
+## 0.136.2 — authenticate the gate on the path that actually runs it
+
+0.136.1 removed the self-deadlock and the release workflow got further: every
+gate check passed, then `npm publish` failed with *"the forge CLI is not
+authenticated"*.
+
+The gate runs **twice** in that job. Once as an explicit `Release gate` step,
+which sets `GH_TOKEN` and passed. Then again inside `npm publish`, which
+triggers `prepublishOnly` — and that step set only `NODE_AUTH_TOKEN`. The second
+run had no token, could not read hosted CI, and refused.
+
+The gate was right to refuse: unverifiable is not green, and the remedy it
+suggests — `--allow-unverified-ci` — would have published without proving the
+commit was green at all. So the fix is to authenticate it, not to relax it.
+`checks: read` is now declared explicitly too; the gate passed without it on the
+default token, but depending on an undeclared default is how a tightened default
+becomes a mystery failure a year later.
+
+The same shape as the deadlock it follows: a control that works on the path
+that was tested, and fails on the path that actually ships.
+
+## 0.136.1 — the release gate stops deadlocking on itself
+
+`npm publish` had been impossible since 0.135.0, and the cause was not the one I
+reported. Not a missing npm token: **the release gate was waiting for itself.**
+
+The release workflow's job is named `publish`. The gate runs inside that job,
+queries hosted CI for HEAD, sees a check run named `publish` that is
+`in_progress` — itself — and requires it to finish before allowing the release.
+It never can. Every tag push failed this way with all nine real checks green, and
+the resulting `publish: failure` then blocked local publishes too, which is the
+error that finally surfaced it.
+
+`.github/required-checks.json` gains a third category, **self**, beside blocking
+and informational. A self check is EXCLUDED, not trusted: it cannot report a
+conclusion until the gate it contains has already passed, so requiring it is a
+deadlock and believing it would be believing a check that has not run. The file
+already insisted every check be classified deliberately — this is the category
+that was missing, and its absence meant `publish` fell through to the safe
+default of blocking, which was exactly wrong here.
+
+Proven in three directions, because excluding a check must not weaken a gate:
+a pending self check no longer blocks, a FAILED self check no longer blocks, and
+a genuinely red blocking check still does. Verified against live CI state:
+`PASS Hosted CI is green for HEAD`.
+
+v0.135.0 and v0.136.0 were tagged but never reached npm for this reason. The
+tags stay where they are — a public tag is not moved — so this ships as 0.136.1.
+
+## 0.136.0 — NIST Privacy Framework 1.1, and the frameworks that never shipped
+
+### A privacy compliance scan that says what it did not check
+
+All 104 PF 1.1 controls, assessed on every scan, artifacts at
+`.agentic-security/privacy-framework.{json,md}`. Each gap is emitted as an
+ordinary finding (`family: privacy-compliance`, `CWE-359`) carrying an actionable
+remediation, so `/fix` handles it like anything else.
+
+The design turns on one column in NIST's own workbook. PF 1.1 rates each control
+for code-testability — **23 yes, 33 partial, 48 no** — and that rating says a
+control *could* be assessed from source, not that this engine assesses it.
+Collapsing the two is how a privacy report marks "the organizational mission is
+communicated" as PASSED because no rule fired against it, and someone hands that
+to an auditor. So every control lands in exactly one stated bucket:
+
+| Bucket | Meaning |
+|---|---|
+| gap | mapped to an engine signal, and that signal is failing — the only bucket that emits a finding |
+| not assessed | NIST rates it code-testable, this engine has no signal — named, never a pass |
+| manual | NIST rates it not code-testable — governance, outside any scanner's reach |
+| satisfied | mapped, and the signal is clean |
+
+Measured on a live fixture: 9 gaps, 20 satisfied, 27 not assessed, 48 manual.
+The satisfied rate is reported over the **29 assessed** controls, never over 104.
+
+A **vacuous-satisfaction guard** was added after the module's own test caught it:
+a `family:`-mapped control clears when no findings of that family are open, which
+is equally true of a scan that read zero files. Pointing the tool at an empty
+directory was reporting privacy controls as satisfied on the strength of having
+looked at nothing. Now every mapped control degrades to *not assessed* and the
+summary says so.
+
+Findings are opt-in (`AGENTIC_SECURITY_PRIVACY_FRAMEWORK=1`). The assessment
+always runs and persists; appending findings by default would change every
+severity count and gate verdict downstream, and a compliance opinion should not
+silently become someone's build failure.
+
+### A real `compliance` subcommand
+
+```
+agentic-security compliance [--gap] [--list] [--walkthrough <id>]
+                            [--format cli|json|md] [--fail-on gap]
+```
+
+`/compliance --privacy` was documented as a mode with no binary behind it. It
+reads `last-scan.json` rather than re-scanning — a compliance answer is a
+statement about a scan that happened. With no scan to read it exits **2** rather
+than assessing an empty project. Exit codes: 0 report produced, 1 only with
+`--fail-on gap` and a failing control, 2 nothing to assess.
+
+### Every bundled framework had been invisible from the published artifact
+
+Running the new subcommand from the shipped bundle printed nothing and exited 0.
+`auditor-walkthrough` resolves its data directory from `import.meta.url` — inside
+the bundle that is `dist/`, and `dist/compliance-frameworks/` never existed
+because the build only emitted the `.mjs`. `listFrameworks` catches the readdir
+failure and returns `[]`.
+
+So GDPR, ASVS, NIST AI 600-1 — all nine — worked perfectly from source and were
+**silently absent from the npm package**. Nothing caught it because every test
+ran against `src/`. The build now copies the data next to the bundle, the data is
+tracked in git for the same reason the bundle is, and a test drives the BUNDLE so
+it cannot regress.
+
+Twice in this release, testing the shipped artifact rather than the source found
+something the whole suite was blind to.
+
+### Also
+
+Removed all remaining PRD documents, repairing 18 files that referenced them.
+`bench/proof-corpus`'s section-level citations (parse-coverage rule, acceptance
+criterion 2, criterion 4, the disclosure boundary) are now stated inline and
+owned by the files that depend on them — the rationale outlived the document.
+
+## 0.135.0 — a scan stops modifying what it scans, and the benchmark stops flattering
+
+Two findings, one of which reverses something this project previously reported.
+
+### `--no-state`: a scan is an observation again
+
+Pointing the engine at a directory used to write **10 files** into it. For a user
+that means CI asserting a clean tree fails after a scan, and scanning a
+dependency or a customer's code leaves artifacts in a tree they own. Worse, our
+own output contains CWE identifiers, so a second scan could read the first
+scan's conclusions as source.
+
+`AGENTIC_SECURITY_NO_STATE=1` (and `--no-state`) now adds **zero paths** while
+reporting byte-identical findings. Both halves are asserted, and the test was
+proven to FAIL with the switch off — the switch must change what is written,
+never what is found. The engine also skips `.agentic-security/` when walking, so
+our output can never become our input.
+
+Three defects were found in the guard meant to enforce this, each worth more
+than the line that fixed it:
+
+- **`git status` is not sufficient evidence.** Git does not track empty
+  directories, so an earlier revision reported a CLEAN tree while still creating
+  `sbom-history/` and `fix-history/`. Directory creation is mutation: it fails on
+  a read-only mount and is litter in someone else's repo. The acceptance test
+  compares full path listings.
+- **The guard was blind to `bin/`** — where the three largest artifacts
+  (`findings.json`, `last-scan.json`, `.sig`) are written. A seam guard that
+  cannot see the CLI entry point misses the primary writer.
+- **The detector counted documentation as a violation**, then over-corrected: a
+  glob inside `// .agentic-security/rules/*.yml` opened a block comment that
+  consumed 12,198 characters and hid a real violation. Comment-strip order is
+  now load-bearing and asserted in both directions.
+
+Stated plainly: 55 modules still build state paths by hand and remain on a
+migration ledger. What changed permanently is that a 56th cannot be added.
+
+### The independent recall figure was wrong, and the correction is downward
+
+Benchmark trees had been contaminated by the engine's own state files (220
+polluted trees, 544 carrying `CWE-` strings). Fixing that coincided with a second
+change — restricting matches to the files the advisory's fix commit touched — and
+recall fell from a previously reported 33.6% to 12.7%.
+
+Attributing that fall to the wrong cause would have been the same reasoning error
+as the contamination, pointed the other way, so the runner now scores both ways
+in one pass:
+
+| | advisory-local (**the claim**) | wide (diagnostic) |
+|---|---|---|
+| recall | **12.7%** (14/110) | 33.6% (37/110) |
+| precision | **50.0%** (14/28) | 50.0% (37/74) |
+| F1 | **0.203** | 0.402 |
+
+The wide figure is identical to the pre-purge number. **The contamination was
+real and had to be fixed, but it was not inflating the measurement** — every one
+of the 20.9 points comes from the benchmark becoming honest about *where* a
+finding has to be. 12.7% is the true recall, and it always was.
+
+It is published as a low number rather than quietly requalified, because the
+point of owning the instrument is to be able to trust it when it disagrees.
+
+
 ## 0.134.0 — the loop closes, and the logic tier learns how to be wrong
 
 The remaining PRD epics. Two of them are new capability; the other two are the

package/bin/agentic-security-audit.js

@@ -23,6 +23,7 @@ import * as fs from 'node:fs';
 import * as path from 'node:path';
 import { verifyAuditLog } from '../src/mcp/audit.js';
 
+import { statePath } from '../src/posture/state-dir.js';
 function args() {
   const a = process.argv.slice(2);
   const sub = a[0] || 'review';
@@ -48,7 +49,7 @@ function _parseDuration(s) {
   return n * u;
 }
 
-function _logPath(root) { return path.join(root, '.agentic-security', 'mcp-audit.log'); }
+function _logPath(root) { return statePath(root, 'mcp-audit.log'); }
 
 function _readEntries(root) {
   const fp = _logPath(root);

package/bin/agentic-security-consistency.js

@@ -26,6 +26,7 @@ import * as fs from 'node:fs';
 import * as path from 'node:path';
 import { measureConsistency, summarize } from '../src/llm-validator/consistency.js';
 
+import { statePath } from '../src/posture/state-dir.js';
 function args() {
   const a = process.argv.slice(2);
   const out = { trials: 5, top: 5, json: false, root: process.cwd() };
@@ -40,7 +41,7 @@ function args() {
 
 async function main() {
   const opts = args();
-  const scanFile = path.join(opts.root, '.agentic-security', 'last-scan.json');
+  const scanFile = statePath(opts.root, 'last-scan.json');
   if (!fs.existsSync(scanFile)) {
     console.error(`no last-scan.json at ${scanFile} — run a scan first`);
     process.exit(2);