@clear-capabilities/agentic-security-scanner 0.133.0 → 0.136.2

package/CHANGELOG.md

@@ -1,5 +1,304 @@
 # Changelog
 
+> **A note on `docs/*_PRD.md` references below.** Product requirement documents in
+> this project are removed once their workstreams land — the durable parts move
+> into the code and tests that enforce them, where they cannot drift out of date.
+> Entries below cite PRD files that were live at the time of that release and no
+> longer exist. They are left as written: a changelog records what was true when
+> it was written, and rewriting past entries to hide a since-deleted file would
+> make the history less accurate, not more.
+
+
+## 0.136.2 — authenticate the gate on the path that actually runs it
+
+0.136.1 removed the self-deadlock and the release workflow got further: every
+gate check passed, then `npm publish` failed with *"the forge CLI is not
+authenticated"*.
+
+The gate runs **twice** in that job. Once as an explicit `Release gate` step,
+which sets `GH_TOKEN` and passed. Then again inside `npm publish`, which
+triggers `prepublishOnly` — and that step set only `NODE_AUTH_TOKEN`. The second
+run had no token, could not read hosted CI, and refused.
+
+The gate was right to refuse: unverifiable is not green, and the remedy it
+suggests — `--allow-unverified-ci` — would have published without proving the
+commit was green at all. So the fix is to authenticate it, not to relax it.
+`checks: read` is now declared explicitly too; the gate passed without it on the
+default token, but depending on an undeclared default is how a tightened default
+becomes a mystery failure a year later.
+
+The same shape as the deadlock it follows: a control that works on the path
+that was tested, and fails on the path that actually ships.
+
+## 0.136.1 — the release gate stops deadlocking on itself
+
+`npm publish` had been impossible since 0.135.0, and the cause was not the one I
+reported. Not a missing npm token: **the release gate was waiting for itself.**
+
+The release workflow's job is named `publish`. The gate runs inside that job,
+queries hosted CI for HEAD, sees a check run named `publish` that is
+`in_progress` — itself — and requires it to finish before allowing the release.
+It never can. Every tag push failed this way with all nine real checks green, and
+the resulting `publish: failure` then blocked local publishes too, which is the
+error that finally surfaced it.
+
+`.github/required-checks.json` gains a third category, **self**, beside blocking
+and informational. A self check is EXCLUDED, not trusted: it cannot report a
+conclusion until the gate it contains has already passed, so requiring it is a
+deadlock and believing it would be believing a check that has not run. The file
+already insisted every check be classified deliberately — this is the category
+that was missing, and its absence meant `publish` fell through to the safe
+default of blocking, which was exactly wrong here.
+
+Proven in three directions, because excluding a check must not weaken a gate:
+a pending self check no longer blocks, a FAILED self check no longer blocks, and
+a genuinely red blocking check still does. Verified against live CI state:
+`PASS Hosted CI is green for HEAD`.
+
+v0.135.0 and v0.136.0 were tagged but never reached npm for this reason. The
+tags stay where they are — a public tag is not moved — so this ships as 0.136.1.
+
+## 0.136.0 — NIST Privacy Framework 1.1, and the frameworks that never shipped
+
+### A privacy compliance scan that says what it did not check
+
+All 104 PF 1.1 controls, assessed on every scan, artifacts at
+`.agentic-security/privacy-framework.{json,md}`. Each gap is emitted as an
+ordinary finding (`family: privacy-compliance`, `CWE-359`) carrying an actionable
+remediation, so `/fix` handles it like anything else.
+
+The design turns on one column in NIST's own workbook. PF 1.1 rates each control
+for code-testability — **23 yes, 33 partial, 48 no** — and that rating says a
+control *could* be assessed from source, not that this engine assesses it.
+Collapsing the two is how a privacy report marks "the organizational mission is
+communicated" as PASSED because no rule fired against it, and someone hands that
+to an auditor. So every control lands in exactly one stated bucket:
+
+| Bucket | Meaning |
+|---|---|
+| gap | mapped to an engine signal, and that signal is failing — the only bucket that emits a finding |
+| not assessed | NIST rates it code-testable, this engine has no signal — named, never a pass |
+| manual | NIST rates it not code-testable — governance, outside any scanner's reach |
+| satisfied | mapped, and the signal is clean |
+
+Measured on a live fixture: 9 gaps, 20 satisfied, 27 not assessed, 48 manual.
+The satisfied rate is reported over the **29 assessed** controls, never over 104.
+
+A **vacuous-satisfaction guard** was added after the module's own test caught it:
+a `family:`-mapped control clears when no findings of that family are open, which
+is equally true of a scan that read zero files. Pointing the tool at an empty
+directory was reporting privacy controls as satisfied on the strength of having
+looked at nothing. Now every mapped control degrades to *not assessed* and the
+summary says so.
+
+Findings are opt-in (`AGENTIC_SECURITY_PRIVACY_FRAMEWORK=1`). The assessment
+always runs and persists; appending findings by default would change every
+severity count and gate verdict downstream, and a compliance opinion should not
+silently become someone's build failure.
+
+### A real `compliance` subcommand
+
+```
+agentic-security compliance [--gap] [--list] [--walkthrough <id>]
+                            [--format cli|json|md] [--fail-on gap]
+```
+
+`/compliance --privacy` was documented as a mode with no binary behind it. It
+reads `last-scan.json` rather than re-scanning — a compliance answer is a
+statement about a scan that happened. With no scan to read it exits **2** rather
+than assessing an empty project. Exit codes: 0 report produced, 1 only with
+`--fail-on gap` and a failing control, 2 nothing to assess.
+
+### Every bundled framework had been invisible from the published artifact
+
+Running the new subcommand from the shipped bundle printed nothing and exited 0.
+`auditor-walkthrough` resolves its data directory from `import.meta.url` — inside
+the bundle that is `dist/`, and `dist/compliance-frameworks/` never existed
+because the build only emitted the `.mjs`. `listFrameworks` catches the readdir
+failure and returns `[]`.
+
+So GDPR, ASVS, NIST AI 600-1 — all nine — worked perfectly from source and were
+**silently absent from the npm package**. Nothing caught it because every test
+ran against `src/`. The build now copies the data next to the bundle, the data is
+tracked in git for the same reason the bundle is, and a test drives the BUNDLE so
+it cannot regress.
+
+Twice in this release, testing the shipped artifact rather than the source found
+something the whole suite was blind to.
+
+### Also
+
+Removed all remaining PRD documents, repairing 18 files that referenced them.
+`bench/proof-corpus`'s section-level citations (parse-coverage rule, acceptance
+criterion 2, criterion 4, the disclosure boundary) are now stated inline and
+owned by the files that depend on them — the rationale outlived the document.
+
+## 0.135.0 — a scan stops modifying what it scans, and the benchmark stops flattering
+
+Two findings, one of which reverses something this project previously reported.
+
+### `--no-state`: a scan is an observation again
+
+Pointing the engine at a directory used to write **10 files** into it. For a user
+that means CI asserting a clean tree fails after a scan, and scanning a
+dependency or a customer's code leaves artifacts in a tree they own. Worse, our
+own output contains CWE identifiers, so a second scan could read the first
+scan's conclusions as source.
+
+`AGENTIC_SECURITY_NO_STATE=1` (and `--no-state`) now adds **zero paths** while
+reporting byte-identical findings. Both halves are asserted, and the test was
+proven to FAIL with the switch off — the switch must change what is written,
+never what is found. The engine also skips `.agentic-security/` when walking, so
+our output can never become our input.
+
+Three defects were found in the guard meant to enforce this, each worth more
+than the line that fixed it:
+
+- **`git status` is not sufficient evidence.** Git does not track empty
+  directories, so an earlier revision reported a CLEAN tree while still creating
+  `sbom-history/` and `fix-history/`. Directory creation is mutation: it fails on
+  a read-only mount and is litter in someone else's repo. The acceptance test
+  compares full path listings.
+- **The guard was blind to `bin/`** — where the three largest artifacts
+  (`findings.json`, `last-scan.json`, `.sig`) are written. A seam guard that
+  cannot see the CLI entry point misses the primary writer.
+- **The detector counted documentation as a violation**, then over-corrected: a
+  glob inside `// .agentic-security/rules/*.yml` opened a block comment that
+  consumed 12,198 characters and hid a real violation. Comment-strip order is
+  now load-bearing and asserted in both directions.
+
+Stated plainly: 55 modules still build state paths by hand and remain on a
+migration ledger. What changed permanently is that a 56th cannot be added.
+
+### The independent recall figure was wrong, and the correction is downward
+
+Benchmark trees had been contaminated by the engine's own state files (220
+polluted trees, 544 carrying `CWE-` strings). Fixing that coincided with a second
+change — restricting matches to the files the advisory's fix commit touched — and
+recall fell from a previously reported 33.6% to 12.7%.
+
+Attributing that fall to the wrong cause would have been the same reasoning error
+as the contamination, pointed the other way, so the runner now scores both ways
+in one pass:
+
+| | advisory-local (**the claim**) | wide (diagnostic) |
+|---|---|---|
+| recall | **12.7%** (14/110) | 33.6% (37/110) |
+| precision | **50.0%** (14/28) | 50.0% (37/74) |
+| F1 | **0.203** | 0.402 |
+
+The wide figure is identical to the pre-purge number. **The contamination was
+real and had to be fixed, but it was not inflating the measurement** — every one
+of the 20.9 points comes from the benchmark becoming honest about *where* a
+finding has to be. 12.7% is the true recall, and it always was.
+
+It is published as a low number rather than quietly requalified, because the
+point of owning the instrument is to be able to trust it when it disagrees.
+
+
+## 0.134.0 — the loop closes, and the logic tier learns how to be wrong
+
+The remaining PRD epics. Two of them are new capability; the other two are the
+same idea applied twice — a claim nobody can disagree with is the weakest thing
+this engine emits, so both new tiers ship with the machinery to refute
+themselves.
+
+### Two more proof classes, and neither needed a running application
+
+`sql-injection` and `path-traversal` join the three existing classes. The PRD
+assumed both would wait on a running-app harness; they did not, because the
+harness was never where the proof lived.
+
+- **SQL injection is settled at the driver boundary.** Either the payload
+  arrives inside the query TEXT or it arrives as a bound parameter — the first
+  is the vulnerability by definition and the second is the fix by definition,
+  and no schema, rows or live server are needed to tell them apart. The PoC
+  stubs the driver with a recorder and writes the marker only when the payload
+  shows up inside something recognisably SQL. A parameterised query reaches
+  `proof-failed` by **execution**, not by reading the source.
+- **Path traversal is settled by what comes back.** A sentinel is planted
+  outside the served directory and the marker is written only if its content —
+  or, for `sendFile`, its resolved path — comes back out of the handler. A
+  `basename` guard is refuted by running it.
+
+Both directions are pinned by tests that execute in the real sandbox. Classes
+still absent (IDOR, SSRF, XSS) are now documented as decisions with reasons
+rather than gaps: a PoC built on invented application state proves something
+about the invention.
+
+Fixed along the way: the webhook PoC wrote its marker after a top-level `await`
+guarded by an **unref'd** timer, so a handler that never replied let Node exit
+with the promise pending and the marker check never ran. The test asserting
+"no decision writes no marker" had been passing without reaching the line it
+was testing. Timer is now ref'd and cleared, with a positive control asserting
+the process reaches exit 0.
+
+### The autonomous loop, wired to real stages
+
+`scripts/autopilot.mjs` connects the loop to a real scan, a real sandboxed
+exploit, a deterministic-then-model fix, and the real gate. End-to-end tests run
+the whole thing against a live HTTP endpoint, including the one that matters: a
+patch that changes the file, reads like a fix, and would satisfy any "did the
+scanner go quiet?" check is **refused and never written**, because the exploit
+still fires against it.
+
+The CLI refuses to start without a confinement backend (the verdict requires
+executing something) and refuses a dirty git tree by default (the test leg
+writes the candidate patch to disk and restores it in a `finally`; a clean tree
+is what makes a crash recoverable). A verified fix reached with no test runner
+detected is counted and reported separately — the exploit stopped firing, but
+nothing checked the application still works.
+
+### The business-logic tier learns how to be wrong
+
+The deterministic half already existed. The reviewing agent's half was prose:
+it asserted that a handler lets one user act on another's resource, and nothing
+in the finding gave a second party anything to disagree with. It was the only
+tier in this engine with no way to be wrong.
+
+`posture/logic-claims.js` adds three offline lenses that can refute one —
+citation (the file exists and the line is inside it), quotation (the quoted
+snippet is at the cited line), corroboration (a "no authentication" claim
+against a handler that plainly authenticates). Verdicts go through the existing
+producer/verifier separation, so a lens can never vote on a claim it produced;
+that is why the lenses are deterministic code and not another prompt. Refuted
+claims are quarantined, never deleted and never severity-touched.
+
+### A comparison harness that ships no opinion about who the competition is
+
+`posture/comparison.js` + `scripts/comparison.mjs` score this engine
+head-to-head against participants **the operator supplies**. The repository
+ships the harness and the answer key and names no tool — a test enforces that.
+
+Two properties are the entire module. Every rate is computed over the
+**intersection** of corpus entries *all* participants completed, because a tool
+that crashed on the forty hardest entries and was scored over the remaining
+hundred and seventy looks like it beat one that completed everything, and the
+difference is invisible in the output. And an entry a participant could not run
+is **unscored**, never counted as a miss: counting a crash as a false negative
+penalises a tool for a harness problem, counting it as a pass rewards it for
+one. Matching is CWE-only so nobody is scored on this engine's vocabulary.
+
+No comparison figures are published in this repository. Running other vendors'
+tools and publishing the numbers is the operator's call, not the harness's.
+
+### The suppression pragma never worked
+
+Found while suppressing a false positive in this release's own new code.
+`// agentic-security-ignore: <rule-id>` is documented in `CLAUDE.md` and
+`pr-comment.js` tells every reviewer to use it — and **nothing implemented it**.
+It has been advertised and inert. A dead suppression mechanism is worse than an
+absent one: the developer writes the pragma, sees the finding again, and
+concludes the scanner is noisy rather than that the pragma is dead.
+
+Now implemented in `engine.js`, applied after dedupe and after every cross-file
+pass so it covers a finding whichever analysis produced it. Line-scoped, matched
+against the finding's id / vuln / CWE / family, and **logged** to the same
+ledger custom rules use so `--include-suppressed` can show it. Every test
+carries a positive control — the same file without the pragma must still
+produce the finding, or "0 findings" would prove the suppression works and
+equally prove the detector stopped firing.
+
 ## 0.133.0 — two ways findings could be silently deleted, both closed
 
 Four rounds of adversarial premortem against this repository's own artifacts.

package/bin/agentic-security-audit.js

@@ -23,6 +23,7 @@ import * as fs from 'node:fs';
 import * as path from 'node:path';
 import { verifyAuditLog } from '../src/mcp/audit.js';
 
+import { statePath } from '../src/posture/state-dir.js';
 function args() {
   const a = process.argv.slice(2);
   const sub = a[0] || 'review';
@@ -48,7 +49,7 @@ function _parseDuration(s) {
   return n * u;
 }
 
-function _logPath(root) { return path.join(root, '.agentic-security', 'mcp-audit.log'); }
+function _logPath(root) { return statePath(root, 'mcp-audit.log'); }
 
 function _readEntries(root) {
   const fp = _logPath(root);

package/bin/agentic-security-consistency.js

@@ -26,6 +26,7 @@ import * as fs from 'node:fs';
 import * as path from 'node:path';
 import { measureConsistency, summarize } from '../src/llm-validator/consistency.js';
 
+import { statePath } from '../src/posture/state-dir.js';
 function args() {
   const a = process.argv.slice(2);
   const out = { trials: 5, top: 5, json: false, root: process.cwd() };
@@ -40,7 +41,7 @@ function args() {
 
 async function main() {
   const opts = args();
-  const scanFile = path.join(opts.root, '.agentic-security', 'last-scan.json');
+  const scanFile = statePath(opts.root, 'last-scan.json');
   if (!fs.existsSync(scanFile)) {
     console.error(`no last-scan.json at ${scanFile} — run a scan first`);
     process.exit(2);