@clear-capabilities/agentic-security-scanner 0.145.0 → 0.147.5

package/CHANGELOG.md

@@ -11,6 +11,272 @@
 
 
 
+## 0.147.5 — Fix: `bench:provenance:check`'s cold-memory sample size was unreliable on GitHub Actions
+
+`v0.147.4`'s tag was pushed but its hosted release-gate run failed before
+reaching `npm publish` — the `dataflow-watch` fix held (the full test suite
+passed cleanly), but `bench/provenance/runner.mjs`'s `bench:provenance:check`
+failed: "only 8/20 iterations produced a usable sample (need >= 10)" for
+`cold.memOverheadRatio`. This is the SAME failure `v0.147.3`'s release run
+also hit (7/20 that time) — two consecutive, reproducible failures on
+GitHub's runners specifically, while this bench passes cleanly (~20/20)
+every time locally, so this is a real platform gap, not one-off noise.
+
+Root cause: `cold.memOverheadRatio` is a per-iteration ratio against the
+without-provenance arm's own peak-RSS delta, and `measure()` skips the
+ratio entirely for an iteration where that delta is `0`
+(`if (without.rssDeltaBytes > 0) { ... }`) — the without-provenance scan is
+only ~40-60ms, short enough that the 10ms-interval peak-RSS poller can
+genuinely observe no net growth for it, especially under a shared runner's
+different GC pacing. `BASELINE.json` itself already records this as
+expected (`memOverheadRatio.n: 19`, one dropped iteration out of 20 even on
+the machine that captured it) — GitHub Actions just drops far more of them
+(a ~35-40% survival rate against RELIABLE_N=10 at N=20 leaves no margin).
+
+Raises `N` from 20 to 60 in `bench/provenance/runner.mjs` — the same
+already-established remedy this exact file used once before for a related
+small-sample fragility (doubling 10→20), extended further given the new,
+worse-than-expected survival rate. `RELIABLE_N` (10) and the regression
+thresholds are unchanged; `BASELINE.json`'s reference values were not
+touched (the measured overhead is well within budget — this was purely a
+sample-count reliability failure, not a regression). Verified locally:
+passes with real margin (cold memory p95 11.42x vs a 24.38x limit), at a
+real but bounded cost — this one bench step now takes ~2:23 instead of
+~40-60s, a release-time-only cost. `v0.147.4` was never published to any
+registry (confirmed via `npm view`). Carries the identical fixes from
+0.147.1-0.147.4 forward.
+
+## 0.147.4 — Fix: close a real fs.watch startup race in `dataflow watch` (test-only; no product behavior change)
+
+`v0.147.3`'s tag was pushed but its hosted release-gate run failed before
+reaching `npm publish` — again `cli/dataflow-watch-2`, but this time raising
+the wait timeout (0.147.3's own fix) did NOT help: it timed out at the
+*full* 90000ms budget with zero rescan output, which the real rescan
+pipeline (~1-2s once triggered) rules out as "merely slow." Root cause: `bin/
+agentic-security.js`'s `cmdDataflowWatch` prints its "watching ... Ctrl-C to
+stop" banner — the exact string these tests poll for as the "watcher is
+live" signal — *before* calling `watchProject()`, which is where
+`fs.watch()` actually arms its (possibly recursive, whole-tree-walking on
+Linux) inotify descriptors. A test file-write landing in that gap produces
+an event inotify never delivers at all — a permanent miss, not a delay —
+which a longer timeout cannot fix by construction.
+
+Fixed with a 1000ms settle delay in the test, inserted between seeing the
+startup banner and writing the trigger file, in both `dataflow-watch-1` and
+`dataflow-watch-2` (`test/cli/dataflow-watch.test.js`). No product code
+changed — `cmdDataflowWatch`'s banner-before-watchProject ordering is real,
+pre-existing behavior, unrelated to anything in 0.147.1-0.147.3, and left
+alone here rather than reordered under release time pressure without full
+confidence in every caller depending on today's ordering. `v0.147.3` was
+never published to any registry (confirmed via `npm view`). Carries the
+identical fixes from 0.147.1-0.147.3 forward.
+
+## 0.147.3 — CI fix: raise dataflow-watch's live-subprocess wait timeout (no functional change from 0.147.2)
+
+Same story as 0.147.2, different test: `v0.147.2`'s hosted release-gate run
+also failed before reaching `npm publish` — `test/cli/dataflow-watch.test.js`'s
+`cli/dataflow-watch-2` polls a real subprocess's live stderr for a
+`DRIFT POLICY VIOLATION` line after triggering a file-watch-driven rescan,
+with a 30000ms budget. That step completes in ~1.4s on a normal machine (a
+~20x margin) but still timed out on this run — this repo's watch mode uses
+Node's native `fs.watch(..., { recursive: true })`
+(`src/posture/watch-mode.js`), documented as slower/less reliable under
+Linux CI runners' inotify handling than on a dev machine. `v0.147.2` was
+never published to any registry (confirmed via `npm view`).
+
+Raises the wait budget from 30000ms to a shared `WAIT_TIMEOUT_MS = 90000`
+across both real-subprocess tests in that file (the ones that depend on the
+file-watch path — an unrelated fast-fail test in the same file was left
+untouched). No product code changed; this carries the identical fixes from
+0.147.1 and 0.147.2 forward.
+
+## 0.147.2 — CI fix: raise the headless-Chrome render timeout (no functional change from 0.147.1)
+
+`v0.147.1`'s tag was pushed but its hosted release-gate run failed before
+ever reaching `npm publish` — `scripts/export-image.mjs`'s `exportPng` test
+hit its own hardcoded 15000ms Chrome render timeout on a slower draw of the
+GitHub Actions runner (the same render took 7983ms on the immediately
+preceding v0.147.0 release run — a ~2x runner-speed swing well within
+normal CI variance, not a regression). `v0.147.1` was never published to
+any registry; this release carries the identical `explore`/`dataflow
+export --format html` fix described below under 0.147.1, plus this one
+timeout change: `RENDER_TIMEOUT_MS` default raised from 15000ms to 30000ms
+in `scripts/export-image.mjs`, giving roughly 4x headroom over the observed
+good-case render time. No other code changed.
+
+## 0.147.1 — Fix: `explore` and `dataflow export --format html` were broken for every real npm/npx install
+
+Both commands located the Data Flow Explorer frontend (`frontend/`, a
+repo-root sibling of `scanner/` in this monorepo) via a fixed relative path
+computed from each module's own `import.meta.url`
+(`path.resolve(HERE, '../../../frontend')` in `static-assets.js`,
+`'../../frontend'` in `generate-html-report.mjs`). That math was correct
+only for the unbundled dev checkout. Once `npm run build` (ncc) split those
+modules into their own `dist/*.index.js` chunks, `import.meta.url` inside
+them reflected `dist/`'s own, shallower location, and the same fixed
+relative math resolved *outside the installed package entirely* — and
+`frontend/` was never part of the published npm tarball to begin with. The
+practical effect: `agentic-security explore .` 404'd on `GET /` for every
+real `npx`/`npm install` user (`{"error":"not found"}`, never an
+index.html), and `dataflow export . --format html` failed outright
+(`ENOENT: no such file or directory, scandir '.../frontend/styles'`). Both
+only ever worked when run from a full monorepo clone.
+
+Fixed by shipping the frontend's servable files inside the published
+package: `scanner/scripts/copy-frontend.mjs` (new) copies `frontend/`'s
+allowlisted files (`index.html`, `src/**/*.js`, top-level `styles/*.css` —
+the exact same allowlist `static-assets.js` already enforced for serving)
+into `scanner/dist/frontend/` as part of `npm run build`; a new shared
+resolver, `scanner/src/shared/frontend-root.js`, replaces both hardcoded
+relative paths with a search-upward strategy that finds the frontend
+correctly regardless of bundling depth. Verified by packing a real tarball
+(`npm pack`) and installing it into an isolated directory with no monorepo
+present: `explore`'s `GET /` now returns 200 with the real `index.html`
+(previously 404), static JS/CSS assets and the token-authenticated
+`/api/v1/graph` endpoint both 200, and `dataflow export --format html`
+exits 0 and writes a real self-contained report (previously exited 2).
+`dpia`/`ropa` export formats were already unaffected (a separate,
+already-self-contained code path) and are unchanged.
+
+## 0.147.0 — Documentation overhaul: a cohesive assurance-platform story, verified against the shipped code
+
+The 0.144.0 Assurance Hardening release shipped a real 3-state ship verdict,
+a model-egress policy gate, risk-dollar scenario disclosure, and per-artifact
+retention/encryption — but the docs describing it, and the README's whole
+front-door story, hadn't caught up. This release rewrites both, built
+exclusively from real, re-verified source rather than the original
+documentation brief's assumptions (which turned out to contain several
+fabricated CLI flags and vocabularies — a `scan --assurance` flag that
+doesn't exist, `--assurance` is `ci`-only; a fictional `--explain-health`
+flag; nine invented verify-verdict strings; a fictional unified compliance
+state model where three separate, differently-cased vocabularies actually
+exist).
+
+**README** rewritten around a 5-capability story (Find It / Prove It / Fix
+It Safely / Govern It / Explain It), a corrected 5-minute quickstart, the
+real 3-state ship verdict as a running example, and persona-based
+navigation (Developer / AppSec / Privacy / Compliance / Platform
+Engineering).
+
+**16 new docs**: `docs/walkthroughs/{assurance-modes,scan-health,
+finding-evidence,verified-remediation,privacy-data-flow,model-egress}.md`;
+`docs/architecture/finding-lifecycle.md`; `docs/concepts.md`
+(evidence-before-severity, deterministic-vs-model-assisted);
+`docs/governance/state-and-retention.md`; `docs/guides/risk-dollars.md`;
+`docs/examples/README.md` (a 13-entry gallery); `docs/reference/
+{output-schema,glossary}.md`; `docs/troubleshooting/scan-health.md`; and two
+real, captured Data Flow Explorer browser screenshots at `docs/assets/`.
+
+**A new doc-testing tool**, `scripts/verify-doc-examples.mjs`: parses the
+real CLI dispatch table out of `bin/agentic-security.js` at runtime (so it
+can't drift out of sync with the CLI the way a hand-maintained list would)
+to catch fabricated commands and the specific `<subcommand> --help`
+landmine (passing `--help` to a subcommand silently runs the real command
+instead of printing help — only bare `agentic-security help` is valid),
+plus JSON-block and Mermaid-syntax sanity checks. Deliberately not wired
+into any gate — a standalone `npm run docs:verify-examples`.
+
+**Existing docs corrected**, not just extended: `docs/guides/
+{quickstart,scanning,compliance,ci-setup,finding-provenance}.md`,
+`docs/reference/{cli,configuration}.md`, `docs/ARCHITECTURE.md`,
+`docs/README.md`, and several `commands/`/`skills/` files carried stale
+claims — the most consequential being that `scan` was documented as
+defaulting finding-provenance ON, when it actually defaults OFF (`ci`
+defaults it ON; this asymmetry exists because plain provenance resolution
+cost a measured 4.5s→45s regression on a 207-file tree, the same
+measurement behind this project's own TTFF benchmark).
+
+## 0.146.0 — Data Flow Explorer: trace sensitive data through your architecture, not just through one finding
+
+Every prior release answers "is this line of code dangerous?" This one adds a
+different question: "where does this piece of sensitive data actually go, and
+what protects it along the way?" `agentic-security dataflow` builds a
+`DataFlowGraph` from a real scan — sources, sinks, the data elements that
+flow between them, and a verdict per hop on transit/at-rest/handling
+protection — and a new local, read-only, loopback-only server
+(`agentic-security explore`) serves it to a clickable UI. Six milestones,
+built in dependency order over this release cycle:
+
+**The graph itself (Milestones 0-2).** A stable-ID'd, schema-versioned graph
+contract; an interprocedural field-identity taint engine purpose-built for
+this graph (deliberately separate from the SAST engine's own taint
+analysis — different state shape, different precision goals) with
+context-sensitive summaries and a compact path-provenance DAG that tracks
+*why* a field reached a sink, not just *that* it did; source/sink/
+transformation registries reclassifying this project's existing detection
+catalogs into the graph's own vocabulary; and protection-verdict analyzers
+for transit (TLS presence and validity), at-rest (encryption-before-store),
+and handling (masking/hashing/tokenization — never treated as synonyms).
+
+**A local server and reference UI (Milestone 3).** `agentic-security explore`
+— the first HTTP server anywhere in this codebase (the MCP and LSP servers
+are both stdio) — serves the signed, already-scanned graph read-only,
+loopback-only, to a zero-build-step frontend prototype with architecture,
+privacy-lifecycle, and evidence-trace views.
+
+**Export and decision-intelligence surfaces (Milestone 4).** JSON/CSV/HTML/
+PNG/PDF/SVG export; a Regulatory Obligation Overlay mapping graph facts
+against bundled compliance frameworks (never claiming "compliant" from a
+single mapping); an Executive Risk Story mode ranking flows on nine
+transparent, disclosed factors — never a blended, uncalibrated score; a
+Data-Flow Time Machine (`dataflow diff`) comparing two scans with real
+drift-policy alerting on newly-introduced disclosures; and Third-Party and
+Cross-Border Intelligence, keeping legal/jurisdiction facts honestly
+`unknown` rather than inferred from brand recognition.
+
+**Simulation, corroboration, and cross-repo linkage (Milestone 5).** A
+What-If Architecture Simulator (`dataflow scenario apply`) that clones and
+overrides a graph to test a hypothetical fix, never mutating the real scan,
+with every simulated verdict marked `HYPOTHETICAL` and `assumed`-graded so
+it can never be mistaken for a real one; Blast-Radius Impact Assessment and
+a Remediation Command Center with a hash-chained ledger enforcing that
+closing an item requires either a clean rescan or an explicit, permitted
+manual attestation — and that a later regression reopens it automatically;
+a Runtime-Corroborated Digital Twin (`dataflow observations import`) that
+layers operator-supplied runtime telemetry onto the static graph through a
+closed-world contract that rejects, not merely redacts, any field capable
+of carrying a payload value; per-language coverage-tier disclosure so a
+partially-supported language is labeled as such rather than silently
+under-reported; and `federate declare` — an operator-declared edge between
+two independently-scanned repositories' own graphs, with each graph staying
+its own separate, unmodified artifact.
+
+**New top-level commands:** `explore`, `dataflow`, `governance`,
+`remediation`, `federate` — the latter three write operator-declared
+config through the same reviewable contract (preview, version guard,
+backup, audit event) established for exactly this class of write.
+
+**On rigor, again.** As with 0.145.0's own provenance work, a dedicated
+final-review pass — after every task-level review and a green test
+suite — found real Blocking or Important bugs in nearly every one of
+these sub-projects: a remediation item that could be marked verified from
+a snapshot chronologically *older* than the incident it was supposed to
+fix; a runtime-observation payload-refusal grammar that closed single-value
+smuggling but missed the same payload split across array elements; a
+cross-repo link's own self-consistency digest compared against the wrong
+half of a redacted export, so an untampered file permanently read as
+tampered; and — found only in this release's own post-completion
+re-audit, prompted by checking every named acceptance criterion's text
+against the actual shipped code rather than trusting a summary — one
+acceptance criterion (`AC-26`, on the What-If Simulator) that had never
+been checked against its own literal wording by any prior document, and
+was quietly unmet until this audit closed it. Every one of these was
+found, fixed, and independently re-verified — including personally
+reproducing the bug and its fix live, not just re-reading a diff — before
+this release.
+
+**Disclosed, not built.** Full graph-merge across repositories (only
+declared, single-edge cross-repo links ship); synthetic node/edge
+insertion for the What-If Simulator (every node/edge today derives from a
+real registry decision or call site — inventing one has no precedent yet);
+automatic destination/schema correlation for cross-repo edges (needs
+prerequisite work — schema-derived edges, fuller destination resolution —
+that isn't built); an HTML-table stress-test pass for the largest graphs;
+config-declared (as opposed to runtime-observed) corroboration for the
+Digital Twin, folded into already-scoped backlog rather than duplicated.
+None of these block the acceptance criteria this release's milestones are
+gated on.
+
 ## 0.145.0 — Finding Provenance ships, and two audits find what the first one missed
 
 Every finding now carries a `findingProvenance` record answering "when did this