@clear-capabilities/agentic-security-scanner 0.150.2 → 0.151.1

package/CHANGELOG.md

@@ -9,6 +9,141 @@
 > make the history less accurate, not more.
 
 
+## 0.151.1 - Adversarial premortem on the SARD benchmarking subsystem: 12 real findings, 12 real fixes
+
+A structured adversarial premortem ("assume this subsystem has completely failed six months
+from now — work backwards to why") was run against 0.151.0's SARD benchmarking work, producing
+19 findings across data/leakage, methodology, security, MLOps, and governance. All 12 findings
+judged actionable were fixed and verified with real runs — none deferred, none papered over. Full
+account, including one deliberate self-correction, in `bench/sard/IMPLEMENTATION_STATUS.md`'s
+"Adversarial premortem + full remediation pass" section.
+
+**Data & leakage:**
+- Train/dev/test splits (`bench/sard/splits/*.json`) were computed and self-verified but never
+  consumed by scoring — every prior macro-F1 number was measured over the full corpus, not a
+  held-out split. `bench-realworld.js` now has a real `--split train|dev|test` flag filtering
+  both ground truth and actual findings symmetrically (8 new tests).
+- `leakage-audit.mjs`'s own `CWE` term couldn't match Juliet's real fused naming convention
+  (`CWE89`, no separator) — `'CWE89'.match(/\bCWE\b/i)` returns null. Added a dedicated
+  `cwe-number` check independent of the plain term (2 new tests).
+- PHP's identifier neutralization renamed exactly one hardcoded variable (`$tainted`) — an ad hoc
+  point-fix, not a designed protection. Generalized to the same hash-based rule pattern Java/C#
+  already use.
+
+**Methodology:**
+- The C# "26/32 CWE families, root-caused, not fixed" claim was re-investigated and found wrong
+  as stated: catalog entries already exist for several of the "uncovered" families. A first
+  re-investigation pass concluded the interprocedural taint engine was broken — before writing
+  that down, a controlled test matrix caught that the "evidence" for it was a coincidental
+  structural-detector hit, not real taint (a lowercase `request` parameter never matched the
+  catalog's case-sensitive source entry, in either the passing or failing fixture). With the
+  casing fixed, interprocedural taint propagation works correctly for this shape. The real
+  root cause of the C# corpus gap remains genuinely open. 3 new regression tests.
+- Added the previously-missing adversarial (must-flip) mutation side to `mutate.mjs` —
+  `ADVERSARIAL_SOURCE_LITERALIZATION` replaces a tainted source's initializer with a hardcoded
+  literal and scores the opposite polarity from the existing metamorphic mutators. Found a real
+  detector precision gap on its first use: a structural Java SQL-injection detector fires on
+  "string built via concatenation" regardless of whether the value is genuinely tainted, while
+  the real taint engine correctly stays silent (5 new tests; the detector gap itself is
+  disclosed, not fixed here).
+- `compare-baseline.mjs`'s flat 2-percentage-point regression tolerance either hid a real
+  regression on a high-support metric or failed on ordinary noise for a low-support one — exactly
+  the double-bind measured live this session (an unrelated holdout app's F1 moved between two
+  identical runs). Added `adaptiveTolerance()`, widening the band for low-support metrics only,
+  one-directionally (10 new tests, including the first automated end-to-end CLI proof for this
+  script in either direction).
+- Headline metrics (100% Fully Verified Fix Rate, 100% Semantic Robustness Rate) now carry
+  explicit scope qualifiers in the implementation ledger, so neither can be quoted out of
+  context as a whole-corpus claim.
+
+**Security:**
+- PRD §56 "LLM Isolation" had zero implementation and zero test anywhere in this repo. Added
+  `test/sard-llm-isolation.test.js` against the real LLM-validator prompt builder: an
+  already-neutralized input produces a leakage-clean prompt; the scanner's own CWE classification
+  is legitimate content, not a leak; and, disclosed rather than hidden, the prompt builder has no
+  independent redaction of its own — isolation is entirely inherited from upstream neutralization.
+
+**MLOps/CI:**
+- `realworld-bench`'s `needs: synthetic-bench` meant a real, unrelated regression in
+  `synthetic-bench` (91.3%→85.9% F1, first observed 2026-09-08) silently skipped
+  `realworld-bench` for 5+ consecutive scheduled runs with zero alert. Removed the dependency —
+  both are already independently-tiered informational jobs.
+- `compare-baseline.mjs` was local-only by design (no committed scores), leaving no CI-enforced
+  regression gate at all. Wired a GitHub Actions cache (never a git commit) into `sard-blind-smoke`
+  so a genuine regression fails the job for real, without touching the "no scores committed"
+  policy.
+- The external-holdout generalization gate could never fail: every existing curated app's ground
+  truth was bootstrapped from a past scanner run, not built independently. Added
+  `bench/holdout-independent/tinymart/` — a small, hand-written app whose ground truth was
+  authored from its own source before the scanner ever ran against it. Proved the gate has real
+  teeth end to end: clean baseline, deliberately removed a vulnerability (gate correctly failed,
+  naming the app and the exact regression), reverted (gate returned to clean, numbers matched
+  byte-for-byte). 5 new tests.
+
+No detection-engine changes in this release — every fix above is to the benchmarking/CI
+infrastructure itself, not to `scanner/src/`.
+
+## 0.151.0 - SARD/Juliet benchmarking: leakage-clean scoring, macro-F1, mutation testing, and fix verification (SARD_AGENTIC_SECURITY_PRD.md)
+
+Builds a full benchmarking subsystem against NIST SARD's Juliet (Java/C#) and PHP Vulnerability
+Test Suite corpora, extending the pre-existing `bench-realworld.js` harness rather than
+duplicating it. New: `bench/sard/` (dataset lockfile, PHP ingestion, leakage audit, macro-F1 +
+per-CWE + CWE-confusion-matrix + localization-accuracy scoring, structural train/dev/test
+splitting with a duplicate-crossing audit, semantic mutation testing, and independent fix
+verification), plus a `--cwe` targeted/smoke-run flag and a new CI job (`sard-blind-smoke`)
+measuring genuine leakage-clean detection quality separately from the pre-existing
+non-blind `sard-juliet-java` job (which tracks a different thing: whether the corpus-shape-aware
+fallback code still works, not detection quality).
+
+**Real, measured numbers** (all reproduced, none quoted without a fresh run — see
+`bench/sard/IMPLEMENTATION_STATUS.md` for the full ledger and every command that produced them):
+Java macro-F1 45.6% (P=66.5%, R=36.9%, leakage-clean, vulnerability-level scoring); C# macro-F1
+8.4% (26/32 CWE families found to have zero detector coverage at all — a real, root-caused gap,
+not yet fixed); Semantic Robustness Rate 100% (84/84 real mutations survived, sanity-checked
+against a deliberately-safe negative control); Fully Verified Fix Rate 100% (32/32 real Java
+weak-hash fixes); leakage reduced from 907,516 to 3 residual hits (99.9997%).
+
+**Real engine fixes found via this benchmarking work** (general capability improvements, not
+SARD-specific shortcuts — each verified with `bench:layer-recall`/`test:dataflow` before and
+after):
+- PHP: `$_SESSION`/`$_ENV` added as taint sources (previously entirely absent from the catalog),
+  `mysql_query()` given a real taint-dataflow sink (previously only a same-line structural
+  regex, missing the dominant assign-then-call-later shape). `bench:layer-recall` PHP taint
+  recall moved 12→13; baseline re-recorded (a real improvement, not silently left stale — this
+  gate compares for equality, not a floor, precisely to prevent that).
+- Java: `BufferedReader.readLine()`/`Console.readLine()` and `ResultSet.getString`/`getObject`
+  added as taint sources, root-caused from real, evidence-based error-cluster analysis (grouping
+  false negatives by Juliet's own filename descriptor) rather than guessed.
+- `deterministic-fix.js`'s weak-hash rule had a real Java coverage gap: its `applies()` gate
+  already matched Java CWE-327/328/916 findings, but `transform()` had no Java branch at all, so
+  every Java weak-hash finding silently produced no fix. Added
+  `MessageDigest.getInstance("MD5"|"SHA1")` → `"SHA-256"`.
+- `bench-realworld.js` and `leakage-audit.mjs` both had an unconditional top-level `main()` (and,
+  in `bench-realworld.js`'s case, a separate top-level usage-check) with no `import.meta.url`
+  guard — importing either file's helper functions as a module silently ran the ENTIRE CLI
+  against the *importer's* own `process.argv`. Found the hard way when a new mutation-testing
+  script's own `--app`/`--cwe` flags happened to also be valid `bench-realworld.js` flags,
+  triggering an unwanted ~160s benchmark run as a side effect of a function import. Both files
+  now guard their top-level side effects the same way every other multi-purpose script in
+  `bench/sard/scripts/` already did.
+- Root `CLAUDE.md` described `bench:layer-recall:check` as "a FLOOR, not an equality check...
+  silent on a rise" — stale documentation of a design the gate itself had already moved past
+  (its own code comment says "PRD F12.2 — compares for EQUALITY, not against a floor"). Fixed.
+- `src/lineage/source-registry.js`'s `NO_PROVENANCE_OVERRIDES` table (Data Flow Explorer) had no
+  entries for the new `java-io-readline`/`java-resultset-getstring`/`java-resultset-getobject`
+  catalog sources, or for the pre-existing `php-session`/`php-env` sources — all five failed the
+  registry's own completeness guard (`source-registry.test.js`), which fails loudly rather than
+  silently producing an uncategorized graph node. Added (`user-input`/`database-read`/
+  `http-cookie`/`env-value` respectively); pinned coverage counts re-measured and updated
+  (185 total source entries, 87 `candidate`). Two `bench/sard/scripts/` helpers
+  (`analyze-errors.mjs`, `score-php.mjs`) were also wired into `scanner/package.json` after
+  `no-orphan-scripts.test.js` flagged them as unreachable from any npm script.
+
+14 new automated tests (`java-taint-flow.test.js` +3, `deterministic-fix.test.js` +3, new
+`sard-leakage-pipeline.test.js` +8 covering PRD §62's literal leakage-injection list and a
+synthetic-fixture integration test through the real neutralization pipeline).
+
+
 
 ## 0.150.2 - Two adversarial-review passes on 0.150.0's Ollama support: redaction, disclosure, and robustness fixes
 

package/dist/9560.index.js

@@ -2864,6 +2864,7 @@ const _internals = { CACHE_DIR, CVE_RE, _stalenessTier };
 
 const JS_EXT = /\.(?:js|jsx|ts|tsx|mjs|cjs)$/i;
 const PY_EXT = /\.py$/i;
+const JAVA_EXT = /\.java$/i;
 
 // Each rule gates on the finding's cwe/family, then rewrites the whole-file
 // content. transform() returns the new content, or null when nothing changed
@@ -2875,12 +2876,22 @@ const RULES = [
     // swapping them all is safe; the verifier confirms the weak-hash finding is
     // gone and nothing worse appeared.
     applies: (f) => /CWE-(?:327|328|916)/.test(f.cwe || '') || /weak.?hash/i.test(f.family || ''),
+    // SARD_AGENTIC_SECURITY_PRD.md Phase 8 bench work found this rule's
+    // `applies()` gate matched Java findings (CWE-327/328) by cwe/family, but
+    // `transform()` had no Java branch at all — every Java weak-hash finding
+    // silently produced `null` (no fix), a coverage gap invisible from the
+    // gate alone. `MessageDigest.getInstance("MD5"|"SHA1"|"SHA-1")` ->
+    // `"SHA-256"` is the same class of context-independent literal swap as
+    // the existing JS/Python branches (the algorithm name is a string
+    // literal, not something requiring surrounding-code understanding).
     transform: (content, file) => {
       let out = content;
       if (JS_EXT.test(file)) {
         out = out.replace(/(\bcreateHash\s*\(\s*['"`])(?:md5|sha1)(['"`])/gi, '$1sha256$2');
       } else if (PY_EXT.test(file)) {
         out = out.replace(/\bhashlib\.(?:md5|sha1)\s*\(/g, 'hashlib.sha256(');
+      } else if (JAVA_EXT.test(file)) {
+        out = out.replace(/(\bMessageDigest\.getInstance\s*\(\s*")(?:MD5|SHA-?1)(")/gi, '$1SHA-256$2');
       }
       return out !== content ? out : null;
     },