@clear-capabilities/agentic-security-scanner 0.151.0 → 0.151.2

package/CHANGELOG.md

@@ -9,6 +9,153 @@
 > make the history less accurate, not more.
 
 
+## 0.151.2 - Real C# SAST accuracy improvements: interprocedural call-graph fixes + new detector coverage
+
+Follow-up to 0.151.1's SARD work: a direct effort to improve the scanner's actual code-scanning
+accuracy (not benchmark-shape gaming), using self-authored probe fixtures rather than reading the
+deny-listed SARD corpus. Found and fixed two general engine bugs plus several detector gaps, all
+verified with real self-authored fixtures and zero regressions. Full account, including the
+benchmark-scoring limitation that makes the bigger fix's true impact invisible on the SARD
+number specifically, in `bench/sard/IMPLEMENTATION_STATUS.md`'s "Real accuracy improvement pass"
+section.
+
+**Engine fixes (general, not SARD-specific):**
+- **C# `this.method(...)` same-instance calls never resolved in the call graph.** A same-class
+  helper call written as `this.badSink(data)` silently blocked all interprocedural taint into
+  the callee, for every sink inside it. `parser-cs.js` lowers this to the flat callee string
+  `"this.foo"`, but the target method's registered name is bare (`"foo"`) — the opposite
+  asymmetry from Java's documented class-qualified-name gap, and unhandled by the existing
+  bare-tail fallback (which explicitly skips any callee containing a dot). Fixed in
+  `callgraph.js` with a new resolution branch, unconditional (not a guess — `this.` unambiguously
+  means "a member of the current instance"). 4 new tests.
+- **C# cross-class calls never resolved at all — same file or not.** `parser-cs.js`'s function
+  IDs never recorded which class a method belongs to, so `callgraph.js`'s cross-class index
+  (which resolves `new Helper().Sink(x)`, `Helper h = new Helper(); h.Sink(x)`, and bare
+  `Helper.Sink(x)`) was permanently empty for C#. Fixed by having the parser track each method's
+  enclosing class (matching Java's/JS's existing `"ClassName.method"` convention) plus a
+  companion local variable-type-inference pass for the `Helper h = new Helper(); h.Sink(x)`
+  shape (refuses to guess when a variable holds more than one distinct constructed type in the
+  same function). Also closed a related gap found along the way: a bare
+  `new Helper().Sink(data);` statement with no assignment previously dropped entirely. 8 new
+  tests, zero regressions across the full test suite.
+- Fixing the class-qualified naming above required also hardening the `this.`-call fix from the
+  same session to fall back to the existing bare-tail index, so the two fixes compose correctly
+  rather than one silently breaking the other.
+
+**Detector coverage:**
+- **CWE-94 code injection: `CSharpCodeProvider.CompileAssemblyFromSource` had zero coverage.**
+  The only existing C# code-injection patterns were Roslyn's `CSharpScript` (2014+) and
+  `DataTable.Compute` — missing `System.CodeDom.Compiler`'s `CSharpCodeProvider`, the
+  historically standard .NET dynamic-compile API, predating Roslyn scripting by a decade and
+  still the most commonly documented one. Largest single measured improvement of this release.
+- **CWE-78 command injection: `Process.Start` with a non-shell-literal filename was explicitly
+  out of scope.** Missed a real, well-documented .NET Framework gotcha:
+  `ProcessStartInfo.UseShellExecute` defaults to `true` on .NET Framework (only .NET Core/5+
+  default it to `false`), so `Process.Start("ping", tainted)` goes through the OS shell
+  regardless of the filename. Added as a companion sink at `high` (not `critical`) severity to
+  reflect the framework-version caveat, alongside the existing shell-literal-gated entry.
+- **LDAP injection attribute matching was a hardcoded 10-item enum** (`uid`/`cn`/`mail`/...).
+  Real LDAP/Active Directory schemas define far more attributes than any fixed list can
+  enumerate, including custom extensions — a real, general precision/recall bug for production
+  LDAP code, not a benchmark-specific one. Widened to a general LDAP-attribute-name shape.
+
+**Self-inflicted bug caught by this project's own gate:**
+- The new C# class-boundary-detection regex was a genuine, timing-confirmed ReDoS (30,000
+  non-matching modifier-keyword repeats took 5+ real seconds) — caught by `bench:self-scan:check`
+  exactly as designed. A first fix (bounding the repetition count) was measurably linear but
+  still tripped the static ReDoS detector's "nested quantifier" heuristic; unrolled into explicit
+  non-nested optional groups to satisfy both real safety and the detector.
+
+**What this release honestly does NOT close:**
+- The cross-class fix is proven correct via direct fixtures and unit tests, but the real SARD
+  corpus's macro-F1 for C# barely moved (8.4% → 10.8%, almost entirely from the CWE-94 fix
+  above) — traced to a benchmark-scoring limitation, not a detection gap: `bench-realworld.js`'s
+  scorer only credits a finding located inside the *expected method's own line range*, and an
+  interprocedural finding is, by definition, located in the callee it flows into. The scanner is
+  now more accurate; the benchmark's strict per-method scoring can't see it yet. Scoped as a
+  concrete next step, not fixed here.
+- Java's own version of the `this.`-call bug (`parser-java.js` lowers it to the literal string
+  `"unknown"` before it ever reaches the call graph) is unfixed — a separate, parser-level gap.
+- Weak-crypto/weak-RNG families (`new Random()`, `MD5`/`SHA1` for passwords) were investigated
+  and deliberately left alone: the existing precision heuristic (requires a security-suggestive
+  identifier nearby) is real and general, and loosening it just to score higher on this one
+  benchmark would trade away real-world precision — the exact shortcut this work was asked not
+  to take.
+
+## 0.151.1 - Adversarial premortem on the SARD benchmarking subsystem: 12 real findings, 12 real fixes
+
+A structured adversarial premortem ("assume this subsystem has completely failed six months
+from now — work backwards to why") was run against 0.151.0's SARD benchmarking work, producing
+19 findings across data/leakage, methodology, security, MLOps, and governance. All 12 findings
+judged actionable were fixed and verified with real runs — none deferred, none papered over. Full
+account, including one deliberate self-correction, in `bench/sard/IMPLEMENTATION_STATUS.md`'s
+"Adversarial premortem + full remediation pass" section.
+
+**Data & leakage:**
+- Train/dev/test splits (`bench/sard/splits/*.json`) were computed and self-verified but never
+  consumed by scoring — every prior macro-F1 number was measured over the full corpus, not a
+  held-out split. `bench-realworld.js` now has a real `--split train|dev|test` flag filtering
+  both ground truth and actual findings symmetrically (8 new tests).
+- `leakage-audit.mjs`'s own `CWE` term couldn't match Juliet's real fused naming convention
+  (`CWE89`, no separator) — `'CWE89'.match(/\bCWE\b/i)` returns null. Added a dedicated
+  `cwe-number` check independent of the plain term (2 new tests).
+- PHP's identifier neutralization renamed exactly one hardcoded variable (`$tainted`) — an ad hoc
+  point-fix, not a designed protection. Generalized to the same hash-based rule pattern Java/C#
+  already use.
+
+**Methodology:**
+- The C# "26/32 CWE families, root-caused, not fixed" claim was re-investigated and found wrong
+  as stated: catalog entries already exist for several of the "uncovered" families. A first
+  re-investigation pass concluded the interprocedural taint engine was broken — before writing
+  that down, a controlled test matrix caught that the "evidence" for it was a coincidental
+  structural-detector hit, not real taint (a lowercase `request` parameter never matched the
+  catalog's case-sensitive source entry, in either the passing or failing fixture). With the
+  casing fixed, interprocedural taint propagation works correctly for this shape. The real
+  root cause of the C# corpus gap remains genuinely open. 3 new regression tests.
+- Added the previously-missing adversarial (must-flip) mutation side to `mutate.mjs` —
+  `ADVERSARIAL_SOURCE_LITERALIZATION` replaces a tainted source's initializer with a hardcoded
+  literal and scores the opposite polarity from the existing metamorphic mutators. Found a real
+  detector precision gap on its first use: a structural Java SQL-injection detector fires on
+  "string built via concatenation" regardless of whether the value is genuinely tainted, while
+  the real taint engine correctly stays silent (5 new tests; the detector gap itself is
+  disclosed, not fixed here).
+- `compare-baseline.mjs`'s flat 2-percentage-point regression tolerance either hid a real
+  regression on a high-support metric or failed on ordinary noise for a low-support one — exactly
+  the double-bind measured live this session (an unrelated holdout app's F1 moved between two
+  identical runs). Added `adaptiveTolerance()`, widening the band for low-support metrics only,
+  one-directionally (10 new tests, including the first automated end-to-end CLI proof for this
+  script in either direction).
+- Headline metrics (100% Fully Verified Fix Rate, 100% Semantic Robustness Rate) now carry
+  explicit scope qualifiers in the implementation ledger, so neither can be quoted out of
+  context as a whole-corpus claim.
+
+**Security:**
+- PRD §56 "LLM Isolation" had zero implementation and zero test anywhere in this repo. Added
+  `test/sard-llm-isolation.test.js` against the real LLM-validator prompt builder: an
+  already-neutralized input produces a leakage-clean prompt; the scanner's own CWE classification
+  is legitimate content, not a leak; and, disclosed rather than hidden, the prompt builder has no
+  independent redaction of its own — isolation is entirely inherited from upstream neutralization.
+
+**MLOps/CI:**
+- `realworld-bench`'s `needs: synthetic-bench` meant a real, unrelated regression in
+  `synthetic-bench` (91.3%→85.9% F1, first observed 2026-09-08) silently skipped
+  `realworld-bench` for 5+ consecutive scheduled runs with zero alert. Removed the dependency —
+  both are already independently-tiered informational jobs.
+- `compare-baseline.mjs` was local-only by design (no committed scores), leaving no CI-enforced
+  regression gate at all. Wired a GitHub Actions cache (never a git commit) into `sard-blind-smoke`
+  so a genuine regression fails the job for real, without touching the "no scores committed"
+  policy.
+- The external-holdout generalization gate could never fail: every existing curated app's ground
+  truth was bootstrapped from a past scanner run, not built independently. Added
+  `bench/holdout-independent/tinymart/` — a small, hand-written app whose ground truth was
+  authored from its own source before the scanner ever ran against it. Proved the gate has real
+  teeth end to end: clean baseline, deliberately removed a vulnerability (gate correctly failed,
+  naming the app and the exact regression), reverted (gate returned to clean, numbers matched
+  byte-for-byte). 5 new tests.
+
+No detection-engine changes in this release — every fix above is to the benchmarking/CI
+infrastructure itself, not to `scanner/src/`.
+
 ## 0.151.0 - SARD/Juliet benchmarking: leakage-clean scoring, macro-F1, mutation testing, and fix verification (SARD_AGENTIC_SECURITY_PRD.md)
 
 Builds a full benchmarking subsystem against NIST SARD's Juliet (Java/C#) and PHP Vulnerability