@clear-capabilities/agentic-security-scanner 0.141.0 → 0.143.0

package/CHANGELOG.md

@@ -10,6 +10,236 @@
 
 
 
+
+## 0.143.0 — OSCAL output, and the finding an OSCAL document must refuse to make
+
+`--format oscal` was documented in `commands/compliance.md` long before anything
+implemented it. In 0.139.0 that was corrected to an explicit refusal rather than
+left aspirational. This release makes it real, in both places the request asked
+for: any scan, and any framework assessment.
+
+```bash
+agentic-security scan . --format oscal
+agentic-security compliance --report <framework> --format oscal
+agentic-security compliance --format oscal          # NIST Privacy Framework 1.1
+```
+
+Both emit NIST [OSCAL](https://pages.nist.gov/OSCAL-Reference/models/) 1.1.2
+`assessment-results`. `--report <fw>` is also now a real CLI synonym for
+`--walkthrough <fw>`; the slash command had always spelled it that way, and it
+previously reached the frameworks only through an inlined script in `commands/`.
+
+**The interesting part is what these documents refuse to say.** An OSCAL
+`finding` is a statement about a control, and its `status.state` is binary:
+satisfied or not-satisfied. There is no "unknown" and no "we did not look". So:
+
+- **A raw scan emits no findings at all.** Observations (what the scanner saw)
+  and risks (what it would mean), and a `reviewed-controls` block that says
+  plainly that no catalog was in scope. A SQL-injection hit is not an opinion
+  about a control; emitting one would publish a CWE→control mapping nobody
+  wrote. `include-all` is likewise absent — it would assert this scan reviewed
+  every control of an unnamed catalog.
+- **A control the engine could not decide carries no finding.** `manual`
+  controls, and on the privacy path `engine-gap` controls — where NIST rates the
+  control code-testable and *this scanner has no check for it* — become
+  observations with method `EXAMINE`. Calling them satisfied would be a false
+  compliance claim; calling them not-satisfied would blame the assessed system
+  for a hole in the tool.
+
+The distinction OSCAL cannot express rides along as an `assessment-status`
+property, so nothing is lost by the conversion. Full mapping table in
+`docs/OSCAL.md`.
+
+**A bug this found in itself.** The first adapter mapped `present` to satisfied,
+`partial` to not-satisfied, and everything else to unassessed. `evaluateFramework`
+also returns `absent` — signals exist and not one cleared, the strongest failure
+it can express — and the catch-all silently relabelled it "requires human
+judgement", deleting real control failures from the document and attaching a
+remark that was false. It was caught by running the exporter against a bundled
+framework and reading the output. There is now no catch-all: the mapping is
+exhaustive, and an unrecognised upstream status is reported *as* unrecognised,
+naming itself as an exporter defect rather than making a claim about the control.
+
+**Two more defects the checking found, both invisible at emit time.** OSCAL's
+`assessment-assets` — the block that identifies what performed the assessment —
+is scoped to the *result*, not the document (`assessment-results/local-definitions`
+carries objectives-and-methods and activities, and nothing else), and it requires
+at least one `assessment-platforms` entry. The first draft had it at document
+level with no platform. Both produce a document that emits cleanly and fails
+validation, which is precisely the failure mode of a format claim nobody ran
+through a validator.
+
+**Control identifiers are rewritten, and the originals kept.** OSCAL's `token`
+datatype is an NCName. The CCPA catalog bundled with this engine uses ids like
+`§1798.100`, which is not one — emitting it raw produces a document a validator
+rejects at the first control, the usual failure mode of an OSCAL export that was
+never run through one. Ids are sanitised to legal tokens and the publisher's
+original is carried on every observation and finding as `source-control-id`.
+
+**Checked, and the checks were checked.** `test/oscal-conformance.test.js` (15
+tests) validates required fields, the constrained datatypes (`uuid`, `token`,
+`dateTime-with-timezone`), the closed value sets, and referential integrity —
+every `*-uuid` and `#fragment` must resolve inside the document. It also pins the
+doctrine above, which is not a schema property and would otherwise be one
+refactor from reversing. Three deliberate regressions (neutering the token
+sanitiser, the deterministic uuid shaping, and the no-findings rule) were each
+confirmed to fail the suite before the source was restored. Scope is stated in
+the file: structural validation, not full JSON-Schema validation against NIST's
+published schema — fetching it at test time breaks the no-network rule and
+vendoring it adds a file that rots silently. Same call, same reasoning, as
+`test/sbom-conformance.test.js`.
+
+`oscal` joins the `format-determinism` gate, so two emits of one scan are
+byte-identical and an attestation over the document still verifies. That gate
+also exposed a real gap: every conformance test ran the `crypto.randomUUID()`
+branch, leaving the `--deterministic` branch — the one an attestation is
+actually taken over — untested. A digest slice is not a legal uuid; roughly 15
+in 16 fail on the variant nibble alone. It is now covered explicitly.
+
+**Also:** the two load-bearing caveats (ordinal scores are not probabilities;
+benchmark-tuned F1 does not generalize) now come from one exported constant
+rather than being spelled out inside `toSARIF`. SARIF carries them as run
+notifications, OSCAL as back-matter resources, from the same source — two copies
+of a caveat is one copy that goes stale, and the stale one is always the one
+somebody reads. The three inline "is this a machine format" lists in the CLI
+became one set, for the same reason: each new format previously had to be added
+to all three or it got human chatter interleaved into its output.
+
+
+## 0.142.0 — The last five PRD items, and three criteria that now fail on evidence
+
+0.141.0 built instruments for the surfaces that had none. This closes the
+remainder of `docs/WORLD_CLASS_HARNESS_PRD.md`: four of the five items found
+live engine bugs, and the fifth re-derived the headline the whole document rests
+on. Three success criteria now **fail** — measured, rather than unknown.
+
+### A sanitizer could be undone and the flow still read as clean
+
+`he.decode(escapeHtml(req.query.name))` reaching an HTML sink was reported
+**sanitized**. The decode puts back exactly what the escape removed, so this was
+a missed XSS presented as a clean flow.
+
+Nothing modelled reversal: the catalog holds sanitizers, a decoder is the
+opposite, so it was never even recorded on the path. Fixed across three layers —
+the taint walk now collects un-sanitizer callees, the gate maps them to the
+family they reverse (percent-decoding does **not** undo HTML escaping, so a flat
+list would be wrong), and the finding-projection allowlist had to learn the new
+field. That last one is why the fix looked inert through three rounds of
+debugging.
+
+Found by the mutation gate, which went from 12 to **34 cases** — the five
+detector families 0.141.0 shipped had owed a metamorphic pair and an adversarial
+near-miss and had none.
+
+### The CloudFormation ingress rule was keyed on YAML key order
+
+`- CidrIp:` first and `- IpProtocol:` first are the same template — YAML mappings
+are unordered — and only the second matched. The rule was keyed on the author's
+formatting, which is syntax, not meaning. Also caught by the mutation gate, by
+the metamorphic case that exists for exactly this.
+
+### Every reachability demotion we could adjudicate was wrong
+
+Reachability was reported as a demotion *rate*, and that rate was **0 for every
+entry by construction**: `bench/sca-replay` fetched lockfiles, so the analysis
+had no source to walk. A number that is structurally zero looks like a
+measurement and is not one.
+
+With source fetched and scored against an import-level oracle: **3 adjudicable
+demotions, 3 false-unreachable — 100% wrong, in the missed-exploit direction.**
+`express`/`cookie`, `express`/`send`, `poetry`/`requests` — each demoted to
+`info` and out of the report, each verified by hand as genuinely imported.
+
+Three defects behind it:
+
+- **Failure to prove reachability was reported as proof of unreachability.** A
+  site the analysis could not reason about became a positive claim. It is now
+  `unknown`, a state the code already used elsewhere.
+- **A project with no routes was still asked "reachable from a route?"** For a
+  library that question has no answer — its callers are its users, not in the
+  tree.
+- **`_enclosingFn` knew two of four declaration forms.** `res.cookie = function
+  (…)` — how most of the JS ecosystem defines a public method — was invisible, so
+  the scan attributed call sites to unrelated functions further up the file. It
+  now also tracks whether the enclosing function is **exported**, because a
+  public-API function with no in-tree caller is the normal case, not dead code.
+
+**If you use SCA reachability to triage, this affected you**: findings were being
+demoted out of your report on projects that genuinely import the vulnerable
+package.
+
+### The VS Code extension had never been type-checked
+
+`typescript` is now a devDependency and `npm run typecheck` runs in CI. It found
+errors immediately: the tsconfig declared no `types` at all, so `process`,
+`setTimeout` and `NodeJS.Timeout` were every one of them unresolved — `@types/node`
+was installed and nothing consumed it — plus two implicit `any` parameters. The
+build is esbuild, which strips types without checking them, so none of this had
+ever surfaced.
+
+Neovim and JetBrains are now smoke-tested in CI too. JetBrains is classified
+**informational**: it downloads a full IntelliJ distribution, so red there is more
+often the network than this code, and a habitually-red gate stops being read.
+
+## The population re-measured — and it is worse
+
+991 scored entries at engine 0.141.0 (the code that ships here), against 315 at
+0.138.0 before:
+
+| | before | **now** |
+|---|---:|---:|
+| localized recall | 3.56% | **2.83%** |
+| localized precision | 44.00% | **36.36%** |
+| fix-discrimination | 81.8% | **71.43%** |
+| held-out recall | — | **2.93%** |
+
+**The headline fell because the question got harder.** The corpus tripled once
+the advisory miner could page past the first hundred entries per ecosystem, and
+what it pulled in is recent, TypeScript-heavy and dominated by authorization
+classes. Held-out tracks development almost exactly, so nothing is fitted.
+
+**Ruby 0% → 3.20%. Go 0% → 1.19%.** Both measured zeros are off zero, on
+populations 8× and 1.2× their old size. **PHP is the new zero, at 0/73.**
+
+Two of this release's own fixes are confirmed on real code rather than their own
+fixtures: the Ruby `File.join` rule earns `lsegal/yard`, and `CONVENTION` earns
+its first localized true positive ever on GitPython — the family recorded as
+permanently silent until it turned out to be mislocalized by five lines.
+
+### Three success criteria now fail, on evidence
+
+- **Fix-discrimination is 71.43%**, below its 80% floor. 8 of 28 findings still
+  fire on the code the fix produced — those detected an API, not a vulnerability.
+- **Taint contributes 1 of 28** localized true positives. It was 1 of 12; the
+  count did not move while the population tripled. Deep mode also costs 5 extra
+  false positives and **loses** a Go finding that pattern-only makes.
+- **Compliance still has no accuracy instrument** — the last feature measured by
+  nothing.
+
+### Measured and deliberately not fixed
+
+- **The agent trust-boundary delta is 0 of 0 — undefined, not zero.** The
+  population now holds 28 entries of real MCP-server code; the engine produces no
+  localized true positives on any of them, so there is nothing for the boundary
+  modelling to have contributed to. It is not silent there — 21 findings per
+  entry, all on the advisory's own files, none of the labelled class. WRONG-CWE,
+  on the most differentiated surface in the product.
+- **Reachability now demotes nothing** on the source-bearing corpus. The false
+  demotions are gone; whether the new caution is correct needs applications with
+  real routes, which those four entries are not.
+
+## Also
+
+- A whole-population benchmark run **wedged** at 0.0% CPU after 4.5 hours, taking
+  every already-scored entry with it. `bench/independent/runner.mjs` now takes
+  `--offset`/`--limit` and `merge-chunks.mjs` reassembles the slices,
+  **recomputing** every aggregate from the per-entry rows and refusing to write
+  unless that reproduces each chunk's own numbers exactly. It caught two of its
+  own bugs that way.
+- A guard-shaped method NAME on a declaration line (`def check_static_cache(`) no
+  longer counts as a containment guard, which had been silently dropping path
+  findings inside any method called `check*` / `validate*` / `ensure*`.
+
 ## 0.141.0 — Six new instruments, and the seven live bugs they found
 
 The previous release fixed five bugs found by measuring. This one builds the

package/bin/agentic-security.js

@@ -39,6 +39,17 @@ function writeStdout(s) {
   }
 }
 import { toJSON, toMarkdown, toSARIF, toSTIX, toCSV, toJUnit, toCLI, toCLIByProfile, toShipVerdict, toProTable, toHTML, toSummary, toVex, exitCodeFor, normalizeFindings } from '../src/report/index.js';
+import { toOSCAL } from '../src/report/oscal.js';
+
+// Formats whose output is a machine artifact, not something a human reads in a
+// terminal. Three separate copies of this list used to be spelled out inline —
+// the MTTR line, the fix-duration line and the streak line — and each new
+// format had to be added to all three or it silently got human chatter
+// interleaved into its stdout/stderr. One list, one place to update.
+const MACHINE_FORMATS = new Set([
+  'json', 'sarif', 'oscal', 'cyclonedx', 'sbom', 'spdx', 'vex', 'openvex', 'pbom', 'aibom',
+]);
+function isMachineFormat(fmt) { return MACHINE_FORMATS.has(String(fmt)); }
 import { toCycloneDX, toSPDX } from '../src/posture/sbom.js';
 import { toPBOM } from '../src/sast/pipeline.js';
 import { buildAIBOM, aibomToMarkdown } from '../src/posture/aibom.js';
@@ -96,8 +107,13 @@ Commands:
   compliance [--privacy]       Assess the last scan against NIST Privacy Framework 1.1
                                --list                 show bundled + BYO frameworks
                                --walkthrough <id>     auditor narrative for any framework
+                               --report <id>          synonym for --walkthrough
                                --gap                  only the failing controls
-                               --format cli|json|md   (default cli)
+                               --format cli|json|md|oscal   (default cli)
+                               --format oscal         NIST OSCAL assessment-results.
+                                                      Controls the engine could not
+                                                      decide carry NO finding — see
+                                                      docs/OSCAL.md.
                                --fail-on gap          exit 1 when a control is failing
                                Reads .agentic-security/last-scan.json — run a scan first.
   version                      Print version
@@ -113,7 +129,7 @@ Commands:
 Options:
   --profile vibecoder|pro      Override profile for this run
   --only sast|sca|secrets      Limit scan to one pillar
-  --format <fmt>               cli | json | md | sarif | stix | junit | csv | html | cyclonedx | spdx | pbom | aibom | aibom-md
+  --format <fmt>               cli | json | md | sarif | oscal | stix | junit | csv | html | cyclonedx | spdx | pbom | aibom | aibom-md
   --pack <name>                Focus on a curated rule pack (repeatable): owasp-top-10 | cwe-top-25 | llm-security | supply-chain
   --baseline <ref>             Diff against a git ref; only findings new vs. that ref count (ci subcommand)
   --fail-on critical|high|medium|low|none  ci-mode exit policy (default: critical)
@@ -678,6 +694,10 @@ async function cmdScan(args) {
   if (format === 'json') body = JSON.stringify(toJSON(scan, meta, { includeSuppressed }), null, 2);
   else if (format === 'md' || format === 'markdown') body = toMarkdown(scan, meta);
   else if (format === 'sarif') body = JSON.stringify(toSARIF(scan, meta), null, 2);
+  // OSCAL assessment-results. Observations and risks only, never findings — an
+  // OSCAL finding is a statement about a control, and a source scan reviews no
+  // control catalog. See src/report/oscal.js.
+  else if (format === 'oscal') body = JSON.stringify(toOSCAL(scan, meta), null, 2);
   else if (format === 'stix') body = JSON.stringify(toSTIX(scan, meta), null, 2);
   else if (format === 'junit') body = toJUnit(scan, meta);
   else if (format === 'csv')   body = toCSV(scan);
@@ -759,7 +779,7 @@ async function cmdScan(args) {
         const removed = prevAll.filter(f => !currentFps.has(fingerprintFinding(f)));
         persistedScan.mttr = computeMTTR(removed);
         // Surface the SLA-breach line on human-readable formats (not JSON/CI pipes).
-        const isJson = format === 'json' || format === 'sarif' || format === 'cyclonedx' || format === 'sbom' || format === 'spdx' || format === 'vex' || format === 'openvex' || format === 'pbom' || format === 'aibom';
+        const isJson = isMachineFormat(format);
         if (!isJson) {
           const sla = renderSlaSummary(persistedScan.findings || []);
           if (sla) process.stderr.write(`⏰ agentic-security: ${sla}\n`);
@@ -789,7 +809,7 @@ async function cmdScan(args) {
         const fixMetrics = fixDurationReport(path.resolve(target));
         if (fixMetrics.attempts > 0) {
           persistedScan.fixMetrics = fixMetrics;
-          const isJsonFmt = format === 'json' || format === 'sarif' || format === 'cyclonedx' || format === 'sbom' || format === 'spdx' || format === 'vex' || format === 'openvex' || format === 'pbom' || format === 'aibom';
+          const isJsonFmt = isMachineFormat(format);
           const line = renderFixDurationSummary(fixMetrics);
           if (!isJsonFmt && line) process.stderr.write(`🔧 agentic-security: ${line}\n`);
         }
@@ -832,7 +852,7 @@ async function cmdScan(args) {
   try {
     const streak = persistedScan !== null ? recordScan(stateDirPath, persistedScan) : null;
     // Print celebration / streak line to stderr so it doesn't pollute --format json
-    if (streak && process.stderr.isTTY && format !== 'json' && format !== 'sarif') {
+    if (streak && process.stderr.isTTY && !isMachineFormat(format)) {
       const delta = formatGradeDelta(streak);
       const line = formatStreakLine(streak);
       if (delta) process.stderr.write('\n' + delta + '\n');
@@ -1791,14 +1811,27 @@ async function cmdCompliance(args) {
   // The engine records this two ways depending on the emit path; take either.
   scan.filesScanned = scan._scanMeta?.filesScanned ?? scan.scanned?.files ?? 0;
 
-  const wt = args.flags.walkthrough;
+  // `--report <fw>` is accepted as a synonym for `--walkthrough <fw>`: the
+  // slash-command surface has always spelled it that way, and it previously
+  // reached the frameworks only through an inlined node call in commands/
+  // rather than through this CLI. One code path, two spellings.
+  const wt = args.flags.walkthrough || args.flags.report;
   if (wt && wt !== true) {
     const fw = loadFramework(scanRoot, String(wt));
     if (!fw) {
       console.error(`Unknown framework "${wt}". Try --list.`);
       return 2;
     }
-    console.log(renderWalkthrough(fw, evaluateFramework(scanRoot, fw, scan), {}));
+    const evaluation = evaluateFramework(scanRoot, fw, scan);
+    if (fmt === 'oscal') {
+      const { toOSCALCompliance, complianceRowsFromEvaluation } = await import('../src/report/oscal.js');
+      writeStdout(JSON.stringify(
+        toOSCALCompliance(fw, complianceRowsFromEvaluation(evaluation), { startedAt: scan._scanMeta?.startedAt }),
+        null, 2) + '\n');
+      return 0;
+    }
+    if (fmt === 'json') { writeStdout(JSON.stringify(evaluation, null, 2) + '\n'); return 0; }
+    console.log(renderWalkthrough(fw, evaluation, {}));
     return 0;
   }
 
@@ -1808,6 +1841,19 @@ async function cmdCompliance(args) {
   if (!r) { console.error(`Framework ${PRIVACY_FRAMEWORK_ID} could not be loaded.`); return 2; }
 
   const gapsOnly = !!args.flags.gap;
+  if (fmt === 'oscal') {
+    // The privacy assessment's own bucket model, not evaluateFramework's — see
+    // complianceRowsFromPrivacy for why `engine-gap` must not become a control
+    // failure. `--gap` is deliberately NOT applied here: an OSCAL document that
+    // silently omitted the satisfied controls would understate what was
+    // reviewed, and reviewed-controls would then disagree with the findings.
+    const { toOSCALCompliance, complianceRowsFromPrivacy } = await import('../src/report/oscal.js');
+    const fwMeta = loadFramework(scanRoot, PRIVACY_FRAMEWORK_ID) || { id: PRIVACY_FRAMEWORK_ID, name: r.frameworkName };
+    writeStdout(JSON.stringify(
+      toOSCALCompliance(fwMeta, complianceRowsFromPrivacy(r), { startedAt: scan._scanMeta?.startedAt }),
+      null, 2) + '\n');
+    return args.flags['fail-on'] === 'gap' && r.summary.gap > 0 ? 1 : 0;
+  }
   if (fmt === 'json') {
     writeStdout(JSON.stringify(gapsOnly ? { ...r, controls: r.controls.filter(c => c.bucket === 'gap') } : r, null, 2) + '\n');
   } else if (fmt === 'md') {