circle-ir-ai 2.67.0 → 2.75.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +327 -0
- package/dist/agents/mastra/workflow.d.ts +23 -1
- package/dist/agents/mastra/workflow.d.ts.map +1 -1
- package/dist/agents/mastra/workflow.js +76 -3
- package/dist/agents/mastra/workflow.js.map +1 -1
- package/dist/llm/ax-client.d.ts +8 -0
- package/dist/llm/ax-client.d.ts.map +1 -1
- package/dist/llm/ax-client.js +29 -2
- package/dist/llm/ax-client.js.map +1 -1
- package/dist/llm/verification.d.ts +42 -13
- package/dist/llm/verification.d.ts.map +1 -1
- package/dist/llm/verification.js +88 -41
- package/dist/llm/verification.js.map +1 -1
- package/dist/security-scan/scanner.d.ts +83 -4
- package/dist/security-scan/scanner.d.ts.map +1 -1
- package/dist/security-scan/scanner.js +155 -17
- package/dist/security-scan/scanner.js.map +1 -1
- package/dist/utils/file-skip.d.ts +51 -0
- package/dist/utils/file-skip.d.ts.map +1 -1
- package/dist/utils/file-skip.js +92 -0
- package/dist/utils/file-skip.js.map +1 -1
- package/dist/utils/scan-heartbeat.d.ts +52 -0
- package/dist/utils/scan-heartbeat.d.ts.map +1 -0
- package/dist/utils/scan-heartbeat.js +85 -0
- package/dist/utils/scan-heartbeat.js.map +1 -0
- package/package.json +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,333 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [2.75.0] - 2026-07-31
|
|
9
|
+
|
|
10
|
+
### Fixed — bound verifier fan-out per file (#252 problem B)
|
|
11
|
+
|
|
12
|
+
Closes #252's last substantive item: *"verification cannot emit unbounded
|
|
13
|
+
15×15×N chunk storms without hitting a documented budget."*
|
|
14
|
+
|
|
15
|
+
`verifyBatchChunkSplit` has always capped its **sink** partitioning at
|
|
16
|
+
`VERIFY_MAX_CHUNKS` (default 4). The **source**-chunking path had no cap at
|
|
17
|
+
all, so a single file could emit an unbounded number of verifier calls — and it
|
|
18
|
+
bites hardest exactly where it hurts. `chunkSize` is
|
|
19
|
+
`floor(effectiveMaxPairs / sinks)`, so a sink-heavy file drives it to 1 and
|
|
20
|
+
produces **one LLM call per source**. #252 recorded
|
|
21
|
+
`15 sources × 15 sinks = 225 pairs → 15 chunks`: fifteen calls for one file,
|
|
22
|
+
against a provider already returning 429s.
|
|
23
|
+
|
|
24
|
+
Source chunks are now capped at `VERIFY_MAX_CHUNKS`. Sources arrive ordered by
|
|
25
|
+
the upstream `prioritized` selection strategy, so truncating the tail keeps the
|
|
26
|
+
highest-value candidates. Raise `LLM_VERIFY_MAX_CHUNKS` to widen.
|
|
27
|
+
|
|
28
|
+
The dropped pairs are **counted, not silently discarded** — new
|
|
29
|
+
`pairsDroppedByBudget` on `ScanMeta.verification`, distinct from `pairsOmitted`:
|
|
30
|
+
|
|
31
|
+
| field | who declined |
|
|
32
|
+
|---|---|
|
|
33
|
+
| `pairsOmitted` | the model, shown the pair, chose not to mention it |
|
|
34
|
+
| `pairsDroppedByBudget` | **us**, never showed it the pair |
|
|
35
|
+
|
|
36
|
+
Collapsing the two would make our own fan-out cap read as model behaviour —
|
|
37
|
+
the same conflation that made `Finding.exploitable` useless in #246. A budget
|
|
38
|
+
drop also must not inflate `omissionRate`, which describes only pairs actually
|
|
39
|
+
submitted; a test pins that.
|
|
40
|
+
|
|
41
|
+
Harness rolls the field up alongside the rest. 1 test added. Suite 1585 pass +
|
|
42
|
+
3 skipped.
|
|
43
|
+
|
|
44
|
+
## [2.74.0] - 2026-07-31
|
|
45
|
+
|
|
46
|
+
### Added — progress logs that name the stuck file (#252 problem E)
|
|
47
|
+
|
|
48
|
+
Closes #252's progress-logging item. Two defects, one of them silently wrong for
|
|
49
|
+
as long as parallel scanning has existed:
|
|
50
|
+
|
|
51
|
+
**The `[N/M]` index was meaningless under concurrency.** It was logged as
|
|
52
|
+
`filesProcessed + 1` when a worker *picked up* a file, but `filesProcessed` only
|
|
53
|
+
advances when a file *finishes*. With four workers, four different files all
|
|
54
|
+
logged `[1/812]`. Now a dispatch counter, so every line has a distinct index
|
|
55
|
+
that identifies its file.
|
|
56
|
+
|
|
57
|
+
**Nothing said which file was stuck.** `ScanHeartbeat` (`src/utils/scan-heartbeat.ts`)
|
|
58
|
+
tracks what is actually in flight and names anything running past a threshold:
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
⏱ still analyzing after 180s: site/bisheng.common.config.js (1 file(s) in flight). cognium-ai#252
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
That is the line whose absence made #252's own evidence require attaching to a
|
|
65
|
+
running process to find the offending 4 KB file.
|
|
66
|
+
|
|
67
|
+
Silent on a healthy scan — nothing is emitted unless a file exceeds
|
|
68
|
+
`SCAN_STALL_WARN_MS`, and each stalled file reports **once**, so a 3-hour wedge
|
|
69
|
+
produces one line rather than hundreds. Tunable via `SCAN_HEARTBEAT_MS`
|
|
70
|
+
(default 30000; `0` disables) and `SCAN_STALL_WARN_MS` (default 60000).
|
|
71
|
+
|
|
72
|
+
**Known limit, verified rather than assumed:** the heartbeat is timer-driven, so
|
|
73
|
+
a worker wedged in CPU-bound synchronous work (a Tree-sitter parse) blocks it —
|
|
74
|
+
the same constraint that defeats `withTimeout` and the scan deadline. Confirmed
|
|
75
|
+
on the #252 repro fixture: silent by default, and correct once
|
|
76
|
+
`SCAN_PARSE_ISOLATION=1` moves the parse off the main thread. The two features
|
|
77
|
+
compose; either alone leaves a gap.
|
|
78
|
+
|
|
79
|
+
8 tests. Suite 1584 pass + 3 skipped.
|
|
80
|
+
|
|
81
|
+
## [2.73.0] - 2026-07-31
|
|
82
|
+
|
|
83
|
+
### Added — skip mirrored CDN/vendor trees (#252 problem C)
|
|
84
|
+
|
|
85
|
+
Closes #252's CDN-mirror acceptance item. #243 catches build artifacts by the
|
|
86
|
+
`.min.` convention, but site-ripper mirrors carry neither `.min.` nor a giant
|
|
87
|
+
line — they flatten a remote URL into a filename:
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
js/ajax.googleapis.com__ajax__libs__dojo__1.4.3__dojo__dojo.js
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
poisontap ships 249 such copies. They are third-party code the repo owner does
|
|
94
|
+
not maintain, so findings in them are noise — and in #252 these were the trees
|
|
95
|
+
that walked discovery into a 7200s timeout with no artifact.
|
|
96
|
+
|
|
97
|
+
`isCdnMirrorPath()` refuses two shapes, either sufficient:
|
|
98
|
+
|
|
99
|
+
1. a hostname-shaped token ending in a real TLD, immediately followed by the
|
|
100
|
+
`__` flattening marker;
|
|
101
|
+
2. a known CDN host anywhere in the path, for mirrors that keep real
|
|
102
|
+
directories (`vendor/cdnjs.cloudflare.com/ajax/libs/...`).
|
|
103
|
+
|
|
104
|
+
Wired into `preFlightSkip` next to the `#243` check — matched by name, so the
|
|
105
|
+
file is never stat'd or read.
|
|
106
|
+
|
|
107
|
+
**Deliberately conservative**, because the failure mode of a filename heuristic
|
|
108
|
+
is silently dropping first-party source. Both conditions are required for shape
|
|
109
|
+
1: the `__` marker keeps ordinary dotted names out (`utils.test.js`,
|
|
110
|
+
`app.config.js`), and the TLD allowlist keeps first-party names that merely
|
|
111
|
+
contain `__` out (`foo.bar__baz.js`, `data.v2__snapshot.json` — the first of
|
|
112
|
+
which an earlier draft of the pattern did match). A hostname-shaped name with
|
|
113
|
+
no marker (`foo.example.com.js`) is treated as first-party.
|
|
114
|
+
|
|
115
|
+
Opt out with `SKIP_CDN_MIRRORS=0`. 17 tests, 11 of them asserting what must NOT
|
|
116
|
+
be skipped. Suite 1576 pass + 3 skipped.
|
|
117
|
+
|
|
118
|
+
## [2.72.0] - 2026-07-31
|
|
119
|
+
|
|
120
|
+
### Changed — one minified predicate across every stage (#252 problem D)
|
|
121
|
+
|
|
122
|
+
Closes the fourth acceptance item on #252: *"content skipped by verification
|
|
123
|
+
`#102` thresholds is also skipped at discovery / `preFlightSkip`."*
|
|
124
|
+
|
|
125
|
+
Verification has refused minified/bundled content since #102 — but that decision
|
|
126
|
+
was made at the **end** of the pipeline. A bundle had already paid for a parse,
|
|
127
|
+
the SAST passes, and both discovery LLM calls before anything looked at its
|
|
128
|
+
shape. The walker's own caps don't cover this class: `preFlightSkip` refuses a
|
|
129
|
+
line over **50,000** chars, while the minified predicate trips at **5,000**, so
|
|
130
|
+
everything in between sailed through the expensive stages only to be dropped at
|
|
131
|
+
the last one.
|
|
132
|
+
|
|
133
|
+
`isMinifiedContent` now lives in `src/utils/file-skip.ts` — the module that owns
|
|
134
|
+
the other skip heuristics — and `verification.ts` re-exports it, so the two
|
|
135
|
+
stages share one binding and cannot drift. `runEnrich` applies it before the
|
|
136
|
+
`#145` pattern pre-filter, emitting the same synthetic `discoverSources` /
|
|
137
|
+
`discoverSinks` skip rows so `analyze-llm-log.ts` still reconciles.
|
|
138
|
+
|
|
139
|
+
Contract is unchanged from #102: this gates **LLM enrichment only**. Static
|
|
140
|
+
findings on minified files are unaffected, and ordinary source is untouched —
|
|
141
|
+
the predicate needs size > 50KB *and* a line > 5,000 chars *and* an average line
|
|
142
|
+
over 500, so a large well-formatted file does not trip it.
|
|
143
|
+
|
|
144
|
+
5 tests, including one that asserts the two stages reference the same function
|
|
145
|
+
rather than merely agreeing today. Suite 1559 pass + 3 skipped.
|
|
146
|
+
|
|
147
|
+
## [2.71.0] - 2026-07-31
|
|
148
|
+
|
|
149
|
+
### Added — `LLM_RATE_INTERVAL_MS`: throttle below 60 RPM
|
|
150
|
+
|
|
151
|
+
`LLM_RATE_LIMIT` shared a hardcoded 1000ms p-queue window, so it meant
|
|
152
|
+
"requests per second" and the slowest rate expressible was `LLM_RATE_LIMIT=1` →
|
|
153
|
+
**60 RPM**. Providers enforcing a lower requests-per-minute budget could not be
|
|
154
|
+
respected by any combination of settings — #252 recorded Novita returning 429s
|
|
155
|
+
even at `LLM_CONCURRENCY=1`.
|
|
156
|
+
|
|
157
|
+
Concurrency is the wrong lever for an RPM cap: the AIMD controller (#229) floors
|
|
158
|
+
at one in-flight call and a single call every few seconds can still exceed a
|
|
159
|
+
30 RPM budget. The result was scans burning retries against a limit they had no
|
|
160
|
+
way to obey.
|
|
161
|
+
|
|
162
|
+
The interval is now configurable, so the two knobs together express any rate:
|
|
163
|
+
|
|
164
|
+
| target | `LLM_RATE_LIMIT` | `LLM_RATE_INTERVAL_MS` |
|
|
165
|
+
|---|---|---|
|
|
166
|
+
| 30 RPM (Novita) | `1` | `2000` |
|
|
167
|
+
| 60 RPM | `1` | `1000` (default) |
|
|
168
|
+
| 120 RPM | `2` | `1000` |
|
|
169
|
+
| 1200 RPM | `20` | `1000` (default) |
|
|
170
|
+
|
|
171
|
+
Default stays 1000ms, so existing deployments are byte-identical. Non-positive
|
|
172
|
+
or unparseable values fall back to 1000 rather than disabling the throttle — a
|
|
173
|
+
0ms p-queue interval would busy-loop. A one-line startup banner reports the
|
|
174
|
+
effective RPM whenever the interval is non-default.
|
|
175
|
+
|
|
176
|
+
Both `LLM_RATE_LIMIT` and `LLM_RATE_INTERVAL_MS` are read once at module load,
|
|
177
|
+
so they must be exported before the process starts. 5 tests; suite 1554 pass +
|
|
178
|
+
3 skipped.
|
|
179
|
+
|
|
180
|
+
## [2.70.1] - 2026-07-31
|
|
181
|
+
|
|
182
|
+
### Changed — bump circle-ir 3.194.0 → 3.195.0
|
|
183
|
+
|
|
184
|
+
Uniform fleet pin. No circle-ir-ai source change; typecheck clean, full suite
|
|
185
|
+
1549 pass + 3 skipped on 3.195.0.
|
|
186
|
+
|
|
187
|
+
## [2.70.0] - 2026-07-31
|
|
188
|
+
|
|
189
|
+
### Fixed — `maxScanMs` now bounds in-flight work, not just the next file (#252)
|
|
190
|
+
|
|
191
|
+
`--max-scan-time` (2.65.0) checked the budget only *before a worker pulled its
|
|
192
|
+
next file*. If every worker was blocked inside an in-flight file when the budget
|
|
193
|
+
expired, none returned to the top of the loop, `Promise.all` never settled, and
|
|
194
|
+
the scan ran until an outer orchestrator SIGKILLed it **with no output** — the
|
|
195
|
+
exact failure the budget exists to prevent.
|
|
196
|
+
|
|
197
|
+
Found while running the #246 measurement against a local Ollama endpoint, which
|
|
198
|
+
serializes requests: 4 workers each grabbed a file at t=0 and blocked on the
|
|
199
|
+
queue for longer than the whole budget. Two consecutive corpus runs lost ~4 h of
|
|
200
|
+
compute across 8 repos and produced **zero** `scan.json` files between them.
|
|
201
|
+
|
|
202
|
+
New exported `awaitWorkersWithDeadline()` races the worker pool against the
|
|
203
|
+
deadline: on expiry the queue drains, the result is marked degraded, and the
|
|
204
|
+
scan returns with whatever completed. In-flight workers are abandoned rather than
|
|
205
|
+
awaited, and their late rejections are swallowed so an unhandled rejection can't
|
|
206
|
+
kill the process before the partial result is written.
|
|
207
|
+
|
|
208
|
+
Same repo, same 5-minute stage timeout, before and after:
|
|
209
|
+
|
|
210
|
+
| | before | after |
|
|
211
|
+
|---|---|---|
|
|
212
|
+
| outcome | `timeout (300019ms, SIGKILL)` | `ok (241510ms)` |
|
|
213
|
+
| `scan.json` | **absent** | present, `degraded: true` |
|
|
214
|
+
| files analyzed | — | 23 |
|
|
215
|
+
| findings | — | 164 |
|
|
216
|
+
|
|
217
|
+
Known limit, documented on the function: the race only fires while the event
|
|
218
|
+
loop is free. A worker wedged in CPU-bound synchronous work (a Tree-sitter
|
|
219
|
+
parse) still blocks the timer — that case needs `SCAN_PARSE_ISOLATION=1`
|
|
220
|
+
(2.66.0). The two mechanisms cover different halves of the same failure.
|
|
221
|
+
|
|
222
|
+
Unchanged when `maxScanMs` is unset: a plain `Promise.all`. 5 tests; suite 1549
|
|
223
|
+
pass + 3 skipped.
|
|
224
|
+
|
|
225
|
+
## [2.69.1] - 2026-07-31
|
|
226
|
+
|
|
227
|
+
### Fixed — verifier summary counts, and the chunked-path denominator (#246)
|
|
228
|
+
|
|
229
|
+
Two defects found by running 2.69.0's own instrumentation against real models.
|
|
230
|
+
Both made the verifier look more permissive than it is.
|
|
231
|
+
|
|
232
|
+
**1. The summary counted raw model strings.** `pairs[]` stored
|
|
233
|
+
`normalizeVerdict(p.verdict)` while `summary` filtered the *unnormalized*
|
|
234
|
+
`p.verdict`, so the two disagreed for any model that phrased a verdict its own
|
|
235
|
+
way. `gemma3:12b` returns `"Likely Vulnerable"`; others return `"tp"` or
|
|
236
|
+
`"false-positive"`. `normalizeVerdict` handles all of these for `pairs[]`
|
|
237
|
+
(unknown → `UNCERTAIN`), but the summary scored them as neither TP, FP, nor
|
|
238
|
+
uncertain. The summary is now derived from the normalized array.
|
|
239
|
+
|
|
240
|
+
Related: when the model supplied its own `summary` block it was preferred over
|
|
241
|
+
counting the pairs (`...(response.summary || {…})`). A model's arithmetic about
|
|
242
|
+
its own answer is not a fact about our data; the self-report is now ignored.
|
|
243
|
+
|
|
244
|
+
**2. `pairs_submitted` was dropped on the chunked path.** The parallel
|
|
245
|
+
chunk loop (`verification.ts` ~765) accumulated pairs and verdicts across chunks
|
|
246
|
+
but not the submitted denominator, so any file large enough to chunk contributed
|
|
247
|
+
its pairs to the numerator and **zero** to the denominator. Caught on
|
|
248
|
+
`lionsoul2014/ip2region`, which reported `pairsSubmitted: 20` against 74 verdicts
|
|
249
|
+
plus 163 unmapped pairs — an impossible ratio that only showed up because
|
|
250
|
+
2.69.0 records all three.
|
|
251
|
+
|
|
252
|
+
Effect on 2.69.0 readings: `pairsOmitted` clamped low and `omissionRate` read far
|
|
253
|
+
below the truth on any chunked file. `dropRate` is computed from verdict counts
|
|
254
|
+
alone and was **not** affected by (2), but was affected by (1) on models with
|
|
255
|
+
non-canonical verdict strings. Sweeps run on 2.69.0 should be re-run.
|
|
256
|
+
|
|
257
|
+
3 regression tests pin verdict-phrasing normalization, self-report rejection, and
|
|
258
|
+
the submitted denominator on an all-declined batch. Suite 1544 pass + 3 skipped.
|
|
259
|
+
|
|
260
|
+
## [2.69.0] - 2026-07-30
|
|
261
|
+
|
|
262
|
+
### Added — `pairsSubmitted` / `pairsOmitted`: the verifier's silent declines (#246)
|
|
263
|
+
|
|
264
|
+
The 2.67.0 and 2.68.0 counters exposed what the verifier *said*. They still could
|
|
265
|
+
not express what it was *asked* — and the gap turns out to be where almost all of
|
|
266
|
+
its behaviour lives.
|
|
267
|
+
|
|
268
|
+
`verification.ts` (~line 376) instructs the model:
|
|
269
|
+
|
|
270
|
+
> Only include pairs where data CAN flow from source to sink.
|
|
271
|
+
|
|
272
|
+
So a pair the verifier declines is **omitted from the response**, not returned as
|
|
273
|
+
`FALSE_POSITIVE`. Nothing downstream could distinguish "declined silently" from
|
|
274
|
+
"never submitted". Worse, `verification.ts` set `total_analyzed:
|
|
275
|
+
response.pairs.length` — *analyzed* was defined as *returned*, so a model
|
|
276
|
+
answering about 1 of 200 pairs reported "1 analyzed".
|
|
277
|
+
|
|
278
|
+
Observed on `jwtk/jjwt` (gemma-3-27b, 2.68.0): **46 `verifyBatch` calls, every one
|
|
279
|
+
`success: true` with a clean parse and ~344 output tokens — 15 verdicts total.**
|
|
280
|
+
|
|
281
|
+
`BatchVerificationResult.summary` now carries `pairs_submitted` (the already-computed
|
|
282
|
+
`sources.length * sinks.length`, summed across chunks), threaded through `runVerify`
|
|
283
|
+
→ `runReport` → `ScanMeta.verification`, which gains:
|
|
284
|
+
|
|
285
|
+
| field | meaning |
|
|
286
|
+
|---|---|
|
|
287
|
+
| `pairsSubmitted` | pairs actually sent to the verifier |
|
|
288
|
+
| `pairsUnmapped` | pairs it returned that matched no known source/sink (drifted/hallucinated lines) |
|
|
289
|
+
| `pairsOmitted` | `submitted - verdicts - unmapped` — asked about, never mentioned |
|
|
290
|
+
| `omissionRate` | `pairsOmitted / pairsSubmitted` |
|
|
291
|
+
|
|
292
|
+
Minimal live demonstration (single Java file, 2 sources × 3 sinks):
|
|
293
|
+
|
|
294
|
+
```json
|
|
295
|
+
{ "truePositives": 3, "falsePositives": 0, "uncertain": 0,
|
|
296
|
+
"pairsSubmitted": 6, "pairsOmitted": 3, "omissionRate": 0.5, "dropRate": 0 }
|
|
297
|
+
```
|
|
298
|
+
|
|
299
|
+
`dropRate: 0` reads as "affirmed everything, declined nothing" — the verifier
|
|
300
|
+
actually declined **half** the pairs, silently. That is #246's illusion in one file.
|
|
301
|
+
|
|
302
|
+
**Read `dropRate` only alongside `omissionRate`.** dropRate is over returned
|
|
303
|
+
verdicts alone; on its own it is the same trap as `Finding.exploitable`.
|
|
304
|
+
|
|
305
|
+
Harness updated to match: `baseline-stats.json.verification` gains the four fields,
|
|
306
|
+
`summary.csv` gains `verifyPairsSubmitted` / `verifyPairsOmitted` /
|
|
307
|
+
`verifyOmissionRate`. 7 tests; suite 1541 pass + 3 skipped.
|
|
308
|
+
|
|
309
|
+
## [2.68.0] - 2026-07-30
|
|
310
|
+
|
|
311
|
+
### Changed — `meta.verification`: split "verification enabled" from "verifier spoke" (#246)
|
|
312
|
+
|
|
313
|
+
**Replaces `filesVerified` (2.67.0) with `filesVerificationEnabled` + `filesWithVerdicts`.**
|
|
314
|
+
Breaking for anyone who read the 2.67.0 field — published the same day, no known
|
|
315
|
+
consumer, and the old name asserted something untrue, so it is corrected rather
|
|
316
|
+
than aliased.
|
|
317
|
+
|
|
318
|
+
Found by running the new counters through the Java harness on `apache/commons-csv`:
|
|
319
|
+
**170 findings emitted, 12 files with verification enabled, 0 verdicts returned.**
|
|
320
|
+
|
|
321
|
+
`runVerify` short-circuits to an empty result when a file has no merged sources
|
|
322
|
+
or no merged sinks (`workflow.ts:669`), and `runReport` then falls through to the
|
|
323
|
+
unverified `generateFindings` branch. Those findings ship having never been seen
|
|
324
|
+
by the verifier. 2.67.0's `filesVerified` counted the *request*, not the
|
|
325
|
+
*judgement*, so it reported 12 for a file set the verifier never judged.
|
|
326
|
+
|
|
327
|
+
This is a fourth structural reason #246 never observed a downgraded critical,
|
|
328
|
+
alongside the three in 2.67.0's entry: a large share of findings are not
|
|
329
|
+
verifier output at all. The gap between the two new fields measures it directly.
|
|
330
|
+
|
|
331
|
+
`dropRate: 0` is now readable only in company with `filesWithVerdicts` — 0 drops
|
|
332
|
+
out of 0 verdicts means "nothing was judged", not "nothing was dropped". 5 tests;
|
|
333
|
+
suite 1539 pass + 3 skipped.
|
|
334
|
+
|
|
8
335
|
## [2.67.0] - 2026-07-30
|
|
9
336
|
|
|
10
337
|
### Added — `ScanMeta.verification`: the verifier's own verdict tallies (#246)
|
|
@@ -43,6 +43,16 @@ export interface WorkflowOutput {
|
|
|
43
43
|
truePositives: number;
|
|
44
44
|
falsePositives: number;
|
|
45
45
|
uncertain: number;
|
|
46
|
+
/**
|
|
47
|
+
* cognium-ai#246 — pairs submitted to the verifier, and pairs it returned
|
|
48
|
+
* that failed to map back to a known source/sink. Together with the
|
|
49
|
+
* verdict counts these make the verifier's silent losses measurable:
|
|
50
|
+
* omitted = pairsSubmitted - (TP + FP + uncertain) - pairsUnmapped
|
|
51
|
+
*/
|
|
52
|
+
pairsSubmitted: number;
|
|
53
|
+
pairsUnmapped: number;
|
|
54
|
+
/** #252 B — pairs never sent because per-file verifier fan-out hit its cap. */
|
|
55
|
+
pairsDroppedByBudget: number;
|
|
46
56
|
totalTimeMs: number;
|
|
47
57
|
};
|
|
48
58
|
/**
|
|
@@ -64,7 +74,19 @@ export declare function runMerge(patternSources: any[], patternSinks: any[], enr
|
|
|
64
74
|
/**
|
|
65
75
|
* Run report generation step
|
|
66
76
|
*/
|
|
67
|
-
export declare function runReport(filePath: string, mergedSources: any[], mergedSinks: any[], verificationResults: any[], patternMatchTimeMs: number, enrichTimeMs: number, verifyTimeMs: number, verificationEnabled?: boolean, sastFindings?: any[], dfg?: DFG, types?: any[], sourceCode?: string
|
|
77
|
+
export declare function runReport(filePath: string, mergedSources: any[], mergedSinks: any[], verificationResults: any[], patternMatchTimeMs: number, enrichTimeMs: number, verifyTimeMs: number, verificationEnabled?: boolean, sastFindings?: any[], dfg?: DFG, types?: any[], sourceCode?: string,
|
|
78
|
+
/**
|
|
79
|
+
* cognium-ai#246 — verifier submission accounting from `runVerify`.
|
|
80
|
+
* `verificationResults` only carries pairs the model returned AND that
|
|
81
|
+
* mapped back to a known source/sink, so on its own it cannot express how
|
|
82
|
+
* much the verifier was asked about. Optional: legacy callers (tests) omit
|
|
83
|
+
* it and the stats fall back to the returned counts.
|
|
84
|
+
*/
|
|
85
|
+
verifyAccounting?: {
|
|
86
|
+
pairsSubmitted: number;
|
|
87
|
+
pairsUnmapped: number;
|
|
88
|
+
pairsDroppedByBudget: number;
|
|
89
|
+
}): WorkflowOutput;
|
|
68
90
|
/**
|
|
69
91
|
* Run security analysis on a single file
|
|
70
92
|
*
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"workflow.d.ts","sourceRoot":"","sources":["../../../src/agents/mastra/workflow.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAGH,OAAO,KAAK,EAAoC,GAAG,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAOvF,OAAO,KAAK,EACV,SAAS,EAEV,MAAM,0CAA0C,CAAC;AAKlD,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,qBAAqB,CAAC;
|
|
1
|
+
{"version":3,"file":"workflow.d.ts","sourceRoot":"","sources":["../../../src/agents/mastra/workflow.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAGH,OAAO,KAAK,EAAoC,GAAG,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAOvF,OAAO,KAAK,EACV,SAAS,EAEV,MAAM,0CAA0C,CAAC;AAKlD,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,qBAAqB,CAAC;AAarD,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,2BAA2B,CAAC;AACnE,OAAO,EAAwB,KAAK,aAAa,EAAE,MAAM,0BAA0B,CAAC;AAGpF,OAAO,EACL,gBAAgB,EAChB,UAAU,EACV,SAAS,EACT,UAAU,EACV,UAAU,GACX,MAAM,YAAY,CAAC;AAMpB,MAAM,MAAM,oBAAoB,GAAG,UAAU,GAAG,SAAS,GAAG,aAAa,CAAC;AAE1E,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,OAAO,CAAC,EAAE;QACR,gBAAgB,CAAC,EAAE,OAAO,CAAC;QAC3B,kBAAkB,CAAC,EAAE,OAAO,CAAC;QAC7B,mBAAmB,CAAC,EAAE,MAAM,CAAC;QAC7B,sBAAsB,CAAC,EAAE,MAAM,CAAC;QAChC,oBAAoB,CAAC,EAAE,oBAAoB,CAAC;KAC7C,CAAC;CACH;AAED,MAAM,WAAW,cAAc;IAC7B,eAAe,EAAE,GAAG,EAAE,CAAC;IACvB;;;;;;OAMG;IACH,YAAY,EAAE,GAAG,EAAE,CAAC;IACpB,KAAK,EAAE;QACL,YAAY,EAAE,MAAM,CAAC;QACrB,UAAU,EAAE,MAAM,CAAC;QACnB,uBAAuB,EAAE,MAAM,CAAC;QAChC,aAAa,EAAE,MAAM,CAAC;QACtB,cAAc,EAAE,MAAM,CAAC;QACvB,SAAS,EAAE,MAAM,CAAC;QAClB;;;;;WAKG;QACH,cAAc,EAAE,MAAM,CAAC;QACvB,aAAa,EAAE,MAAM,CAAC;QACtB,+EAA+E;QAC/E,oBAAoB,EAAE,MAAM,CAAC;QAC7B,WAAW,EAAE,MAAM,CAAC;KACrB,CAAC;IACF;;;;;;;;;OASG;IACH,aAAa,CAAC,EAAE,aAAa,CAAC;CAC/B;AA+XD,wBAAgB,QAAQ,CACtB,cAAc,EAAE,GAAG,EAAE,EACrB,YAAY,EAAE,GAAG,EAAE,EACnB,gBAAgB,EAAE,GAAG,GAAG,IAAI,EAC5B,mBAAmB,EAAE,MAAM,EAC3B,UAAU,CAAC,EAAE,MAAM,EACnB,QAAQ,CAAC,EAAE,MAAM,EACjB,KAAK,CAAC,EAAE,aAAa,CAAC,SAAS,CAAC,EAChC,QAAQ,CAAC,EAAE,MAAM;;;EAwHlB;AAgQD;;GAEG;AAIH,wBAAgB,SAAS,CACvB,QAAQ,EAAE,MAAM,EAChB,aAAa,EAAE,GAAG,EAAE,EACpB,WAAW,EAAE,GAAG,EAAE,EAClB,mBAAmB,EAAE,GAAG,EAAE,EAC1B,kBAAkB,EAAE,MAAM,EAC1B,YAAY,EAAE,MAAM,EACpB,YAAY,EAAE,MAAM,EACpB,mBAAmB,GAAE,OAAc,EACnC,YAAY,GAAE,GAAG,EAAO,EAQxB,GAAG,CAAC,EAAE,GAAG,EAKT,KAAK,CAAC,EAAE,GAAG,EAAE,EAMb,UAAU,CAAC,EAAE,MAAM;AACnB;;;;;;GAMG;AACH,gBAAgB,CAAC,EAAE;IACjB,cAAc,EAAE,MAAM,CAAC;IACvB,aAAa,EAAE,MAAM,CAAC;IACtB,oBAAoB,EAAE,MAAM,CAAC;CAC9B,GACA,cAAc,CAkThB;AAMD;;;;;;;;;;;;;;;;;GAiBG;AACH,wBAAsB,WAAW,CAC/B,QAAQ,EAAE,MAAM,EAChB,UAAU,EAAE,MAAM,EAClB,OAAO,CAAC,EAAE;IACR,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,gBAAgB,CAAC,EAAE,OAAO,CAAC;IAC3B,kBAAkB,CAAC,EAAE,OAAO,CAAC;IAC7B,mBAAmB,CAAC,EAAE,MAAM,CAAC;IAC7B,sBAAsB,CAAC,EAAE,MAAM,CAAC;IAChC,oBAAoB,CAAC,EAAE,oBAAoB,CAAC;IAC5C,SAAS,CAAC,EAAE,OAAO,CAAC,SAAS,CAAC,CAAC;IAC/B,cAAc,CAAC,EAAE,MAAM,EAAE,CAAC;IAC1B;;;;;OAKG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB;;;;;;;;OAQG;IACH,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB;;;;;;;;OAQG;IACH,cAAc,CAAC,EAAE,cAAc,GAAG,GAAG,CAAC,MAAM,EAAE,cAAc,CAAC,CAAC;IAC9D;;;;;;OAMG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB;;;;OAIG;IACH,UAAU,CAAC,EAAE,iBAAiB,CAAC;IAC/B;;;;;;;OAOG;IACH,cAAc,CAAC,EAAE,aAAa,GAAG,SAAS,GAAG,WAAW,GAAG,MAAM,CAAC;CACnE,GACA,OAAO,CAAC,cAAc,CAAC,CAgKzB;AAED;;GAEG;AACH,wBAAuB,iBAAiB,CACtC,QAAQ,EAAE,MAAM,EAChB,UAAU,EAAE,MAAM,EAClB,OAAO,CAAC,EAAE;IACR,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,gBAAgB,CAAC,EAAE,OAAO,CAAC;IAC3B,kBAAkB,CAAC,EAAE,OAAO,CAAC;IAC7B,sBAAsB,CAAC,EAAE,MAAM,CAAC;IAChC,oBAAoB,CAAC,EAAE,oBAAoB,CAAC;IAC5C,SAAS,CAAC,EAAE,OAAO,CAAC,SAAS,CAAC,CAAC;IAC/B,yDAAyD;IACzD,cAAc,CAAC,EAAE,aAAa,GAAG,SAAS,GAAG,WAAW,GAAG,MAAM,CAAC;CACnE,GACA,cAAc,CAAC;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,MAAM,CAAC;IAAC,IAAI,CAAC,EAAE,GAAG,CAAA;CAAE,CAAC,CAgF9D;AAGD,eAAO,MAAM,wBAAwB;;;;CAIpC,CAAC"}
|
|
@@ -13,6 +13,7 @@ import { classifySource, applyClassificationGate, } from '../../security-scan/so
|
|
|
13
13
|
import { shouldGateInterproceduralParam, classifyEntryPointTier, } from '../../analysis/entry-point-detection.js';
|
|
14
14
|
import { withLLMContext } from '../../llm/debug-context.js';
|
|
15
15
|
import { fileNeedsEnrichment } from '../../llm/pre-filter.js';
|
|
16
|
+
import { isMinifiedContent } from '../../utils/file-skip.js';
|
|
16
17
|
import { llmCallLogger } from '../../llm/call-logger.js';
|
|
17
18
|
import { resetLLMCallLimiter, isSwarmActive } from '../../llm/call-limiter.js';
|
|
18
19
|
import { resetLLMPerFileBreaker, resetLLMBreakerForScan } from '../../llm/ax-client.js';
|
|
@@ -186,6 +187,36 @@ async function runEnrich(filePath, sourceCode, types, imports, patternSources, p
|
|
|
186
187
|
// the workflow proceeds with `additionalSources: [], additionalSinks: []`
|
|
187
188
|
// and static-analysis sources/sinks flow through merge/verify unchanged.
|
|
188
189
|
// Disable via `LLM_PREFILTER=0` to restore legacy 2-calls-per-file shape.
|
|
190
|
+
// cognium-ai#252 (problem D) — unify the skip thresholds across stages.
|
|
191
|
+
// Verification has refused minified/bundled content since #102, but that
|
|
192
|
+
// decision was made at the END of the pipeline: the file had already paid
|
|
193
|
+
// for a parse, the SAST passes, and both discovery calls before anything
|
|
194
|
+
// looked at its shape. The walker's own caps don't catch this class either
|
|
195
|
+
// — `preFlightSkip` refuses a line over 50k chars, while the minified
|
|
196
|
+
// predicate trips at 5k, so everything in between sailed through the
|
|
197
|
+
// expensive stages only to be dropped at the last one.
|
|
198
|
+
//
|
|
199
|
+
// Same predicate, applied earlier. Static findings are unaffected (this
|
|
200
|
+
// gates LLM enrichment only), matching #102's contract exactly.
|
|
201
|
+
if (isMinifiedContent(sourceCode)) {
|
|
202
|
+
llmCallLogger.logSkip({
|
|
203
|
+
signature: 'discoverSources',
|
|
204
|
+
phase: 'enrichment',
|
|
205
|
+
context: { file: filePath, method: moduleName },
|
|
206
|
+
});
|
|
207
|
+
llmCallLogger.logSkip({
|
|
208
|
+
signature: 'discoverSinks',
|
|
209
|
+
phase: 'enrichment',
|
|
210
|
+
context: { file: filePath, method: moduleName },
|
|
211
|
+
});
|
|
212
|
+
const sizeKb = Math.round(sourceCode.length / 1024);
|
|
213
|
+
console.error(` ⚠ enrichment: skipping minified content (${sizeKb}kb). ` +
|
|
214
|
+
`Static findings unchanged. cognium-ai#252`);
|
|
215
|
+
return {
|
|
216
|
+
enrichmentResult: null,
|
|
217
|
+
processingTimeMs: Date.now() - startTime,
|
|
218
|
+
};
|
|
219
|
+
}
|
|
189
220
|
if (!fileNeedsEnrichment(sourceCode)) {
|
|
190
221
|
// Synthetic JSONL entries so `analyze-llm-log.ts` can surface the
|
|
191
222
|
// skip rate. Two entries (one per call we *would* have fired) keep
|
|
@@ -518,6 +549,11 @@ contextProfile) {
|
|
|
518
549
|
truePositives: 0,
|
|
519
550
|
falsePositives: 0,
|
|
520
551
|
uncertain: 0,
|
|
552
|
+
// #246 — nothing submitted: verification disabled, or the file had no
|
|
553
|
+
// merged sources or sinks. Distinct from "submitted and all omitted".
|
|
554
|
+
pairsSubmitted: 0,
|
|
555
|
+
pairsUnmapped: 0,
|
|
556
|
+
pairsDroppedByBudget: 0,
|
|
521
557
|
processingTimeMs: 0,
|
|
522
558
|
};
|
|
523
559
|
}
|
|
@@ -627,6 +663,17 @@ contextProfile) {
|
|
|
627
663
|
truePositives: batchResult.summary.true_positives,
|
|
628
664
|
falsePositives: batchResult.summary.false_positives,
|
|
629
665
|
uncertain: batchResult.summary.uncertain,
|
|
666
|
+
// cognium-ai#246 — the denominator. Without it, "the verifier declined
|
|
667
|
+
// nothing" is unfalsifiable: the prompt tells the model to omit
|
|
668
|
+
// non-flowing pairs, so a decline leaves no trace in the response.
|
|
669
|
+
pairsSubmitted: batchResult.summary.pairs_submitted ?? 0,
|
|
670
|
+
// #246 — pairs the model DID return that we then discarded because
|
|
671
|
+
// `pair.source_line` / `pair.sink_line` matched no known source/sink
|
|
672
|
+
// (hallucinated or shifted line numbers). A second silent loss, distinct
|
|
673
|
+
// from the model's own omissions.
|
|
674
|
+
pairsUnmapped: Math.max(0, batchResult.pairs.length - results.length),
|
|
675
|
+
// #252 B — pairs our own fan-out budget refused to send.
|
|
676
|
+
pairsDroppedByBudget: batchResult.summary.pairs_dropped_by_budget ?? 0,
|
|
630
677
|
processingTimeMs: batchResult.processingTimeMs,
|
|
631
678
|
};
|
|
632
679
|
}
|
|
@@ -637,6 +684,9 @@ contextProfile) {
|
|
|
637
684
|
truePositives: 0,
|
|
638
685
|
falsePositives: 0,
|
|
639
686
|
uncertain: mergedSources.length * mergedSinks.length,
|
|
687
|
+
pairsSubmitted: mergedSources.length * mergedSinks.length,
|
|
688
|
+
pairsUnmapped: 0,
|
|
689
|
+
pairsDroppedByBudget: 0,
|
|
640
690
|
processingTimeMs: Date.now() - startTime,
|
|
641
691
|
};
|
|
642
692
|
}
|
|
@@ -666,7 +716,15 @@ types,
|
|
|
666
716
|
// "application/json…")` before an XSS-typed sink. Optional — when
|
|
667
717
|
// omitted the JSON-content-type check is a no-op and the existing
|
|
668
718
|
// suppression set is unchanged.
|
|
669
|
-
sourceCode
|
|
719
|
+
sourceCode,
|
|
720
|
+
/**
|
|
721
|
+
* cognium-ai#246 — verifier submission accounting from `runVerify`.
|
|
722
|
+
* `verificationResults` only carries pairs the model returned AND that
|
|
723
|
+
* mapped back to a known source/sink, so on its own it cannot express how
|
|
724
|
+
* much the verifier was asked about. Optional: legacy callers (tests) omit
|
|
725
|
+
* it and the stats fall back to the returned counts.
|
|
726
|
+
*/
|
|
727
|
+
verifyAccounting) {
|
|
670
728
|
const vulnerabilities = [];
|
|
671
729
|
if (verificationEnabled && verificationResults.length > 0) {
|
|
672
730
|
// cognium-ai#213 — collapse verified findings by (sink.line,
|
|
@@ -953,6 +1011,9 @@ sourceCode) {
|
|
|
953
1011
|
truePositives: verificationResults.filter((v) => v.result?.verdict === 'TRUE_POSITIVE').length,
|
|
954
1012
|
falsePositives: verificationResults.filter((v) => v.result?.verdict === 'FALSE_POSITIVE').length,
|
|
955
1013
|
uncertain: verificationResults.filter((v) => v.result?.verdict === 'UNCERTAIN').length,
|
|
1014
|
+
pairsSubmitted: verifyAccounting?.pairsSubmitted ?? verificationResults.length,
|
|
1015
|
+
pairsUnmapped: verifyAccounting?.pairsUnmapped ?? 0,
|
|
1016
|
+
pairsDroppedByBudget: verifyAccounting?.pairsDroppedByBudget ?? 0,
|
|
956
1017
|
totalTimeMs: patternMatchTimeMs + enrichTimeMs + verifyTimeMs,
|
|
957
1018
|
},
|
|
958
1019
|
};
|
|
@@ -1043,7 +1104,13 @@ export async function analyzeFile(filePath, sourceCode, options) {
|
|
|
1043
1104
|
}
|
|
1044
1105
|
const report = runReport(filePath, mergeResult.mergedSources, mergeResult.mergedSinks, verifyResult.verificationResults, patternMatchTimeMs, enrichResult.processingTimeMs, verifyResult.processingTimeMs, enableVerification, patternResult.sastFindings, patternResult.dfg, // #26
|
|
1045
1106
|
patternResult.types, // #111
|
|
1046
|
-
sourceCode
|
|
1107
|
+
sourceCode, // #172
|
|
1108
|
+
// cognium-ai#246 — submission denominator + unmapped-pair loss.
|
|
1109
|
+
{
|
|
1110
|
+
pairsSubmitted: verifyResult.pairsSubmitted,
|
|
1111
|
+
pairsUnmapped: verifyResult.pairsUnmapped,
|
|
1112
|
+
pairsDroppedByBudget: verifyResult.pairsDroppedByBudget,
|
|
1113
|
+
});
|
|
1047
1114
|
// cognium-ai#154 (L5c) — surface the cluster vector extracted while the
|
|
1048
1115
|
// IR was live in `runPatternMatch`. Keeps `runReport`'s signature
|
|
1049
1116
|
// untouched (its job is source×sink → vuln synthesis; the vector is
|
|
@@ -1081,7 +1148,13 @@ export async function* analyzeFileStream(filePath, sourceCode, options) {
|
|
|
1081
1148
|
yield { step: 'report', status: 'running' };
|
|
1082
1149
|
const reportResult = runReport(filePath, mergeResult.mergedSources, mergeResult.mergedSinks, verifyResult.verificationResults, patternMatchTimeMs, enrichResult.processingTimeMs, verifyResult.processingTimeMs, enableVerification, patternResult.sastFindings, patternResult.dfg, // #26
|
|
1083
1150
|
patternResult.types, // #111
|
|
1084
|
-
sourceCode
|
|
1151
|
+
sourceCode, // #172
|
|
1152
|
+
// cognium-ai#246 — submission denominator + unmapped-pair loss.
|
|
1153
|
+
{
|
|
1154
|
+
pairsSubmitted: verifyResult.pairsSubmitted,
|
|
1155
|
+
pairsUnmapped: verifyResult.pairsUnmapped,
|
|
1156
|
+
pairsDroppedByBudget: verifyResult.pairsDroppedByBudget,
|
|
1157
|
+
});
|
|
1085
1158
|
yield { step: 'report', status: 'completed', data: reportResult };
|
|
1086
1159
|
}
|
|
1087
1160
|
// Legacy export for compatibility
|