circle-ir-ai 2.8.5 → 2.8.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +48 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,54 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [2.8.6] - 2026-06-10
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- **#84 postmortem: parse-retry budget was entangled with HTTP retry
|
|
13
|
+
budget in v2.8.5.** Investigation after the v2.8.5 gemma3:12b
|
|
14
|
+
benchmark re-run showed an unchanged score (94/120, same 2 parse
|
|
15
|
+
errors). Inspection of the v2.8.5 verbose trail revealed:
|
|
16
|
+
|
|
17
|
+
```
|
|
18
|
+
[LLM] JSON parse error (array) — parse-retry 1/1
|
|
19
|
+
[LLM] Error: This operation was aborted — retry 2/3 in 10s
|
|
20
|
+
[LLM] JSON parse error (array) — parse-retries exhausted
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
The `parseRetries` counter in `benchmarks/runners/run-cwe-bench-java.ts`
|
|
24
|
+
was declared outside the `for (attempt...)` loop and was consumed by
|
|
25
|
+
the first parse error. When an AbortError subsequently triggered an
|
|
26
|
+
HTTP retry, the retry consumed an `attempt` slot but did not refresh
|
|
27
|
+
the parse-retry budget — so the next parse error had no budget left.
|
|
28
|
+
|
|
29
|
+
Fix: decoupled the two counters. Converted the loop to `while
|
|
30
|
+
(attempt <= MAX_RETRIES)` and now only HTTP-retry sites
|
|
31
|
+
(`attempt++`) consume the HTTP budget. Parse-retry `continue`
|
|
32
|
+
statements leave `attempt` untouched. Each error type now has its
|
|
33
|
+
own independent budget (max 4 HTTP attempts + 1 parse retry per
|
|
34
|
+
call). Maximum loop iterations bounded at 5.
|
|
35
|
+
|
|
36
|
+
### Notes
|
|
37
|
+
|
|
38
|
+
- The v2.8.6 gemma3:12b benchmark (re-run 2026-06-10) scored the
|
|
39
|
+
same 94/120 with 2 parse errors. The retry-logic fix is correct
|
|
40
|
+
but didn't recover the lost detection because gemma3:12b
|
|
41
|
+
consistently fails on the same 2 specific prompts under the
|
|
42
|
+
benchmark's actual conditions (sustained corpus load + cross-file
|
|
43
|
+
context augmentation). Standalone repros with `num_ctx=32768`
|
|
44
|
+
succeed 5/5 and 6/6 respectively, indicating the failure is
|
|
45
|
+
contextual to the runner's full augmented prompt + position in
|
|
46
|
+
the sequence rather than a logic bug.
|
|
47
|
+
|
|
48
|
+
- The fix is the right semantic to ship regardless — removes a
|
|
49
|
+
latent bug from the retry path that would have masked any future
|
|
50
|
+
retry-recoverable failure.
|
|
51
|
+
|
|
52
|
+
- For users seeking zero parse errors, recommend `qwen3-coder:30b`
|
|
53
|
+
(0/113 failures, 91/120 score, ~2x faster per call) over
|
|
54
|
+
`gemma3:12b` (2/109 failures, 94/120 score).
|
|
55
|
+
|
|
8
56
|
## [2.8.5] - 2026-06-09
|
|
9
57
|
|
|
10
58
|
### Fixed
|