@hoanghainh/komagu 2.1.1 → 2.1.2

package/CHANGELOG.md

@@ -2,6 +2,13 @@
 
 All notable changes to `@hoanghainh/komagu` (published as `komainu-guard` up to v1.7.0) are documented here. From v1.2.2 on, entries are generated by [release-please](https://github.com/googleapis/release-please) from [Conventional Commits](https://www.conventionalcommits.org/). Earlier entries are copied from the [GitHub Releases](https://github.com/hoanghainh1188/komainu/releases).
 
+## [2.1.2](https://github.com/hoanghainh1188/komagu/compare/v2.1.1...v2.1.2) (2026-11-01)
+
+
+### Bug Fixes
+
+* **ai:** stop code fields breaking JSON and write findings for flagged verdicts ([#334](https://github.com/hoanghainh1188/komagu/issues/334)) ([b1aab60](https://github.com/hoanghainh1188/komagu/commit/b1aab60ced26d3776bf5d12af2bffb1300cd7668))
+
 ## [2.1.1](https://github.com/hoanghainh1188/komagu/compare/v2.1.0...v2.1.1) (2026-11-01)
 
 

package/assess/lsa/tests/EVAL-RESULTS.md

@@ -308,3 +308,28 @@ Fixture markers were left unchanged so Run 4 and Run 5 share the same ground tru
 3. **Fixtures:** review the `register.php` MASS marker; add BOLA markers for the routes judged real above.
 4. **qwen3.5:9b** still unmeasured: 32K context does not fit in 24 GB next to the OS; needs 16K context or
    a larger machine.
+
+Harness changes for gaps 1 and 2 (#323), not yet measured in a new run:
+- The 4a / 4b templates ask for `recommendation` as one string (no `code_before` / `code_after`). A reply
+  that is still invalid is parsed again after escaping quotes that cannot end their string; such batches
+  are listed under `repaired` in `ai-run-stats.json` and their raw reply is kept in `ai-raw/`.
+- Every `vulnerable` / `needs_review` verdict without a finding gets a `confidence: low` finding anchored
+  to its Tier 1 route or candidate (`operate/ai/verdicts.py`), listed under `findings_from_verdicts`. An
+  offline replay of run A turns the four flagged candidates into located findings (mass assignment 14 / 15
+  before verification) with no SAFE marker hit (`operate/tests/test_ai_verdicts.py`).
+
+### Ground truth change after Run 5 (#323)
+
+The fixtures were changed after Run 5, so runs from Run 6 on are scored against a new ground truth
+(117 → 121 items). The archived `eval-run-4/` and `eval-run-5/` ground-truth files are kept as they were,
+and their tests still score against them.
+
+| Change | Reason |
+|---|---|
+| `php-plain/register.php`: `$role` defaults to `customer`, `extract($_POST)` can overwrite it, and it is inserted (MASS marker kept on line 4) | As written, the INSERT used only `email` and `name`, so the verifier's rejection was sound |
+| New `BOLA` markers: `AccountController.java:21` (`/account/rename`), `CustomerController.cs:29` (`Refresh`), `python-flask/profile.py:18` (`/users/<uid>/name`) and `:26` (`PATCH /users/<uid>`) | The record id comes from the request and is not bound to the caller; checked by hand |
+| No marker for Laravel `account/rename` / `account/profile` or .NET `Customer/EditName` | They act on `$request->user()`, or touch no record, so the Run 5 triage was wrong about them |
+
+Run 5 rescored against the new ground truth (for comparison only; findings unchanged): BOLA 28 / 29
+(run A) and 16 / 29 (run B), precision 0.859 and 0.826, still no false positive on SAFE markers. Every
+new marker was already reported by both runs; Tier 1 lists all of them as BOLA route candidates.

package/assess/lsa/tests/fixtures/dotnet-legacy/Web/Controllers/CustomerController.cs

@@ -26,7 +26,7 @@ namespace Web.Controllers
         [HttpPost]
         public ActionResult Refresh(int id)
         {
-            var customer = db.Customers.Find(id);
+            var customer = db.Customers.Find(id); // LSA-EXPECT:BOLA
             TryUpdateModel(customer); // LSA-EXPECT:MASS
             db.SaveChanges();
             return RedirectToAction("Index");

package/assess/lsa/tests/fixtures/java-legacy/src/com/acme/web/AccountController.java

@@ -18,7 +18,7 @@ public class AccountController {
 
     @RequestMapping(value = "/rename", method = RequestMethod.POST)
     public String rename(@ModelAttribute RenameForm form) { // LSA-EXPECT:SAFE-MASS
-        accountDao.rename(form.getId(), form.getName());
+        accountDao.rename(form.getId(), form.getName()); // LSA-EXPECT:BOLA
         return "redirect:/account";
     }
 

package/assess/lsa/tests/fixtures/php-plain/register.php

@@ -1,6 +1,6 @@
 <?php
 require_once 'inc/db.php';
-
+$role = 'customer';
 extract($_POST); // LSA-EXPECT:MASS
-$stmt = $db->prepare("INSERT INTO users (email, name) VALUES (?, ?)");
-$stmt->execute(array($email, $name));
+$stmt = $db->prepare("INSERT INTO users (email, name, role) VALUES (?, ?, ?)");
+$stmt->execute(array($email, $name, $role));

package/assess/lsa/tests/fixtures/python-flask/profile.py

@@ -15,7 +15,7 @@ def create_user():
 
 @app.route("/users/<int:uid>/name", methods=["POST"])
 def rename(uid):
-    user = User.query.get(uid)
+    user = User.query.get(uid)  # LSA-EXPECT:BOLA
     user.name = request.json["name"]  # LSA-EXPECT:SAFE-MASS
     db.session.commit()
     return jsonify(ok=True)
@@ -23,7 +23,7 @@ def rename(uid):
 
 @app.route("/users/<int:uid>", methods=["PATCH"])
 def patch(uid):
-    user = User.query.get(uid)
+    user = User.query.get(uid)  # LSA-EXPECT:BOLA
     for key, value in request.json.items():  # LSA-EXPECT:MASS
         setattr(user, key, value)
     db.session.commit()

package/operate/ai/client.py

@@ -10,7 +10,7 @@ import re
 import time
 import urllib.error
 import urllib.request
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Tuple
 
 
 class LLMClient:
@@ -107,3 +107,71 @@ class LLMClient:
                 if end_bracket != -1:
                     return json.loads(text[start_bracket : end_bracket + 1])
             raise RuntimeError(f"Could not parse valid JSON from LLM response:\n{content}") from exc
+
+
+def _next_char(text: str, i: int) -> Tuple[str, int]:
+    """First non-space character at or after i, and its index ("" at the end)."""
+    while i < len(text) and text[i].isspace():
+        i += 1
+    return (text[i], i) if i < len(text) else ("", i)
+
+
+def _closes_string(text: str, i: int, container: str, is_key: bool) -> bool:
+    """Whether the quote at text[i] can close the current string, judged by what follows it."""
+    ch, j = _next_char(text, i + 1)
+    if is_key:
+        return ch == ":"
+    if container == "{":
+        return ch == "}" or (ch == "," and _next_char(text, j + 1)[0] == '"')
+    if container == "[":
+        return ch in ",]"
+    return ch == ""
+
+
+def repair_json_quotes(content: str) -> str:
+    """Escape `"` characters that cannot close the string they appear in (#323).
+
+    Small models copy code into string values without escaping it, for example
+    `"code_before": "SELECT * FROM orders WHERE id = " + id",`. A quote inside a string is kept as
+    the closing quote only when what follows fits the enclosing object or array: `:` after a key;
+    `}` or `, "` in an object; `,` or `]` in an array.
+    Only the JSON part of the reply (fenced block, or first `{` to last `}`) is repaired and returned.
+    """
+    text = content.strip()
+    match = re.search(r"```(?:json)?\s*([\s\S]*?)\s*```", text)
+    if match:
+        text = match.group(1).strip()
+    start, end = text.find("{"), text.rfind("}")
+    if start != -1 and end > start:
+        text = text[start : end + 1]
+
+    out: List[str] = []
+    stack: List[str] = []
+    in_string = escaped = is_key = expecting_key = False
+    for i, ch in enumerate(text):
+        if in_string:
+            if escaped:
+                escaped = False
+            elif ch == "\\":
+                escaped = True
+            elif ch == '"':
+                if _closes_string(text, i, stack[-1] if stack else "", is_key):
+                    in_string = False
+                else:
+                    out.append('\\"')
+                    continue
+        elif ch == '"':
+            in_string, is_key = True, bool(stack) and stack[-1] == "{" and expecting_key
+        elif ch in "{[":
+            stack.append(ch)
+            expecting_key = ch == "{"
+        elif ch in "}]":
+            if stack:
+                stack.pop()
+            expecting_key = False
+        elif ch == ",":
+            expecting_key = bool(stack) and stack[-1] == "{"
+        elif ch == ":":
+            expecting_key = False
+        out.append(ch)
+    return "".join(out)

package/operate/ai/runner.py

@@ -22,17 +22,15 @@ import time
 from typing import Any, Callable, Dict, List, Optional
 
 from operate.ai.anchoring import Anchor, AnchorReport, anchor_findings, load_anchors
-from operate.ai.client import LLMClient
+from operate.ai.client import LLMClient, repair_json_quotes
 from operate.ai.context import _included_files, _model_source, _read_snippet, _verification_context  # noqa: F401
+from operate.ai.verdicts import DEFAULT_CWE, SUBTYPES, verdict_findings
 
 REPO_ROOT = Path(__file__).resolve().parents[2]
 LSA_PROMPTS = REPO_ROOT / "assess" / "lsa" / "prompts"
-DEFAULT_CWE = {"bola": ["CWE-284"], "bfla": ["CWE-285"], "mass_assignment": ["CWE-915"],
-               "sql_injection": ["CWE-89"], "nosql_injection": ["CWE-943"]}
 LSA_SCRIPTS = REPO_ROOT / "assess" / "lsa" / "scripts"
 # Valid subtypes per category (finding.schema.json); the first one is used when a model writes an invalid value
-CATEGORY_SUBTYPES = {"bfla": ["missing_role_check", "role_from_request", "client_side_role_check"],
-                     "mass_assignment": ["whole_object_binding", "privileged_field_writable"]}
+CATEGORY_SUBTYPES = {c: SUBTYPES[c] for c in ("bfla", "mass_assignment")}
 FINDING_STATUSES = {"unverified", "confirmed", "likely", "rejected", "needs_review"}
 
 
@@ -62,6 +60,10 @@ class HeadlessAIRunner:
         self.call_errors: Dict[str, Dict[str, str]] = {}
         # Per-phase batches whose first reply was empty or unparseable and were asked again
         self.retries: Dict[str, List[str]] = {}
+        # Per-phase batches that parsed only after escaping stray quotes in string values (#323)
+        self.repaired: Dict[str, List[str]] = {}
+        # Per-phase finding ids generated from flagged verdicts the model wrote no finding for (#323)
+        self.findings_from_verdicts: Dict[str, List[str]] = {}
         # Findings whose location was taken from the Tier 1 inventory (#319)
         self.anchor_report = AnchorReport()
         self._anchors: Optional[Dict[str, Anchor]] = None
@@ -103,19 +105,47 @@ class HeadlessAIRunner:
         if not (reply or "").strip():
             return False
         try:
-            self.client.extract_json(reply)
+            self._extract(reply)
             return True
         except Exception:
             return False
 
+    def _extract(self, reply: Optional[str], phase: Optional[str] = None, batch_id: Optional[str] = None) -> Any:
+        """Parse a reply; if that fails, escape stray quotes in string values and parse again (#323).
+
+        With phase and batch_id, a repaired batch is recorded and its raw reply kept in OUT/ai-raw/.
+        """
+        try:
+            return self.client.extract_json(reply)
+        except Exception:
+            parsed = self.client.extract_json(repair_json_quotes(reply or ""))
+        if phase and batch_id:
+            self.repaired.setdefault(phase, []).append(batch_id)
+            self._save_raw(phase, batch_id, reply or "")
+            self._log("warn", f"{phase}: reply for {batch_id} parsed after escaping stray quotes")
+        return parsed
+
+    def _save_raw(self, phase: str, batch_id: str, reply: str) -> None:
+        raw_dir = self.out_dir / "ai-raw"
+        raw_dir.mkdir(parents=True, exist_ok=True)
+        (raw_dir / f"{phase}-{batch_id}.txt").write_text(reply, encoding="utf-8")
+
+    def _with_verdict_findings(self, phase: str, findings: List[Dict[str, Any]], verdicts: Any, key: str,
+                               records: Dict[str, Dict[str, Any]], category: str) -> List[Dict[str, Any]]:
+        """Add an anchored low-confidence finding for each flagged verdict without one (#323)."""
+        extra = verdict_findings(verdicts, findings, key, records, category)
+        if not extra:
+            return findings
+        self.findings_from_verdicts.setdefault(phase, []).extend(f["finding_id"] for f in extra)
+        self._log("warn", f"{phase}: {len(extra)} flagged verdict(s) had no finding; generated low-confidence findings")
+        return findings + self._anchor(extra)
+
     def _parse_failed(self, phase: str, batch_id: str, reply: Optional[str] = None) -> None:
         if batch_id in self.call_errors.get(phase, {}):
             return
         self.parse_failures.setdefault(phase, []).append(batch_id)
         if reply is not None:
-            raw_dir = self.out_dir / "ai-raw"
-            raw_dir.mkdir(parents=True, exist_ok=True)
-            (raw_dir / f"{phase}-{batch_id}.txt").write_text(reply, encoding="utf-8")
+            self._save_raw(phase, batch_id, reply)
         self._log("warn", f"{phase}: could not parse the model reply for {batch_id}; batch left without verdicts")
 
     def _usage(self) -> Dict[str, Any]:
@@ -140,6 +170,8 @@ class HeadlessAIRunner:
             "parse_failures": self.parse_failures,
             "call_errors": self.call_errors,
             "retries": self.retries,
+            "repaired": self.repaired,
+            "findings_from_verdicts": self.findings_from_verdicts,
             "anchoring": {"moved": self.anchor_report.moved, "unanchored": self.anchor_report.unanchored},
         }
         path = self.out_dir / "ai-run-stats.json"
@@ -408,7 +440,7 @@ Respond STRICTLY with a valid JSON object matching:
       "severity": "high",
       "confidence": "high",
       "cwe": ["CWE-284", "CWE-639"],
-      "recommendation": {{"summary": "Bind query to current session user", "code_before": "...", "code_after": "..."}},
+      "recommendation": "<one string: the fix in words, no code>",
       "status": "unverified"
     }}
   ],
@@ -416,6 +448,7 @@ Respond STRICTLY with a valid JSON object matching:
     {{"route_id": "<route_id>", "verdict": "vulnerable|safe"}}
   ]
 }}
+Write one finding for every "vulnerable" verdict.
 """
             messages = [
                 {"role": "system", "content": auditor_prompt},
@@ -424,10 +457,13 @@ Respond STRICTLY with a valid JSON object matching:
 
             reply = self._chat("4a-bola-auditor", r_file.stem, messages)
             try:
-                parsed = self.client.extract_json(reply)
+                parsed = self._extract(reply, "4a-bola-auditor", r_file.stem)
                 raw_findings = parsed.get("findings", [])
                 findings = self._anchor([self._sanitize_finding(f) for f in raw_findings])
                 route_verdicts = parsed.get("route_verdicts", [])
+                findings = self._with_verdict_findings(
+                    "4a-bola-auditor", findings, route_verdicts, "route_id",
+                    {r.get("route_id"): r for r in routes_to_audit}, "bola")
             except Exception:
                 self._parse_failed("4a-bola-auditor", r_file.stem, reply)
                 findings = []
@@ -495,7 +531,7 @@ Respond STRICTLY with a valid JSON object matching:
       "severity": "critical",
       "confidence": "high",
       "cwe": ["CWE-89"],
-      "recommendation": {{"summary": "Use parameterized queries", "code_before": "...", "code_after": "..."}},
+      "recommendation": "<one string: the fix in words, no code>",
       "status": "unverified"
     }}
   ],
@@ -503,6 +539,7 @@ Respond STRICTLY with a valid JSON object matching:
     {{"candidate_id": "<candidate_id>", "verdict": "vulnerable|safe"}}
   ]
 }}
+Write one finding for every "vulnerable" verdict.
 """
             messages = [
                 {"role": "system", "content": auditor_prompt},
@@ -511,10 +548,13 @@ Respond STRICTLY with a valid JSON object matching:
 
             reply = self._chat("4b-injection-auditor", b_file.stem, messages)
             try:
-                parsed = self.client.extract_json(reply)
+                parsed = self._extract(reply, "4b-injection-auditor", b_file.stem)
                 raw_findings = parsed.get("findings", [])
                 findings = self._anchor([self._sanitize_finding(f) for f in raw_findings])
                 candidate_verdicts = parsed.get("candidate_verdicts", [])
+                findings = self._with_verdict_findings(
+                    "4b-injection-auditor", findings, candidate_verdicts, "candidate_id",
+                    {c.get("candidate_id"): c for c in candidates}, "sql_injection")
             except Exception:
                 self._parse_failed("4b-injection-auditor", b_file.stem, reply)
                 findings = []
@@ -621,10 +661,16 @@ For mass assignment, judge the fields from "model_definition". If it says the de
             ]
             reply = self._chat("4c-privilege-auditor", b_file.stem, messages)
             try:
-                parsed = self.client.extract_json(reply)
+                parsed = self._extract(reply, "4c-privilege-auditor", b_file.stem)
                 findings = self._anchor([self._sanitize_finding(f) for f in parsed.get("findings", [])])
                 route_verdicts = parsed.get("privilege_route_verdicts", [])
                 candidate_verdicts = parsed.get("candidate_verdicts", [])
+                findings = self._with_verdict_findings(
+                    "4c-privilege-auditor", findings, route_verdicts, "route_id",
+                    {r.get("route_id"): r for r in routes}, "bfla")
+                findings = self._with_verdict_findings(
+                    "4c-privilege-auditor", findings, candidate_verdicts, "candidate_id",
+                    {c.get("candidate_id"): c for c in mass}, "mass_assignment")
             except Exception:
                 self._parse_failed("4c-privilege-auditor", b_file.stem, reply)
                 findings, route_verdicts, candidate_verdicts = [], [], []
@@ -712,7 +758,7 @@ Respond STRICTLY with a valid JSON object matching verification.schema.json:
 
             reply = self._chat("5-finding-verifier", f"verified-{(idx // batch_size) + 1}", messages)
             try:
-                parsed = self.client.extract_json(reply)
+                parsed = self._extract(reply, "5-finding-verifier", f"verified-{(idx // batch_size) + 1}")
                 verifs = parsed.get("verifications", [])
             except Exception:
                 # No verdicts: merge_findings.py lists these findings under

package/operate/ai/verdicts.py

@@ -0,0 +1,113 @@
+"""Write a minimal finding for every flagged auditor verdict the model left without one (#323).
+
+Small local models sometimes judge a route or candidate correctly ("vulnerable" / "needs_review")
+but write no finding for it (Evaluation Run 5, run A: four mass assignment candidates). Those count
+as missed although the judgment was right, so a finding is generated from the verdict and the
+Tier 1 record it refers to:
+- category and subtype come from the record (`category_hint` / `subtype_hint` / `subtype`), or the
+  phase's default;
+- location is the record's handler or candidate site (the runner then anchors it like any finding);
+- `confidence: low`, `status: unverified` (the verifier judges it like any finding), and `impact`
+  names the verdict it was generated from.
+
+Standard-library only (Python 3.8+ compatible).
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List
+
+FLAGGED = ("vulnerable", "needs_review")
+# Valid subtypes per category (finding.schema.json); the first one is the default
+SUBTYPES = {
+    "bola": ["missing_ownership_check", "ownership_check_bypassable", "id_from_request_used_as_identity",
+             "no_authentication", "parent_checked_child_not", "check_result_ignored"],
+    "sql_injection": ["string_concatenation", "string_interpolation", "format_string", "template_placeholder",
+                      "dynamic_identifier", "second_order", "stored_procedure_dynamic_sql"],
+    "nosql_injection": ["operator_injection", "js_injection", "string_concatenation", "string_interpolation"],
+    "bfla": ["missing_role_check", "role_from_request", "client_side_role_check"],
+    "mass_assignment": ["whole_object_binding", "privileged_field_writable"],
+}
+DEFAULT_CWE = {"bola": ["CWE-284"], "bfla": ["CWE-285"], "mass_assignment": ["CWE-915"],
+               "sql_injection": ["CWE-89"], "nosql_injection": ["CWE-943"]}
+PREFIX = {"bola": "BOLA", "sql_injection": "INJ", "nosql_injection": "INJ", "bfla": "BFLA", "mass_assignment": "MASS"}
+LABEL = {"bola": "Missing ownership check", "sql_injection": "SQL injection", "nosql_injection": "NoSQL injection",
+         "bfla": "Missing role check", "mass_assignment": "Mass assignment"}
+RECOMMENDATION = {
+    "bola": "Bind the object lookup to the current user or tenant, or check ownership before using it.",
+    "sql_injection": "Use parameterized queries; allow-list any dynamic identifier.",
+    "nosql_injection": "Pass typed values to the query; reject operator objects and server-side JavaScript.",
+    "bfla": "Check the caller's role on the server before the privileged action.",
+    "mass_assignment": "Bind only an explicit allow-list of fields; never bind role, owner or balance fields.",
+}
+
+
+def _category(record: Dict[str, Any], default: str) -> str:
+    hint = record.get("category_hint")
+    return hint if hint in SUBTYPES else default
+
+
+def _subtype(record: Dict[str, Any], category: str) -> str:
+    hints = str(record.get("subtype_hint") or record.get("subtype") or "").split("|")
+    return next((h for h in hints if h in SUBTYPES[category]), SUBTYPES[category][0])
+
+
+def _site(record: Dict[str, Any]) -> Dict[str, Any]:
+    """file / start_line of a route handler or a candidate."""
+    handler = record.get("handler") if isinstance(record.get("handler"), dict) else {}
+    loc = record.get("location") if isinstance(record.get("location"), dict) else {}
+    line = handler.get("line") or record.get("line") or loc.get("start_line")
+    return {"file": handler.get("file") or record.get("file") or loc.get("file") or "unknown",
+            "start_line": line if isinstance(line, int) else 1}
+
+
+def _finding(key: str, record_id: str, verdict: str, record: Dict[str, Any], default_category: str) -> Dict[str, Any]:
+    category = _category(record, default_category)
+    site = _site(record)
+    finding = {
+        "finding_id": f"{PREFIX[category]}-{record_id}-verdict",
+        "category": category,
+        "subtype": _subtype(record, category),
+        "title": f"{LABEL[category]} at {site['file']}:{site['start_line']} (from auditor verdict)",
+        key: record_id,
+        "location": site,
+        "evidence": [{"file": site["file"], "line": site["start_line"],
+                      "why": f"The auditor judged this {'route' if key == 'route_id' else 'candidate'} '{verdict}' but wrote no finding"}],
+        "impact": f"Generated from the auditor's '{verdict}' verdict; the model wrote no finding for it.",
+        "severity": "medium",
+        "confidence": "low",
+        "cwe": list(DEFAULT_CWE[category]),
+        "recommendation": RECOMMENDATION[category],
+        # the verifier (phase 5) decides; the original verdict is kept in impact
+        "status": "unverified",
+    }
+    if key == "route_id" and record.get("path"):
+        method = record.get("method")
+        finding["endpoint"] = {"method": method[0] if isinstance(method, list) and method else method or "GET",
+                               "path": record["path"]}
+    return finding
+
+
+def verdict_findings(verdicts: Any, findings: List[Dict[str, Any]], key: str,
+                     records: Dict[str, Dict[str, Any]], default_category: str) -> List[Dict[str, Any]]:
+    """Findings for flagged verdicts (by `key`: route_id or candidate_id) that no finding covers.
+
+    A verdict is covered by a finding that carries its id in `key`, that the verdict lists in
+    `finding_ids`, or whose finding_id names the record (`BFLA-R-0001-1`). Verdicts whose id is not a
+    known Tier 1 record are skipped: they cannot be located.
+    """
+    dicts = [f for f in findings if isinstance(f, dict)]
+    covered = {f.get(key) for f in dicts}
+    finding_ids = {str(f.get("finding_id")) for f in dicts}
+    out: List[Dict[str, Any]] = []
+    for v in verdicts if isinstance(verdicts, list) else []:
+        if not isinstance(v, dict) or v.get("verdict") not in FLAGGED:
+            continue
+        record_id = v.get(key)
+        listed = v.get("finding_ids") if isinstance(v.get("finding_ids"), list) else []
+        if (record_id in covered or record_id not in records or finding_ids.intersection(map(str, listed))
+                or any(f"-{record_id}-" in f"{fid}-" for fid in finding_ids)):
+            continue
+        covered.add(record_id)
+        out.append(_finding(key, record_id, v["verdict"], records[record_id], default_category))
+    return out

package/operate/cli/__init__.py

@@ -2,4 +2,4 @@
 
 from __future__ import annotations
 
-__version__ = "2.1.1"  # x-release-please-version
+__version__ = "2.1.2"  # x-release-please-version

package/operate/tests/test_ai_verdicts.py

@@ -0,0 +1,200 @@
+"""Issue #323: replies with stray quotes in code fields parse, and flagged verdicts without a finding
+become low-confidence anchored findings (offline replay of Evaluation Run 5, run A)."""
+
+from __future__ import annotations
+
+import json
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+from operate.ai.anchoring import AnchorReport, anchor_findings, load_anchors
+from operate.ai.client import LLMClient, repair_json_quotes
+from operate.ai.runner import HeadlessAIRunner
+from operate.ai.verdicts import verdict_findings
+from operate.tests.test_ai_anchoring import _ScriptedClient
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+EVAL_RUN_5 = REPO_ROOT / "assess" / "lsa" / "examples" / "eval-run-5"
+SCORER = REPO_ROOT / "assess" / "lsa" / "tests" / "score_findings.py"
+
+# The slip that cost batch routes-001 in Run 4 and Run 5, run B (raw reply was not archived)
+SLIPPED_REPLY = """Here is the audit:
+```json
+{
+  "batch_id": "routes-001",
+  "findings": [
+    {
+      "finding_id": "BOLA-R-0001",
+      "category": "bola",
+      "subtype": "missing_ownership_check",
+      "title": "BOLA in GET /orders/{id}",
+      "route_id": "R-0001",
+      "location": {"file": "app/orders.php", "start_line": 3},
+      "evidence": [{"file": "app/orders.php", "start_line": 3, "code": "$id = $_GET["id"];", "why": "id from request"}],
+      "severity": "high",
+      "confidence": "high",
+      "cwe": ["CWE-639"],
+      "recommendation": {"summary": "Bind to the session user", "code_before": "SELECT * FROM orders WHERE id = " + id", "code_after": "... AND user_id = ?"},
+      "status": "unverified"
+    }
+  ],
+  "route_verdicts": [{"route_id": "R-0001", "verdict": "vulnerable"}]
+}
+```"""
+
+
+class TestRepairJsonQuotes(unittest.TestCase):
+    def test_run_5_slip_parses_after_repair(self) -> None:
+        with self.assertRaises((RuntimeError, ValueError)):
+            LLMClient.extract_json(SLIPPED_REPLY)
+        parsed = json.loads(repair_json_quotes(SLIPPED_REPLY))
+        rec = parsed["findings"][0]["recommendation"]
+        self.assertEqual(rec["code_before"], 'SELECT * FROM orders WHERE id = " + id')
+        self.assertEqual(parsed["findings"][0]["evidence"][0]["code"], '$id = $_GET["id"];')
+        self.assertEqual(parsed["route_verdicts"], [{"route_id": "R-0001", "verdict": "vulnerable"}])
+
+    def test_valid_json_is_unchanged(self) -> None:
+        doc = {"a": 'say "hi"', "b": ["x", {"c": "back\\slash"}], "d": 1}
+        text = json.dumps(doc, indent=2)
+        self.assertEqual(repair_json_quotes(text), text)
+        self.assertEqual(json.loads(repair_json_quotes("Answer: " + json.dumps(doc) + " done")), doc)
+
+
+class TestRunnerParsesRepairedReply(unittest.TestCase):
+    def setUp(self) -> None:
+        self.tmp = Path(tempfile.mkdtemp(prefix="komagu_repair_"))
+        self.target, self.out = self.tmp / "app", self.tmp / "out"
+        (self.target / "app").mkdir(parents=True)
+        (self.target / "app" / "orders.php").write_text("<?php\nsession_start();\n$id = $_GET['id'];\n", encoding="utf-8")
+        (self.out / "routes").mkdir(parents=True)
+        route = {"route_id": "R-0001", "method": ["GET"], "path": "/orders/{id}",
+                 "handler": {"file": "app/orders.php", "line": 3}, "needs_bola_audit": True}
+        (self.out / "routes" / "routes-001.json").write_text(json.dumps({"routes": [route]}), encoding="utf-8")
+
+    def tearDown(self) -> None:
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    def test_slipped_bola_reply_keeps_its_findings_without_retry(self) -> None:
+        runner = HeadlessAIRunner(self.target, self.out, _ScriptedClient([SLIPPED_REPLY]))
+        runner.audit_bola_routes()
+        doc = json.loads((self.out / "findings" / "bola-routes-001.json").read_text(encoding="utf-8"))
+        self.assertEqual([f["finding_id"] for f in doc["findings"]], ["BOLA-R-0001"])
+        self.assertEqual(doc["findings"][0]["recommendation"], "Bind to the session user")
+        self.assertEqual((runner.retries, runner.parse_failures), ({}, {}))
+        self.assertEqual(runner.repaired, {"4a-bola-auditor": ["routes-001"]})
+        self.assertTrue((self.out / "ai-raw" / "4a-bola-auditor-routes-001.txt").is_file())
+        stats = json.loads(runner.write_run_stats().read_text(encoding="utf-8"))
+        self.assertEqual(stats["repaired"], {"4a-bola-auditor": ["routes-001"]})
+
+    def test_vulnerable_verdict_without_finding_becomes_a_finding(self) -> None:
+        reply = json.dumps({"batch_id": "routes-001", "findings": [],
+                            "route_verdicts": [{"route_id": "R-0001", "verdict": "vulnerable"}]})
+        runner = HeadlessAIRunner(self.target, self.out, _ScriptedClient([reply]))
+        runner.audit_bola_routes()
+        doc = json.loads((self.out / "findings" / "bola-routes-001.json").read_text(encoding="utf-8"))
+        (f,) = doc["findings"]
+        self.assertEqual((f["category"], f["subtype"], f["confidence"], f["status"]),
+                         ("bola", "missing_ownership_check", "low", "unverified"))
+        self.assertEqual(f["location"], {"file": "app/orders.php", "start_line": 3})
+        self.assertEqual(f["endpoint"], {"method": "GET", "path": "/orders/{id}"})
+        self.assertEqual(runner.findings_from_verdicts, {"4a-bola-auditor": [f["finding_id"]]})
+
+    def test_the_4a_template_no_longer_asks_for_code_fields(self) -> None:
+        client = _ScriptedClient(['{"findings": [], "route_verdicts": []}'])
+        HeadlessAIRunner(self.target, self.out, client).audit_bola_routes()
+        prompt = client.calls[0][-1]["content"]
+        self.assertNotIn("code_before", prompt)
+        self.assertNotIn("code_after", prompt)
+
+
+class TestVerdictFindings(unittest.TestCase):
+    RECORDS = {
+        "M-1": {"candidate_id": "M-1", "file": "a.php", "line": 6, "subtype": "privileged_field_writable"},
+        "I-1": {"candidate_id": "I-1", "file": "q.js", "line": 9, "category_hint": "nosql_injection",
+                "subtype_hint": "operator_injection|dynamic_identifier"},
+    }
+
+    def test_only_flagged_uncovered_known_ids(self) -> None:
+        verdicts = [{"candidate_id": "M-1", "verdict": "needs_review"}, {"candidate_id": "I-1", "verdict": "safe"},
+                    {"candidate_id": "M-9", "verdict": "vulnerable"}, "noise"]
+        (f,) = verdict_findings(verdicts, [], "candidate_id", self.RECORDS, "mass_assignment")
+        self.assertEqual((f["finding_id"], f["subtype"], f["cwe"]), ("MASS-M-1-verdict", "privileged_field_writable", ["CWE-915"]))
+        self.assertIn("needs_review", f["impact"])
+        covered = [{"candidate_id": "M-1"}]
+        self.assertEqual(verdict_findings(verdicts, covered, "candidate_id", self.RECORDS, "mass_assignment"), [])
+
+    def test_finding_linked_by_finding_ids_or_finding_id_covers_the_verdict(self) -> None:
+        findings = [{"finding_id": "MASS-M-1-1"}]
+        self.assertEqual(verdict_findings([{"candidate_id": "M-1", "verdict": "vulnerable"}], findings,
+                                          "candidate_id", self.RECORDS, "mass_assignment"), [])
+        listed = [{"candidate_id": "M-1", "verdict": "vulnerable", "finding_ids": ["F-9"]}]
+        self.assertEqual(verdict_findings(listed, [{"finding_id": "F-9"}], "candidate_id", self.RECORDS, "mass_assignment"), [])
+        # M-1 must not be covered by a finding for M-10
+        (f,) = verdict_findings([{"candidate_id": "M-1", "verdict": "vulnerable"}], [{"finding_id": "MASS-M-10-1"}],
+                                "candidate_id", self.RECORDS, "mass_assignment")
+        self.assertEqual(f["candidate_id"], "M-1")
+
+    def test_category_and_subtype_hints_from_the_record(self) -> None:
+        (f,) = verdict_findings([{"candidate_id": "I-1", "verdict": "vulnerable"}], [], "candidate_id",
+                                self.RECORDS, "sql_injection")
+        self.assertEqual((f["category"], f["subtype"], f["cwe"]), ("nosql_injection", "operator_injection", ["CWE-943"]))
+
+    def test_generated_findings_match_the_finding_schema(self) -> None:
+        sys.path.insert(0, str(REPO_ROOT / "assess" / "lsa" / "scripts"))
+        from merge_findings import load_validator
+        validate = load_validator("finding.schema.json")
+        records = {**self.RECORDS, "R-1": {"route_id": "R-1", "method": ["POST"], "path": "/users/{id}/role",
+                                           "handler": {"file": "admin.php", "line": 4}}}
+        for key, record_id, category in (("route_id", "R-1", "bola"), ("route_id", "R-1", "bfla"),
+                                         ("candidate_id", "M-1", "mass_assignment"), ("candidate_id", "I-1", "sql_injection")):
+            with self.subTest(category=category):
+                (f,) = verdict_findings([{key: record_id, "verdict": "vulnerable"}], [], key, records, category)
+                self.assertEqual(validate(f), [])
+
+
+class TestReplayRun5RunA(unittest.TestCase):
+    """The four flagged mass assignment candidates of run A become located findings; no SAFE marker is hit."""
+
+    def setUp(self) -> None:
+        self.tmp = Path(tempfile.mkdtemp(prefix="komagu_replay_"))
+        self.out = self.tmp / "gptoss-a"
+        shutil.copytree(EVAL_RUN_5 / "gptoss-a", self.out)
+
+    def tearDown(self) -> None:
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    def score(self) -> dict:
+        proc = subprocess.run([sys.executable, str(SCORER), "--verdicts", str(EVAL_RUN_5 / "ground-truth.json"), str(self.out)],
+                              capture_output=True, text=True, check=True)
+        return json.loads(proc.stdout)
+
+    def test_replay_recovers_the_four_mass_assignment_items(self) -> None:
+        cands = json.loads((self.out / "mass-assignment.candidates.json").read_text(encoding="utf-8"))["candidates"]
+        records = {c["candidate_id"]: c for c in cands}
+        # the fixtures are not in the archive; anchoring only needs the Tier 1 records
+        anchors, report = load_anchors(self.out, self.tmp), AnchorReport()
+        generated = []
+        for path in sorted((self.out / "findings").glob("priv-*.json")):
+            doc = json.loads(path.read_text(encoding="utf-8"))
+            extra = verdict_findings(doc["candidate_verdicts"], doc["findings"], "candidate_id", records, "mass_assignment")
+            generated += anchor_findings(extra, anchors, self.tmp, report)
+            generated += verdict_findings(doc["privilege_route_verdicts"], doc["findings"], "route_id", {}, "bfla")
+        self.assertEqual(sorted(f["candidate_id"] for f in generated), ["M-0001", "M-0002", "M-0011", "M-0012"])
+        self.assertEqual(report.unanchored, [])
+
+        merged = json.loads((self.out / "findings.json").read_text(encoding="utf-8"))
+        merged["findings"] += generated
+        (self.out / "findings.json").write_text(json.dumps(merged), encoding="utf-8")
+        mass = self.score()["per_category"]["mass_assignment"]
+        # before: 10 / 15; register.php:4 stays missed (rejected by the verifier, fixture under review)
+        self.assertEqual((mass["found"], mass["missed"]), (14, ["php-plain/register.php:4"]))
+        for cat, row in self.score()["per_category"].items():
+            self.assertEqual(row["false_positive_on_safe_code"], [], cat)
+
+
+if __name__ == "__main__":
+    unittest.main()

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hoanghainh/komagu",
-  "version": "2.1.1",
+  "version": "2.1.2",
   "description": "Security-First Legacy Modernization Platform CLI (LSA Kit, IAM Perimeter, Cross-User Testing)",
   "main": "remediate/authz/nodejs/src/index.js",
   "types": "remediate/authz/nodejs/src/index.d.ts",