@hoanghainh/komagu 2.0.0 → 2.0.1

package/assess/lsa/prompts/35-privilege-auditor.md

@@ -69,7 +69,9 @@ privileged, and a hinted route can be safe.
 2. **Find the model's fields.** Read the model / entity / table. List the
    privileged fields it holds (role, admin flag, permissions, owner, user,
    tenant, price, amount, balance, status, verified, approved, plan).
-   No privileged field → `safe`, reason "no privileged field".
+   No privileged field → `safe`, reason "no privileged field". If you cannot
+   find or read the model's definition, you do not know its fields: answer
+   `needs_review` (reason "model fields not visible"), never `safe`.
 3. **Look for what limits the fields**, with `file:line`: explicit field
    copy, allow-list (`$fillable` without privileged fields, `permit(...)`,
    `[Bind(Include=...)]`, DTO with only editable fields), validation that

package/assess/lsa/tests/EVAL-RESULTS.md

@@ -152,3 +152,159 @@ Based on Qwen 3.5's findings, the following concrete improvements were implement
 | **False Positives on Safe Code** | 0 | 0 | 0 on safe markers | **0** |
 | **Precision (matched / active)** | 0.965 (55 / 57) | 0.965 (55 / 57) | 0.966 | **1.000 (57 / 57)** |
 | **Legacy Stack Shims** | None | None | Recommended | **4 Stacks (`remediate/legacy_shims/`)** |
+
+---
+
+## Run 4 — 2026-10-31, kit 0.3.1 (BFLA & mass assignment Tier 2, local Ollama)
+
+Issue #286. First end-to-end measurement of phase 4c (`35-privilege-auditor`) with real models, and a
+re-check of BOLA / injection on the same harness. All runs are **local only**: fixture code never left
+the machine, and no third-party LLM was used.
+
+### Setup
+
+| Item | Value |
+|---|---|
+| Eval repo | `tests/build_eval_repo.py` (10 fixture apps, markers stripped); ground truth 117 items: 25 BOLA, 34 SQL/NoSQL injection, 10 BFLA, 15 mass assignment, plus 33 SAFE markers |
+| Pipeline | `komagu scan <repo> --no-cache --headless-ai http://localhost:11434/v1 --model <model> --ai-timeout 3600` (`operate/ai/runner.py`, Tier 1 pre-scan + phases 2–6) |
+| Hosting | Ollama 0.40.13 on an Apple M4 Pro, 24 GB RAM, 100% GPU; context 32,768 tokens |
+| Models | `gpt-oss:20b` (MXFP4, reasoning effort default `medium`), `qwen3.5:9b` |
+| Scoring | `tests/score_findings.py --verdicts` (tolerance 3 lines); archived per run in [`assess/lsa/examples/eval-run-4/`](../examples/eval-run-4/) |
+
+**This is a floor, not a ceiling.** A 9–20B model on a laptop is much weaker than the models used in
+Runs 1–2. The numbers below say what a fully local deployment achieves today, not what the prompts can
+do with a stronger model.
+
+**Not comparable with Run 3.** Run 3 does not record its model, and the archived Run 1 / Run 2 / Qwen
+numbers were produced through agent runtimes that can open files, not through the headless runner.
+The BOLA / injection columns below are therefore a new baseline for the headless harness, not a
+regression check against Run 3.
+
+### Harness fixes found by this run
+
+Measuring with real models exposed defects in the headless runner that inflated or lost results.
+They were fixed before the numbers were trusted, in two iterations (the limit set in #286):
+
+| Iteration | Fix | Effect seen |
+|---|---|---|
+| 0 → 1 | Verifier no longer marks a whole batch `confirmed` when its reply cannot be parsed (it used to, with a made-up "consensus" note); such findings stay `unverified` | Results stopped being inflated by broken replies |
+| 0 → 1 | A failed call (timeout) is recorded per batch in `ai-run-stats.json` instead of aborting the run | A 25-candidate injection batch took > 15 min and used to kill the run |
+| 0 → 1 | Phase 4c gets the same JSON output template as 4a / 4b; sanitizer joins list `impact` values and moves a verdict written into `subtype` to `status`; unparseable replies saved to `OUT/ai-raw/` | Mass assignment findings stopped failing schema validation |
+| 1 → 2 | Verifier now receives the code it judges (finding site, file header, evidence lines, server-side includes) and the auth model, and may only cite controls that are visible; otherwise `likely` + `code_not_visible` | In iteration 1 it rejected 10 of 11 real BFLA findings by inventing an admin base class and a role check in a login-only `auth.php` |
+| 1 → 2 | Verifier batches of 5 instead of 10; sanitizer fills a `null` evidence line and a missing `status` | No verifier parse failures in iteration 2 |
+
+### Results after verification
+
+"Marker" columns use the normal location match. "Verdict" columns (BFLA / mass assignment only) count a
+truth item as found when the privilege auditor judged the route or candidate that contains it
+`vulnerable` or `needs_review`, whatever path and line the model then wrote into the finding.
+
+| Run | BOLA | Injection | BFLA marker | Mass marker | BFLA verdict | Mass verdict | FP on safe code | Precision | Rejected | Unverified | Parse fail | Min · calls · tokens |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| iter 0, gpt-oss A | 19/25 | 20/34 | 8/10 | 2/15 | 10/10 | 12/15 | 0 | 0.689 | 10 | 11 | 3 | 81 · 22 · 311K |
+| iter 1, gpt-oss A | 21/25 | 34/34 | 1/10 | 2/15 | 10/10 | 14/15 | 0 | 0.765 | 19 | 17 | 2 | 75 · 21 · 288K |
+| iter 1, gpt-oss B | 19/25 | 28/34 | 5/10 | 5/15 | 10/10 | 14/15 | 1 | 0.667 | 4 | 8 | 3 | 80 · 22 · 273K |
+| iter 1, qwen3.5 (incomplete) | 23/25 | 33/34 | 0/10 | 0/15 | 0/10 | 8/15 | 1 | 0.778 | 0 | 52 | — | stopped after 2 h 50 min |
+| **iter 2, gpt-oss A** | **24/25** | **34/34** | **8/10** | **6/15** | **10/10** | **10/15** | **0** | 0.729 | 1 | 0 | 0 | 57 · 32 · 309K |
+| **iter 2, gpt-oss B** | **11/25** | **32/34** | **5/10** | **4/15** | **10/10** | **9/15** | **0** | 0.693 | 1 | 0 | 1 | 56 · 29 · 318K |
+
+Final harness = iteration 2. Stated plainly for `gpt-oss:20b` on it:
+
+- **BFLA:** the auditor flagged the right route for **10 / 10** BFLA markers in both runs, with no SAFE-BFLA
+  route flagged. Only **8 / 10 and 5 / 10** survive as correctly located findings, because the model
+  writes wrong or non-existent paths into the finding (`dotnet-mvc/Controllers/AdminController.cs`,
+  `routes.js`).
+- **Mass assignment:** **10 / 15 and 9 / 15** at verdict level, **6 / 15 and 4 / 15** as located findings.
+  Five candidates in .NET (`CustomerController.cs`) and Java (`AccountController.java`,
+  `ProfileServlet.java`) were judged `safe` in both runs: real model misses.
+- **False positives on SAFE markers:** **0** in both iteration-2 runs, for all four categories.
+- **BOLA / injection:** 24 / 25 and 34 / 34 in run A. In run B, 11 of its 14 missed BOLA items sit in
+  one batch whose reply was empty (`routes-001`: `gpt-oss` returned no content, so 15 routes got no
+  verdict); the other 3 are real misses (`perl-cgi/cgi-bin/order.cgi:19` is missed in both runs).
+- **Run-to-run variance is large.** Two runs with the same model, harness and temperature 0.1 differ by
+  up to 13 BOLA items and 3 BFLA findings. Single-run numbers from a local model should not be quoted
+  as product capability.
+- **Precision ~0.7** is measured against markers only. Of the ~25 unmatched findings per run, most are BOLA
+  findings in files that carry BFLA / mass-assignment markers but no BOLA markers
+  (`CustomerController.cs`, `AccountController.java`, `AdminController.cs`), plus the misplaced BFLA /
+  mass findings above. They have not been triaged by a person and are not counted as false positives.
+
+**qwen3.5:9b** could not be measured on this machine: with a 32K context the MLX runner used 22 GB of the
+24 GB RAM, the system swapped (29.7 / 30 GB), and two calls hit the 1-hour timeout (the privilege batch
+holding all BFLA routes, and a verifier batch). The run was stopped after 2 h 50 min with 52 findings
+unverified. Its partial output is archived as `iter1-qwen-a-incomplete/` and says nothing about the
+model's ability.
+
+### Known gaps (follow-up: #319)
+
+1. **Misplaced findings.** Take `file` / `line` of BFLA and mass assignment findings from the Tier 1 route
+   or candidate the auditor judged (`route_id` / `candidate_id`), not from the model's text. Expected to
+   close most of the gap between the marker and verdict columns.
+2. **Empty replies.** Retry a batch once when the reply is empty or unparseable (seen with `gpt-oss`).
+3. **Mass assignment on .NET / Java binding.** Review `knowledge/mass-assignment-patterns.md` for
+   `UpdateModel` / `TryUpdateModel` and Spring `@ModelAttribute` binding into entities.
+4. **Re-run qwen3.5:9b** with 16K context or on a machine with more memory.
+5. **Triage unmatched BOLA findings** and add markers to the fixtures where they are real.
+
+---
+
+## Run 5 — 2026-10-31, kit 0.3.1 (Run 4 follow-up: anchored locations, retries, model context)
+
+Issue #319. Same eval repo, ground truth, hosting and model as Run 4 (`gpt-oss:20b`, Ollama 0.40.13,
+M4 Pro 24 GB, 32K context, local only). Outputs are archived in
+[`assess/lsa/examples/eval-run-5/`](../examples/eval-run-5/).
+
+### Changes since Run 4 iteration 2
+
+| Change | Why (from Run 4) |
+|---|---|
+| Finding `file` / `line` come from the Tier 1 route handler span or candidate the finding references (`operate/ai/anchoring.py`); evidence is re-pointed or dropped; unknown references get `confidence: low` | The auditor judged the right route for 10 / 10 BFLA markers, but wrote paths such as `routes.js`, `app.py`, `dotnet-mvc/...` |
+| An empty or unparseable batch reply is retried once with a JSON-only reminder | One empty reply cost 15 BOLA routes |
+| Each mass assignment candidate carries the bound model's definition (same app and language); an unknown model means `needs_review`, never `safe` (prompt `35-privilege-auditor.md` and the headless 4c prompt) | Five .NET / Java candidates were judged `safe` in both runs because the auditor never saw `Account.java` (`role`, `balance`) |
+
+### Results after verification
+
+| Run | BOLA | Injection | BFLA marker | Mass marker | BFLA verdict | Mass verdict | FP on safe code | Precision | Rejected | Unverified | Retries · parse fail | Min · calls · tokens |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| Run 4 iter 2, A | 24/25 | 34/34 | 8/10 | 6/15 | 10/10 | 10/15 | 0 | 0.729 | 1 | 0 | — · 0 | 57 · 32 · 309K |
+| Run 4 iter 2, B | 11/25 | 32/34 | 5/10 | 4/15 | 10/10 | 9/15 | 0 | 0.693 | 1 | 0 | — · 1 | 56 · 29 · 318K |
+| **Run 5, A** | **24/25** | **34/34** | **10/10** | **10/15** | **10/10** | **15/15** | **0** | **0.818** | 2 | 0 | 0 · 0 | 83 · 32 · 328K |
+| **Run 5, B** | **12/25** | **32/34** | **9/10** | **14/15** | **10/10** | **15/15** | **0** | **0.779** | 3 | 0 | 1 · 1 | 86 · 31 · 369K |
+
+- **BFLA:** 10 / 10 and 9 / 10 as located findings (from 8 and 5). Every anchored finding pointed at a
+  known route or candidate (0 unanchored).
+- **Mass assignment:** the auditor flagged **15 / 15** candidates in both runs (from 10 and 9), including the
+  five .NET / Java bindings. As located findings 10 / 15 and 14 / 15: in run A the model gave a
+  `vulnerable` verdict for four candidates (`php-plain/profile.php` ×2, `CustomerController.cs` ×2)
+  without writing a finding for them.
+- **`php-plain/register.php:4` (MASS marker)** was rejected by the verifier in both runs: `extract($_POST)`
+  is followed by an INSERT of only `email` and `name`. The argument is sound for this fixture as written,
+  so the marker should be reviewed rather than counted as a verifier error.
+- **BOLA in run B (12 / 25)** again lost batch `routes-001` (15 routes). This time the reply was not empty
+  but invalid JSON: the model put code with unescaped `"` into `recommendation.code_before`; the retry
+  failed the same way. The same batch failed in Run 4 run B.
+- **No false positives** on SAFE markers in either run; precision rose to 0.82 / 0.78.
+- Runs are ~45% slower than Run 4 iteration 2 (83–86 min): the privilege and verifier prompts are longer
+  and there are more findings to verify.
+
+### Unmatched BOLA findings (triage, not yet reflected in fixtures)
+
+About 20 BOLA findings per run sit in files with no BOLA marker. By route:
+
+| Route(s) | Assessment |
+|---|---|
+| `Customer/EditName`, `account/rename`, `python-flask` `/users/<uid>/name`, `account/profile` | Likely **real BOLA** not marked: the record ID comes from the request and is not bound to the caller |
+| `Customer/Edit`, `Customer/Refresh`, `account/save`, `account/api`, `/users/<uid>` | Mass-assignment fixtures that also take the ID from the request: plausibly BOLA as well |
+| `Admin/DeleteUser`, `Admin/Export`, `admin/users/{id}/delete`, `delete_user.php`, `promote.php`, `manage_users.php` | **BFLA routes also reported as BOLA**: same flaw seen from a second category; duplicate rather than wrong |
+
+Fixture markers were left unchanged so Run 4 and Run 5 share the same ground truth.
+
+### Remaining gaps
+
+1. **Invalid JSON from code fields:** drop `code_before` / `code_after` from the 4a / 4b output templates
+   (the sanitizer already reduces `recommendation` to a string), or repair unescaped quotes before parsing.
+2. **Verdict without finding:** write a minimal finding for every `vulnerable` / `needs_review` verdict
+   the model did not turn into one.
+3. **Fixtures:** review the `register.php` MASS marker; add BOLA markers for the routes judged real above.
+4. **qwen3.5:9b** still unmeasured: 32K context does not fit in 24 GB next to the OS; needs 16K context or
+   a larger machine.

package/assess/lsa/tests/score_findings.py

@@ -7,13 +7,20 @@ A vulnerable truth item is found (TP) when an active finding (status != rejected
 same category (sql_injection and nosql_injection are treated as one group), and the truth
 line is within --tolerance lines of the finding location or of any evidence line in the same file.
 A safe truth item hit by an active finding counts as a false positive (FP-safe).
+
+--verdicts adds a verdict-level score for bfla and mass_assignment: a truth item counts as found when
+the privilege auditor judged the route or candidate that contains it "vulnerable" or "needs_review",
+whatever file and line the model then wrote into the finding. Routes cover their handler line up to
+the next inventoried route in the same file; mass assignment candidates match their exact line.
 """
 from __future__ import annotations
 
 import argparse
+import glob
 import json
 import sys
 from pathlib import Path
+from typing import Dict, List, Optional, Tuple
 
 
 def group(cat: str) -> str:
@@ -32,11 +39,60 @@ def hits(finding: dict, item: dict, tol: int) -> bool:
     return False
 
 
+FLAGGED = ("vulnerable", "needs_review")
+
+
+def _same_file(a: str, b: str) -> bool:
+    return bool(a) and bool(b) and (a.endswith(b) or b.endswith(a))
+
+
+def _route_for(item: dict, routes: List[Tuple[str, int, str]]) -> Optional[str]:
+    """route_id whose handler span (handler line .. next route in the file) contains the truth line."""
+    in_file = sorted((line, rid) for f, line, rid in routes if _same_file(f, item["file"]) and line <= item["line"])
+    return in_file[-1][1] if in_file else None
+
+
+def verdict_scores(out: Path, truth: List[dict]) -> Dict[str, dict]:
+    routes: List[Tuple[str, int, str]] = []
+    for p in glob.glob(str(out / "routes" / "*.json")):
+        for r in json.loads(Path(p).read_text()).get("routes", []):
+            h = r.get("handler") or {}
+            if h.get("file") and isinstance(h.get("line"), int):
+                routes.append((h["file"], h["line"], r.get("route_id")))
+    cands_path = out / "mass-assignment.candidates.json"
+    cands = json.loads(cands_path.read_text()).get("candidates", []) if cands_path.is_file() else []
+    route_v: Dict[str, str] = {}
+    cand_v: Dict[str, str] = {}
+    for p in glob.glob(str(out / "findings" / "priv-*.json")):
+        doc = json.loads(Path(p).read_text())
+        route_v.update({v.get("route_id"): v.get("verdict") for v in doc.get("privilege_route_verdicts", []) if isinstance(v, dict)})
+        cand_v.update({v.get("candidate_id"): v.get("verdict") for v in doc.get("candidate_verdicts", []) if isinstance(v, dict)})
+
+    def flagged(item: dict) -> bool:
+        if item["category"] == "bfla":
+            return route_v.get(_route_for(item, routes)) in FLAGGED
+        return any(_same_file(c.get("file", ""), item["file"]) and c.get("line") == item["line"]
+                   and cand_v.get(c.get("candidate_id")) in FLAGGED for c in cands)
+
+    rows = {}
+    for cat in ("bfla", "mass_assignment"):
+        vul = [t for t in truth if t["category"] == cat and t["vulnerable"]]
+        safe = [t for t in truth if t["category"] == cat and not t["vulnerable"]]
+        if not vul:
+            continue
+        found = [t for t in vul if flagged(t)]
+        rows[cat] = {"truth_vulnerable": len(vul), "found": len(found), "recall": round(len(found) / len(vul), 3),
+                     "flagged_safe": [f"{t['file']}:{t['line']}" for t in safe if flagged(t)],
+                     "missed": [f"{t['file']}:{t['line']}" for t in vul if t not in found]}
+    return rows
+
+
 def main() -> int:
     ap = argparse.ArgumentParser(description=__doc__)
     ap.add_argument("ground_truth")
     ap.add_argument("out")
     ap.add_argument("--tolerance", type=int, default=3)
+    ap.add_argument("--verdicts", action="store_true", help="also score bfla / mass_assignment by auditor verdicts")
     a = ap.parse_args()
     truth = json.loads(Path(a.ground_truth).read_text())["items"]
     # fixture paths are relative to tests/fixtures; eval repo keeps the same layout
@@ -69,6 +125,8 @@ def main() -> int:
     report = {"per_category": rows, "active_findings": len(g_findings), "matched_findings": len(matched_findings),
               "precision": precision, "unmatched_findings": unmatched,
               "needs_review_not_scored": needs_review}
+    if a.verdicts:
+        report["verdict_level"] = verdict_scores(Path(a.out), truth)
     print(json.dumps(report, indent=2))
     return 0
 

package/assess/lsa/tests/test_eval_run_4.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+"""Keeps the archived Evaluation Run 4 numbers (Issue #286) in step with the scorer and EVAL-RESULTS.md."""
+
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+TESTS_DIR = Path(__file__).resolve().parent
+LSA_ROOT = TESTS_DIR.parent
+EVAL_RUN_4_DIR = LSA_ROOT / "examples" / "eval-run-4"
+GROUND_TRUTH = EVAL_RUN_4_DIR / "ground-truth.json"
+RUNS = ("iter0-gptoss-a", "iter1-gptoss-a", "iter1-gptoss-b", "iter1-qwen-a-incomplete", "iter2-gptoss-a", "iter2-gptoss-b")
+# (BOLA, injection, BFLA, mass assignment) found by location, then BFLA / mass found by auditor verdict
+EXPECTED = {
+    "iter2-gptoss-a": {"marker": (24, 34, 8, 6), "verdict": (10, 10)},
+    "iter2-gptoss-b": {"marker": (11, 32, 5, 4), "verdict": (10, 9)},
+}
+
+
+def score(run: str) -> dict:
+    proc = subprocess.run(
+        [sys.executable, str(TESTS_DIR / "score_findings.py"), "--verdicts", str(GROUND_TRUTH), str(EVAL_RUN_4_DIR / run)],
+        capture_output=True, text=True, check=True,
+    )
+    return json.loads(proc.stdout)
+
+
+class TestEvaluationRun4(unittest.TestCase):
+    def test_artifacts_exist(self) -> None:
+        self.assertTrue(GROUND_TRUTH.is_file())
+        for run in RUNS:
+            for name in ("findings.json", "coverage.json", "score.json"):
+                self.assertTrue((EVAL_RUN_4_DIR / run / name).is_file(), f"{run}/{name}")
+
+    def test_archived_scores_match_the_scorer(self) -> None:
+        for run in RUNS:
+            with self.subTest(run=run):
+                archived = json.loads((EVAL_RUN_4_DIR / run / "score.json").read_text(encoding="utf-8"))
+                self.assertEqual(score(run), archived)
+
+    def test_final_harness_numbers_match_eval_results(self) -> None:
+        for run, want in EXPECTED.items():
+            with self.subTest(run=run):
+                report = score(run)
+                pc, vl = report["per_category"], report["verdict_level"]
+                got = tuple(pc[c]["found"] for c in ("bola", "injection", "bfla", "mass_assignment"))
+                self.assertEqual(got, want["marker"])
+                self.assertEqual((vl["bfla"]["found"], vl["mass_assignment"]["found"]), want["verdict"])
+
+    def test_final_harness_has_no_false_positive_on_safe_code(self) -> None:
+        for run in EXPECTED:
+            with self.subTest(run=run):
+                report = score(run)
+                for cat, row in report["per_category"].items():
+                    self.assertEqual(row["false_positive_on_safe_code"], [], cat)
+                for cat, row in report["verdict_level"].items():
+                    self.assertEqual(row["flagged_safe"], [], cat)
+
+    def test_final_harness_records_run_stats(self) -> None:
+        for run in EXPECTED:
+            with self.subTest(run=run):
+                stats = json.loads((EVAL_RUN_4_DIR / run / "ai-run-stats.json").read_text(encoding="utf-8"))
+                self.assertEqual(stats["model"], "gpt-oss:20b")
+                self.assertGreater(stats["totals"]["prompt_tokens"], 0)
+                self.assertEqual(stats["call_errors"], {})
+
+
+if __name__ == "__main__":
+    unittest.main()

package/assess/lsa/tests/test_eval_run_5.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+"""Keeps the archived Evaluation Run 5 numbers (Issue #319) in step with the scorer and EVAL-RESULTS.md."""
+
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+TESTS_DIR = Path(__file__).resolve().parent
+LSA_ROOT = TESTS_DIR.parent
+EVAL_RUN_5_DIR = LSA_ROOT / "examples" / "eval-run-5"
+GROUND_TRUTH = EVAL_RUN_5_DIR / "ground-truth.json"
+RUNS = ("gptoss-a", "gptoss-b")
+# (BOLA, injection, BFLA, mass assignment) found by location, then BFLA / mass found by auditor verdict
+EXPECTED = {
+    "gptoss-a": {"marker": (24, 34, 10, 10), "verdict": (10, 15)},
+    "gptoss-b": {"marker": (12, 32, 9, 14), "verdict": (10, 15)},
+}
+
+def score(run: str) -> dict:
+    proc = subprocess.run(
+        [sys.executable, str(TESTS_DIR / "score_findings.py"), "--verdicts", str(GROUND_TRUTH), str(EVAL_RUN_5_DIR / run)],
+        capture_output=True, text=True, check=True,
+    )
+    return json.loads(proc.stdout)
+
+
+class TestEvaluationRun5(unittest.TestCase):
+    def test_artifacts_exist(self) -> None:
+        self.assertTrue(GROUND_TRUTH.is_file())
+        for run in RUNS:
+            for name in ("findings.json", "coverage.json", "score.json"):
+                self.assertTrue((EVAL_RUN_5_DIR / run / name).is_file(), f"{run}/{name}")
+
+    def test_archived_scores_match_the_scorer(self) -> None:
+        for run in RUNS:
+            with self.subTest(run=run):
+                archived = json.loads((EVAL_RUN_5_DIR / run / "score.json").read_text(encoding="utf-8"))
+                self.assertEqual(score(run), archived)
+
+    def test_final_harness_numbers_match_eval_results(self) -> None:
+        for run, want in EXPECTED.items():
+            with self.subTest(run=run):
+                report = score(run)
+                pc, vl = report["per_category"], report["verdict_level"]
+                got = tuple(pc[c]["found"] for c in ("bola", "injection", "bfla", "mass_assignment"))
+                self.assertEqual(got, want["marker"])
+                self.assertEqual((vl["bfla"]["found"], vl["mass_assignment"]["found"]), want["verdict"])
+
+    def test_final_harness_has_no_false_positive_on_safe_code(self) -> None:
+        for run in EXPECTED:
+            with self.subTest(run=run):
+                report = score(run)
+                for cat, row in report["per_category"].items():
+                    self.assertEqual(row["false_positive_on_safe_code"], [], cat)
+                for cat, row in report["verdict_level"].items():
+                    self.assertEqual(row["flagged_safe"], [], cat)
+
+    def test_final_harness_records_run_stats(self) -> None:
+        for run in EXPECTED:
+            with self.subTest(run=run):
+                stats = json.loads((EVAL_RUN_5_DIR / run / "ai-run-stats.json").read_text(encoding="utf-8"))
+                self.assertEqual(stats["model"], "gpt-oss:20b")
+                self.assertGreater(stats["totals"]["prompt_tokens"], 0)
+                self.assertEqual(stats["call_errors"], {})
+                self.assertEqual(stats["anchoring"]["unanchored"], [])
+
+
+if __name__ == "__main__":
+    unittest.main()

package/assess/lsa/tests/test_privilege_tier2.py

@@ -242,6 +242,36 @@ class TestScoring(unittest.TestCase):
             self.assertEqual(report["per_category"]["mass_assignment"]["recall"], 0.0)
             self.assertNotIn("bola", report["per_category"])
 
+    def test_verdict_level_ignores_misplaced_findings(self) -> None:
+        """--verdicts credits the auditor's route / candidate verdict even when the finding has a wrong path (#286)."""
+        with tempfile.TemporaryDirectory() as tmp:
+            truth = Path(tmp) / "gt.json"
+            write_json(truth, {"items": [
+                {"file": "admin.js", "line": 9, "category": "bfla", "vulnerable": True, "label": "BFLA"},
+                {"file": "admin.js", "line": 13, "category": "bfla", "vulnerable": False, "label": "SAFE-BFLA"},
+                {"file": "p.php", "line": 6, "category": "mass_assignment", "vulnerable": True, "label": "MASS"},
+                {"file": "p.php", "line": 9, "category": "mass_assignment", "vulnerable": False, "label": "SAFE-MASS"},
+            ]})
+            out = Path(tmp) / "out"
+            write_json(out / "findings.json", {"findings": [
+                {"finding_id": "BFLA-R-1-1", "category": "bfla", "status": "confirmed",
+                 "location": {"file": "routes.js", "start_line": 1}, "evidence": []}]})
+            write_json(out / "routes" / "routes-001.json", {"routes": [
+                {"route_id": "R-1", "handler": {"file": "admin.js", "line": 8}},
+                {"route_id": "R-2", "handler": {"file": "admin.js", "line": 12}}]})
+            write_json(out / "mass-assignment.candidates.json", {"candidates": [
+                {"candidate_id": "M-1", "file": "p.php", "line": 6}]})
+            write_json(out / "findings" / "priv-privilege-001.json", {
+                "privilege_route_verdicts": [{"route_id": "R-1", "verdict": "vulnerable"}],
+                "candidate_verdicts": [{"candidate_id": "M-1", "verdict": "needs_review"}], "findings": []})
+            res = subprocess.run([sys.executable, str(SCORE), "--verdicts", str(truth), str(out)], capture_output=True, text=True)
+            report = json.loads(res.stdout)
+            self.assertEqual(report["per_category"]["bfla"]["found"], 0)
+            self.assertEqual(report["verdict_level"]["bfla"], {"truth_vulnerable": 1, "found": 1, "recall": 1.0,
+                                                              "flagged_safe": [], "missed": []})
+            self.assertEqual(report["verdict_level"]["mass_assignment"]["found"], 1)
+            self.assertEqual(report["verdict_level"]["mass_assignment"]["flagged_safe"], [])
+
 
 if __name__ == "__main__":
     unittest.main()

package/operate/ai/anchoring.py

@@ -0,0 +1,130 @@
+"""Anchor model-written finding locations to the Tier 1 routes and candidates they refer to (#319).
+
+Small local models judge the right route or candidate but often write a wrong or non-existent path
+into the finding (Evaluation Run 4: `routes.js`, `app.py`, `dotnet-mvc/...`). Tier 1 already knows
+where every route handler and query / binding candidate is, so the location comes from there.
+
+Rules:
+- route findings (`route_id`): the file is the route handler's file; the model's line is kept when
+  it falls inside the handler span (handler line up to the next route in the same file, or the end
+  of the file), otherwise the handler line is used.
+- candidate findings (`candidate_id`): file and line are the candidate's.
+- evidence entries are re-pointed the same way; entries whose file does not exist and cannot be
+  re-pointed are dropped.
+- findings with an unknown `route_id` / `candidate_id` keep their location, get `confidence: low`
+  and are reported as unanchored.
+
+Standard-library only (Python 3.8+ compatible).
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple
+
+CANDIDATE_FILES = ("injection.candidates.json", "mass-assignment.candidates.json")
+
+
+@dataclass(frozen=True)
+class Anchor:
+    file: str
+    line: int
+    end_line: Optional[int]  # last line of the handler span; None for an exact candidate site
+
+
+@dataclass
+class AnchorReport:
+    moved: List[str] = field(default_factory=list)
+    unanchored: List[str] = field(default_factory=list)
+
+
+def _read(path: Path) -> Dict[str, Any]:
+    try:
+        doc = json.loads(path.read_text(encoding="utf-8"))
+        return doc if isinstance(doc, dict) else {}
+    except (OSError, ValueError):
+        return {}
+
+
+def _line_count(target_dir: Path, rel_file: str) -> int:
+    try:
+        return len((target_dir / rel_file).read_text(encoding="utf-8", errors="replace").splitlines())
+    except OSError:
+        return 0
+
+
+def load_anchors(out_dir: Path, target_dir: Path) -> Dict[str, Anchor]:
+    """route_id / candidate_id -> Anchor, from OUT/routes/*.json and the Tier 1 candidate files."""
+    handlers: List[Tuple[str, int, str]] = []
+    for path in sorted((out_dir / "routes").glob("*.json")):
+        for route in _read(path).get("routes") or []:
+            if not isinstance(route, dict):
+                continue
+            handler = route.get("handler") or {}
+            if route.get("route_id") and handler.get("file") and isinstance(handler.get("line"), int):
+                handlers.append((handler["file"], handler["line"], route["route_id"]))
+
+    anchors: Dict[str, Anchor] = {}
+    for rel_file, line, route_id in handlers:
+        later = sorted(ln for f, ln, _ in handlers if f == rel_file and ln > line)
+        end = later[0] - 1 if later else max(_line_count(target_dir, rel_file), line)
+        anchors[route_id] = Anchor(rel_file, line, end)
+
+    for name in CANDIDATE_FILES:
+        for cand in _read(out_dir / name).get("candidates") or []:
+            if isinstance(cand, dict) and cand.get("candidate_id") and cand.get("file") and isinstance(cand.get("line"), int):
+                anchors[cand["candidate_id"]] = Anchor(cand["file"], cand["line"], None)
+    return anchors
+
+
+def _anchored_line(anchor: Anchor, line: Any) -> int:
+    if anchor.end_line is None or not isinstance(line, int):
+        return anchor.line
+    return line if anchor.line <= line <= anchor.end_line else anchor.line
+
+
+def _repoint_evidence(evidence: Any, anchor: Anchor, target_dir: Path) -> List[Dict[str, Any]]:
+    kept = []
+    for ev in evidence if isinstance(evidence, list) else []:
+        if not isinstance(ev, dict):
+            continue
+        rel = str(ev.get("file") or "")
+        if rel == anchor.file or (rel and (target_dir / rel).is_file()):
+            kept.append(ev)
+        elif anchor.end_line is not None and isinstance(ev.get("line"), int) and anchor.line <= ev["line"] <= anchor.end_line:
+            kept.append({**ev, "file": anchor.file})
+    return kept or [{"file": anchor.file, "line": anchor.line, "why": "Location taken from the Tier 1 inventory"}]
+
+
+def anchor_finding(finding: Dict[str, Any], anchors: Dict[str, Anchor], target_dir: Path) -> Tuple[Dict[str, Any], Optional[bool]]:
+    """Return (finding with Tier 1 location, moved?); moved is None when the finding has no known anchor."""
+    key = finding.get("route_id") or finding.get("candidate_id")
+    anchor = anchors.get(key) if key else None
+    if anchor is None:
+        return {**finding, "confidence": "low"}, None
+    loc = finding.get("location") if isinstance(finding.get("location"), dict) else {}
+    # a wrong path with a line inside the handler span is most likely the right line in the wrong file
+    new_loc = {**loc, "file": anchor.file, "start_line": _anchored_line(anchor, loc.get("start_line"))}
+    if anchor.end_line is None:
+        new_loc.pop("end_line", None)
+    moved = (loc.get("file"), loc.get("start_line")) != (new_loc["file"], new_loc["start_line"])
+    evidence = _repoint_evidence(finding.get("evidence"), anchor, target_dir)
+    return {**finding, "location": new_loc, "evidence": evidence}, moved
+
+
+def anchor_findings(findings: List[Dict[str, Any]], anchors: Dict[str, Anchor], target_dir: Path,
+                    report: AnchorReport) -> List[Dict[str, Any]]:
+    out = []
+    for f in findings:
+        if not isinstance(f, dict):
+            out.append(f)
+            continue
+        anchored, moved = anchor_finding(f, anchors, target_dir)
+        if moved is None:
+            report.unanchored.append(str(f.get("finding_id")))
+        elif moved:
+            report.moved.append(str(f.get("finding_id")))
+        out.append(anchored)
+    return out

package/operate/ai/client.py

@@ -7,6 +7,7 @@ from __future__ import annotations
 
 import json
 import re
+import time
 import urllib.error
 import urllib.request
 from typing import Any, Dict, List, Optional
@@ -30,6 +31,8 @@ class LLMClient:
         self.model = model
         self.api_key = api_key or "ollama"
         self.timeout = timeout
+        # Running totals across chat() calls, read by the runner for ai-run-stats.json
+        self.usage: Dict[str, float] = {"calls": 0, "prompt_tokens": 0, "completion_tokens": 0, "seconds": 0.0}
 
     def chat(
         self,
@@ -54,10 +57,12 @@ class LLMClient:
         }
 
         req = urllib.request.Request(self.endpoint, data=data, headers=headers, method="POST")
+        started = time.monotonic()
         try:
             with urllib.request.urlopen(req, timeout=self.timeout) as resp:
                 body = resp.read().decode("utf-8")
                 res_json = json.loads(body)
+                self._record_usage(res_json.get("usage") or {}, time.monotonic() - started)
                 choices = res_json.get("choices", [])
                 if not choices:
                     raise RuntimeError(f"LLM API returned no choices: {body}")
@@ -69,6 +74,14 @@ class LLMClient:
         except urllib.error.URLError as e:
             raise RuntimeError(f"Failed to connect to LLM endpoint at {self.endpoint}: {e.reason}") from e
 
+    def _record_usage(self, usage: Dict[str, Any], seconds: float) -> None:
+        self.usage = {
+            "calls": self.usage["calls"] + 1,
+            "prompt_tokens": self.usage["prompt_tokens"] + int(usage.get("prompt_tokens") or 0),
+            "completion_tokens": self.usage["completion_tokens"] + int(usage.get("completion_tokens") or 0),
+            "seconds": round(self.usage["seconds"] + seconds, 3),
+        }
+
     @staticmethod
     def extract_json(content: str) -> Any:
         """Extract and parse JSON from Markdown code blocks or raw text string."""