@hoanghainh/komagu 2.0.0 → 2.1.0

package/assess/lsa/tests/EVAL-RESULTS.md

@@ -152,3 +152,159 @@ Based on Qwen 3.5's findings, the following concrete improvements were implement
 | **False Positives on Safe Code** | 0 | 0 | 0 on safe markers | **0** |
 | **Precision (matched / active)** | 0.965 (55 / 57) | 0.965 (55 / 57) | 0.966 | **1.000 (57 / 57)** |
 | **Legacy Stack Shims** | None | None | Recommended | **4 Stacks (`remediate/legacy_shims/`)** |
+
+---
+
+## Run 4 — 2026-10-31, kit 0.3.1 (BFLA & mass assignment Tier 2, local Ollama)
+
+Issue #286. First end-to-end measurement of phase 4c (`35-privilege-auditor`) with real models, and a
+re-check of BOLA / injection on the same harness. All runs are **local only**: fixture code never left
+the machine, and no third-party LLM was used.
+
+### Setup
+
+| Item | Value |
+|---|---|
+| Eval repo | `tests/build_eval_repo.py` (10 fixture apps, markers stripped); ground truth 117 items: 25 BOLA, 34 SQL/NoSQL injection, 10 BFLA, 15 mass assignment, plus 33 SAFE markers |
+| Pipeline | `komagu scan <repo> --no-cache --headless-ai http://localhost:11434/v1 --model <model> --ai-timeout 3600` (`operate/ai/runner.py`, Tier 1 pre-scan + phases 2–6) |
+| Hosting | Ollama 0.40.13 on an Apple M4 Pro, 24 GB RAM, 100% GPU; context 32,768 tokens |
+| Models | `gpt-oss:20b` (MXFP4, reasoning effort default `medium`), `qwen3.5:9b` |
+| Scoring | `tests/score_findings.py --verdicts` (tolerance 3 lines); archived per run in [`assess/lsa/examples/eval-run-4/`](../examples/eval-run-4/) |
+
+**This is a floor, not a ceiling.** A 9–20B model on a laptop is much weaker than the models used in
+Runs 1–2. The numbers below say what a fully local deployment achieves today, not what the prompts can
+do with a stronger model.
+
+**Not comparable with Run 3.** Run 3 does not record its model, and the archived Run 1 / Run 2 / Qwen
+numbers were produced through agent runtimes that can open files, not through the headless runner.
+The BOLA / injection columns below are therefore a new baseline for the headless harness, not a
+regression check against Run 3.
+
+### Harness fixes found by this run
+
+Measuring with real models exposed defects in the headless runner that inflated or lost results.
+They were fixed before the numbers were trusted, in two iterations (the limit set in #286):
+
+| Iteration | Fix | Effect seen |
+|---|---|---|
+| 0 → 1 | Verifier no longer marks a whole batch `confirmed` when its reply cannot be parsed (it used to, with a made-up "consensus" note); such findings stay `unverified` | Results stopped being inflated by broken replies |
+| 0 → 1 | A failed call (timeout) is recorded per batch in `ai-run-stats.json` instead of aborting the run | A 25-candidate injection batch took > 15 min and used to kill the run |
+| 0 → 1 | Phase 4c gets the same JSON output template as 4a / 4b; sanitizer joins list `impact` values and moves a verdict written into `subtype` to `status`; unparseable replies saved to `OUT/ai-raw/` | Mass assignment findings stopped failing schema validation |
+| 1 → 2 | Verifier now receives the code it judges (finding site, file header, evidence lines, server-side includes) and the auth model, and may only cite controls that are visible; otherwise `likely` + `code_not_visible` | In iteration 1 it rejected 10 of 11 real BFLA findings by inventing an admin base class and a role check in a login-only `auth.php` |
+| 1 → 2 | Verifier batches of 5 instead of 10; sanitizer fills a `null` evidence line and a missing `status` | No verifier parse failures in iteration 2 |
+
+### Results after verification
+
+"Marker" columns use the normal location match. "Verdict" columns (BFLA / mass assignment only) count a
+truth item as found when the privilege auditor judged the route or candidate that contains it
+`vulnerable` or `needs_review`, whatever path and line the model then wrote into the finding.
+
+| Run | BOLA | Injection | BFLA marker | Mass marker | BFLA verdict | Mass verdict | FP on safe code | Precision | Rejected | Unverified | Parse fail | Min · calls · tokens |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| iter 0, gpt-oss A | 19/25 | 20/34 | 8/10 | 2/15 | 10/10 | 12/15 | 0 | 0.689 | 10 | 11 | 3 | 81 · 22 · 311K |
+| iter 1, gpt-oss A | 21/25 | 34/34 | 1/10 | 2/15 | 10/10 | 14/15 | 0 | 0.765 | 19 | 17 | 2 | 75 · 21 · 288K |
+| iter 1, gpt-oss B | 19/25 | 28/34 | 5/10 | 5/15 | 10/10 | 14/15 | 1 | 0.667 | 4 | 8 | 3 | 80 · 22 · 273K |
+| iter 1, qwen3.5 (incomplete) | 23/25 | 33/34 | 0/10 | 0/15 | 0/10 | 8/15 | 1 | 0.778 | 0 | 52 | — | stopped after 2 h 50 min |
+| **iter 2, gpt-oss A** | **24/25** | **34/34** | **8/10** | **6/15** | **10/10** | **10/15** | **0** | 0.729 | 1 | 0 | 0 | 57 · 32 · 309K |
+| **iter 2, gpt-oss B** | **11/25** | **32/34** | **5/10** | **4/15** | **10/10** | **9/15** | **0** | 0.693 | 1 | 0 | 1 | 56 · 29 · 318K |
+
+Final harness = iteration 2. Stated plainly for `gpt-oss:20b` on it:
+
+- **BFLA:** the auditor flagged the right route for **10 / 10** BFLA markers in both runs, with no SAFE-BFLA
+  route flagged. Only **8 / 10 and 5 / 10** survive as correctly located findings, because the model
+  writes wrong or non-existent paths into the finding (`dotnet-mvc/Controllers/AdminController.cs`,
+  `routes.js`).
+- **Mass assignment:** **10 / 15 and 9 / 15** at verdict level, **6 / 15 and 4 / 15** as located findings.
+  Five candidates in .NET (`CustomerController.cs`) and Java (`AccountController.java`,
+  `ProfileServlet.java`) were judged `safe` in both runs: real model misses.
+- **False positives on SAFE markers:** **0** in both iteration-2 runs, for all four categories.
+- **BOLA / injection:** 24 / 25 and 34 / 34 in run A. In run B, 11 of its 14 missed BOLA items sit in
+  one batch whose reply was empty (`routes-001`: `gpt-oss` returned no content, so 15 routes got no
+  verdict); the other 3 are real misses (`perl-cgi/cgi-bin/order.cgi:19` is missed in both runs).
+- **Run-to-run variance is large.** Two runs with the same model, harness and temperature 0.1 differ by
+  up to 13 BOLA items and 3 BFLA findings. Single-run numbers from a local model should not be quoted
+  as product capability.
+- **Precision ~0.7** is measured against markers only. Of the ~25 unmatched findings per run, most are BOLA
+  findings in files that carry BFLA / mass-assignment markers but no BOLA markers
+  (`CustomerController.cs`, `AccountController.java`, `AdminController.cs`), plus the misplaced BFLA /
+  mass findings above. They have not been triaged by a person and are not counted as false positives.
+
+**qwen3.5:9b** could not be measured on this machine: with a 32K context the MLX runner used 22 GB of the
+24 GB RAM, the system swapped (29.7 / 30 GB), and two calls hit the 1-hour timeout (the privilege batch
+holding all BFLA routes, and a verifier batch). The run was stopped after 2 h 50 min with 52 findings
+unverified. Its partial output is archived as `iter1-qwen-a-incomplete/` and says nothing about the
+model's ability.
+
+### Known gaps (follow-up: #319)
+
+1. **Misplaced findings.** Take `file` / `line` of BFLA and mass assignment findings from the Tier 1 route
+   or candidate the auditor judged (`route_id` / `candidate_id`), not from the model's text. Expected to
+   close most of the gap between the marker and verdict columns.
+2. **Empty replies.** Retry a batch once when the reply is empty or unparseable (seen with `gpt-oss`).
+3. **Mass assignment on .NET / Java binding.** Review `knowledge/mass-assignment-patterns.md` for
+   `UpdateModel` / `TryUpdateModel` and Spring `@ModelAttribute` binding into entities.
+4. **Re-run qwen3.5:9b** with 16K context or on a machine with more memory.
+5. **Triage unmatched BOLA findings** and add markers to the fixtures where they are real.
+
+---
+
+## Run 5 — 2026-10-31, kit 0.3.1 (Run 4 follow-up: anchored locations, retries, model context)
+
+Issue #319. Same eval repo, ground truth, hosting and model as Run 4 (`gpt-oss:20b`, Ollama 0.40.13,
+M4 Pro 24 GB, 32K context, local only). Outputs are archived in
+[`assess/lsa/examples/eval-run-5/`](../examples/eval-run-5/).
+
+### Changes since Run 4 iteration 2
+
+| Change | Why (from Run 4) |
+|---|---|
+| Finding `file` / `line` come from the Tier 1 route handler span or candidate the finding references (`operate/ai/anchoring.py`); evidence is re-pointed or dropped; unknown references get `confidence: low` | The auditor judged the right route for 10 / 10 BFLA markers, but wrote paths such as `routes.js`, `app.py`, `dotnet-mvc/...` |
+| An empty or unparseable batch reply is retried once with a JSON-only reminder | One empty reply cost 15 BOLA routes |
+| Each mass assignment candidate carries the bound model's definition (same app and language); an unknown model means `needs_review`, never `safe` (prompt `35-privilege-auditor.md` and the headless 4c prompt) | Five .NET / Java candidates were judged `safe` in both runs because the auditor never saw `Account.java` (`role`, `balance`) |
+
+### Results after verification
+
+| Run | BOLA | Injection | BFLA marker | Mass marker | BFLA verdict | Mass verdict | FP on safe code | Precision | Rejected | Unverified | Retries · parse fail | Min · calls · tokens |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| Run 4 iter 2, A | 24/25 | 34/34 | 8/10 | 6/15 | 10/10 | 10/15 | 0 | 0.729 | 1 | 0 | — · 0 | 57 · 32 · 309K |
+| Run 4 iter 2, B | 11/25 | 32/34 | 5/10 | 4/15 | 10/10 | 9/15 | 0 | 0.693 | 1 | 0 | — · 1 | 56 · 29 · 318K |
+| **Run 5, A** | **24/25** | **34/34** | **10/10** | **10/15** | **10/10** | **15/15** | **0** | **0.818** | 2 | 0 | 0 · 0 | 83 · 32 · 328K |
+| **Run 5, B** | **12/25** | **32/34** | **9/10** | **14/15** | **10/10** | **15/15** | **0** | **0.779** | 3 | 0 | 1 · 1 | 86 · 31 · 369K |
+
+- **BFLA:** 10 / 10 and 9 / 10 as located findings (from 8 and 5). Every anchored finding pointed at a
+  known route or candidate (0 unanchored).
+- **Mass assignment:** the auditor flagged **15 / 15** candidates in both runs (from 10 and 9), including the
+  five .NET / Java bindings. As located findings 10 / 15 and 14 / 15: in run A the model gave a
+  `vulnerable` verdict for four candidates (`php-plain/profile.php` ×2, `CustomerController.cs` ×2)
+  without writing a finding for them.
+- **`php-plain/register.php:4` (MASS marker)** was rejected by the verifier in both runs: `extract($_POST)`
+  is followed by an INSERT of only `email` and `name`. The argument is sound for this fixture as written,
+  so the marker should be reviewed rather than counted as a verifier error.
+- **BOLA in run B (12 / 25)** again lost batch `routes-001` (15 routes). This time the reply was not empty
+  but invalid JSON: the model put code with unescaped `"` into `recommendation.code_before`; the retry
+  failed the same way. The same batch failed in Run 4 run B.
+- **No false positives** on SAFE markers in either run; precision rose to 0.82 / 0.78.
+- Runs are ~45% slower than Run 4 iteration 2 (83–86 min): the privilege and verifier prompts are longer
+  and there are more findings to verify.
+
+### Unmatched BOLA findings (triage, not yet reflected in fixtures)
+
+About 20 BOLA findings per run sit in files with no BOLA marker. By route:
+
+| Route(s) | Assessment |
+|---|---|
+| `Customer/EditName`, `account/rename`, `python-flask` `/users/<uid>/name`, `account/profile` | Likely **real BOLA** not marked: the record ID comes from the request and is not bound to the caller |
+| `Customer/Edit`, `Customer/Refresh`, `account/save`, `account/api`, `/users/<uid>` | Mass-assignment fixtures that also take the ID from the request: plausibly BOLA as well |
+| `Admin/DeleteUser`, `Admin/Export`, `admin/users/{id}/delete`, `delete_user.php`, `promote.php`, `manage_users.php` | **BFLA routes also reported as BOLA**: same flaw seen from a second category; duplicate rather than wrong |
+
+Fixture markers were left unchanged so Run 4 and Run 5 share the same ground truth.
+
+### Remaining gaps
+
+1. **Invalid JSON from code fields:** drop `code_before` / `code_after` from the 4a / 4b output templates
+   (the sanitizer already reduces `recommendation` to a string), or repair unescaped quotes before parsing.
+2. **Verdict without finding:** write a minimal finding for every `vulnerable` / `needs_review` verdict
+   the model did not turn into one.
+3. **Fixtures:** review the `register.php` MASS marker; add BOLA markers for the routes judged real above.
+4. **qwen3.5:9b** still unmeasured: 32K context does not fit in 24 GB next to the OS; needs 16K context or
+   a larger machine.

package/assess/lsa/tests/score_findings.py

@@ -7,13 +7,20 @@ A vulnerable truth item is found (TP) when an active finding (status != rejected
 same category (sql_injection and nosql_injection are treated as one group), and the truth
 line is within --tolerance lines of the finding location or of any evidence line in the same file.
 A safe truth item hit by an active finding counts as a false positive (FP-safe).
+
+--verdicts adds a verdict-level score for bfla and mass_assignment: a truth item counts as found when
+the privilege auditor judged the route or candidate that contains it "vulnerable" or "needs_review",
+whatever file and line the model then wrote into the finding. Routes cover their handler line up to
+the next inventoried route in the same file; mass assignment candidates match their exact line.
 """
 from __future__ import annotations
 
 import argparse
+import glob
 import json
 import sys
 from pathlib import Path
+from typing import Dict, List, Optional, Tuple
 
 
 def group(cat: str) -> str:
@@ -32,11 +39,60 @@ def hits(finding: dict, item: dict, tol: int) -> bool:
     return False
 
 
+FLAGGED = ("vulnerable", "needs_review")
+
+
+def _same_file(a: str, b: str) -> bool:
+    return bool(a) and bool(b) and (a.endswith(b) or b.endswith(a))
+
+
+def _route_for(item: dict, routes: List[Tuple[str, int, str]]) -> Optional[str]:
+    """route_id whose handler span (handler line .. next route in the file) contains the truth line."""
+    in_file = sorted((line, rid) for f, line, rid in routes if _same_file(f, item["file"]) and line <= item["line"])
+    return in_file[-1][1] if in_file else None
+
+
+def verdict_scores(out: Path, truth: List[dict]) -> Dict[str, dict]:
+    routes: List[Tuple[str, int, str]] = []
+    for p in glob.glob(str(out / "routes" / "*.json")):
+        for r in json.loads(Path(p).read_text()).get("routes", []):
+            h = r.get("handler") or {}
+            if h.get("file") and isinstance(h.get("line"), int):
+                routes.append((h["file"], h["line"], r.get("route_id")))
+    cands_path = out / "mass-assignment.candidates.json"
+    cands = json.loads(cands_path.read_text()).get("candidates", []) if cands_path.is_file() else []
+    route_v: Dict[str, str] = {}
+    cand_v: Dict[str, str] = {}
+    for p in glob.glob(str(out / "findings" / "priv-*.json")):
+        doc = json.loads(Path(p).read_text())
+        route_v.update({v.get("route_id"): v.get("verdict") for v in doc.get("privilege_route_verdicts", []) if isinstance(v, dict)})
+        cand_v.update({v.get("candidate_id"): v.get("verdict") for v in doc.get("candidate_verdicts", []) if isinstance(v, dict)})
+
+    def flagged(item: dict) -> bool:
+        if item["category"] == "bfla":
+            return route_v.get(_route_for(item, routes)) in FLAGGED
+        return any(_same_file(c.get("file", ""), item["file"]) and c.get("line") == item["line"]
+                   and cand_v.get(c.get("candidate_id")) in FLAGGED for c in cands)
+
+    rows = {}
+    for cat in ("bfla", "mass_assignment"):
+        vul = [t for t in truth if t["category"] == cat and t["vulnerable"]]
+        safe = [t for t in truth if t["category"] == cat and not t["vulnerable"]]
+        if not vul:
+            continue
+        found = [t for t in vul if flagged(t)]
+        rows[cat] = {"truth_vulnerable": len(vul), "found": len(found), "recall": round(len(found) / len(vul), 3),
+                     "flagged_safe": [f"{t['file']}:{t['line']}" for t in safe if flagged(t)],
+                     "missed": [f"{t['file']}:{t['line']}" for t in vul if t not in found]}
+    return rows
+
+
 def main() -> int:
     ap = argparse.ArgumentParser(description=__doc__)
     ap.add_argument("ground_truth")
     ap.add_argument("out")
     ap.add_argument("--tolerance", type=int, default=3)
+    ap.add_argument("--verdicts", action="store_true", help="also score bfla / mass_assignment by auditor verdicts")
     a = ap.parse_args()
     truth = json.loads(Path(a.ground_truth).read_text())["items"]
     # fixture paths are relative to tests/fixtures; eval repo keeps the same layout
@@ -69,6 +125,8 @@ def main() -> int:
     report = {"per_category": rows, "active_findings": len(g_findings), "matched_findings": len(matched_findings),
               "precision": precision, "unmatched_findings": unmatched,
               "needs_review_not_scored": needs_review}
+    if a.verdicts:
+        report["verdict_level"] = verdict_scores(Path(a.out), truth)
     print(json.dumps(report, indent=2))
     return 0
 

package/assess/lsa/tests/test_default_scope.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+"""Default scan scope skips framework build output, caches and source maps (Issue #325)."""
+
+from __future__ import annotations
+
+import json
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+LSA_SCRIPTS = REPO_ROOT / "assess" / "lsa" / "scripts"
+sys.path.insert(0, str(LSA_SCRIPTS))
+
+from find_injection import find_injection  # noqa: E402
+from lsa_common import DEFAULT_CONFIG, deep_merge, iter_source_files, load_config  # noqa: E402
+
+TEMPLATE = REPO_ROOT / "assess" / "lsa" / "templates" / "audit-config.json"
+
+VULN_JS = (
+    "const id = req.query.id;\n"
+    "db.query(`SELECT * FROM orders WHERE id = ${id}`);\n"
+)
+
+GENERATED = [
+    ".next/server/chunks/x.js",
+    ".next/dev/server/chunks/y.js",
+    ".nuxt/dist/server/a.js",
+    ".output/server/index.mjs",
+    ".svelte-kit/output/server/b.js",
+    ".vercel/output/functions/c.js",
+    ".turbo/cache/d.js",
+    "out/_next/e.js",
+    "coverage/lcov-report/f.js",
+    "test-results/g.js",
+    "playwright-report/h.js",
+    "storybook-static/i.js",
+    ".venv/lib/site.py",
+    "__pycache__/m.py",
+    "src/app/page.js.map",
+]
+
+
+def _write(root: Path, rel: str, text: str) -> None:
+    p = root / rel
+    p.parent.mkdir(parents=True, exist_ok=True)
+    p.write_text(text, encoding="utf-8")
+
+
+class TestDefaultScope(unittest.TestCase):
+    def _repo(self, td: str) -> Path:
+        root = Path(td)
+        for rel in GENERATED:
+            _write(root, rel, VULN_JS)
+        _write(root, "src/lib/orders.js", VULN_JS)
+        return root
+
+    def test_generated_output_not_scanned(self) -> None:
+        with tempfile.TemporaryDirectory() as td:
+            root = self._repo(td)
+            rels = {rel for _, rel in iter_source_files(root, load_config(None))}
+            self.assertEqual(rels, {"src/lib/orders.js"})
+
+    def test_injection_candidates_only_from_source(self) -> None:
+        with tempfile.TemporaryDirectory() as td:
+            root = self._repo(td)
+            res = find_injection(root, load_config(None))
+            files = {c["file"] for c in res["candidates"]}
+            self.assertEqual(files, {"src/lib/orders.js"})
+
+    def test_user_exclude_override_brings_folder_back(self) -> None:
+        with tempfile.TemporaryDirectory() as td:
+            root = self._repo(td)
+            excludes = [e for e in DEFAULT_CONFIG["scope"]["exclude"] if e != ".next/"]
+            cfg = deep_merge(DEFAULT_CONFIG, {"scope": {"exclude": excludes}})
+            rels = {rel for _, rel in iter_source_files(root, cfg)}
+            self.assertIn(".next/server/chunks/x.js", rels)
+
+    def test_template_matches_default(self) -> None:
+        template = json.loads(TEMPLATE.read_text(encoding="utf-8"))
+        self.assertEqual(template["scope"]["exclude"], DEFAULT_CONFIG["scope"]["exclude"])
+
+
+if __name__ == "__main__":
+    unittest.main()

package/assess/lsa/tests/test_eval_run_4.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+"""Keeps the archived Evaluation Run 4 numbers (Issue #286) in step with the scorer and EVAL-RESULTS.md."""
+
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+TESTS_DIR = Path(__file__).resolve().parent
+LSA_ROOT = TESTS_DIR.parent
+EVAL_RUN_4_DIR = LSA_ROOT / "examples" / "eval-run-4"
+GROUND_TRUTH = EVAL_RUN_4_DIR / "ground-truth.json"
+RUNS = ("iter0-gptoss-a", "iter1-gptoss-a", "iter1-gptoss-b", "iter1-qwen-a-incomplete", "iter2-gptoss-a", "iter2-gptoss-b")
+# (BOLA, injection, BFLA, mass assignment) found by location, then BFLA / mass found by auditor verdict
+EXPECTED = {
+    "iter2-gptoss-a": {"marker": (24, 34, 8, 6), "verdict": (10, 10)},
+    "iter2-gptoss-b": {"marker": (11, 32, 5, 4), "verdict": (10, 9)},
+}
+
+
+def score(run: str) -> dict:
+    proc = subprocess.run(
+        [sys.executable, str(TESTS_DIR / "score_findings.py"), "--verdicts", str(GROUND_TRUTH), str(EVAL_RUN_4_DIR / run)],
+        capture_output=True, text=True, check=True,
+    )
+    return json.loads(proc.stdout)
+
+
+class TestEvaluationRun4(unittest.TestCase):
+    def test_artifacts_exist(self) -> None:
+        self.assertTrue(GROUND_TRUTH.is_file())
+        for run in RUNS:
+            for name in ("findings.json", "coverage.json", "score.json"):
+                self.assertTrue((EVAL_RUN_4_DIR / run / name).is_file(), f"{run}/{name}")
+
+    def test_archived_scores_match_the_scorer(self) -> None:
+        for run in RUNS:
+            with self.subTest(run=run):
+                archived = json.loads((EVAL_RUN_4_DIR / run / "score.json").read_text(encoding="utf-8"))
+                self.assertEqual(score(run), archived)
+
+    def test_final_harness_numbers_match_eval_results(self) -> None:
+        for run, want in EXPECTED.items():
+            with self.subTest(run=run):
+                report = score(run)
+                pc, vl = report["per_category"], report["verdict_level"]
+                got = tuple(pc[c]["found"] for c in ("bola", "injection", "bfla", "mass_assignment"))
+                self.assertEqual(got, want["marker"])
+                self.assertEqual((vl["bfla"]["found"], vl["mass_assignment"]["found"]), want["verdict"])
+
+    def test_final_harness_has_no_false_positive_on_safe_code(self) -> None:
+        for run in EXPECTED:
+            with self.subTest(run=run):
+                report = score(run)
+                for cat, row in report["per_category"].items():
+                    self.assertEqual(row["false_positive_on_safe_code"], [], cat)
+                for cat, row in report["verdict_level"].items():
+                    self.assertEqual(row["flagged_safe"], [], cat)
+
+    def test_final_harness_records_run_stats(self) -> None:
+        for run in EXPECTED:
+            with self.subTest(run=run):
+                stats = json.loads((EVAL_RUN_4_DIR / run / "ai-run-stats.json").read_text(encoding="utf-8"))
+                self.assertEqual(stats["model"], "gpt-oss:20b")
+                self.assertGreater(stats["totals"]["prompt_tokens"], 0)
+                self.assertEqual(stats["call_errors"], {})
+
+
+if __name__ == "__main__":
+    unittest.main()

package/assess/lsa/tests/test_eval_run_5.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+"""Keeps the archived Evaluation Run 5 numbers (Issue #319) in step with the scorer and EVAL-RESULTS.md."""
+
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+TESTS_DIR = Path(__file__).resolve().parent
+LSA_ROOT = TESTS_DIR.parent
+EVAL_RUN_5_DIR = LSA_ROOT / "examples" / "eval-run-5"
+GROUND_TRUTH = EVAL_RUN_5_DIR / "ground-truth.json"
+RUNS = ("gptoss-a", "gptoss-b")
+# (BOLA, injection, BFLA, mass assignment) found by location, then BFLA / mass found by auditor verdict
+EXPECTED = {
+    "gptoss-a": {"marker": (24, 34, 10, 10), "verdict": (10, 15)},
+    "gptoss-b": {"marker": (12, 32, 9, 14), "verdict": (10, 15)},
+}
+
+def score(run: str) -> dict:
+    proc = subprocess.run(
+        [sys.executable, str(TESTS_DIR / "score_findings.py"), "--verdicts", str(GROUND_TRUTH), str(EVAL_RUN_5_DIR / run)],
+        capture_output=True, text=True, check=True,
+    )
+    return json.loads(proc.stdout)
+
+
+class TestEvaluationRun5(unittest.TestCase):
+    def test_artifacts_exist(self) -> None:
+        self.assertTrue(GROUND_TRUTH.is_file())
+        for run in RUNS:
+            for name in ("findings.json", "coverage.json", "score.json"):
+                self.assertTrue((EVAL_RUN_5_DIR / run / name).is_file(), f"{run}/{name}")
+
+    def test_archived_scores_match_the_scorer(self) -> None:
+        for run in RUNS:
+            with self.subTest(run=run):
+                archived = json.loads((EVAL_RUN_5_DIR / run / "score.json").read_text(encoding="utf-8"))
+                self.assertEqual(score(run), archived)
+
+    def test_final_harness_numbers_match_eval_results(self) -> None:
+        for run, want in EXPECTED.items():
+            with self.subTest(run=run):
+                report = score(run)
+                pc, vl = report["per_category"], report["verdict_level"]
+                got = tuple(pc[c]["found"] for c in ("bola", "injection", "bfla", "mass_assignment"))
+                self.assertEqual(got, want["marker"])
+                self.assertEqual((vl["bfla"]["found"], vl["mass_assignment"]["found"]), want["verdict"])
+
+    def test_final_harness_has_no_false_positive_on_safe_code(self) -> None:
+        for run in EXPECTED:
+            with self.subTest(run=run):
+                report = score(run)
+                for cat, row in report["per_category"].items():
+                    self.assertEqual(row["false_positive_on_safe_code"], [], cat)
+                for cat, row in report["verdict_level"].items():
+                    self.assertEqual(row["flagged_safe"], [], cat)
+
+    def test_final_harness_records_run_stats(self) -> None:
+        for run in EXPECTED:
+            with self.subTest(run=run):
+                stats = json.loads((EVAL_RUN_5_DIR / run / "ai-run-stats.json").read_text(encoding="utf-8"))
+                self.assertEqual(stats["model"], "gpt-oss:20b")
+                self.assertGreater(stats["totals"]["prompt_tokens"], 0)
+                self.assertEqual(stats["call_errors"], {})
+                self.assertEqual(stats["anchoring"]["unanchored"], [])
+
+
+if __name__ == "__main__":
+    unittest.main()

package/assess/lsa/tests/test_js_tagged_sql.py

@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+"""Parameterizing JS/TS sql tagged templates are not injection candidates (Issue #326).
+
+Drizzle, postgres.js, @vercel/postgres, Slonik and Prisma ($queryRaw, Prisma.sql) send ${...}
+inside a tagged template as a bound parameter. Raw-text escape hatches (sql.raw, Prisma.raw,
+sql.unsafe, $queryRawUnsafe) and untagged templates are still flagged.
+"""
+
+from __future__ import annotations
+
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+from typing import Dict, List
+
+SCRIPTS_DIR = Path(__file__).resolve().parents[1] / "scripts"
+if str(SCRIPTS_DIR) not in sys.path:
+    sys.path.insert(0, str(SCRIPTS_DIR))
+
+from detect_stack import detect  # noqa: E402
+from find_injection import find_injection  # noqa: E402
+from lsa_common import load_config  # noqa: E402
+
+SAFE_TAGGED = '''\
+import { sql } from "drizzle-orm";
+import { Prisma } from "@prisma/client";
+
+export async function counts(db, prisma, orderId, req) {
+  const id = req.query.id;
+  const lineCount = sql<number>`(select count(*)::int from ${orderLine} where ${orderLine.orderId} = ${orderId})`;
+  const rows = await db.execute(sql`select * from orders where id = ${id}`);
+  const p = await prisma.$queryRaw`SELECT * FROM users WHERE id = ${id}`;
+  await prisma.$executeRaw`UPDATE users SET name = ${req.body.name} WHERE id = ${id}`;
+  const frag = Prisma.sql`SELECT * FROM users WHERE email = ${req.body.email}`;
+  const pg = await pool.sql`select * from items where owner_id = ${id}`;
+  return [lineCount, rows, p, frag, pg];
+}
+'''
+
+UNSAFE = '''\
+import { sql } from "drizzle-orm";
+
+export async function unsafe(db, prisma, req) {
+  const id = req.query.id;
+  const a = await db.execute(`SELECT * FROM orders WHERE id = ${id}`);
+  const b = await prisma.$queryRawUnsafe(`SELECT * FROM users WHERE id = ${id}`);
+  const c = await db.execute(sql`select * from orders order by ${sql.raw(req.query.sort)}`);
+  const d = await db.execute(sql`select nextval(${sql.raw(`'${customerSeq.seqName}'`)}) as value`);
+  const e = await db.execute(sql`select * from t order by ${sql.raw("created_at")}`);
+  const f = await prisma.$queryRaw`SELECT * FROM t ORDER BY ${Prisma.raw(req.query.sort)}`;
+  return [a, b, c, d, e, f];
+}
+'''
+
+NO_INPUT = '''\
+import { sql } from "drizzle-orm";
+
+export async function nextCode(tx, sequence) {
+  return tx.execute(sql`select nextval(${sql.raw(`'${sequence.seqName}'`)}) as value`);
+}
+'''
+
+
+def _scan(files: Dict[str, str]) -> List[Dict]:
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        for rel, text in files.items():
+            p = root / rel
+            p.parent.mkdir(parents=True, exist_ok=True)
+            p.write_text(text, encoding="utf-8")
+        return find_injection(root, load_config(None))["candidates"]
+
+
+def _by_line(cands: List[Dict]) -> Dict[int, Dict]:
+    return {c["line"]: c for c in cands}
+
+
+class TestJsTaggedSql(unittest.TestCase):
+    def test_parameterizing_tags_not_flagged(self) -> None:
+        self.assertEqual(_scan({"src/safe.ts": SAFE_TAGGED}), [])
+
+    def test_untagged_and_unsafe_calls_flagged(self) -> None:
+        got = _by_line(_scan({"src/unsafe.ts": UNSAFE}))
+        self.assertEqual(got[5]["rule"], "sql-interpolation")  # untagged template to db.execute
+        self.assertEqual(got[6]["rule"], "sql-interpolation")  # $queryRawUnsafe
+
+    def test_raw_fragments(self) -> None:
+        got = _by_line(_scan({"src/unsafe.ts": UNSAFE}))
+        self.assertEqual(got[7]["rule"], "sql-raw-fragment")  # sql.raw(req.query.sort)
+        self.assertEqual(got[7]["priority"], "high")
+        self.assertEqual(got[8]["rule"], "sql-raw-fragment")  # sql.raw of a schema property
+        self.assertNotIn(9, got)  # sql.raw("literal") is a constant
+        self.assertEqual(got[10]["rule"], "sql-raw-fragment")  # Prisma.raw(req.query.sort)
+        self.assertEqual(got[10]["priority"], "high")
+
+    def test_raw_fragment_without_request_input_is_low(self) -> None:
+        got = _scan({"src/seq.ts": NO_INPUT})
+        self.assertEqual([(c["rule"], c["priority"]) for c in got], [("sql-raw-fragment", "low")])
+
+    def test_data_access_names_library(self) -> None:
+        with tempfile.TemporaryDirectory() as td:
+            root = Path(td)
+            (root / "a.ts").write_text('import { sql } from "drizzle-orm";\n', encoding="utf-8")
+            (root / "b.ts").write_text('import postgres from "postgres";\n', encoding="utf-8")
+            da = detect(root, load_config(None))["data_access"]
+            self.assertIn("Drizzle ORM (Node/TS)", da)
+            self.assertIn("postgres.js (Node/TS)", da)
+
+
+if __name__ == "__main__":
+    unittest.main()