@kernlang/agon-dedup 0.1.0

package/README.md

@@ -0,0 +1,88 @@
+# @kernlang/agon-dedup
+
+Brainstorm dedup sidecar — collapses near-duplicate engine drafts so you don't read the same idea three times.
+
+## Why Python
+
+- Semantic similarity needs sentence embeddings, not bag-of-words. TF-IDF was tried first and scored 0.06 between drafts that say the same thing in different words. Useless for this task.
+- `fastembed` (ONNX-based, ~30MB) gives proper paraphrase detection at ~500ms cold start without dragging in torch.
+- TS has no comparable library — `transformers.js` works but the WASM cold-start is multi-second and the model is 80MB+ on first download.
+
+This is the boundary where Python is *actually better*, not just different.
+
+## Install
+
+```bash
+python3 -m pip install --user -r packages/dedup/requirements.txt
+# or:
+npm run install:python -w packages/dedup
+```
+
+Model is downloaded once on first run (~80MB) and cached under `~/.cache/fastembed/`.
+
+## Use
+
+```bash
+# JSONL on stdin, JSON on stdout
+echo '{"id":"claude","text":"Ship A"}
+{"id":"codex","text":"Pick option A"}
+{"id":"gemini","text":"Hold off entirely"}' | python3 packages/dedup/sidecar.py
+```
+
+Output:
+
+```json
+{
+  "groups": [
+    {"members": ["claude", "codex"], "representative": "codex", "similarity": 0.7},
+    {"members": ["gemini"], "representative": "gemini", "similarity": 1.0}
+  ],
+  "threshold": 0.55,
+  "method": "minilm-cosine"
+}
+```
+
+## Test
+
+```bash
+npm run test:sidecar -w packages/dedup
+```
+
+The smoke test feeds three drafts (two paraphrases + one dissent) and asserts the paraphrases group while the dissent stays alone.
+
+## Tuning
+
+`THRESHOLD = 0.55` in `sidecar.py`. Calibration:
+
+| Pair                                  | MiniLM cosine |
+| ------------------------------------- | ------------- |
+| Identical text                        | 1.0           |
+| Same idea, different wording          | 0.7 – 0.9     |
+| Related topic                         | 0.4 – 0.7     |
+| Unrelated                             | < 0.3         |
+
+Lower = more aggressive merging (risk: collapsing real disagreement). Higher = more conservative (risk: missing obvious paraphrases).
+
+## Spawn model
+
+Per-call subprocess. Agon spawns `python3 sidecar.py` only when it has drafts to dedupe, matching how engine adapters spawn their CLIs. Cold-start is the model load (~500ms). For 6 drafts the full call lands in well under 2s — negligible against an 8-12 minute brainstorm.
+
+## Status
+
+- [x] Sidecar built and smoke-tested
+- [x] Workspace registered
+- [x] Wired into `runBrainstorm` (`packages/forge/src/kern/dedup-bridge.kern` spawns the sidecar; result attached as `BrainstormResult.groups`)
+- [x] CLI `agon brainstorm` shows `(N engines agree)` tag in the bids table
+- [ ] Integration test against a real `agon brainstorm` call (manual for now — brainstorms run 8-12 min with 6 engines)
+- [ ] Optional: cache embeddings per session so identical drafts across re-runs reuse vectors
+
+## Failure modes (graceful)
+
+The bridge in `packages/forge` returns `null` and the CLI falls back to the un-deduped bids table when:
+
+- `python3` is missing (or `AGON_PYTHON` env var points at something broken)
+- `packages/dedup/sidecar.py` is missing (e.g., production install only shipped TS)
+- `fastembed` not installed (`exit 2`) — a one-line warning prints with the install command
+- Sidecar emits malformed JSON
+
+In every case agon brainstorm still completes — dedup is purely additive.

package/classifier.py

@@ -0,0 +1,134 @@
+#!/usr/bin/env python3
+"""
+Task classifier sidecar — zero-shot classification via fastembed.
+
+Fed when the regex classifier in `task-classifier.kern` falls through to
+'other'. Embeds the input text, embeds each candidate label's description,
+returns the label with highest cosine similarity.
+
+Protocol:
+  stdin  — single JSON line: {"text": "<task description>"}
+  stdout — single JSON: {"class": "<TaskClass>",
+                          "confidence": 0.74,
+                          "scores": {"feature": 0.74, "bugfix": 0.31, ...}}
+
+Returns 'other' if no class scores above MIN_CONFIDENCE.
+
+Exit codes:
+  0  — success
+  1  — bad input
+  2  — fastembed not installed
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+
+
+MIN_CONFIDENCE = 0.10
+MARGIN = 0.05  # top class must beat #2 by this much to commit; otherwise 'other'
+MODEL = "sentence-transformers/all-MiniLM-L6-v2"
+
+# Few-shot examples per label. We embed concrete task phrasings rather than
+# abstract definitions because all-MiniLM scores concrete-vs-concrete cosine
+# much higher than concrete-vs-abstract. Empirically this lifts true-positive
+# rates on Agon's actual prompt distribution from ~30% to ~85%.
+LABELS: dict[str, str] = {
+    "docs":      ("Update the README. Document the API surface. Add comments "
+                  "explaining the algorithm. Write a changelog entry. "
+                  "Rationale for the migration. Update the docs."),
+    "test":      ("Add a unit test. Cover the edge case with assertions. "
+                  "Write integration tests for the auth flow. Increase test "
+                  "coverage. Add fixtures. Snapshot test."),
+    "bugfix":    ("Fix the off-by-one error. Why is the cache evicting? "
+                  "Resolve the crash on startup. Patch the regression. "
+                  "Off-by-one. Race condition. Memory leak. Stuck process. "
+                  "Broken behavior. Unexpected output."),
+    "refactor":  ("Rename across the codebase. Extract the helper. Simplify "
+                  "this method. Reorganize the file structure. Clean up dead "
+                  "code. Move modules. Restructure without changing behavior."),
+    "algorithm": ("Implement Glicko-2 ratings. Compute the rolling median. "
+                  "Optimize the sort. Score the engines. Calculate confidence "
+                  "intervals. Compute distances. Numerical computation. "
+                  "Data structure. Sorting algorithm."),
+    "feature":   ("Add support for streaming. Build a streaming JSON parser. "
+                  "Create a new dashboard. Implement a new API endpoint. "
+                  "Build a CLI command. Add a new capability. Ship a new "
+                  "feature. Extend the engine adapter."),
+}
+
+
+def _read_input() -> str:
+    raw = sys.stdin.read().strip()
+    if not raw:
+        print("classifier-sidecar: empty stdin", file=sys.stderr)
+        sys.exit(1)
+    try:
+        obj = json.loads(raw)
+    except json.JSONDecodeError as err:
+        print(f"classifier-sidecar: invalid JSON: {err}", file=sys.stderr)
+        sys.exit(1)
+    if not isinstance(obj, dict) or "text" not in obj:
+        print("classifier-sidecar: missing 'text' field", file=sys.stderr)
+        sys.exit(1)
+    text = str(obj["text"]).strip()
+    if not text:
+        print("classifier-sidecar: 'text' is empty", file=sys.stderr)
+        sys.exit(1)
+    return text
+
+
+def _classify(text: str) -> dict:
+    try:
+        from fastembed import TextEmbedding
+        import numpy as np
+    except ImportError:
+        print("classifier-sidecar: fastembed not installed — install via "
+              "`pip install -r packages/dedup/requirements.txt`",
+              file=sys.stderr)
+        sys.exit(2)
+
+    embedder = TextEmbedding(MODEL)
+    label_keys = list(LABELS.keys())
+    label_descriptions = [LABELS[k] for k in label_keys]
+    all_texts = [text] + label_descriptions
+    embs = np.array(list(embedder.embed(all_texts)))
+
+    norms = np.linalg.norm(embs, axis=1, keepdims=True)
+    normed = embs / np.where(norms == 0, 1, norms)
+    text_vec = normed[0]
+    label_vecs = normed[1:]
+    sims = label_vecs @ text_vec
+
+    scores = {label_keys[i]: round(float(sims[i]), 3)
+              for i in range(len(label_keys))}
+    sorted_indices = sorted(range(len(sims)), key=lambda i: -sims[i])
+    best_idx = sorted_indices[0]
+    second_idx = sorted_indices[1]
+    best_label = label_keys[best_idx]
+    best_score = float(sims[best_idx])
+    margin = best_score - float(sims[second_idx])
+
+    if best_score < MIN_CONFIDENCE or margin < MARGIN:
+        chosen = "other"
+    else:
+        chosen = best_label
+
+    return {
+        "class": chosen,
+        "confidence": round(best_score, 3),
+        "margin": round(margin, 3),
+        "scores": scores,
+    }
+
+
+def main() -> None:
+    text = _read_input()
+    result = _classify(text)
+    json.dump(result, sys.stdout)
+    sys.stdout.write("\n")
+
+
+if __name__ == "__main__":
+    main()

package/history-search.py

@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+"""
+History search sidecar — semantic ranking of forge run manifests by query.
+
+Reuses the MiniLM tax already paid by sidecar.py / classifier.py — same
+fastembed model, same ~30MB on disk, same ~500ms cold / ~50ms warm.
+
+Why Python: TS has no production-grade local embedding runtime. The dedup
+sidecar already proved MiniLM/cosine beats substring grep (0.06 → 0.83+ on
+paraphrases). The current `agon history` lookup is exact substring on the
+forgeId, so paraphrased queries ("the SaaS API thing" vs "FastAPI shim")
+get nothing back. Cosine over MiniLM fixes that.
+
+Protocol:
+  stdin  — single JSON:
+    {
+      "query": "<search text>",
+      "items": [{"id": "<runId>", "text": "<task + fitnessCmd + ...>"}, ...],
+      "top_k": 10        // optional, default 10
+    }
+  stdout — single JSON:
+    {
+      "results": [{"id": "<runId>", "similarity": 0.82}, ...],
+      "method":  "minilm-cosine",
+      "model":   "sentence-transformers/all-MiniLM-L6-v2"
+    }
+
+Exit codes:
+  0 — success
+  1 — bad input (malformed JSON, missing fields, empty)
+  2 — fastembed not installed (caller should fall back to chronological)
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from typing import Any
+
+
+MODEL = "sentence-transformers/all-MiniLM-L6-v2"
+DEFAULT_TOP_K = 10
+MIN_SIMILARITY = 0.15  # below this, the match is effectively noise — drop it
+
+
+def _read_input() -> dict[str, Any]:
+    raw = sys.stdin.read().strip()
+    if not raw:
+        print("history-search: empty stdin", file=sys.stderr)
+        sys.exit(1)
+    try:
+        obj = json.loads(raw)
+    except json.JSONDecodeError as err:
+        print(f"history-search: invalid JSON: {err}", file=sys.stderr)
+        sys.exit(1)
+    if not isinstance(obj, dict):
+        print("history-search: input must be a JSON object", file=sys.stderr)
+        sys.exit(1)
+    query = obj.get("query")
+    items = obj.get("items")
+    if not isinstance(query, str) or not query.strip():
+        print("history-search: 'query' must be a non-empty string",
+              file=sys.stderr)
+        sys.exit(1)
+    if not isinstance(items, list) or not items:
+        print("history-search: 'items' must be a non-empty array",
+              file=sys.stderr)
+        sys.exit(1)
+    normalized: list[dict[str, str]] = []
+    for i, raw_item in enumerate(items):
+        if (not isinstance(raw_item, dict)
+                or "id" not in raw_item or "text" not in raw_item):
+            print(f"history-search: item[{i}] missing 'id' or 'text'",
+                  file=sys.stderr)
+            sys.exit(1)
+        text = str(raw_item["text"]).strip()
+        if not text:
+            # Skip empty-text items rather than fail — manifests without a
+            # task description shouldn't poison the whole query.
+            continue
+        normalized.append({"id": str(raw_item["id"]), "text": text})
+    if not normalized:
+        print("history-search: no items with non-empty text", file=sys.stderr)
+        sys.exit(1)
+    top_k = obj.get("top_k", DEFAULT_TOP_K)
+    if not isinstance(top_k, int) or top_k <= 0:
+        top_k = DEFAULT_TOP_K
+    return {"query": query.strip(), "items": normalized, "top_k": top_k}
+
+
+def _rank(payload: dict[str, Any]) -> list[dict[str, Any]]:
+    try:
+        from fastembed import TextEmbedding
+        import numpy as np
+    except ImportError:
+        print("history-search: fastembed not installed — install via "
+              "`pip install -r packages/dedup/requirements.txt`",
+              file=sys.stderr)
+        sys.exit(2)
+
+    query: str = payload["query"]
+    items: list[dict[str, str]] = payload["items"]
+    top_k: int = payload["top_k"]
+
+    embedder = TextEmbedding(MODEL)
+    texts = [query] + [item["text"] for item in items]
+    embs = np.array(list(embedder.embed(texts)))
+
+    norms = np.linalg.norm(embs, axis=1, keepdims=True)
+    normed = embs / np.where(norms == 0, 1, norms)
+    query_vec = normed[0]
+    item_vecs = normed[1:]
+
+    sims = item_vecs @ query_vec  # shape: (n_items,)
+
+    scored = [
+        {"id": items[i]["id"], "similarity": float(sims[i])}
+        for i in range(len(items))
+        if float(sims[i]) >= MIN_SIMILARITY
+    ]
+    scored.sort(key=lambda r: -r["similarity"])
+    top = scored[:top_k]
+    for r in top:
+        r["similarity"] = round(r["similarity"], 3)
+    return top
+
+
+def main() -> None:
+    payload = _read_input()
+    results = _rank(payload)
+    json.dump(
+        {"results": results, "method": "minilm-cosine", "model": MODEL},
+        sys.stdout,
+    )
+    sys.stdout.write("\n")
+
+
+if __name__ == "__main__":
+    main()

package/package.json

@@ -0,0 +1,29 @@
+{
+  "name": "@kernlang/agon-dedup",
+  "version": "0.1.0",
+  "description": "Python sidecars for Agon AI — semantic embeddings (fastembed/MiniLM) and tree-sitter syntax validation. Bridged from KERN via stdin/stdout JSON. Ships .py files that the @kernlang/agon-core bridges spawn at runtime.",
+  "type": "module",
+  "files": [
+    "*.py",
+    "requirements.txt",
+    "README.md"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "scripts": {
+    "install:python": "python3 -m pip install --user -r requirements.txt",
+    "test:sidecar": "node tests/smoke.mjs",
+    "test:classifier": "node tests/classifier-smoke.mjs",
+    "test:history-search": "node tests/history-search-smoke.mjs",
+    "test:syntax-validator": "node tests/syntax-validator-smoke.mjs",
+    "test:syntax-validator-bridge": "node tests/syntax-validator-bridge-smoke.mjs",
+    "test:all-sidecars": "npm run test:sidecar && npm run test:classifier && npm run test:history-search && npm run test:syntax-validator && npm run test:syntax-validator-bridge"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/KERNlang/agon.git",
+    "directory": "packages/dedup"
+  },
+  "license": "MIT"
+}

package/requirements.txt

@@ -0,0 +1,7 @@
+fastembed>=0.4,<0.8
+numpy>=1.26,<3
+tree-sitter>=0.23,<0.25
+tree-sitter-python>=0.23,<0.25
+tree-sitter-typescript>=0.23,<0.25
+tree-sitter-javascript>=0.23,<0.25
+tree-sitter-json>=0.24,<0.26

package/sidecar.py

@@ -0,0 +1,145 @@
+#!/usr/bin/env python3
+"""
+Brainstorm dedup sidecar — clusters near-duplicate engine drafts.
+
+Uses sentence embeddings (MiniLM via fastembed/ONNX, ~30MB) so paraphrases
+group together. TF-IDF was tried first and scored 0.06 between drafts that
+say the same thing in different words — useless for this task.
+
+Protocol:
+  stdin  — JSONL, one per line: {"id": "<engineId>", "text": "<draft>"}
+  stdout — single JSON:
+    {
+      "groups": [
+        {"members": ["claude", "codex"], "representative": "claude",
+         "similarity": 0.83},
+        ...
+      ],
+      "threshold": 0.55,
+      "method": "minilm-cosine"
+    }
+
+Two engines are in the same group iff cosine similarity of their MiniLM
+embeddings >= threshold. Representative = engine with the longest text in
+the group (most detail). Singleton groups are emitted too.
+
+Exit codes:
+  0  — success
+  1  — bad input (malformed JSON, no items)
+  2  — fastembed not installed (caller should fall back to no-dedup)
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from typing import Any
+
+
+THRESHOLD = 0.55
+MODEL = "sentence-transformers/all-MiniLM-L6-v2"
+
+
+def _read_input() -> list[dict[str, str]]:
+    items: list[dict[str, str]] = []
+    for line_num, raw_line in enumerate(sys.stdin, 1):
+        line = raw_line.strip()
+        if not line:
+            continue
+        try:
+            obj = json.loads(line)
+        except json.JSONDecodeError as err:
+            print(f"dedup-sidecar: line {line_num} is not valid JSON: {err}",
+                  file=sys.stderr)
+            sys.exit(1)
+        if not isinstance(obj, dict) or "id" not in obj or "text" not in obj:
+            print(f"dedup-sidecar: line {line_num} missing 'id' or 'text'",
+                  file=sys.stderr)
+            sys.exit(1)
+        items.append({"id": str(obj["id"]), "text": str(obj["text"])})
+    return items
+
+
+def _cluster(items: list[dict[str, str]]) -> list[dict[str, Any]]:
+    if len(items) == 1:
+        return [{
+            "members": [items[0]["id"]],
+            "representative": items[0]["id"],
+            "similarity": 1.0,
+        }]
+
+    try:
+        from fastembed import TextEmbedding
+        import numpy as np
+    except ImportError:
+        print("dedup-sidecar: fastembed not installed — install via "
+              "`pip install -r packages/dedup/requirements.txt`",
+              file=sys.stderr)
+        sys.exit(2)
+
+    embedder = TextEmbedding(MODEL)
+    texts = [item["text"] for item in items]
+    embs = np.array(list(embedder.embed(texts)))
+
+    norms = np.linalg.norm(embs, axis=1, keepdims=True)
+    normed = embs / np.where(norms == 0, 1, norms)
+    sim = normed @ normed.T
+
+    n = len(items)
+    parent = list(range(n))
+
+    def find(i: int) -> int:
+        while parent[i] != i:
+            parent[i] = parent[parent[i]]
+            i = parent[i]
+        return i
+
+    def union(i: int, j: int) -> None:
+        ri, rj = find(i), find(j)
+        if ri != rj:
+            parent[ri] = rj
+
+    for i in range(n):
+        for j in range(i + 1, n):
+            if sim[i][j] >= THRESHOLD:
+                union(i, j)
+
+    clusters: dict[int, list[int]] = {}
+    for i in range(n):
+        clusters.setdefault(find(i), []).append(i)
+
+    groups = []
+    for indices in clusters.values():
+        members_items = [items[i] for i in indices]
+        rep = max(members_items, key=lambda x: len(x["text"]))
+        if len(indices) > 1:
+            pair_sims = [float(sim[i][j])
+                         for i in indices for j in indices if i < j]
+            avg_sim = sum(pair_sims) / len(pair_sims)
+        else:
+            avg_sim = 1.0
+        groups.append({
+            "members": [item["id"] for item in members_items],
+            "representative": rep["id"],
+            "similarity": round(avg_sim, 3),
+        })
+
+    groups.sort(key=lambda g: (-len(g["members"]), g["representative"]))
+    return groups
+
+
+def main() -> None:
+    items = _read_input()
+    if not items:
+        print("dedup-sidecar: no items on stdin", file=sys.stderr)
+        sys.exit(1)
+    groups = _cluster(items)
+    json.dump(
+        {"groups": groups, "threshold": THRESHOLD, "method": "minilm-cosine"},
+        sys.stdout,
+    )
+    sys.stdout.write("\n")
+
+
+if __name__ == "__main__":
+    main()

package/syntax-validator.py

@@ -0,0 +1,331 @@
+#!/usr/bin/env python3
+"""
+Syntax validator sidecar — parses files via tree-sitter and reports errors.
+
+Why Python: tree-sitter has mature grammar packages on PyPI with prebuilt
+wheels (tree-sitter-python, tree-sitter-typescript, tree-sitter-javascript).
+TypeScript has its own tree-sitter bindings via node-tree-sitter, but they
+require native module compilation at install time and add a heavyweight
+dependency to @kernlang/agon-core for a workflow that runs after patch-apply (not
+per keystroke).
+
+The current forge `validate` mode in packages/forge/src/generated/stages.ts
+only inspects engine stdout (regex match for "validated"/"looks good"/etc.).
+Nothing actually parses the resulting code. Tree-sitter fills that gap.
+
+Protocol:
+  stdin  — single JSON:
+    {
+      "files": [
+        {"path": "src/foo.ts", "content": "...", "language": "typescript"},
+        ...
+      ]
+    }
+  stdout — single JSON:
+    {
+      "results": [
+        {
+          "path": "src/foo.ts",
+          "valid": true,
+          "language": "typescript",
+          "errors": []
+        },
+        {
+          "path": "src/bad.ts",
+          "valid": false,
+          "language": "typescript",
+          "errors": [
+            {"row": 12, "column": 4, "message": "MISSING ;"}
+          ]
+        }
+      ],
+      "method": "tree-sitter",
+      "supported_languages": ["typescript", "tsx", "javascript", ...]
+    }
+
+Languages: typescript, tsx, javascript, jsx, python, json
+  (additional names accepted: ts, ty, py — normalized internally)
+
+Exit codes:
+  0 — success
+  1 — bad input
+  2 — tree-sitter or a grammar package not installed
+  3 — at least one file's language is unsupported (still produces output;
+      caller can decide what to do with unsupported)
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from typing import Any
+
+
+LANGUAGE_ALIASES: dict[str, str] = {
+    "ts": "typescript",
+    "typescript": "typescript",
+    "tsx": "tsx",
+    "js": "javascript",
+    "javascript": "javascript",
+    "jsx": "jsx",
+    "py": "python",
+    "python": "python",
+    "json": "json",
+}
+
+
+SUPPORTED_LANGUAGES: frozenset[str] = frozenset({
+    "typescript", "tsx", "javascript", "jsx", "python", "json",
+})
+
+
+def _read_input() -> list[dict[str, str]]:
+    raw = sys.stdin.read().strip()
+    if not raw:
+        print("syntax-validator: empty stdin", file=sys.stderr)
+        sys.exit(1)
+    try:
+        obj = json.loads(raw)
+    except json.JSONDecodeError as err:
+        print(f"syntax-validator: invalid JSON: {err}", file=sys.stderr)
+        sys.exit(1)
+    if not isinstance(obj, dict) or "files" not in obj:
+        print("syntax-validator: input must be {'files': [...]}",
+              file=sys.stderr)
+        sys.exit(1)
+    files = obj["files"]
+    if not isinstance(files, list) or not files:
+        print("syntax-validator: 'files' must be a non-empty array",
+              file=sys.stderr)
+        sys.exit(1)
+    normalized: list[dict[str, str]] = []
+    for i, raw_item in enumerate(files):
+        if not isinstance(raw_item, dict):
+            print(f"syntax-validator: files[{i}] must be an object",
+                  file=sys.stderr)
+            sys.exit(1)
+        for key in ("path", "content", "language"):
+            if key not in raw_item:
+                print(f"syntax-validator: files[{i}] missing '{key}'",
+                      file=sys.stderr)
+                sys.exit(1)
+            # Reject non-string values explicitly. `str(None)` is "None",
+            # which silently lies about the input — surface the bug here.
+            if not isinstance(raw_item[key], str):
+                print(f"syntax-validator: files[{i}].{key} must be a string "
+                      f"(got {type(raw_item[key]).__name__})",
+                      file=sys.stderr)
+                sys.exit(1)
+        normalized.append({
+            "path": raw_item["path"],
+            "content": raw_item["content"],
+            "language": raw_item["language"].lower().strip(),
+        })
+    return normalized
+
+
+def _load_parsers() -> dict[str, Any]:
+    """Lazy-load tree-sitter grammars. Returns dict of language → Parser."""
+    try:
+        from tree_sitter import Language, Parser
+    except ImportError:
+        print("syntax-validator: tree-sitter not installed — install via "
+              "`pip install -r packages/dedup/requirements.txt`",
+              file=sys.stderr)
+        sys.exit(2)
+
+    parsers: dict[str, Any] = {}
+
+    def _try(name: str, loader) -> None:
+        try:
+            lang = Language(loader())
+        except Exception as err:  # noqa: BLE001  — grammar import is fragile
+            print(f"syntax-validator: failed to load grammar {name}: {err}",
+                  file=sys.stderr)
+            return
+        parsers[name] = Parser(lang)
+
+    # Each grammar package is OPTIONAL — if one isn't installed, the
+    # validator just refuses to parse files of that language (the bridge
+    # then flags them grammar_unavailable). Log the absence to stderr so
+    # `agon doctor` and curious users can see which grammars are missing
+    # rather than silently degrading.
+    def _log_missing_grammar(name: str, err: ImportError) -> None:
+        print(
+            f"syntax-validator: grammar package not installed: {name} ({err})",
+            file=sys.stderr,
+        )
+
+    try:
+        import tree_sitter_python as tspy
+        _try("python", lambda: tspy.language())
+    except ImportError as err:
+        _log_missing_grammar("tree_sitter_python", err)
+
+    try:
+        import tree_sitter_typescript as tsts
+        _try("typescript", lambda: tsts.language_typescript())
+        _try("tsx", lambda: tsts.language_tsx())
+    except ImportError as err:
+        _log_missing_grammar("tree_sitter_typescript", err)
+
+    try:
+        import tree_sitter_javascript as tsjs
+        _try("javascript", lambda: tsjs.language())
+        # JS and JSX share a grammar in tree-sitter-javascript.
+        _try("jsx", lambda: tsjs.language())
+    except ImportError as err:
+        _log_missing_grammar("tree_sitter_javascript", err)
+
+    try:
+        import tree_sitter_json as tsjson
+        _try("json", lambda: tsjson.language())
+    except ImportError as err:
+        _log_missing_grammar("tree_sitter_json", err)
+
+    if not parsers:
+        print("syntax-validator: no grammar packages installed — install via "
+              "`pip install -r packages/dedup/requirements.txt`",
+              file=sys.stderr)
+        sys.exit(2)
+
+    return parsers
+
+
+def _collect_errors(root, cap: int = 10) -> list[dict[str, Any]]:
+    """Iteratively walk the AST collecting ERROR/MISSING nodes up to `cap`.
+    Iterative form prevents RecursionError on deeply-nested invalid input
+    (e.g. heavily-nested JSON, long arithmetic chains, large method chains)."""
+    errors: list[dict[str, Any]] = []
+    stack: list[Any] = [root]
+    while stack and len(errors) < cap:
+        node = stack.pop()
+        if node.is_error or node.is_missing:
+            msg = "MISSING " + node.type if node.is_missing else "ERROR"
+            errors.append({
+                "row": node.start_point[0],
+                "column": node.start_point[1],
+                "message": msg,
+            })
+            if len(errors) >= cap:
+                break
+        # Reverse so the depth-first order matches the source order.
+        for child in reversed(node.children):
+            stack.append(child)
+    return errors
+
+
+def _python_indentation_check(content: str) -> list[dict[str, Any]]:
+    """Tree-sitter's Python grammar is forgiving on indentation — it will
+    accept malformed indentation that the CPython parser rejects (e.g.
+    `def f():\nreturn 1`). Supplement tree-sitter with `compile(...)` for
+    Python files so indentation errors are caught."""
+    try:
+        compile(content, "<input>", "exec")
+        return []
+    except SyntaxError as err:
+        return [{
+            "row": (err.lineno or 1) - 1,
+            "column": (err.offset or 1) - 1,
+            "message": f"PYTHON {err.msg or 'syntax error'}".strip(),
+        }]
+    except (ValueError, TypeError) as err:
+        # ValueError: null bytes in source; TypeError: non-str source.
+        return [{
+            "row": 0,
+            "column": 0,
+            "message": f"PYTHON {type(err).__name__}: {err}",
+        }]
+
+
+def _validate(
+    files: list[dict[str, str]],
+    parsers: dict[str, Any],
+) -> tuple[list[dict[str, Any]], bool]:
+    """Returns (results, any_unsupported). Files whose language we don't
+    know at all get valid=true with `language_unsupported: true` so the
+    caller can skip them. Files whose language IS known but whose grammar
+    failed to load get valid=false with `grammar_unavailable: true` so the
+    caller doesn't mistake a degraded sidecar for clean code."""
+    results: list[dict[str, Any]] = []
+    any_unsupported = False
+    for f in files:
+        lang_raw = f["language"]
+        lang = LANGUAGE_ALIASES.get(lang_raw)
+        if lang is None or lang not in SUPPORTED_LANGUAGES:
+            any_unsupported = True
+            results.append({
+                "path": f["path"],
+                "valid": True,  # cannot prove invalid without a parser
+                "language": lang_raw,
+                "errors": [],
+                "language_unsupported": True,
+            })
+            continue
+        parser = parsers.get(lang)
+        if parser is None:
+            # Known language, but the grammar package isn't installed in
+            # this environment. Surface it as a per-file failure rather
+            # than letting it pass — silent pass-through is the bug
+            # codex flagged at 0.86 confidence.
+            results.append({
+                "path": f["path"],
+                "valid": False,
+                "language": lang,
+                "errors": [{
+                    "row": 0,
+                    "column": 0,
+                    "message": f"grammar-unavailable: install tree-sitter-{lang}",
+                }],
+                "grammar_unavailable": True,
+            })
+            continue
+        try:
+            tree = parser.parse(bytes(f["content"], "utf-8"))
+        except Exception as err:  # noqa: BLE001
+            results.append({
+                "path": f["path"],
+                "valid": False,
+                "language": lang,
+                "errors": [{
+                    "row": 0,
+                    "column": 0,
+                    "message": f"parser-threw: {type(err).__name__}",
+                }],
+            })
+            continue
+        errors: list[dict[str, Any]] = []
+        if tree.root_node.has_error:
+            errors = _collect_errors(tree.root_node)
+        # Python: tree-sitter accepts malformed indentation that the real
+        # parser rejects. Run CPython's parser as a second pass; merge.
+        if lang == "python" and not errors:
+            errors = _python_indentation_check(f["content"])
+        results.append({
+            "path": f["path"],
+            "valid": not errors,
+            "language": lang,
+            "errors": errors,
+        })
+    return results, any_unsupported
+
+
+def main() -> None:
+    files = _read_input()
+    parsers = _load_parsers()
+    results, any_unsupported = _validate(files, parsers)
+    json.dump(
+        {
+            "results": results,
+            "method": "tree-sitter",
+            "supported_languages": sorted(parsers.keys()),
+        },
+        sys.stdout,
+    )
+    sys.stdout.write("\n")
+    if any_unsupported:
+        sys.exit(3)
+
+
+if __name__ == "__main__":
+    main()