hexcli 2.10.0__tar.gz → 2.11.0__tar.gz

{hexcli-2.10.0 → hexcli-2.11.0}/CHANGELOG.md

@@ -6,6 +6,41 @@ the Hexagon NPU, not single-run anecdotes.
 
 ## Unreleased
 
+## 2.11.0 — 2026-09-14
+
+A minor release: tool result text the model reads changed (`verify_syntax`
+reports its rung) and a new nudge was added. Gate: extended suite at 5 runs, seed 20260914, gate PASS after a 6-run recheck (five cases missed once, all 5/6 or better on recheck; run-level 156/205 vs 165/208, p=0.48); multi-turn at 3 runs on a fresh server with 0 invalid runs, no 3/3 case lost, uc3-t7 gained (38/48 vs 33/44, p=0.80); smoke 10/10; stall probe clean; CI green on main and the tag.
+
+- Claims need evidence. A finish that says it ran something, that it
+  works, that the buttons respond or that the tests pass, needs a run this
+  turn (run_code or run_command); reading the file back proves the bytes,
+  not the behaviour. The owner's 2026-09-13 calculator session made three
+  such claims after read_file, with nothing ever run and nothing wired.
+  One nudge names the claim and asks for a run or a plain statement that
+  it was not run; a second unbacked claim goes out with a dim "Nothing was
+  run this turn." under it. The tests-claim nudge is the same rule for one
+  phrase; this is the class.
+- `verify_syntax` is a ladder that reports the rung it reached. A page is
+  parsed and cross-referenced: handlers the markup calls must be functions
+  a script defines, ids the scripts look up must be elements the markup
+  has, buttons must be wired to something, and a script after `</body>` is
+  noted (the calculator: no handler on any button, an id no element had, a
+  function nothing called). Python is parsed and cross-referenced for
+  names it reads but never binds (a star import turns that off). JSON and
+  PowerShell say "parsed". A file with no checker is `NOT CHECKED`, never
+  `OK: skipped`, and says so, so the claim above cannot rest on it.
+- Two eval cases: `claims-1`, the calculator prompt graded on the page
+  being wired and the finish not claiming a run that never happened, and
+  `claims-2`, a Python edit-and-claim variant. The 2026-09-13 page is a
+  fixture.
+- The eval runner waits for the inference slot after a client timeout.
+  An abandoned request keeps the server's one slot until it finishes; the
+  next run queued behind it, timed out too, and took a whole scenario
+  down as invalid (uc3, 2026-09-13, twice). After a timed-out run the
+  runner probes until the backend answers again, up to ten minutes, and
+  the timeout no longer counts toward the abort streak once the slot is
+  back.
+
 ## 2.10.0 — 2026-09-14
 
 A minor release: the retry feedback the model reads changed. Gate: extended

{hexcli-2.10.0 → hexcli-2.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hexcli
-Version: 2.10.0
+Version: 2.11.0
 Summary: Local Hexagon NPU terminal agent for Snapdragon X Elite Windows ARM64
 Project-URL: Homepage, https://github.com/NathanL15/Hex-CLI
 Project-URL: Repository, https://github.com/NathanL15/Hex-CLI

{hexcli-2.10.0 → hexcli-2.11.0}/hexcli/__init__.py

@@ -3,4 +3,4 @@
 # The one place the version is written. pyproject.toml reads it (hatch
 # dynamic version), agent.VERSION re-exports it, and CI refuses a release
 # tag that does not match it.
-__version__ = "2.10.0"
+__version__ = "2.11.0"

{hexcli-2.10.0 → hexcli-2.11.0}/hexcli/agent.py

@@ -1193,6 +1193,9 @@ def _run_autopilot_turn(
     _tests_requested = _asks_to_run_tests(query)
     _tests_nudge_used = False
     _run_targets: list[str] = []
+    _ran_anything = False          # any run_code / run_command this turn: the evidence a behaviour claim needs
+    _claim_nudge_used = False
+    _unbacked_claim = False
     # The last test run's failure output (None once a run passes), so a
     # finish right after a failing run can be sent back once more.
     _last_test_failure: str | None = None
@@ -1338,6 +1341,21 @@ def _run_autopilot_turn(
                 messages.append({"role": "assistant", "content": strip_thinking(raw)})
                 messages.append({"role": "user", "content": _retest_nudge_text(_last_test_failure)})
                 continue
+            # Claims need evidence. "Ran it successfully", "the buttons now
+            # respond", "tests pass": a behaviour claim in the finish needs
+            # a run this turn; reading a file back proves the bytes, not the
+            # behaviour (the calculator session, 2026-09-13: three such
+            # claims, nothing ever run, nothing wired). One nudge: run it, or
+            # say plainly that it was not run. A second unbacked claim goes
+            # out with a one-line notice under it, so the user knows.
+            claim = _behaviour_claim(msg)
+            if (claim and not _ran_anything and config.get("require_verification", True)):
+                if not _claim_nudge_used:
+                    _claim_nudge_used = True
+                    messages.append({"role": "assistant", "content": strip_thinking(raw)})
+                    messages.append({"role": "user", "content": _claim_nudge_text(claim)})
+                    continue
+                _unbacked_claim = True
             # Escalation trigger B — the verification nudge was ignored: the
             # model finished a second time without checking its own mutation.
             if (_unverified_mutation and _verify_nudge_used
@@ -1367,6 +1385,8 @@ def _run_autopilot_turn(
                 })
                 continue
             result = msg or last_tool_output or "Done."
+            if _unbacked_claim:
+                ui.cprint("  Nothing was run this turn.", C.DIM)
             memory.maybe_index_turn(config, query, tools_used, touched_paths, outcome="completed")
             if session:
                 _record_undo_snapshots(session, _turn_snapshots)
@@ -1398,6 +1418,7 @@ def _run_autopilot_turn(
             touched_paths.append(str(tool_path))
         if tool_name in ("run_code", "run_command") and isinstance(action.get("args"), dict):
             _run_targets.append(str(action["args"].get("path") or action["args"].get("command") or ""))
+            _ran_anything = True
 
         # Capture file state before first mutation so /undo can restore it.
         if tool_name in {"edit_file", "write_file", "append_file"} and tool_path:
@@ -1685,6 +1706,30 @@ _RUN_TESTS_RE = re.compile(
 )
 
 
+_BEHAVIOUR_CLAIM_RE = re.compile(
+    r"\b(ran (?:it|the [a-z]+|successfully)(?: successfully)?|runs (?:correctly|fine|successfully|as expected)|"
+    r"(?:it|this|that|everything|the [a-z]+) (?:now |all )?works\b|works (?:as expected|correctly|now|fine)|"
+    r"(?:it|this|that|everything|the [a-z]+) is (?:now )?working(?: correctly| as expected)?|"
+    r"(?:buttons?|it|the app|the page|the script) (?:now )?responds?|"
+    r"calculates? correctly|opens? correctly|executed successfully|"
+    r"tests? (?:pass|passed|passing|succeed)|all tests pass|verified (?:that )?it (?:works|runs)|"
+    r"confirmed (?:that )?it (?:works|runs))\b", re.IGNORECASE)
+
+
+def _behaviour_claim(message: str) -> str:
+    """The first behaviour claim in a finish message, or ''. Deliberately a
+    list of ways to say "I ran it and it works", not a list of verbs: "the
+    function returns the sum" is a description, "it works" is a claim."""
+    m = _BEHAVIOUR_CLAIM_RE.search(message or "")
+    return m.group(0) if m else ""
+
+
+def _claim_nudge_text(claim: str) -> str:
+    return (f"You said \"{claim}\" but nothing was run this turn. Run it with run_code or "
+            "run_command and report what you observed, or say plainly that it was not run. "
+            "Respond with JSON only.")
+
+
 def _asks_to_run_tests(query: str) -> bool:
     """True when the request itself asks for the tests to be run."""
     return bool(_RUN_TESTS_RE.search(query or ""))

{hexcli-2.10.0 → hexcli-2.11.0}/hexcli/tools.py

@@ -581,22 +581,205 @@ def _verify_node_syntax(path: Path) -> tuple[bool, str]:
     return False, f"FAIL: {result.stderr.strip() or result.stdout.strip()}"
 
 
+_PY_IMPLICIT = {"__name__", "__file__", "__doc__", "__builtins__", "__spec__", "__loader__",
+                "__package__", "__path__", "__annotations__", "__debug__"}
+
+
+def undefined_python_names(source: str) -> list[str]:
+    """Names the module reads but never binds anywhere and Python does not
+    provide: the cross-reference rung for Python. Scopes are collapsed to
+    the module (a name bound anywhere counts), so this never flags a name
+    that some function defines; it only catches the calculator-class
+    mistake of calling something that does not exist. A star import turns
+    the check off."""
+    import builtins
+    try:
+        tree = ast.parse(source)
+    except SyntaxError:
+        return []
+    bound: set[str] = set(dir(builtins)) | _PY_IMPLICIT
+    loads: dict[str, int] = {}
+    for node in ast.walk(tree):
+        if isinstance(node, ast.ImportFrom) and any(a.name == "*" for a in node.names):
+            return []
+        if isinstance(node, (ast.Import, ast.ImportFrom)):
+            for a in node.names:
+                bound.add((a.asname or a.name).split(".")[0])
+        elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
+            bound.add(node.name)
+            if not isinstance(node, ast.ClassDef):
+                args = node.args
+                for a in [*args.posonlyargs, *args.args, *args.kwonlyargs]:
+                    bound.add(a.arg)
+                for a in (args.vararg, args.kwarg):
+                    if a is not None:
+                        bound.add(a.arg)
+        elif isinstance(node, ast.Lambda):
+            args = node.args
+            for a in [*args.posonlyargs, *args.args, *args.kwonlyargs]:
+                bound.add(a.arg)
+            for a in (args.vararg, args.kwarg):
+                if a is not None:
+                    bound.add(a.arg)
+        elif isinstance(node, ast.ExceptHandler) and node.name:
+            bound.add(node.name)
+        elif isinstance(node, (ast.Global, ast.Nonlocal)):
+            bound.update(node.names)
+        elif isinstance(node, ast.MatchAs) and node.name:
+            bound.add(node.name)
+        elif isinstance(node, ast.MatchStar) and node.name:
+            bound.add(node.name)
+        elif isinstance(node, ast.MatchMapping) and node.rest:
+            bound.add(node.rest)
+        elif isinstance(node, ast.Name):
+            if isinstance(node.ctx, (ast.Store, ast.Del)):
+                bound.add(node.id)
+            else:
+                loads.setdefault(node.id, node.lineno)
+    return [f"{name} (line {line})" for name, line in sorted(loads.items(), key=lambda kv: kv[1])
+            if name not in bound]
+
+
+_JS_BUILTIN_CALLS = {"alert", "confirm", "prompt", "eval", "parseInt", "parseFloat", "String",
+                     "Number", "Boolean", "Math", "setTimeout", "setInterval", "console",
+                     "document", "window", "event", "this", "if", "for", "while", "return",
+                     "function", "new", "Array", "Object", "JSON", "isNaN", "Date", "fetch",
+                     "requestAnimationFrame", "clearTimeout", "clearInterval", "Promise"}
+
+
+def html_wiring_report(path: Path) -> tuple[bool, str]:
+    """Cross-reference rung for a page: the handlers the markup calls must be
+    functions the scripts define, the ids the scripts look up must be
+    elements the markup has, and buttons must be wired to something. Reading
+    a page back proves the bytes; this proves the parts are connected
+    (2026-09-14: a calculator with buttons that called nothing, a script that
+    looked up an id no element had, and a function nothing called was
+    "verified" three times by read_file)."""
+    from html.parser import HTMLParser
+
+    class _Walk(HTMLParser):
+        def __init__(self) -> None:
+            super().__init__()
+            self.ids: set[str] = set()
+            self.handlers: list[str] = []
+            self.buttons = 0
+            self.scripts: list[str] = []
+            self.script_srcs: list[str] = []
+            self._in_script = False
+            self._body_closed = False
+            self.script_after_body = False
+            self._buf: list[str] = []
+
+        def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
+            a = {k.lower(): (v or "") for k, v in attrs}
+            if a.get("id"):
+                self.ids.add(a["id"])
+            for k, v in a.items():
+                if k.startswith("on") and v:
+                    self.handlers.extend(m for m in re.findall(r"([A-Za-z_$][\w$]*)\s*\(", v))
+            if tag == "button" or (tag == "input" and a.get("type", "").lower() in {"button", "submit"}):
+                self.buttons += 1
+            if tag == "script":
+                self._in_script = True
+                self._buf = []
+                if a.get("src"):
+                    self.script_srcs.append(a["src"])
+                if self._body_closed:
+                    self.script_after_body = True
+
+        def handle_endtag(self, tag: str) -> None:
+            if tag == "script" and self._in_script:
+                self._in_script = False
+                self.scripts.append("".join(self._buf))
+            if tag in {"body", "html"}:
+                self._body_closed = True
+
+        def handle_data(self, data: str) -> None:
+            if self._in_script:
+                self._buf.append(data)
+
+    source = path.read_text(encoding="utf-8", errors="replace")
+    w = _Walk()
+    try:
+        w.feed(source)
+        w.close()
+    except Exception as exc:  # noqa: BLE001 — html.parser is lenient; anything else is a report
+        return False, f"FAIL: could not parse the page: {exc}"
+    script = "\n".join(w.scripts)
+    for src in w.script_srcs:
+        try:
+            script += "\n" + (path.parent / src).read_text(encoding="utf-8", errors="replace")
+        except OSError:
+            pass
+    defined = set(re.findall(r"\bfunction\s+([A-Za-z_$][\w$]*)\s*\(", script))
+    defined |= set(re.findall(r"\b(?:const|let|var)\s+([A-Za-z_$][\w$]*)\s*=\s*(?:async\s+)?(?:function\b|\(|[A-Za-z_$][\w$]*\s*=>)", script))
+    defined |= set(re.findall(r"\b(?:window\.)?([A-Za-z_$][\w$]*)\s*=\s*(?:async\s+)?function\b", script))
+    looked_up = set(re.findall(r"getElementById\(\s*['\"]([^'\"]+)['\"]", script))
+    looked_up |= set(re.findall(r"querySelector(?:All)?\(\s*['\"]#([\w-]+)", script))
+    listeners = bool(re.search(r"addEventListener\s*\(|\.on[a-z]+\s*=", script))
+    problems: list[str] = []
+    undefined = sorted({h for h in w.handlers if h not in defined and h not in _JS_BUILTIN_CALLS})
+    if undefined:
+        problems.append("handlers the markup calls but no script defines: " + ", ".join(undefined))
+    missing = sorted(i for i in looked_up if i not in w.ids)
+    if missing:
+        problems.append("ids the script looks up but no element has: " + ", ".join(missing))
+    if w.buttons and not w.handlers and not listeners:
+        problems.append(f"{w.buttons} button(s) and none is wired to a script (no on* attribute, no addEventListener)")
+    notes: list[str] = []
+    if w.script_after_body:
+        notes.append("a <script> sits after </body>")
+    if defined and not w.handlers and not listeners:
+        notes.append("functions defined but nothing calls them: " + ", ".join(sorted(defined)))
+    if problems:
+        return False, "FAIL: parsed, cross-referenced: " + "; ".join(problems + notes)
+    summary = f"OK: parsed, cross-referenced ({len(w.handlers)} handler call(s), {len(w.ids)} id(s), {w.buttons} button(s))"
+    if notes:
+        summary += "; " + "; ".join(notes)
+    return True, summary
+
+
 def verify_syntax_tool(path_text: str, language: str, shell_exe: str) -> str:
+    """The verification ladder. Every result says which rung was reached:
+    executed (not here: run_code does that), parsed and cross-referenced,
+    parsed only, or NOT CHECKED. A file this tool cannot check is reported
+    as unchecked, never as OK: the model used to read "OK: skipped" and
+    finish with "verified"."""
     path = resolve_path(path_text)
     if not path.exists():
         raise RuntimeError(f"File not found: {path}")
-    lang = (language or "").strip().lower() or _LANGUAGE_BY_EXT.get(path.suffix.lower(), "")
-    if lang == "python":
+    suffix = path.suffix.lower()
+    lang = (language or "").strip().lower() or _LANGUAGE_BY_EXT.get(suffix, "")
+    if suffix in {".html", ".htm"} or lang in {"html", "htm"}:
+        ok, detail = html_wiring_report(path)
+    elif lang == "python":
         ok, detail = _verify_python_syntax(path)
+        if ok and not detail.startswith("OK: skipped"):
+            undefined = undefined_python_names(path.read_text(encoding="utf-8", errors="replace"))
+            if undefined:
+                ok, detail = False, "FAIL: parsed, cross-referenced: references undefined name(s): " + ", ".join(undefined)
+            else:
+                detail = "OK: parsed, cross-referenced: no syntax errors, every name it reads is defined"
     elif lang == "json":
         ok, detail = _verify_json_syntax(path)
+        if ok:
+            detail = detail.replace("OK: valid JSON", "OK: parsed: valid JSON")
     elif lang == "powershell":
         ok, detail = _verify_powershell_syntax(path, shell_exe)
-    elif lang == "node" or path.suffix.lower() in {".js", ".mjs", ".cjs", ".ts", ".tsx", ".jsx"}:
+        if ok and detail.startswith("OK: skipped"):
+            ok, detail = True, "NOT CHECKED: " + detail[len("OK: skipped ("):].rstrip(")")
+        elif ok:
+            detail = "OK: parsed: no syntax errors"
+    elif lang == "node" or suffix in {".js", ".mjs", ".cjs", ".ts", ".tsx", ".jsx"}:
         ok, detail = _verify_node_syntax(path)
+        if ok and detail.startswith("OK: skipped"):
+            ok, detail = True, "NOT CHECKED: " + detail[len("OK: skipped ("):].rstrip(")")
+        elif ok:
+            detail = "OK: parsed: no syntax errors"
     else:
-        ok, detail = True, f"OK: skipped (no syntax checker for '{path.suffix or language or 'unknown'}')"
-    ui.tool_event("verify", f"{path}  ({'pass' if ok else 'fail'})")
+        ok, detail = True, (f"NOT CHECKED: no checker for '{suffix or language or 'this file'}'. "
+                            "The file was not verified; say so if you report on it.")
+    ui.tool_event("verify", f"{path}  ({'fail' if not ok else ('unchecked' if detail.startswith('NOT CHECKED') else 'pass')})")
     return detail