hexcli 2.12.0__tar.gz → 2.14.0__tar.gz

{hexcli-2.12.0 → hexcli-2.14.0}/CHANGELOG.md

@@ -4,6 +4,71 @@ Full evidence for every claim below — including the experiments that failed
 lives in `docs/V2_PLAN.md` §14. Numbers are pass^k over repeated live runs on
 the Hexagon NPU, not single-run anecdotes.
 
+## 2.14.0 — 2026-09-17
+
+A minor release: one tool behaves differently for the model (a `write_file`
+body that was JSON-escaped twice is decoded), plus a test-hygiene fix.
+
+> **Gate: PASS** on the pinned 24-case set, all 24 held at 5 runs with no
+> recheck. Own arm, fresh server, seed 20260917, 0 invalid of 245 runs.
+> Against the v2.11.1 12-run baseline: run-level 402/514 vs 179/230, −0.4 %,
+> Fisher p = 0.92; pass^k 26 → 30 of 46 shared cases, McNemar p = 0.22. The
+> rule fired in the arm on 3 of 5 `runit-1` writes and on nothing else
+> (one `trap-1` poem hit the older newline shape, as before). `runit-1`:
+> 5/5 in the arm and 15/15 in a follow-up on the same server, **20/20
+> against 5/10 on 2.12.0, p = 0.002**. Ceiling cases that moved: `agentic-3`
+> 3/5 (p = 0.17 against 18/20; both misses are the model writing invalid
+> JSON through `edit_file`, where this rule does not run), `ambiguous-1`
+> 4/5 (p = 0.50). Smoke 10/10 on a fresh server before the arm.
+
+- `write_file` decodes a body whose every quote is escaped. `runit-1`
+  ("Write hello.py that prints Hello, world and then run it") sat at 5/10
+  in the 2.12.0 arm, and not one of the misses was the "and run it" the
+  case was written for: every run wrote the file and ran it. The model
+  escaped its JSON twice, `{"content":"print(\\\"Hello, world\\\")"}`, so
+  the file held `print(\"Hello, world\")`, a SyntaxError; it then "fixed"
+  the line by editing it to itself, twice, until the step limit. The
+  double-escape rule from 2.9.0 only knew the newline shape (literal `\n`
+  and no real line break). It now also knows the quote shape — at least one
+  `\"` and not a single bare `"` — and decodes the body once, as before.
+  Measured over the 511 `write_file` bodies on record: 11 have the shape,
+  all of them this defect in this case, and none has both escaped and bare
+  quotes, so a body with even one bare quote is left exactly as sent.
+  `append_file` is untouched: 0 of its 17 recorded bodies have either shape.
+
+- A test suite was writing into the owner's real chat log. The shell wiring
+  test in `evals/test_product_shell.py` drove `hexcli.agent` with the mock
+  backend and a temp config that left `chat_log_enabled` on, so every run
+  of the suite added a "say hello" session under `~/.shellai/chatlog/` — 93
+  of them by 2026-09-17, beside the 127 real sessions. The 2.12.0 exposure
+  counts were taken over that mix: "306 real turns" was 196 real turns and
+  93 scripted ones (the 11 firings were all on real turns, so the true
+  positives stand; the denominator did not). The test now turns the log
+  off, the corpus loader skips mock-backend sessions, the mock files were
+  moved aside, and the intent nudge's recorded rate is corrected to 9 of
+  196 real turns.
+
+## 2.13.0 — 2026-09-17
+
+- Commands are classified by the name the shell will really run, not by the
+  text as typed. PowerShell resolves aliases before executing, so the
+  pattern tiers were reading a different command from the one that ran:
+  `ri C:\data`, `rmdir C:\data`, `& ('Remove'+'-Item') C:\data` and
+  `sl C:\ ; ri *` all classified as *caution* and executed with **no
+  confirmation at all**. `hexcli/psparse.py` runs PowerShell's own parser
+  (`[Parser]::ParseInput`) in one long-lived `-NoProfile` process, resolves
+  each command name through `Get-Alias`, and hands the real names to the
+  existing tiers; resolving a name applies the existing policy rather than
+  inventing one, so `ri` is destructive because `Remove-Item` always was.
+  346 ms to start, 0.12 ms a parse after that, cached, and the helper exits
+  on EOF when the session does.
+
+  Measured before merging, over every command Hex CLI has actually been
+  asked to run — 353 commands, 119 distinct, from the owner's sessions and
+  every recorded eval run: **0 change tier**. The change cannot alter
+  behaviour on observed traffic; its whole effect is the four bypasses
+  above. Offline suites green, smoke 10/10 on a fresh server.
+
 ## 2.12.0 — 2026-09-17
 
 Ten changes on one theme: the harness now checks that a turn did the kind of

{hexcli-2.12.0 → hexcli-2.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hexcli
-Version: 2.12.0
+Version: 2.14.0
 Summary: Local Hexagon NPU terminal agent for Snapdragon X Elite Windows ARM64
 Project-URL: Homepage, https://github.com/NathanL15/Hex-CLI
 Project-URL: Repository, https://github.com/NathanL15/Hex-CLI

{hexcli-2.12.0 → hexcli-2.14.0}/hexcli/__init__.py

@@ -3,4 +3,4 @@
 # The one place the version is written. pyproject.toml reads it (hatch
 # dynamic version), agent.VERSION re-exports it, and CI refuses a release
 # tag that does not match it.
-__version__ = "2.12.0"
+__version__ = "2.14.0"

{hexcli-2.12.0 → hexcli-2.14.0}/hexcli/agent.py

@@ -1741,8 +1741,11 @@ def _retry_echo(raw: str) -> str:
 # none of the four can fire on it. evals/test_agent_loop.py pins that.
 #
 # Every guard below is here because an earlier draft fired on something
-# that was already RIGHT. They were found by replaying 306 real chat-log
-# turns and 1,366 recorded eval runs against the detectors:
+# that was already RIGHT. They were found by replaying the real chat-log
+# turns and 1,366 recorded eval runs against the detectors (the count
+# first written here, 306 turns, included 93 that a test suite had
+# written into the real log with the mock backend; the owner's own
+# sessions are 196 turns, re-counted 2026-09-17):
 #
 #   * "the condition is checked before the loop" — prose about code the
 #     user pasted, in an answer that never touches the filesystem (two real
@@ -1763,9 +1766,10 @@ def _retry_echo(raw: str) -> str:
 #     because a denied call is not in tools_used. Telling them apart needs
 #     the denial recorded, which is a separate change.
 #
-# After the guards, 11 of the 306 real turns fire and every one is a
-# genuine miss; 1 of the 1,366 eval runs fires, a self-correct-1 run that
-# claimed to have checked and fixed with no tool call at all.
+# After the guards, 9 of the 196 real turns fire and every one is a
+# genuine miss (the calculator, web-app, HiLo and resume sessions); 1 of
+# the 1,366 eval runs fires, a self-correct-1 run that claimed to have
+# checked and fixed with no tool call at all.
 
 _WANTS_RUN_RE = re.compile(
     r"\b(and|then|,)\s+(run|execute|start|launch)\s+(it|them|the\s+\w+)\b"

{hexcli-2.12.0 → hexcli-2.14.0}/hexcli/editing.py

@@ -142,6 +142,7 @@ def unescape_json(text: str) -> str:
 
 _DELTA_TOKEN_RE = re.compile(r"\w+|\s+|[^\w\s]")
 _ESCAPED_RE = re.compile(r'\\["nt\\]')
+_BARE_QUOTE_RE = re.compile(r'(?<!\\)"')
 _DELTA_MAX_HUNKS = 4
 _DELTA_MAX_REPLACED_CHARS = 40
 _DELTA_MAX_INSERTED_CHARS = 400
@@ -149,12 +150,24 @@ _DELTA_MIN_MATCHED = 0.6   # fraction of old_string's tokens found in the window
 
 
 def looks_double_escaped(text: str) -> bool:
-    """A file body with two or more literal backslash-n sequences and not a
-    single real line break is JSON escaped a second time (write_file,
-    persona_guard 2026-09-12: 'import re\\n\\n\\n# Regex …' written as one
-    line). A real one-line file that spells "\\n" on purpose is rarer than the
-    model's habit; anything with a genuine newline is left alone."""
-    return "\n" not in text and text.count("\\n") >= 2
+    """A file body JSON-escaped a second time, in either of the two shapes
+    the model actually produces.
+
+    Newlines: two or more literal backslash-n sequences and not a single
+    real line break (write_file, persona_guard 2026-09-12: 'import
+    re\\n\\n\\n# Regex …' written as one line). A real one-line file that
+    spells "\\n" on purpose is rarer than the model's habit; anything with a
+    genuine newline is left alone.
+
+    Quotes: at least one backslash-quote and not a single bare quote
+    (runit-1, 2026-09-17: `print(\\"Hello, world\\")` as the whole of
+    hello.py, a SyntaxError the model then "fixed" by editing the line to
+    itself until the step limit). Over the 511 write_file bodies on record
+    11 have this shape and every one is that defect; none has both escaped
+    and bare quotes, so a body with even one bare quote is left alone."""
+    if "\n" not in text and text.count("\\n") >= 2:
+        return True
+    return '\\"' in text and _BARE_QUOTE_RE.search(text) is None
 
 
 def _unescape_json(text: str) -> str:

hexcli-2.14.0/hexcli/psparse.py

@@ -0,0 +1,227 @@
+#!/usr/bin/env python3
+"""hexcli.psparse — structural facts about a PowerShell command line.
+
+The safety classifier matched patterns against raw text, which fails in both
+directions (measured 2026-09-15): `ri C:\\data` and `clc notes.txt` ran with no
+confirmation because the alias never appears in a pattern, while
+`Write-Output 'this will erase nothing'` was called destructive because the
+word sat inside a string. Both need to know which text is a command name and
+which is an inert argument, and that needs a parser for the language.
+
+PowerShell has one built in: `[Parser]::ParseInput`. This module runs a small
+helper script inside ONE long-lived `pwsh -NoProfile` process, sends it a
+command per line and reads back a JSON summary. The process starts lazily on
+first use and is reused for the session; results are cached by command string,
+so a repeated command costs nothing. Everything degrades to "unavailable",
+and the caller then keeps its pattern verdict unchanged.
+
+The agent runs commands with `-NoProfile`, so the only aliases that can exist
+are PowerShell's built-ins; the helper resolves them from the live session's
+own alias table rather than a table copied into Python.
+"""
+from __future__ import annotations
+
+import base64
+import json
+import os
+import queue
+import shutil
+import subprocess
+import threading
+from dataclasses import dataclass, field
+
+# One command in, one JSON line out. Base64 keeps newlines and quoting out of
+# the protocol. `Ping` is answered before anything is parsed, so the caller can
+# tell "the helper is up" from "the helper is wedged".
+_HELPER = r"""
+$ErrorActionPreference = 'Stop'
+$aliases = @{}
+foreach ($a in Get-Alias) {
+    # Definition, not ResolvedCommandName: the latter is null for most aliases
+    # unless the target module is already loaded (measured 2026-09-15: 78 of
+    # 135 came back empty and the table was useless).
+    $d = $a.Definition
+    if ($d) { $aliases[$a.Name.ToLowerInvariant()] = $d.ToLowerInvariant() }
+}
+Write-Output 'READY'
+while ($null -ne ($line = [Console]::In.ReadLine())) {
+    try {
+        $cmd = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($line))
+        $errors = $null
+        $ast = [System.Management.Automation.Language.Parser]::ParseInput($cmd, [ref]$null, [ref]$errors)
+        $names = New-Object System.Collections.ArrayList
+        $nonliteral = $false
+        foreach ($c in $ast.FindAll({ param($n) $n -is [System.Management.Automation.Language.CommandAst] }, $true)) {
+            $n = $c.GetCommandName()
+            if ($null -eq $n) { $nonliteral = $true; continue }
+            $n = $n.ToLowerInvariant()
+            if ($aliases.ContainsKey($n)) { $n = $aliases[$n] }
+            [void]$names.Add($n)
+        }
+        $strings = New-Object System.Collections.ArrayList
+        foreach ($s in $ast.FindAll({ param($n) $n -is [System.Management.Automation.Language.StringConstantExpressionAst] }, $true)) {
+            $p = $s.Parent
+            $isName = ($p -is [System.Management.Automation.Language.CommandAst]) -and ($p.CommandElements.Count -gt 0) -and ($p.CommandElements[0] -eq $s)
+            if (-not $isName) {
+                [void]$strings.Add(@($s.Extent.StartOffset, $s.Extent.EndOffset))
+            }
+        }
+        $out = @{ ok = $true; names = @($names); nonliteral = $nonliteral; strings = @($strings) }
+    } catch {
+        $out = @{ ok = $false; names = @(); nonliteral = $false; strings = @() }
+    }
+    Write-Output ($out | ConvertTo-Json -Compress -Depth 4)
+}
+"""
+
+_START_TIMEOUT_S = 10.0
+_PARSE_TIMEOUT_S = 5.0
+_CACHE_MAX = 512
+
+
+@dataclass
+class Facts:
+    """What the parser saw. `ok` False means nothing was learned and the
+    caller must keep whatever verdict it already had."""
+    ok: bool = False
+    names: tuple[str, ...] = ()          # canonical command names, aliases resolved
+    nonliteral: bool = False             # a command whose name is not a literal
+    strings: tuple[tuple[int, int], ...] = field(default=())   # inert string-literal extents
+
+
+UNAVAILABLE = Facts()
+
+_lock = threading.Lock()
+_proc: subprocess.Popen[str] | None = None
+_replies: queue.Queue[str] | None = None
+_disabled = False
+_cache: dict[str, Facts] = {}
+
+
+def _reader(proc: subprocess.Popen[str], q: queue.Queue[str]) -> None:
+    assert proc.stdout is not None
+    for line in iter(proc.stdout.readline, ""):
+        q.put(line.strip())
+    q.put("")
+
+
+def _shell_exe() -> str | None:
+    for candidate in ("pwsh.exe", "powershell.exe", "pwsh"):
+        found = shutil.which(candidate)
+        if found:
+            return found
+    return None
+
+
+def _start() -> bool:
+    """Bring the helper up. False (and disabled for good) when it cannot be."""
+    global _proc, _replies, _disabled
+    if _disabled:
+        return False
+    if _proc is not None and _proc.poll() is None:
+        return True
+    exe = _shell_exe()
+    if exe is None:
+        _disabled = True
+        return False
+    try:
+        env = dict(os.environ, POWERSHELL_TELEMETRY_OPTOUT="1")
+        proc = subprocess.Popen(
+            [exe, "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", "-"],
+            stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
+            text=True, encoding="utf-8", errors="replace", env=env,
+            creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0),
+        )
+        q: queue.Queue[str] = queue.Queue()
+        threading.Thread(target=_reader, args=(proc, q), daemon=True).start()
+        assert proc.stdin is not None
+        proc.stdin.write(_HELPER + "\n")
+        proc.stdin.flush()
+        if q.get(timeout=_START_TIMEOUT_S) != "READY":
+            raise RuntimeError("helper did not announce itself")
+    except Exception:  # noqa: BLE001 — no parser is a supported state, never an error
+        _disabled = True
+        _proc = None
+        return False
+    _proc, _replies = proc, q
+    return True
+
+
+def _ask(cmd: str) -> Facts:
+    global _proc, _disabled
+    if not _start():
+        return UNAVAILABLE
+    assert _proc is not None and _proc.stdin is not None and _replies is not None
+    try:
+        payload = base64.b64encode(cmd.encode("utf-8")).decode("ascii")
+        _proc.stdin.write(payload + "\n")
+        _proc.stdin.flush()
+        line = _replies.get(timeout=_PARSE_TIMEOUT_S)
+        if not line:
+            raise RuntimeError("helper closed")
+        data = json.loads(line)
+    except Exception:  # noqa: BLE001 — a wedged helper must not wedge the agent
+        try:
+            if _proc is not None:
+                _proc.kill()
+        except Exception:  # noqa: BLE001
+            pass
+        _proc = None
+        _disabled = True      # one failure is enough; the patterns still hold
+        return UNAVAILABLE
+    if not data.get("ok"):
+        return UNAVAILABLE
+    strings = tuple((int(a), int(b)) for a, b in (data.get("strings") or []))
+    return Facts(ok=True,
+                 names=tuple(str(n) for n in (data.get("names") or [])),
+                 nonliteral=bool(data.get("nonliteral")),
+                 strings=strings)
+
+
+def facts(cmd: str) -> Facts:
+    """Structural facts about `cmd`, or UNAVAILABLE. Cached by command string:
+    the parse costs a round trip once per distinct command, and an agent that
+    retries the same call pays it once."""
+    key = cmd.strip()
+    if not key:
+        return UNAVAILABLE
+    with _lock:
+        hit = _cache.get(key)
+        if hit is not None:
+            return hit
+        got = _ask(key)
+        if len(_cache) >= _CACHE_MAX:
+            _cache.clear()
+        _cache[key] = got
+        return got
+
+
+def shutdown() -> None:
+    """Stop the helper. Tests call this; production does not need to.
+
+    The helper blocks on OUR stdin pipe, so when this process dies -- exit,
+    crash or taskkill alike -- the OS closes the write end, the helper reads
+    EOF and leaves. Verified 2026-09-17: a session that starts the helper and
+    exits without calling this leaves no PowerShell behind. Do not "fix" that
+    with an atexit hook; there is nothing to clean up, and an exit hook that
+    waits on a pipe is a way to hang a shutdown, not to speed one."""
+    global _proc
+    with _lock:
+        proc, _proc = _proc, None
+        _cache.clear()
+    if proc is not None:
+        try:
+            if proc.stdin is not None:
+                proc.stdin.close()
+            proc.wait(timeout=2)
+        except Exception:  # noqa: BLE001
+            try:
+                proc.kill()
+            except Exception:  # noqa: BLE001
+                pass
+
+
+def _reset_for_tests() -> None:
+    global _disabled
+    shutdown()
+    _disabled = False

{hexcli-2.12.0 → hexcli-2.14.0}/hexcli/safety.py

@@ -7,6 +7,8 @@ import re
 from datetime import UTC, datetime
 from pathlib import Path
 
+from . import psparse
+
 # Patterns checked in order: first match wins.  Destructive > safe > caution.
 
 _DESTRUCTIVE: list[re.Pattern[str]] = [re.compile(p, re.IGNORECASE) for p in [
@@ -84,13 +86,19 @@ _SAFE: list[re.Pattern[str]] = [re.compile(p, re.IGNORECASE) for p in [
 ]]
 
 
-def classify_command(cmd: str) -> str:
-    """Return 'safe', 'caution', 'sensitive', or 'destructive'.
+_SEVERITY = {"safe": 0, "caution": 1, "sensitive": 2, "destructive": 3}
 
-    Priority: destructive > sensitive > safe > caution. Sensitive must outrank
-    the safe list, or read-only cmdlet prefixes whitelist credential access.
-    """
-    s = cmd.strip()
+
+def _worse(a: str, b: str) -> str:
+    """The more severe of two verdicts. An unknown or empty verdict ranks
+    below every real one, so "nothing to say about this name" can never
+    displace `safe` — it did while the default was `caution`'s rank, which
+    turned `Get-ChildItem` into an empty classification."""
+    return a if _SEVERITY.get(a, -1) >= _SEVERITY.get(b, -1) else b
+
+
+def _classify_text(s: str) -> str:
+    """The pattern tiers, matched against the command as written."""
     for pat in _DESTRUCTIVE:
         if pat.search(s):
             return "destructive"
@@ -103,6 +111,52 @@ def classify_command(cmd: str) -> str:
     return "caution"
 
 
+def _tier_for_name(name: str) -> str:
+    """What the tiers would say about a bare command name. Deriving this from
+    the same patterns is deliberate: resolving an alias must not invent a
+    policy, only apply the existing one to the name the shell will really
+    run. `ri` becomes `remove-item` and is destructive because
+    `Remove-Item` always was; `clc` becomes `clear-content` and stays
+    caution because `Clear-Content` is caution today."""
+    probe = name + " "
+    for pat in _DESTRUCTIVE:
+        if pat.search(probe):
+            return "destructive"
+    for pat in _SENSITIVE:
+        if pat.search(probe):
+            return "sensitive"
+    return ""
+
+
+def classify_command(cmd: str) -> str:
+    """Return 'safe', 'caution', 'sensitive', or 'destructive'.
+
+    Priority: destructive > sensitive > safe > caution. Sensitive must outrank
+    the safe list, or read-only cmdlet prefixes whitelist credential access.
+
+    Text alone is not enough. Measured 2026-09-15: `ri C:\\data`,
+    `rmdir C:\\data` and `$c='Remove-Item'; & ($a+$b)` all ran with no
+    confirmation, because an alias or a computed name never appears in a
+    pattern. `hexcli.psparse` resolves the command names the shell will
+    actually run (the agent's shell is `-NoProfile`, so only built-in
+    aliases exist) and reports a name that is not a literal at all. The
+    result can only ever be MORE severe than the text verdict: when the
+    parser is unavailable this is exactly the behaviour it always had.
+    """
+    s = cmd.strip()
+    verdict = _classify_text(s)
+    facts = psparse.facts(s)
+    if not facts.ok:
+        return verdict
+    for name in facts.names:
+        verdict = _worse(verdict, _tier_for_name(name))
+    if facts.nonliteral:
+        # `& ($a + $b)` — the name is computed, so nothing can be checked
+        # about it before it runs. Confirm rather than assume.
+        verdict = _worse(verdict, "sensitive")
+    return verdict
+
+
 def append_audit_log(
     session_id: str | None,
     classification: str,