hexcli 2.13.0__tar.gz → 2.15.0__tar.gz

{hexcli-2.13.0 → hexcli-2.15.0}/CHANGELOG.md

@@ -4,6 +4,80 @@ Full evidence for every claim below — including the experiments that failed
 lives in `docs/V2_PLAN.md` §14. Numbers are pass^k over repeated live runs on
 the Hexagon NPU, not single-run anecdotes.
 
+## 2.15.0 — 2026-09-18
+
+A minor release: one tool error the model reads is new (`edit_file` on
+identical strings).
+
+> **Gate: PASS** on the pinned 24-case set after one recheck (`agentic-5`
+> 4/5, a counting answer with no edit in it, then 6/6). Own arm, fresh
+> server, seed 20260917, 9 invalid of 245 runs. Against the v2.11.1 12-run
+> baseline: run-level 402/514 vs 172/221, −0.4 %, Fisher p = 0.92; pass^k
+> 26 → 28 of 46 shared cases, McNemar p = 0.63. The error fired in the arm
+> on 3 `tests-claim-1` runs and nowhere else; that case went 4/5 against
+> 12/24 (p = 0.34, ceiling). `self-correct-1` read 2/5 in the arm on a
+> server 75 minutes into its traffic (the arm's invalid runs were climbing),
+> and the error had not fired in any of its runs; a paired control on a
+> fresh server, ten runs each on the parent tree and this one, came back
+> **10/10 vs 9/10, p = 1.0**. In the one control run where the error did
+> fire the model had sent a hallucinated `main()` as both strings, was told
+> nothing changed, read the file, and then made the real fix. Smoke miss
+> on `agentic-2` re-measured 9/10 against its recorded 75 %.
+
+- `edit_file` refuses an edit whose `old_string` and `new_string` are the
+  same text, instead of answering "Edited". 37 of the 840 `edit_file` calls
+  on record sent identical strings and were told the edit landed; the model
+  then ran or read the file believing it had changed it (`tests-claim-1`:
+  "The median function was correctly fixed", after an edit of `mean` to
+  itself; `runit-1` before 2.14.0: the broken line edited to itself, twice,
+  to the step limit). The tool now says nothing changed and what to send
+  instead, and leaves the file byte-identical. A missing file still reports
+  the path first.
+
+## 2.14.0 — 2026-09-17
+
+A minor release: one tool behaves differently for the model (a `write_file`
+body that was JSON-escaped twice is decoded), plus a test-hygiene fix.
+
+> **Gate: PASS** on the pinned 24-case set, all 24 held at 5 runs with no
+> recheck. Own arm, fresh server, seed 20260917, 0 invalid of 245 runs.
+> Against the v2.11.1 12-run baseline: run-level 402/514 vs 179/230, −0.4 %,
+> Fisher p = 0.92; pass^k 26 → 30 of 46 shared cases, McNemar p = 0.22. The
+> rule fired in the arm on 3 of 5 `runit-1` writes and on nothing else
+> (one `trap-1` poem hit the older newline shape, as before). `runit-1`:
+> 5/5 in the arm and 15/15 in a follow-up on the same server, **20/20
+> against 5/10 on 2.12.0, p = 0.002**. Ceiling cases that moved: `agentic-3`
+> 3/5 (p = 0.17 against 18/20; both misses are the model writing invalid
+> JSON through `edit_file`, where this rule does not run), `ambiguous-1`
+> 4/5 (p = 0.50). Smoke 10/10 on a fresh server before the arm.
+
+- `write_file` decodes a body whose every quote is escaped. `runit-1`
+  ("Write hello.py that prints Hello, world and then run it") sat at 5/10
+  in the 2.12.0 arm, and not one of the misses was the "and run it" the
+  case was written for: every run wrote the file and ran it. The model
+  escaped its JSON twice, `{"content":"print(\\\"Hello, world\\\")"}`, so
+  the file held `print(\"Hello, world\")`, a SyntaxError; it then "fixed"
+  the line by editing it to itself, twice, until the step limit. The
+  double-escape rule from 2.9.0 only knew the newline shape (literal `\n`
+  and no real line break). It now also knows the quote shape — at least one
+  `\"` and not a single bare `"` — and decodes the body once, as before.
+  Measured over the 511 `write_file` bodies on record: 11 have the shape,
+  all of them this defect in this case, and none has both escaped and bare
+  quotes, so a body with even one bare quote is left exactly as sent.
+  `append_file` is untouched: 0 of its 17 recorded bodies have either shape.
+
+- A test suite was writing into the owner's real chat log. The shell wiring
+  test in `evals/test_product_shell.py` drove `hexcli.agent` with the mock
+  backend and a temp config that left `chat_log_enabled` on, so every run
+  of the suite added a "say hello" session under `~/.shellai/chatlog/` — 93
+  of them by 2026-09-17, beside the 127 real sessions. The 2.12.0 exposure
+  counts were taken over that mix: "306 real turns" was 196 real turns and
+  93 scripted ones (the 11 firings were all on real turns, so the true
+  positives stand; the denominator did not). The test now turns the log
+  off, the corpus loader skips mock-backend sessions, the mock files were
+  moved aside, and the intent nudge's recorded rate is corrected to 9 of
+  196 real turns.
+
 ## 2.13.0 — 2026-09-17
 
 - Commands are classified by the name the shell will really run, not by the

{hexcli-2.13.0 → hexcli-2.15.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hexcli
-Version: 2.13.0
+Version: 2.15.0
 Summary: Local Hexagon NPU terminal agent for Snapdragon X Elite Windows ARM64
 Project-URL: Homepage, https://github.com/NathanL15/Hex-CLI
 Project-URL: Repository, https://github.com/NathanL15/Hex-CLI

{hexcli-2.13.0 → hexcli-2.15.0}/hexcli/__init__.py

@@ -3,4 +3,4 @@
 # The one place the version is written. pyproject.toml reads it (hatch
 # dynamic version), agent.VERSION re-exports it, and CI refuses a release
 # tag that does not match it.
-__version__ = "2.13.0"
+__version__ = "2.15.0"

{hexcli-2.13.0 → hexcli-2.15.0}/hexcli/agent.py

@@ -1741,8 +1741,11 @@ def _retry_echo(raw: str) -> str:
 # none of the four can fire on it. evals/test_agent_loop.py pins that.
 #
 # Every guard below is here because an earlier draft fired on something
-# that was already RIGHT. They were found by replaying 306 real chat-log
-# turns and 1,366 recorded eval runs against the detectors:
+# that was already RIGHT. They were found by replaying the real chat-log
+# turns and 1,366 recorded eval runs against the detectors (the count
+# first written here, 306 turns, included 93 that a test suite had
+# written into the real log with the mock backend; the owner's own
+# sessions are 196 turns, re-counted 2026-09-17):
 #
 #   * "the condition is checked before the loop" — prose about code the
 #     user pasted, in an answer that never touches the filesystem (two real
@@ -1763,9 +1766,10 @@ def _retry_echo(raw: str) -> str:
 #     because a denied call is not in tools_used. Telling them apart needs
 #     the denial recorded, which is a separate change.
 #
-# After the guards, 11 of the 306 real turns fire and every one is a
-# genuine miss; 1 of the 1,366 eval runs fires, a self-correct-1 run that
-# claimed to have checked and fixed with no tool call at all.
+# After the guards, 9 of the 196 real turns fire and every one is a
+# genuine miss (the calculator, web-app, HiLo and resume sessions); 1 of
+# the 1,366 eval runs fires, a self-correct-1 run that claimed to have
+# checked and fixed with no tool call at all.
 
 _WANTS_RUN_RE = re.compile(
     r"\b(and|then|,)\s+(run|execute|start|launch)\s+(it|them|the\s+\w+)\b"

{hexcli-2.13.0 → hexcli-2.15.0}/hexcli/editing.py

@@ -142,6 +142,7 @@ def unescape_json(text: str) -> str:
 
 _DELTA_TOKEN_RE = re.compile(r"\w+|\s+|[^\w\s]")
 _ESCAPED_RE = re.compile(r'\\["nt\\]')
+_BARE_QUOTE_RE = re.compile(r'(?<!\\)"')
 _DELTA_MAX_HUNKS = 4
 _DELTA_MAX_REPLACED_CHARS = 40
 _DELTA_MAX_INSERTED_CHARS = 400
@@ -149,12 +150,24 @@ _DELTA_MIN_MATCHED = 0.6   # fraction of old_string's tokens found in the window
 
 
 def looks_double_escaped(text: str) -> bool:
-    """A file body with two or more literal backslash-n sequences and not a
-    single real line break is JSON escaped a second time (write_file,
-    persona_guard 2026-09-12: 'import re\\n\\n\\n# Regex …' written as one
-    line). A real one-line file that spells "\\n" on purpose is rarer than the
-    model's habit; anything with a genuine newline is left alone."""
-    return "\n" not in text and text.count("\\n") >= 2
+    """A file body JSON-escaped a second time, in either of the two shapes
+    the model actually produces.
+
+    Newlines: two or more literal backslash-n sequences and not a single
+    real line break (write_file, persona_guard 2026-09-12: 'import
+    re\\n\\n\\n# Regex …' written as one line). A real one-line file that
+    spells "\\n" on purpose is rarer than the model's habit; anything with a
+    genuine newline is left alone.
+
+    Quotes: at least one backslash-quote and not a single bare quote
+    (runit-1, 2026-09-17: `print(\\"Hello, world\\")` as the whole of
+    hello.py, a SyntaxError the model then "fixed" by editing the line to
+    itself until the step limit). Over the 511 write_file bodies on record
+    11 have this shape and every one is that defect; none has both escaped
+    and bare quotes, so a body with even one bare quote is left alone."""
+    if "\n" not in text and text.count("\\n") >= 2:
+        return True
+    return '\\"' in text and _BARE_QUOTE_RE.search(text) is None
 
 
 def _unescape_json(text: str) -> str:

{hexcli-2.13.0 → hexcli-2.15.0}/hexcli/tools.py

@@ -423,6 +423,17 @@ def edit_file_tool(path_text: str, old_string: str, new_string: str) -> str:
         raise RuntimeError("edit_file requires a non-empty 'old_string'. Use write_file to overwrite the whole file.")
     if not path.exists():
         raise RuntimeError(f"File not found: {path}.{missing_path_hint(path)}")
+    if old_string == new_string:
+        # 37 of the 840 edit_file calls on record (2026-09-17) sent the same
+        # text as old and new and were told "Edited"; the model then ran or
+        # read the file believing it had changed it (tests-claim-1: "The
+        # median function was correctly fixed", after an edit of mean to
+        # itself). A no-op is not an edit, and saying so is the only way the
+        # next step can be the change that was meant.
+        raise RuntimeError(
+            f"old_string and new_string are identical, so nothing changed in {path}. "
+            "Put the changed text in new_string, or read_file to see what is there now."
+        )
     content = path.read_text(encoding="utf-8")
     tier = "exact"
     if content.count(old_string) == 1: