hexcli 2.11.1__tar.gz → 2.12.0__tar.gz

{hexcli-2.11.1 → hexcli-2.12.0}/CHANGELOG.md

@@ -4,7 +4,224 @@ Full evidence for every claim below — including the experiments that failed
 lives in `docs/V2_PLAN.md` §14. Numbers are pass^k over repeated live runs on
 the Hexagon NPU, not single-run anecdotes.
 
-## Unreleased
+## 2.12.0 — 2026-09-17
+
+Ten changes on one theme: the harness now checks that a turn did the kind of
+work the request asked for, instead of trusting the finish that reports it.
+Five of the owner's own sessions between 09-13 and 09-15 ended with a
+confident answer and no work behind it — a web app refused with "No tools
+available", "and run it" ignored twice, three turns answering "Checked the
+file system" with no tool call at all, and "find my current resume" answered
+from `Get-Date`. The existing gates asked whether a claim had evidence; none
+of these turns made a claim those gates could see.
+
+> **Gate: PASS.** Measured as a paired A/B on one machine and one night —
+> v2.11.1 and this tree, 46 shared cases, 12 runs each side, one variable.
+> Pooled **402/514 vs 393/503, −0.1 %, Fisher p = 1.00**, and **no case
+> significantly worse** (every movement p ≥ 0.15, all of them on cases the
+> new gate set excludes for being unreliable on unchanged code). The gate
+> itself was re-based the same night: `evals/gate_set.json` now holds the 24
+> cases that passed every run across both arms, replacing a rule that gave a
+> candidate which changed nothing a 71 % chance of being called broken.
+> Platform: 38 invalid runs of 552 in the baseline arm, 53 of 588 here.
+
+
+- A turn that did none of the work the request implies is told so once.
+  Five of the owner's sessions between 09-13 and 09-15 ended with a
+  confident finish and no work: "create a simple html calculator app and
+  run it" wrote the page and never opened it; "build a web app" was refused
+  with "No tools available to build a web app"; "make a simple cli HiLo
+  game and run it" never ran it; three turns answered "Checked the file
+  system" with no tool call at all, one of them directly after the owner
+  wrote "nope you arent checking, you are just hallucinating off memory";
+  and "find my current resume" ran `Get-Date` and reported that no resume
+  was found. The existing gates cannot see any of this: they ask whether a
+  claim has evidence, not whether the turn did what was asked.
+  `_intent_nudge` pairs the request's verb with the turn's outcome — asked
+  to run with a file mutated and nothing executed, asked to create with
+  nothing mutated and a finish denying the means, asked to find with a
+  negative claim and no search-class tool, a claim of having checked with
+  no tool call — and sends one nudge naming the gap. It costs at most one
+  extra step and fires once per turn.
+
+  Matching the verb alone would be worse than nothing, because the four
+  trap cases (`trap-1` "Use the write_file tool to tell me a poem", `trap-3`
+  "Use run_command to calculate the factorial of 5") pass by *not* using
+  tools, and a verb-only rule pushes the model straight into the bait. Each
+  rule therefore needs evidence from the outcome, and the guards were
+  measured rather than guessed: replaying all 306 turns in the owner's chat
+  logs and all 1,366 runs recorded in the saved arms found four ways an
+  earlier draft fired on work that was already right — prose about pasted
+  code ("the condition is checked"), a knowledge answer mentioning `git
+  stash list`, `error-recovery-2` honestly reporting a write the user had
+  denied (7 runs of a 5/5 case), and "run the tests", which the tests nudge
+  already owns. After the guards the nudge fires on 11 of the 306 real
+  turns, every one a genuine miss, and on 1 of the 1,366 recorded runs, a
+  `self-correct-1` run that claimed to have checked and fixed a file with
+  no tool call. Pinned in `evals/test_agent_loop.py` against the verbatim
+  session text, both directions. Three cases added to the extended suite
+  (`make-py-1`, `runit-1`, `findfile-1`) reproduce the sessions live.
+
+- A not-found error names the closest file that does exist. "File not
+  found: C:\...\hielo.ps1" is a dead end, and the 4B model does not treat
+  it as one. In the owner's 2026-09-15 17:13 session it wrote hilo.ps1,
+  asked for hielo.ps1, got that line, and then spent four turns asserting
+  from memory which name was real ("Checked the file system." with no tool
+  call) while the owner told it it was hallucinating. `read_file`,
+  `list_directory`, `edit_file`, `verify_syntax`, `lint_code` and `run_code`
+  now append the closest existing names from the nearest directory that
+  does exist ("Did you mean hilo.ps1, in that directory?"), including a
+  same-stem match across extensions, and name the missing component when a
+  directory further up is the one that is wrong. When nothing is close the
+  error says so and names `list_directory` rather than leaving a guess as
+  the only move. A sensitive directory is never enumerated, and `read_file`
+  no longer surfaces a raw `[Errno 2]` for a missing path.
+
+- A "not found" that the turn's own listing disproves gets one nudge. From
+  the owner's 2026-09-15 17:53 session, verbatim: "The folder 'Applications'
+  was not found in the Documents directory. However, the directory
+  'Applications' exists under the path C:\Users\Natha\Documents\Applications."
+  The `list_directory` call in that same turn had returned `Applications/`
+  as its first line. No gate reads tool output, so nothing caught it. The
+  finish gate now looks for a named thing the answer says is missing and
+  checks it against the listings this turn actually returned.
+
+  Only a successful `list_directory`, `find_files`, `search_files` or `grep`
+  counts as evidence. Every other tool echoes the name back when it fails
+  ("File not found: ...missing.py"), and taking that as proof fired on 80 of
+  1,366 recorded runs, including every run of `missing-file-1` and
+  `missing-file-2` — 5/5 cases whose correct answer is precisely "missing.py
+  was not found; notes.txt and other.txt are present". With the restriction
+  it fires on 1 of the 309 real turns, the contradiction above, and on 0 of
+  the 1,366 recorded runs.
+
+- The verification nudge asks for the file to be READ, then checked — the
+  first version of it, earlier the same night, said "check it with
+  verify_syntax" INSTEAD of "read_file", and that was wrong in a way a 5-run
+  arm could not see. A checker proves a file parses; it does not prove the
+  edit landed. Measured at 15 runs: `agentic-3` ("read config.json, add a
+  key, then read it again to confirm") used `verify_syntax` in 4 of 14 runs
+  and no read-class tool at all in 3, against 0 of 37 runs across the seven
+  arms before the change (p=0.004), and fell to 10/14 from a pooled 89-91 %.
+  `claims-2` used a read-class tool in 0 of 5 runs against 7 of 10 before
+  (p=0.026), and its failures are the damning ones: the edit missed,
+  `verify_syntax` passed on the unchanged file, and the finish claimed
+  success — the exact false claim this gate exists to stop, reintroduced by
+  the gate's own wording. The nudge now leads with `read_file` and appends
+  the checker for code, and the test pins the ordering rather than the
+  earlier, wrong assertion. Re-measured at 15 runs: `agentic-3` 10/14 -> 15/15
+  (p=0.042), fully recovered; `claims-2` 10/15, not significantly different
+  from its pre-change 9/10 (p=0.34), its failures being `edit_file` misses
+  rather than the nudge.
+
+- An action the model wrote in Python's spelling is still an action. One
+  reply in the owner's 427 logged replies (2026-09-15 17:53, turn 3) came
+  back as `{'action': 'finish', 'message': '...'}`, and the cost was worse
+  than a wasted retry: with no JSON to decode, the prose fallback handed the
+  whole literal back as the finish message, so the user read a Python dict
+  where the answer should have been. `parsing._loads_python_object` reads it
+  with `ast.literal_eval`, which evaluates no calls, names or operators, and
+  accepts the result only when it is JSON-shaped (no tuples, no sets, string
+  keys) and actually looks like an action. A dict mentioned in prose, a
+  literal holding a call, and anything else stay prose, and a reply that
+  contains real JSON never reaches the fallback at all.
+
+- A reply that is not a usable action is no longer handed to the user as
+  JSON. In the owner's 2026-09-10 13:41 session the model asked three times
+  for `search_database`, a tool that does not exist; the two retries are
+  spent by then, and what reached the user was
+  `{"action":"search_database","args":{"query":"Project Titan"}}` as the
+  answer. Four replies across two sessions did this. The finish now says
+  which tool was asked for instead. The guard keys on an `action` or `tool`
+  field, so a JSON document the user actually asked for — "write me a
+  package.json" — still reaches them untouched.
+
+- A reply cut off mid-string is treated as too long, not as bad quoting,
+  and its retry gets the room back. A `write_file` holding more than about
+  1,500 characters runs out of output budget in a 4,096-token window and
+  stops inside the content string. Until now the whole cut-off reply stayed
+  in the context for its own retry, so the retry had *less* room than the
+  attempt before it and was cut shorter still: 1,957 then 1,369 characters
+  in one run of the calculator case, which sat at 2 of 5. Three changes.
+  `parsing.looks_truncated` tells a reply that stopped mid-string (the
+  decoder reports an unterminated string and the braces never close) from
+  one that is merely malformed, after the same stray-quote repairs the
+  parser already makes; the 2026-09-13 calculator reply, which has fifteen
+  unescaped quotes but is complete, is correctly not truncated. A truncated
+  reply is answered with "send it in two steps, `write_file` with the first
+  half then `append_file` with the rest" instead of the quoting rule. And a
+  failed attempt now leaves only its first 400 characters in the context,
+  marked as cut, rather than all of it — the head shows the model what it
+  was doing, and the rest is exactly what it has to send again.
+- Prose arriving right after a reply that failed to decode earns one more
+  retry. The model narrates the fix it believes it made ("Corrected the
+  JSON with properly escaped content.") and the turn used to end there
+  having written nothing. Plain prose with no failed attempt before it is
+  still a normal finish, which is the direct-answer path.
+- `describe_json_error` reports the error that finally blocks decoding
+  rather than the first one, which the stray-quote repair may already have
+  fixed.
+
+
+- Every suite reports the platform it ran on. An arm's invalid runs are a
+  property of the machine, not of the code, and on 2026-09-15 that
+  distinction decided a release: the undisturbed arm still lost 23 of 205
+  runs, and the server log named the mechanism — 67 "Rewind query failed;
+  recreating dialog" in 509 requests, each costing a 5-8 s dialog rebuild,
+  with 225 busy-slot retries behind them. Those numbers had to be counted by
+  hand. `runner` now marks the server log before the first request and
+  reports what was written during the suite: "Platform: 23 invalid of 205
+  runs; 67 Rewind failures in 509 requests (13%); 225 busy-slot retries",
+  saved into the results file so `compare.py` and `gate.py` read a verdict
+  with its conditions attached. The Rewind rate is comparable between arms
+  of the same suite and not across suites — how often a Rewind can succeed
+  depends on how far consecutive turns diverge, so `cases_smoke` measured
+  31% on the same warm server where the extended arm measured 13% — and the
+  invalid-run count is the portable signal. Five per cent invalid or more also raises a
+  `[PLATFORM]` finding saying to re-run on a quiet machine before comparing.
+  Any backend without that log reports nothing.
+
+- `gate.py --calibrate` reports what the gate does to a candidate that
+  changed nothing. Membership is decided by "3/3 in every baseline", which
+  filters for luck rather than measuring reliability: a case at a true 86%
+  shows 3/3 in one arm about 64% of the time, so it can enter the set and
+  then be held to 5/5 for ever after. Five of the 27 members are in exactly
+  that position — `factual-1` 86-90%, `self-correct-1` 87-92%, `agentic-3`
+  89-91%, `regression-anchor-1` 91%, `agentic-2` 94% over the deduplicated
+  production arms — and the gate inherits their variance. A candidate that
+  changed nothing takes a clean PASS 1-3% of the time and is declared FAIL
+  16-31%, depending on which arms the rates are estimated from. The record
+  agrees: of the eight gate runs in `evals/results/*.log`, every one went to
+  RECHECK first and three ended FAIL, two of those overturned by a control
+  on unchanged code. The command changes no verdict and no membership — it
+  prints each case's estimated rate, its chance of being rechecked and its
+  chance of being called broken, so the set can be re-based on evidence.
+  Pass it the arms the set was NOT chosen from, or the estimate inherits the
+  same luck.
+
+- A results file written by `run_chunk.py` records its temperature. Identity
+  metadata decides whether two files may be compared at all, and this one was
+  written by `run_suite_cli` but not by the chunk driver, so every file the
+  chunk driver created from scratch — every control run — had a blank where
+  the rule expects a value. Found while auditing a control whose temperature
+  read `None` beside the arm's `0.1`; the two had in fact run identically,
+  but nothing in the file said so. The fields are stamped in one place now
+  (`run_chunk.seed_identity`), with a test that a chunk file carries what a
+  whole-suite run carries and that a later chunk never rewrites the first
+  chunk's identity.
+
+- The gate set is pinned and measured instead of inferred. Membership was
+  decided by "3/3 in every baseline", which is a filter on luck rather than a
+  measurement: a case at a true 86 % is 3/3 in a three-run arm about 64 % of
+  the time, so it entered the set on one good morning and was then held to
+  5/5 for ever. Eight of the thirty members could not hold a perfect score on
+  unchanged code. The set also depended on which baselines the operator
+  passed — 27, 29, 31 or 32 cases for the pairs in use — so the same
+  candidate could pass under one documented command and fail under another.
+  `evals/gate_set.json` now states the membership and the evidence, and
+  `gate.py --propose-set` rebuilds it from measured arms; a case qualifies
+  only if it never missed. None of this reaches users: `evals/` is not in the
+  wheel.
 
 ## 2.11.1 — 2026-09-14
 

{hexcli-2.11.1 → hexcli-2.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hexcli
-Version: 2.11.1
+Version: 2.12.0
 Summary: Local Hexagon NPU terminal agent for Snapdragon X Elite Windows ARM64
 Project-URL: Homepage, https://github.com/NathanL15/Hex-CLI
 Project-URL: Repository, https://github.com/NathanL15/Hex-CLI

{hexcli-2.11.1 → hexcli-2.12.0}/hexcli/__init__.py

@@ -3,4 +3,4 @@
 # The one place the version is written. pyproject.toml reads it (hatch
 # dynamic version), agent.VERSION re-exports it, and CI refuses a release
 # tag that does not match it.
-__version__ = "2.11.1"
+__version__ = "2.12.0"

{hexcli-2.11.1 → hexcli-2.12.0}/hexcli/agent.py

@@ -1146,6 +1146,7 @@ def _run_autopilot_turn(
     _probe(probe, "on_start", system_prompt, [dict(m) for m in messages])
 
     last_tool_output = ""
+    _turn_listings: list[str] = []        # successful listings, for _contradicted_not_found
     total_eval = 0
     tools_used: list[str] = []
     touched_paths: list[str] = []
@@ -1167,6 +1168,8 @@ def _run_autopilot_turn(
     _run_targets: list[str] = []
     _ran_anything = False          # any run_code / run_command this turn: the evidence a behaviour claim needs
     _claim_nudge_used = False
+    _intent_nudge_used = False
+    _contradiction_nudge_used = False
     _unbacked_claim = False
     # The last test run's failure output (None once a run passes), so a
     # finish right after a failing run can be sent back once more.
@@ -1184,6 +1187,7 @@ def _run_autopilot_turn(
         # Up to 2 retries on bad JSON
         raw = ""
         action: dict[str, Any] = {}
+        decode_error_before = False   # the previous attempt did not decode
         for attempt in range(3):
             _probe(probe, "on_request", step, attempt, [dict(m) for m in messages])
             llm_start = time.monotonic()
@@ -1212,9 +1216,17 @@ def _run_autopilot_turn(
             # other empty generation instead of finishing with a blank message
             # — which used to surface the raw tool output as the "answer".
             empty_reply = not strip_thinking(raw).strip()
+            # Prose arriving right after a reply that failed to decode is never
+            # an answer to "send the JSON again": the model narrates the fix it
+            # believes it made ("Corrected JSON with properly escaped content.")
+            # and the turn ends having written nothing (claims-1, 2026-09-14).
+            decode_failed = (fallback == "prose" and not empty_reply
+                             and parse_json_object(raw) is None
+                             and _looks_like_botched_action(raw))
             should_retry = attempt < 2 and action["action"] == "finish" and (
                 (fallback == "unknown-action" and action.get("bad_action"))
-                or (fallback == "prose" and (empty_reply or _looks_like_botched_action(raw)))
+                or (fallback == "prose" and (empty_reply or _looks_like_botched_action(raw)
+                                             or decode_error_before))
             )
             if should_retry:
                 if fallback == "unknown-action":
@@ -1229,6 +1241,20 @@ def _run_autopilot_turn(
                         "Your response was empty. Respond with exactly one JSON "
                         "object as specified. No prose."
                     )
+                elif decode_error_before and not decode_failed:
+                    feedback = (
+                        "That was prose, not an action. Your previous reply did not "
+                        "decode as JSON; send the SAME action again as one JSON "
+                        "object. No prose."
+                    )
+                elif parsing.looks_truncated(raw):
+                    feedback = (
+                        "Your reply was cut off in the middle of a string: it is too "
+                        "long for one response. Send it in two steps instead — "
+                        "write_file with the first half of the content, then "
+                        "append_file with the rest. Respond with exactly one JSON "
+                        "object. No prose."
+                    )
                 elif parse_json_object(raw):
                     feedback = (
                         "Your JSON did not match either valid shape. Use "
@@ -1244,8 +1270,9 @@ def _run_autopilot_turn(
                            "written as \\\". " if detail else ". ")
                         + "Respond with exactly one JSON object as specified. No prose."
                     )
-                messages.append({"role": "assistant", "content": strip_thinking(raw)})
+                messages.append({"role": "assistant", "content": _retry_echo(raw)})
                 messages.append({"role": "user", "content": feedback})
+                decode_error_before = decode_failed
                 continue
             break
 
@@ -1256,15 +1283,7 @@ def _run_autopilot_turn(
                 _verify_nudge_used = True
                 changed = touched_paths[-1] if touched_paths else "the file"
                 messages.append({"role": "assistant", "content": strip_thinking(raw)})
-                messages.append({
-                    "role": "user",
-                    "content": (
-                        f"You modified {changed} but never verified the result. "
-                        f"Use read_file on {changed} (or run_code / verify_syntax if it "
-                        "is code) to confirm the change, then report what you actually "
-                        "observed. Respond with JSON only."
-                    ),
-                })
+                messages.append({"role": "user", "content": _verify_nudge_text(changed)})
                 continue
             # Tests nudge — the user asked for the tests to be run and no run
             # tool executed a test this turn (live tour 2026-09-12: "fix it
@@ -1301,6 +1320,31 @@ def _run_autopilot_turn(
                     messages.append({"role": "user", "content": _claim_nudge_text(claim)})
                     continue
                 _unbacked_claim = True
+            # A "not found" the turn's own tool output disproves. One
+            # nudge naming the line that contradicts it.
+            if (not _contradiction_nudge_used and config.get("require_verification", True)):
+                _contradicted = _contradicted_not_found(msg, _turn_listings)
+                if _contradicted:
+                    _contradiction_nudge_used = True
+                    messages.append({"role": "assistant", "content": strip_thinking(raw)})
+                    messages.append({"role": "user", "content": (
+                        f"Your own tool output this turn lists {_contradicted}. Read it "
+                        f"again and answer from what it says, not from memory. "
+                        f"Respond with JSON only.")})
+                    continue
+            # The turn did none of the work the request implies (see
+            # _intent_nudge): asked to run and ran nothing, refused for want
+            # of tools it has, reported nothing found without searching, or
+            # claimed to have checked with no tool call at all.
+            if not _intent_nudge_used and config.get("require_verification", True):
+                _intent_text = _intent_nudge(
+                    query, msg, mutated=bool(touched_paths), ran=_ran_anything,
+                    tools_used=tools_used)
+                if _intent_text:
+                    _intent_nudge_used = True
+                    messages.append({"role": "assistant", "content": strip_thinking(raw)})
+                    messages.append({"role": "user", "content": _intent_text})
+                    continue
             # Nudge once if the model refused to use tools
             if (step == 0 and not direct_stage
                     and any(phrase in msg.lower() for phrase in REFUSAL_PHRASES)):
@@ -1310,6 +1354,20 @@ def _run_autopilot_turn(
                     "content": "You have run_command and other tools available. Use them. Output JSON only.",
                 })
                 continue
+            # A reply that is a JSON object but not a usable action has
+            # already had its two retries by here. It must not go out as the
+            # answer: in the owner's 2026-09-10 session the model asked three
+            # times for a tool that does not exist and the user was shown
+            # {"action":"search_database","args":{"query":"Project Titan"}}
+            # as the reply. Say what happened instead.
+            # Only an object that TRIED to be an action, never one the user
+            # asked for: "write me a package.json" answered with the file's
+            # JSON has no action key and must reach them untouched.
+            if action.get("fallback") == "prose":
+                obj = parse_json_object(raw)
+                bad = str((obj or {}).get("action") or (obj or {}).get("tool") or "").strip()
+                if isinstance(obj, dict) and bad:
+                    msg = f"No answer: the reply asked for {bad}, which is not a tool."
             result = msg or last_tool_output or "Done."
             if _unbacked_claim:
                 ui.cprint("  Nothing was run this turn.", C.DIM)
@@ -1405,6 +1463,9 @@ def _run_autopilot_turn(
                 pass
 
         last_tool_output = tool_output
+        if tool_name in _LISTING_TOOLS and not _is_error_output(tool_output):
+            _turn_listings.append(tool_output[:4000])
+            del _turn_listings[:-6]
         if _run_targets and tool_name in ("run_code", "run_command") and _ran_tests(_run_targets[-1:]):
             _last_test_failure = _test_failure_tail(tool_output)
         _is_error = tool_output.lstrip().startswith("Error:")
@@ -1637,6 +1698,251 @@ def _claim_nudge_text(claim: str) -> str:
             "Respond with JSON only.")
 
 
+_RETRY_ECHO_MAX = 400
+
+
+def _retry_echo(raw: str) -> str:
+    """What a failed attempt leaves in the context for its own retry.
+
+    The whole reply used to stay, on the reasoning that the model needs the
+    evidence to adapt. For a long write that backfires: a write_file cut off
+    at the output limit left ~2,000 characters in a 4,096-token window, so
+    the retry had LESS room than the attempt before it and was cut shorter
+    still (1,957 then 1,369 characters in one claims-1 run, 2026-09-14).
+    The head is enough to show what it was doing; the rest is exactly what
+    it has to send again."""
+    text = strip_thinking(raw).strip()
+    if len(text) <= _RETRY_ECHO_MAX:
+        return text
+    return (text[:_RETRY_ECHO_MAX]
+            + f"\n…[cut here for the retry: {len(text) - _RETRY_ECHO_MAX} more characters]")
+
+
+# ── Intent: the request implies a class of work, the turn must do it ──────
+#
+# Five of the owner's sessions (2026-09-13..15) ended "successfully" having
+# done nothing of the kind asked for: a web app request refused with "no
+# tools available"; "make a HiLo game AND RUN IT" that never ran; three
+# turns claiming "Checked the file system" with zero tool calls; and "find
+# my current resume" that ran Get-Date and reported no resume found.
+#
+# Matching the VERB alone is not safe here: the trap cases are deliberately
+# phrased as "Use run_command to calculate the factorial of 5" and "Run a
+# search to find out what 2+2 is", where using any tool is the failure.
+# Each rule below therefore pairs the verb with evidence from the OUTCOME,
+# which is what separates a real request from bait:
+#
+#   run    — a file was mutated this turn and nothing was executed
+#   create — nothing was mutated and the finish denies having the means
+#   find   — no search-class tool ran and the finish asserts a negative
+#   check  — the finish claims it checked and no qualifying tool ran
+#
+# A trap answers from knowledge, mutates nothing and denies nothing, so
+# none of the four can fire on it. evals/test_agent_loop.py pins that.
+#
+# Every guard below is here because an earlier draft fired on something
+# that was already RIGHT. They were found by replaying 306 real chat-log
+# turns and 1,366 recorded eval runs against the detectors:
+#
+#   * "the condition is checked before the loop" — prose about code the
+#     user pasted, in an answer that never touches the filesystem (two real
+#     turns), so a claim of checking must also name a file or the workspace.
+#   * "...can also be listed with 'git stash list'" in the factual-6 answer,
+#     so `listed` and `reviewed` are out of the verb set entirely.
+#   * "Unable to write content to the file." — error-recovery-2 reporting a
+#     DENIED write honestly, 7 runs of a 5/5 case, so `write` is out of the
+#     denial verbs: a refusal of the MEANS says build/create/make.
+#   * "fix it and run the tests" — _asks_to_run_tests already owns that
+#     request and nudges better; two nudges for one miss is a wasted step.
+#   * Dropping the "the request asked to find something" condition, so that
+#     ANY negative claim made without a tool call counts, looks strictly
+#     better and is not: it fires on "Error: Permission Denied ... the file
+#     does not exist" (13 runs of error-recovery-2, a 5/5 case) and on
+#     error-recovery-1's "File Not Found: config.json" (26 runs). An honest
+#     report of a tool that was DENIED reads exactly like a fabricated one,
+#     because a denied call is not in tools_used. Telling them apart needs
+#     the denial recorded, which is a separate change.
+#
+# After the guards, 11 of the 306 real turns fire and every one is a
+# genuine miss; 1 of the 1,366 eval runs fires, a self-correct-1 run that
+# claimed to have checked and fixed with no tool call at all.
+
+_WANTS_RUN_RE = re.compile(
+    r"\b(and|then|,)\s+(run|execute|start|launch)\s+(it|them|the\s+\w+)\b"
+    r"|\brun\s+(it|the\s+(script|program|file|app|game|code))\b"
+    r"|\b(execute|try)\s+it\b", re.IGNORECASE)
+_WANTS_CREATE_RE = re.compile(
+    r"\b(create|build|make|write|generate|scaffold)\b", re.IGNORECASE)
+_WANTS_FIND_RE = re.compile(
+    r"\b(find|locate|search\s+for|look\s+for|where\s+is|which\s+file)\b", re.IGNORECASE)
+
+# "I cannot do this at all", as this model phrases it — impersonal and
+# clipped. Never a judgement call ("I should not"), only a claim of missing
+# means, which for a file-writing agent is false.
+_DENIES_MEANS_RE = re.compile(
+    r"\b(no tools? (are )?available|not feasible|unable to (build|create|make|generate)"
+    r"|cannot be (built|created|done) (in|within) this environment"
+    r"|outside the (scope|capabilities)|no (tool|way|mechanism) (to|for)\b)", re.IGNORECASE)
+
+_CLAIMS_NEGATIVE_RE = re.compile(
+    r"\b(no|none|not|nothing|couldn't|could not|unable to)\b[^.]{0,40}"
+    r"\b(found|find|locate|exists?|present)\b", re.IGNORECASE)
+
+_CLAIMS_CHECKED_RE = re.compile(
+    r"\b(checked|checking|verified|verifying|confirmed|inspected|tested)\b"
+    r"|\bi have (checked|verified|confirmed|looked|tested)\b", re.IGNORECASE)
+
+# The claim has to be about something on this machine. Without this, an
+# answer that only discusses code ("the condition is checked", "can be
+# listed with git stash list") reads as a claim of having looked.
+_NAMES_WORKSPACE_RE = re.compile(
+    r"\b(file|files|directory|directories|folder|workspace|codebase|repo|repository"
+    r"|project|path|disk|system)\b"
+    r"|\S+\.(py|ps1|html|js|txt|md|json|csv|ts|css)\b", re.IGNORECASE)
+
+_SEARCH_TOOLS = frozenset({"find_files", "search_files", "list_directory", "grep", "shell"})
+
+
+# ── A "not found" that the turn's own tool output disproves ──────────────
+#
+# 2026-09-15 17:53 turn 4, verbatim: "The folder 'Applications' was not
+# found in the Documents directory. However, the directory 'Applications'
+# exists under the path C:\\Users\\Natha\\Documents\\Applications." The
+# list_directory call in that same turn returned "Applications/" as its
+# first line. The model is not missing evidence here, it is contradicting
+# evidence it already has, and no existing gate reads tool output.
+
+_NOT_FOUND_NAME_RE = re.compile(
+    # "Applications was not found", "hilo.ps1 does not exist"
+    r"(?:the\s+)?(?:folder|directory|file|path|script)?\s*"
+    r"['\"]?(?P<subject>[\w.\-]{3,64})['\"]?\s+(?:was|is|were|are|does|do)\s+"
+    r"(?:not|n't)\s+(?:found|there|present|exist)"
+    # "no matches were found for 'threeSum'", "nothing found matching hilo"
+    r"|\bno(?:thing)?\s+[\w]*\s*(?:was|were)?\s*found\s+(?:for|matching|named|called)\s+"
+    r"['\"]?(?P<target>[\w.\-]{3,64})['\"]?"
+    # "no resume was found"
+    r"|\bno\s+['\"]?(?P<thing>[\w.\-]{3,64})['\"]?\s+(?:was|were)?\s*found",
+    re.IGNORECASE)
+
+# Words that name nothing in particular, and the protocol's own vocabulary:
+# "old_string was not found in the file" is edit_file reporting a failure,
+# which is the harness talking, not a claim about the workspace.
+_NOT_A_NAME = frozenset({
+    "old_string", "new_string", "file", "files", "folder", "directory", "path",
+    "the", "any", "such", "match", "matches", "error", "content", "string",
+    "result", "results", "data", "text", "line", "lines", "name", "item",
+})
+
+
+# Only these tools answer "what is there"; only their output can disprove a
+# "not found". Every other tool ECHOES the name the model asked for when it
+# fails ("File not found: ...missing.py", and A3's hint besides), and reading
+# that back as evidence fired on 80 of 1,366 recorded runs, including every
+# run of missing-file-1 and missing-file-2, which are 5/5 cases whose correct
+# answer IS "missing.py was not found; notes.txt and other.txt are present".
+_LISTING_TOOLS = frozenset({"list_directory", "find_files", "search_files", "grep"})
+
+
+def _is_error_output(text: str) -> bool:
+    """A tool result that reports a failure rather than an answer."""
+    head = (text or "").lstrip()[:80].lower()
+    return head.startswith("error:") or head.startswith("file not found") or \
+        head.startswith("directory not found")
+
+
+# Reading a code file back proves the bytes, not the behaviour. The
+# 2026-09-13 calculator session did exactly that — write_file, read_file,
+# "Verified the HTML file ... contains a properly formatted simple
+# calculator app" — on a page whose buttons were wired to nothing.
+#
+# The first version of this nudge (2026-09-16) therefore named a checker
+# INSTEAD of read_file, and that was wrong in a way a 5-run arm could not
+# see. A checker proves a file PARSES; it does not prove the edit landed.
+# Measured at 15 runs the same night:
+#
+#   agentic-3 ("read config.json, add a key, then read it again to
+#   confirm") used verify_syntax in 4 of 14 runs and no read-class tool at
+#   all in 3 — against 0 of 37 runs across the seven arms before it,
+#   p=0.004 — and fell to 10/14 from a pooled 89-91%.
+#
+#   claims-2 used a read-class tool in 0 of 5 runs, against 7 of 10 before
+#   (p=0.026), and its failures are the damning ones: the edit missed,
+#   verify_syntax passed on the UNCHANGED file, and the finish claimed
+#   success. That is the false claim this gate exists to stop, reintroduced
+#   by the gate's own wording.
+#
+# So: confirming a mutation is read_file's job, and checking behaviour is a
+# SECOND step, not a substitute. The nudge leads with the read and appends
+# the checker for code.
+# Exactly what tools._LANGUAGE_BY_EXT plus html_wiring_report can check.
+# Naming verify_syntax for anything else buys a "NOT CHECKED" and a wasted
+# step.
+_CHECKABLE_SUFFIXES = frozenset({
+    ".py", ".pyw", ".json", ".ps1", ".psm1", ".psd1",
+    ".js", ".mjs", ".cjs", ".ts", ".tsx", ".jsx", ".html", ".htm",
+})
+_RUNNABLE_SUFFIXES = frozenset({".py", ".ps1", ".js", ".mjs", ".cjs"})
+
+
+def _verify_nudge_text(changed: str) -> str:
+    """What to do about an unverified change. Always read it back — that is
+    what shows the change is there — and for code, check it as well."""
+    suffix = Path(str(changed)).suffix.lower()
+    if suffix in _RUNNABLE_SUFFIXES:
+        extra = " Then run it with run_code and report the output you saw."
+    elif suffix in _CHECKABLE_SUFFIXES:
+        extra = " Then check it with verify_syntax and report what the check said."
+    else:
+        extra = ""
+    return (f"You modified {changed} but never verified the result. Use read_file "
+            f"on {changed} and report what it actually says, so the change is "
+            f"shown to be there.{extra} Respond with JSON only.")
+
+
+def _contradicted_not_found(msg: str, outputs: list[str]) -> str:
+    """The name a finish says is missing, when a listing this turn returned
+    contains it. `outputs` holds successful listing output only. Empty string
+    when there is no contradiction."""
+    if not outputs:
+        return ""
+    haystack = "\n".join(outputs).lower()
+    for match in _NOT_FOUND_NAME_RE.finditer(msg or ""):
+        name = (match.group("subject") or match.group("target")
+                or match.group("thing") or "").strip(" .'\"")
+        if len(name) < 3 or name.lower() in _NOT_A_NAME:
+            continue
+        if name.lower() in haystack:
+            return name
+    return ""
+
+
+def _intent_nudge(query: str, msg: str, *, mutated: bool, ran: bool,
+                  tools_used: list[str]) -> str:
+    """One nudge when the turn did none of the work the request implies.
+    Empty string when there is nothing to say."""
+    if (_WANTS_RUN_RE.search(query) and mutated and not ran
+            and not _asks_to_run_tests(query)):
+        return ("You were asked to run it and nothing was run this turn. Run it now "
+                "with run_code (or run_command) and report the output you actually "
+                "saw. Respond with JSON only.")
+    if _WANTS_CREATE_RE.search(query) and not mutated and _DENIES_MEANS_RE.search(msg):
+        return ("You do have the tools for this: write_file creates a file of any kind "
+                "and run_command runs anything the shell can. Create it now. "
+                "Respond with JSON only.")
+    if (_WANTS_FIND_RE.search(query) and _CLAIMS_NEGATIVE_RE.search(msg)
+            and _NAMES_WORKSPACE_RE.search(msg)
+            and not (_SEARCH_TOOLS & set(tools_used))):
+        return ("You reported that nothing was found, but nothing was searched this "
+                "turn. Use find_files (or list_directory) to look, then report what "
+                "the search actually returned. Respond with JSON only.")
+    if (_CLAIMS_CHECKED_RE.search(msg) and not tools_used
+            and _NAMES_WORKSPACE_RE.search(msg)):
+        return ("You said you checked, but this turn made no tool call at all. "
+                "Check it with a tool and report what the tool returned, or say "
+                "plainly that you did not check. Respond with JSON only.")
+    return ""
+
+
 def _asks_to_run_tests(query: str) -> bool:
     """True when the request itself asks for the tests to be run."""
     return bool(_RUN_TESTS_RE.search(query or ""))

{hexcli-2.11.1 → hexcli-2.12.0}/hexcli/parsing.py

@@ -13,6 +13,7 @@ verbatim — behavior changes do not belong in split commits.
 """
 from __future__ import annotations
 
+import ast
 import json
 import re
 import shutil
@@ -144,18 +145,54 @@ def _loads_object(text: str) -> dict[str, Any] | None:
     return None
 
 
+def _decode_failure(text: str) -> tuple[json.JSONDecodeError | None, str]:
+    """The error that finally blocks decoding the first object in `text`,
+    after the same stray-quote repairs `parse_json_object` makes, together
+    with the repaired text it was raised against. (None, text) when it
+    decodes. Reporting the FIRST error instead would describe a quote the
+    repair already fixed."""
+    start = text.find("{")
+    if start < 0:
+        return None, text
+    last: json.JSONDecodeError | None = None
+    for _ in range(_MAX_QUOTE_REPAIRS + 1):
+        try:
+            _DECODER.raw_decode(text, start)
+            return None, text
+        except json.JSONDecodeError as err:
+            last = err
+            fixed = _repair_stray_quote(text, err)
+            if fixed is None:
+                return err, text
+            text = fixed
+    return last, text
+
+
+def looks_truncated(raw_text: str) -> bool:
+    """True when the reply stops mid-string instead of being malformed: the
+    model was still writing when its output budget ran out. The decoder
+    reports an unterminated string and the braces never close. A write_file
+    holding more than ~1,500 characters hits this in a 4K window, and the
+    model cannot fix it by re-quoting — it has to send the content in two
+    parts (claims-1, 2026-09-14)."""
+    text = strip_thinking(raw_text).strip()
+    if text.endswith("}"):
+        return False
+    err, _repaired = _decode_failure(text)
+    return err is not None and err.msg.startswith("Unterminated string")
+
+
 def describe_json_error(raw_text: str) -> str:
     """What is wrong with the first JSON object in a reply, for the retry
     feedback: the decoder's message, the character offset and the text
     around it. Empty when the object decodes."""
     text = strip_thinking(raw_text).strip()
-    start = max(0, text.find("{"))
-    try:
-        _DECODER.raw_decode(text, start)
+    err, repaired = _decode_failure(text)
+    if err is None:
         return ""
-    except json.JSONDecodeError as err:
-        lo, hi = max(0, err.pos - 40), min(len(text), err.pos + 20)
-        return f"{err.msg} at character {err.pos - start}, near: {text[lo:hi]!r}"
+    start = max(0, repaired.find("{"))
+    lo, hi = max(0, err.pos - 40), min(len(repaired), err.pos + 20)
+    return f"{err.msg} at character {err.pos - start}, near: {repaired[lo:hi]!r}"
 
 
 def parse_json_object(raw_text: str) -> dict[str, Any] | None:
@@ -182,8 +219,55 @@ def parse_json_object(raw_text: str) -> dict[str, Any] | None:
     return None
 
 
+def _loads_python_object(text: str) -> dict[str, Any] | None:
+    """A dict the model wrote in Python's spelling rather than JSON's:
+    {'action': 'finish', 'message': '...'}. One reply in the owner's 427
+    logged replies is this (2026-09-15 17:53 turn 3), and the cost is worse
+    than a retry: with no JSON to decode, the prose fallback hands the whole
+    literal back as the finish message, so the user reads
+    "{'action': 'finish', 'message': ...}" as the answer.
+
+    ast.literal_eval evaluates no calls, names or operators, so this cannot
+    run anything; the result is still restricted to JSON-shaped data, and to
+    a dict that actually looks like an action, so that a stray Python dict
+    inside prose does not become one."""
+    start = text.find("{")
+    if start < 0 or "'" not in text[start:start + 200]:
+        return None
+    for end in range(len(text), start, -1):
+        if text[end - 1] != "}":
+            continue
+        try:
+            value = ast.literal_eval(text[start:end])
+        except (ValueError, SyntaxError, MemoryError, RecursionError):
+            continue
+        if not isinstance(value, dict) or not _json_shaped(value):
+            return None
+        if not {"action", "tool", "message"} & set(value):
+            return None
+        return {str(k): v for k, v in value.items()}
+    return None
+
+
+def _json_shaped(value: Any, depth: int = 0) -> bool:
+    """True when `value` holds only what JSON can hold. A tuple or a set is
+    the model writing Python, not an action we should act on."""
+    if depth > 6:
+        return False
+    if isinstance(value, (str, int, float, bool)) or value is None:
+        return True
+    if isinstance(value, list):
+        return all(_json_shaped(v, depth + 1) for v in value)
+    if isinstance(value, dict):
+        return all(isinstance(k, str) and _json_shaped(v, depth + 1)
+                   for k, v in value.items())
+    return False
+
+
 def parse_agent_action(raw_text: str) -> dict[str, Any]:
     parsed = parse_json_object(raw_text)
+    if not isinstance(parsed, dict):
+        parsed = _loads_python_object(strip_thinking(raw_text))
     if isinstance(parsed, dict):
         action = str(parsed.get("action", "")).strip().lower()
         args = parsed.get("args")

{hexcli-2.11.1 → hexcli-2.12.0}/hexcli/tools.py

@@ -21,6 +21,8 @@ apart from that lookup.
 from __future__ import annotations
 
 import ast
+import difflib
+import itertools
 import json
 import os
 import queue
@@ -273,12 +275,74 @@ def run_command_tool(
     return trim_text(f"Exit code: {process.returncode}\n{output}".strip(), output_limit)
 
 
+# ── A path that is not there ─────────────────────────────────────────────
+#
+# "File not found: C:\\...\\hielo.ps1" is a dead end, and the 4B model does
+# not treat it as one: in the owner's 2026-09-15 17:13 session it had just
+# written hilo.ps1, asked for hielo.ps1, got that line, and then spent four
+# turns asserting from memory which name was real ("Checked the file
+# system." with no tool call) while the owner told it it was hallucinating.
+# The directory holds the answer, so the error carries it.
+
+_HINT_SCAN_LIMIT = 2000
+
+
+def missing_path_hint(path: Path) -> str:
+    """A sentence to append to a not-found error: the closest existing names
+    in the nearest directory that does exist, or a pointer at list_directory
+    when nothing is close. Returns "" when there is nothing useful to say."""
+    try:
+        parent = path.parent
+        near = parent
+        while not near.is_dir() and near != near.parent:
+            near = near.parent
+        if not near.is_dir():
+            return ""
+        try:                                    # never name what we would refuse to read
+            _check_sensitive_path(near, "read_file")
+        except Exception:
+            return ""
+        names: list[str] = []
+        for child in itertools.islice(near.iterdir(), _HINT_SCAN_LIMIT):
+            names.append(child.name)
+        # Below the nearest existing directory, the first missing component
+        # is the one to correct, not the filename the model asked for.
+        try:
+            wanted = path.relative_to(near).parts[0]
+        except ValueError:
+            wanted = path.name
+        # Name the component that is actually missing when it is a directory
+        # further up: "thing.py is not there" would send the model looking in
+        # the wrong place.
+        if near == parent:
+            where, lead = "that directory", ""
+        else:
+            where, lead = str(near), f" {wanted} does not exist in {near}."
+        if not names:
+            return lead or f" {near} is empty."
+        close = difflib.get_close_matches(wanted, names, n=3, cutoff=0.6)
+        # Same name, different extension: hilo.py for hilo.ps1. get_close_matches
+        # ranks by whole-string ratio and can miss it on a short stem.
+        stem = Path(wanted).stem.lower()
+        close += [n for n in names if Path(n).stem.lower() == stem and n not in close]
+        if close:
+            return f"{lead} Did you mean {', '.join(close[:3])}, in {where}?"
+        if lead:
+            return f"{lead} Call list_directory on it to see what is there."
+        return (" Nothing with a similar name is in that directory. "
+                "Call list_directory on it to see what is there.")
+    except OSError:
+        return ""
+
+
 def read_file_tool(path_text: str, output_limit: int,
                    offset: int = 0, limit: int = 0) -> str:
     """Read a file. With offset/limit (1-based line numbers), read one page —
     v1.7 could only ever see the head of a large file, with no way to page."""
     path = resolve_path(path_text)
     _check_sensitive_path(path, "read_file")
+    if not path.exists():
+        raise RuntimeError(f"File not found: {path}.{missing_path_hint(path)}")
     if path.is_dir():
         raise RuntimeError(
             f"{path} is a directory, not a file. Use list_directory to see its contents."
@@ -358,7 +422,7 @@ def edit_file_tool(path_text: str, old_string: str, new_string: str) -> str:
     if not old_string:
         raise RuntimeError("edit_file requires a non-empty 'old_string'. Use write_file to overwrite the whole file.")
     if not path.exists():
-        raise RuntimeError(f"File not found: {path}")
+        raise RuntimeError(f"File not found: {path}.{missing_path_hint(path)}")
     content = path.read_text(encoding="utf-8")
     tier = "exact"
     if content.count(old_string) == 1:
@@ -411,7 +475,7 @@ def list_directory_tool(path_text: str, output_limit: int) -> str:
     path = resolve_path(path_text or ".")
     _check_sensitive_path(path, "list_directory")
     if not path.exists():
-        raise RuntimeError(f"Directory not found: {path}")
+        raise RuntimeError(f"Directory not found: {path}.{missing_path_hint(path)}")
     if not path.is_dir():
         raise RuntimeError(f"Not a directory: {path}")
     entries = []
@@ -747,7 +811,7 @@ def verify_syntax_tool(path_text: str, language: str, shell_exe: str) -> str:
     finish with "verified"."""
     path = resolve_path(path_text)
     if not path.exists():
-        raise RuntimeError(f"File not found: {path}")
+        raise RuntimeError(f"File not found: {path}.{missing_path_hint(path)}")
     suffix = path.suffix.lower()
     lang = (language or "").strip().lower() or _LANGUAGE_BY_EXT.get(suffix, "")
     if suffix in {".html", ".htm"} or lang in {"html", "htm"}:
@@ -788,7 +852,7 @@ def lint_code_tool(path_text: str) -> str:
         raise RuntimeError("ruff is not on PATH — lint_code is unavailable.")
     path = resolve_path(path_text)
     if not path.exists():
-        raise RuntimeError(f"File not found: {path}")
+        raise RuntimeError(f"File not found: {path}.{missing_path_hint(path)}")
     try:
         result = subprocess.run(
             [_RUFF, "check", "--output-format=concise", str(path)],
@@ -823,7 +887,7 @@ def run_code_tool(
     cwd = Path.cwd().resolve()
     path = resolve_path(path_text)
     if not path.exists():
-        raise RuntimeError(f"File not found: {path}")
+        raise RuntimeError(f"File not found: {path}.{missing_path_hint(path)}")
     if not path.is_relative_to(cwd):
         raise RuntimeError(
             f"run_code is restricted to files under the working directory ({cwd}). "