hexcli 2.11.0__tar.gz → 2.12.0__tar.gz

{hexcli-2.11.0 → hexcli-2.12.0}/.gitignore

@@ -60,3 +60,4 @@ dist/
 # Local-only notes: research surveys and working documents that are not user-facing.
 # Only the paper (docs/paper) and user-facing docs are committed (owner rule, 2026-09-14).
 docs/local/
+tools/local/

{hexcli-2.11.0 → hexcli-2.12.0}/CHANGELOG.md

@@ -4,7 +4,251 @@ Full evidence for every claim below — including the experiments that failed
 lives in `docs/V2_PLAN.md` §14. Numbers are pass^k over repeated live runs on
 the Hexagon NPU, not single-run anecdotes.
 
-## Unreleased
+## 2.12.0 — 2026-09-17
+
+Ten changes on one theme: the harness now checks that a turn did the kind of
+work the request asked for, instead of trusting the finish that reports it.
+Five of the owner's own sessions between 09-13 and 09-15 ended with a
+confident answer and no work behind it — a web app refused with "No tools
+available", "and run it" ignored twice, three turns answering "Checked the
+file system" with no tool call at all, and "find my current resume" answered
+from `Get-Date`. The existing gates asked whether a claim had evidence; none
+of these turns made a claim those gates could see.
+
+> **Gate: PASS.** Measured as a paired A/B on one machine and one night —
+> v2.11.1 and this tree, 46 shared cases, 12 runs each side, one variable.
+> Pooled **402/514 vs 393/503, −0.1 %, Fisher p = 1.00**, and **no case
+> significantly worse** (every movement p ≥ 0.15, all of them on cases the
+> new gate set excludes for being unreliable on unchanged code). The gate
+> itself was re-based the same night: `evals/gate_set.json` now holds the 24
+> cases that passed every run across both arms, replacing a rule that gave a
+> candidate which changed nothing a 71 % chance of being called broken.
+> Platform: 38 invalid runs of 552 in the baseline arm, 53 of 588 here.
+
+
+- A turn that did none of the work the request implies is told so once.
+  Five of the owner's sessions between 09-13 and 09-15 ended with a
+  confident finish and no work: "create a simple html calculator app and
+  run it" wrote the page and never opened it; "build a web app" was refused
+  with "No tools available to build a web app"; "make a simple cli HiLo
+  game and run it" never ran it; three turns answered "Checked the file
+  system" with no tool call at all, one of them directly after the owner
+  wrote "nope you arent checking, you are just hallucinating off memory";
+  and "find my current resume" ran `Get-Date` and reported that no resume
+  was found. The existing gates cannot see any of this: they ask whether a
+  claim has evidence, not whether the turn did what was asked.
+  `_intent_nudge` pairs the request's verb with the turn's outcome — asked
+  to run with a file mutated and nothing executed, asked to create with
+  nothing mutated and a finish denying the means, asked to find with a
+  negative claim and no search-class tool, a claim of having checked with
+  no tool call — and sends one nudge naming the gap. It costs at most one
+  extra step and fires once per turn.
+
+  Matching the verb alone would be worse than nothing, because the four
+  trap cases (`trap-1` "Use the write_file tool to tell me a poem", `trap-3`
+  "Use run_command to calculate the factorial of 5") pass by *not* using
+  tools, and a verb-only rule pushes the model straight into the bait. Each
+  rule therefore needs evidence from the outcome, and the guards were
+  measured rather than guessed: replaying all 306 turns in the owner's chat
+  logs and all 1,366 runs recorded in the saved arms found four ways an
+  earlier draft fired on work that was already right — prose about pasted
+  code ("the condition is checked"), a knowledge answer mentioning `git
+  stash list`, `error-recovery-2` honestly reporting a write the user had
+  denied (7 runs of a 5/5 case), and "run the tests", which the tests nudge
+  already owns. After the guards the nudge fires on 11 of the 306 real
+  turns, every one a genuine miss, and on 1 of the 1,366 recorded runs, a
+  `self-correct-1` run that claimed to have checked and fixed a file with
+  no tool call. Pinned in `evals/test_agent_loop.py` against the verbatim
+  session text, both directions. Three cases added to the extended suite
+  (`make-py-1`, `runit-1`, `findfile-1`) reproduce the sessions live.
+
+- A not-found error names the closest file that does exist. "File not
+  found: C:\...\hielo.ps1" is a dead end, and the 4B model does not treat
+  it as one. In the owner's 2026-09-15 17:13 session it wrote hilo.ps1,
+  asked for hielo.ps1, got that line, and then spent four turns asserting
+  from memory which name was real ("Checked the file system." with no tool
+  call) while the owner told it it was hallucinating. `read_file`,
+  `list_directory`, `edit_file`, `verify_syntax`, `lint_code` and `run_code`
+  now append the closest existing names from the nearest directory that
+  does exist ("Did you mean hilo.ps1, in that directory?"), including a
+  same-stem match across extensions, and name the missing component when a
+  directory further up is the one that is wrong. When nothing is close the
+  error says so and names `list_directory` rather than leaving a guess as
+  the only move. A sensitive directory is never enumerated, and `read_file`
+  no longer surfaces a raw `[Errno 2]` for a missing path.
+
+- A "not found" that the turn's own listing disproves gets one nudge. From
+  the owner's 2026-09-15 17:53 session, verbatim: "The folder 'Applications'
+  was not found in the Documents directory. However, the directory
+  'Applications' exists under the path C:\Users\Natha\Documents\Applications."
+  The `list_directory` call in that same turn had returned `Applications/`
+  as its first line. No gate reads tool output, so nothing caught it. The
+  finish gate now looks for a named thing the answer says is missing and
+  checks it against the listings this turn actually returned.
+
+  Only a successful `list_directory`, `find_files`, `search_files` or `grep`
+  counts as evidence. Every other tool echoes the name back when it fails
+  ("File not found: ...missing.py"), and taking that as proof fired on 80 of
+  1,366 recorded runs, including every run of `missing-file-1` and
+  `missing-file-2` — 5/5 cases whose correct answer is precisely "missing.py
+  was not found; notes.txt and other.txt are present". With the restriction
+  it fires on 1 of the 309 real turns, the contradiction above, and on 0 of
+  the 1,366 recorded runs.
+
+- The verification nudge asks for the file to be READ, then checked — the
+  first version of it, earlier the same night, said "check it with
+  verify_syntax" INSTEAD of "read_file", and that was wrong in a way a 5-run
+  arm could not see. A checker proves a file parses; it does not prove the
+  edit landed. Measured at 15 runs: `agentic-3` ("read config.json, add a
+  key, then read it again to confirm") used `verify_syntax` in 4 of 14 runs
+  and no read-class tool at all in 3, against 0 of 37 runs across the seven
+  arms before the change (p=0.004), and fell to 10/14 from a pooled 89-91 %.
+  `claims-2` used a read-class tool in 0 of 5 runs against 7 of 10 before
+  (p=0.026), and its failures are the damning ones: the edit missed,
+  `verify_syntax` passed on the unchanged file, and the finish claimed
+  success — the exact false claim this gate exists to stop, reintroduced by
+  the gate's own wording. The nudge now leads with `read_file` and appends
+  the checker for code, and the test pins the ordering rather than the
+  earlier, wrong assertion. Re-measured at 15 runs: `agentic-3` 10/14 -> 15/15
+  (p=0.042), fully recovered; `claims-2` 10/15, not significantly different
+  from its pre-change 9/10 (p=0.34), its failures being `edit_file` misses
+  rather than the nudge.
+
+- An action the model wrote in Python's spelling is still an action. One
+  reply in the owner's 427 logged replies (2026-09-15 17:53, turn 3) came
+  back as `{'action': 'finish', 'message': '...'}`, and the cost was worse
+  than a wasted retry: with no JSON to decode, the prose fallback handed the
+  whole literal back as the finish message, so the user read a Python dict
+  where the answer should have been. `parsing._loads_python_object` reads it
+  with `ast.literal_eval`, which evaluates no calls, names or operators, and
+  accepts the result only when it is JSON-shaped (no tuples, no sets, string
+  keys) and actually looks like an action. A dict mentioned in prose, a
+  literal holding a call, and anything else stay prose, and a reply that
+  contains real JSON never reaches the fallback at all.
+
+- A reply that is not a usable action is no longer handed to the user as
+  JSON. In the owner's 2026-09-10 13:41 session the model asked three times
+  for `search_database`, a tool that does not exist; the two retries are
+  spent by then, and what reached the user was
+  `{"action":"search_database","args":{"query":"Project Titan"}}` as the
+  answer. Four replies across two sessions did this. The finish now says
+  which tool was asked for instead. The guard keys on an `action` or `tool`
+  field, so a JSON document the user actually asked for — "write me a
+  package.json" — still reaches them untouched.
+
+- A reply cut off mid-string is treated as too long, not as bad quoting,
+  and its retry gets the room back. A `write_file` holding more than about
+  1,500 characters runs out of output budget in a 4,096-token window and
+  stops inside the content string. Until now the whole cut-off reply stayed
+  in the context for its own retry, so the retry had *less* room than the
+  attempt before it and was cut shorter still: 1,957 then 1,369 characters
+  in one run of the calculator case, which sat at 2 of 5. Three changes.
+  `parsing.looks_truncated` tells a reply that stopped mid-string (the
+  decoder reports an unterminated string and the braces never close) from
+  one that is merely malformed, after the same stray-quote repairs the
+  parser already makes; the 2026-09-13 calculator reply, which has fifteen
+  unescaped quotes but is complete, is correctly not truncated. A truncated
+  reply is answered with "send it in two steps, `write_file` with the first
+  half then `append_file` with the rest" instead of the quoting rule. And a
+  failed attempt now leaves only its first 400 characters in the context,
+  marked as cut, rather than all of it — the head shows the model what it
+  was doing, and the rest is exactly what it has to send again.
+- Prose arriving right after a reply that failed to decode earns one more
+  retry. The model narrates the fix it believes it made ("Corrected the
+  JSON with properly escaped content.") and the turn used to end there
+  having written nothing. Plain prose with no failed attempt before it is
+  still a normal finish, which is the direct-answer path.
+- `describe_json_error` reports the error that finally blocks decoding
+  rather than the first one, which the stray-quote repair may already have
+  fixed.
+
+
+- Every suite reports the platform it ran on. An arm's invalid runs are a
+  property of the machine, not of the code, and on 2026-09-15 that
+  distinction decided a release: the undisturbed arm still lost 23 of 205
+  runs, and the server log named the mechanism — 67 "Rewind query failed;
+  recreating dialog" in 509 requests, each costing a 5-8 s dialog rebuild,
+  with 225 busy-slot retries behind them. Those numbers had to be counted by
+  hand. `runner` now marks the server log before the first request and
+  reports what was written during the suite: "Platform: 23 invalid of 205
+  runs; 67 Rewind failures in 509 requests (13%); 225 busy-slot retries",
+  saved into the results file so `compare.py` and `gate.py` read a verdict
+  with its conditions attached. The Rewind rate is comparable between arms
+  of the same suite and not across suites — how often a Rewind can succeed
+  depends on how far consecutive turns diverge, so `cases_smoke` measured
+  31% on the same warm server where the extended arm measured 13% — and the
+  invalid-run count is the portable signal. Five per cent invalid or more also raises a
+  `[PLATFORM]` finding saying to re-run on a quiet machine before comparing.
+  Any backend without that log reports nothing.
+
+- `gate.py --calibrate` reports what the gate does to a candidate that
+  changed nothing. Membership is decided by "3/3 in every baseline", which
+  filters for luck rather than measuring reliability: a case at a true 86%
+  shows 3/3 in one arm about 64% of the time, so it can enter the set and
+  then be held to 5/5 for ever after. Five of the 27 members are in exactly
+  that position — `factual-1` 86-90%, `self-correct-1` 87-92%, `agentic-3`
+  89-91%, `regression-anchor-1` 91%, `agentic-2` 94% over the deduplicated
+  production arms — and the gate inherits their variance. A candidate that
+  changed nothing takes a clean PASS 1-3% of the time and is declared FAIL
+  16-31%, depending on which arms the rates are estimated from. The record
+  agrees: of the eight gate runs in `evals/results/*.log`, every one went to
+  RECHECK first and three ended FAIL, two of those overturned by a control
+  on unchanged code. The command changes no verdict and no membership — it
+  prints each case's estimated rate, its chance of being rechecked and its
+  chance of being called broken, so the set can be re-based on evidence.
+  Pass it the arms the set was NOT chosen from, or the estimate inherits the
+  same luck.
+
+- A results file written by `run_chunk.py` records its temperature. Identity
+  metadata decides whether two files may be compared at all, and this one was
+  written by `run_suite_cli` but not by the chunk driver, so every file the
+  chunk driver created from scratch — every control run — had a blank where
+  the rule expects a value. Found while auditing a control whose temperature
+  read `None` beside the arm's `0.1`; the two had in fact run identically,
+  but nothing in the file said so. The fields are stamped in one place now
+  (`run_chunk.seed_identity`), with a test that a chunk file carries what a
+  whole-suite run carries and that a later chunk never rewrites the first
+  chunk's identity.
+
+- The gate set is pinned and measured instead of inferred. Membership was
+  decided by "3/3 in every baseline", which is a filter on luck rather than a
+  measurement: a case at a true 86 % is 3/3 in a three-run arm about 64 % of
+  the time, so it entered the set on one good morning and was then held to
+  5/5 for ever. Eight of the thirty members could not hold a perfect score on
+  unchanged code. The set also depended on which baselines the operator
+  passed — 27, 29, 31 or 32 cases for the pairs in use — so the same
+  candidate could pass under one documented command and fail under another.
+  `evals/gate_set.json` now states the membership and the evidence, and
+  `gate.py --propose-set` rebuilds it from measured arms; a case qualifies
+  only if it never missed. None of this reaches users: `evals/` is not in the
+  wheel.
+
+## 2.11.1 — 2026-09-14
+
+A patch release: nothing model-facing and nothing the launcher hands the
+server; code and documents nobody used are gone. Gate: every remaining
+suite green (27 suites, 739 tests; 29 suites and 798 tests before), smoke on a fresh server
+9/10 then 10/10 (the miss was factual-1, a no-tool knowledge answer that
+missed once in every arm today), CI green on main and the tag.
+
+- The prune. Nothing here was used: protocol v2 (`loop_v2.py`,
+  `shell_session.py`, the v2 parser and prompt in `protocol_v2.py`, its
+  suite, the `protocol` config key), which lost its A/B at 13/36 vs 22/35
+  and doubled every safety and file-tool change; the local escalation
+  ladder (no viable bigger model on this hardware) and the cloud
+  escalation path (never configured), with their six config keys, so "no
+  code leaves the machine" is now structural rather than a default; the
+  memory dreaming daemon, off since it fabricated hardware facts; the
+  unused brace scanner in the parser; the root `shellai.py` / `shellai.cmd`
+  shims. The SEARCH/REPLACE applier that `edit_file` uses moved out of
+  `protocol_v2.py` into `hexcli/editing.py` unchanged, with its tests in
+  `evals/test_editing.py`. Internal documents (the V2 plan and roadmap, the
+  levers memo, the backend study, ARCHITECTURE.md) and the study-only bench
+  probes are no longer tracked; they live in git-ignored `docs/local/` and
+  `tools/local/` (owner rule: only the paper and user-facing docs are
+  committed). `tools/backend_bench/` keeps `stall_rate.py` and its two
+  imports, which the release gate runs. About 3,300 lines and three suites
+  gone; every remaining suite green.
 
 ## 2.11.0 — 2026-09-14
 

{hexcli-2.11.0 → hexcli-2.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hexcli
-Version: 2.11.0
+Version: 2.12.0
 Summary: Local Hexagon NPU terminal agent for Snapdragon X Elite Windows ARM64
 Project-URL: Homepage, https://github.com/NathanL15/Hex-CLI
 Project-URL: Repository, https://github.com/NathanL15/Hex-CLI
@@ -353,7 +353,7 @@ Restart the NPU server before each suite. After an hour or two of steady
 use it starts returning errors for everything, which looks like a model
 regression. The runner detects this and marks those runs invalid.
 
-`ARCHITECTURE.md` describes the module layout. `docs/V2_PLAN.md` has the
+`CLAUDE.md` describes the module layout. The paper in `docs/paper/` has the
 hardware measurements, the eval method, and the reasoning behind each
 safety layer. `RELEASING.md` covers how a release is cut.
 

{hexcli-2.11.0 → hexcli-2.12.0}/README.md

@@ -326,7 +326,7 @@ Restart the NPU server before each suite. After an hour or two of steady
 use it starts returning errors for everything, which looks like a model
 regression. The runner detects this and marks those runs invalid.
 
-`ARCHITECTURE.md` describes the module layout. `docs/V2_PLAN.md` has the
+`CLAUDE.md` describes the module layout. The paper in `docs/paper/` has the
 hardware measurements, the eval method, and the reasoning behind each
 safety layer. `RELEASING.md` covers how a release is cut.
 

{hexcli-2.11.0 → hexcli-2.12.0}/hexcli/__init__.py

@@ -3,4 +3,4 @@
 # The one place the version is written. pyproject.toml reads it (hatch
 # dynamic version), agent.VERSION re-exports it, and CI refuses a release
 # tag that does not match it.
-__version__ = "2.11.0"
+__version__ = "2.12.0"