@andorra-labs/code 0.0.0-stage → 0.1.4

package/docs/concepts/jev-actionable-routing.md

@@ -0,0 +1,113 @@
+# Actionable JEV routing and independent optimizations
+
+This extends the guarded check-and-finish implementation from #410. It does not
+replace the executor, approvals, capability checks, completion guards, or provider
+continuation protocol. New tool-menu routing is opt-in; no live performance claim
+is made by this change.
+
+## Separate controls
+
+The installation's `efficiency` object accepts:
+
+```json
+{
+  "deterministicContinuations": true,
+  "jevFactInterpretation": true,
+  "jevPreflight": true,
+  "toolMenuRouting": { "mode": "shadow", "topK": 2, "minSavingsChars": 2048 }
+}
+```
+
+`deterministicContinuations: true` enables guarded prepared checks and completion
+without requiring JEV or sign-in. Explicit false disables that optimization.
+Omission preserves #410's facts-mode behavior. `jevFactInterpretation: false`
+keeps local evidence and continuations but never calls the remote fact service.
+True still requires an active authorized facts-mode decision manager. Neither
+setting grants execution permission or manufactures a verification result.
+
+`jevPreflight: false` disables preflight only. It does not disable the independent
+fact or tool-menu paths. With preflight enabled, facts-mode questions are filtered
+by their actual consumers: unavailable retrieval, disabled/unresolvable model
+routing, and absent delegation roles do not merit a request. If no questions
+remain, `decision.skipped` is recorded without a provider dispatch.
+
+Explicit answer-only/no-tools tasks omit finish-planning machinery and menu
+classification. This is an optimization eligibility rule, not a replacement for
+normal instruction compliance or permission enforcement.
+
+The five older efficiency switches retain their existing defaults and semantics.
+Do not use their all-on combination to attribute a result solely to JEV.
+
+## Tool-menu contract
+
+`src/decision/routeTools.ts` is a pure deterministic consumer of bounded evidence.
+It does not call a provider or execute a tool. The host constructs descriptors
+from the already-permitted menu. Core read/search/edit/shell capabilities, unknown
+tools, hidden compatibility entries, explicitly named node tools and historical
+tool calls are retained. Optional known bundles cover git, codegraph, memory,
+web, browser, subagents and artifacts. Dependencies are host-owned: optional
+roots retain core; a subagent bundle retains an available artifact bundle.
+
+`toolMenuRouting.mode` is `off` (default), `shadow`, or `lean`. A single boundary
+classification sees the redacted bounded task and fixed short bundle descriptions,
+not full schemas, handlers, credentials, costs, or eval labels. It selects the
+requested operation/deliverable, not just the next preliminary read. Ambiguous
+multi-capability requirements retain the normal menu.
+
+The request pins `jev-1.13.0` and question/policy version 1. The host requires the
+exact closed option set, finite probabilities, a distribution summing to one,
+a leading selected option, and the model/version pin. It never normalizes an
+invalid distribution or lowers completion thresholds to force interventions.
+Policy filters by confidence/relevance, orders eligible roots by schema size,
+and applies dependency closure. Top K bounds roots, not prerequisites. Pinned
+capabilities are not evicted to meet a budget.
+
+Uncertainty, clarification, invalid responses, missing prerequisites, inadequate
+schema reduction, service failure and unsupported older deployments all retain
+the original permitted menu. Cancellation and changed task/permission revisions
+cannot apply stale selection. The menu stays stable during the invocation;
+existing `restore_tools` can restore omitted permitted schemas without granting
+new authority. Unknown custom/MCP tools are deliberately not trimmed by name
+heuristics. This first version routes coarse known bundles, not arbitrary tools.
+
+`minSavingsChars` is a conservative schema-size threshold including recovery
+schema/notice allowance. It is not a token count, a monetary model, or proof that
+the extra request pays off. `pinnedTools` is a bounded optional host-supplied list;
+the node executor adds explicit configured tools automatically. Historical calls
+are retained to avoid invalidating provider tool-call history.
+
+## Evidence and accounting
+
+Local checks run before semantic interpretation. Exact owned pass/fail evidence
+needs no JEV request. Unknown, stale, truncated, denied or skipped-only evidence
+cannot become a positive check merely because an optimization is enabled.
+
+Continuation events include `evidenceSource: deterministic | jev`. A prepared
+check is a local continuation; completion is JEV-supported only when the current
+verification/task evidence it consumes actually came from JEV. These are observed
+model-call bypasses, not counterfactual net savings. A bypass can coexist with
+higher total tokens if preparing it cost more or the model took another path.
+
+Hosted transports emit logical decisions, actual client HTTP dispatches, and
+provider-reported usage separately. The proxy retains safe usage even when a
+provider response fails validation. Missing usage/model metadata remains unknown.
+The existing per-run circuit prevents repeated fact failures; menu classification
+also has a per-run circuit and a bounded deadline. No retries are added to repair
+malformed probability distributions.
+
+## Deployment and validation
+
+Deploy the updated account proxy through the existing reviewed process before
+expecting the new `toolMenu` contract to work. This source change does not deploy
+AWS infrastructure. An older proxy's 400/404/422 yields unsupported protocol and
+keeps the full menu. Existing auth, membership, quota and egress boundaries stay
+in force; only host-approved metadata is sent.
+
+Start with shadow mode, then run the paired experiments in
+[`eval/EXPERIMENTS.md`](../../eval/EXPERIMENTS.md). Compare lean against the same
+local-only baseline, preserving success rate and including routing costs,
+restorations, cache changes and failures. Do not enable lean by default until
+full runtime/CLI verification and repeated representative evaluations pass.
+
+See [`tests/jev-actionable-routing-test-plan.md`](../../tests/jev-actionable-routing-test-plan.md)
+for regression, service-compatibility and rollout checks.

package/docs/concepts/jev-decision-thresholds.md

@@ -0,0 +1,412 @@
+# Jev decision thresholds: what the 3D-Dodge run shows
+
+**Status:** analysis of one run, with a second run's numbers, a post-#354 one-pass pair, and the
+Little Harvest run of 2026-11-01 added below. Recommendation: keep
+every threshold as it is, calibrate the hosted service, close three context gaps in the harness,
+and stop awaiting the postflight round when it cannot act. No threshold was changed.
+
+The 3D-Dodge benchmark (`Benchmarks/3D-Dodge/`, added in [Andorra-Labs/Andorra-Code#344](https://github.com/Andorra-Labs/Andorra-Code/pull/344)) ran the Software Engineer graph on
+`glm-5.3-flash` twice, once with hosted Jev and once without. The Jev run asked 172 questions over
+27 rounds, got 142 answers, and applied none of them: the model saw the same 30-tool catalog as the
+no-Jev run (plus `prepare_steps`), and every one of its 27 turns went to the coding model. The
+rounds cost 19.8 s. This page answers whether the thresholds in
+[`DecisionPolicy`](../../src/decision/DecisionPolicy.ts) are too strict for the signal the service
+produces, whether the service's calibration is the problem, or both.
+
+Numbers come from the run's `trajectory/run-events.jsonl` and `meta.json` under
+`Benchmarks/3D-Dodge/runs/2026-10-31_andorra-code_jev_glm-5.3-flash_8fb1b581/`.
+
+## What gates each action
+
+| Question | Decisive at | What a decisive answer does |
+| --- | --- | --- |
+| `tool_family_*` | ≤0.20 | Removes that family's schemas from the catalog (`routeToolFamilies`). A yes does nothing. |
+| `needs_external_docs` | ≤0.20 / ≥0.80 | Removes web schemas / adds a "consult primary docs" directive. |
+| `needs_delegation` | ≤0.20 / ≥0.80 | "Prefer the current agent" directive / role contract in `spawn_subagent` (with `delegation_role` ≥0.90). |
+| `needs_codegraph`, `needs_memory` | 0.80 / 0.20, 0.85 / 0.15 | Skip or request automatic retrieval. Only asked when the retrieval is available. |
+| `security_sensitive`, `architecture_affecting` | ≥0.95 / ≤0.05 | Yes forces full-suite verification. Both must be decisively **no** before a completion shortcut is allowed. |
+| `verification_tier` (choice) | ≥0.90 | Adds the tier's verification directive. |
+| `task_complexity` (score) | confidence ≥0.80 | A score ≥0.8 forces full-suite verification and a milestone directive. |
+| Six post-tool booleans | ≥0.95 / ≤0.05 | Yes adds a one-line directive to the next prompt. Each prepared-step kind also needs its own boolean to be a decisive yes (`permittedContinuation`). |
+| `next_action` (choice) | confidence and probability ≥0.95 | Runs a prepared step without a model turn. The runtime passes 0.95 here, not the 0.90 default. |
+
+A prepared step runs only when all of these hold: the model called `prepare_steps` in the
+batch whose results are being judged (steps are cleared when the model resumes), `next_action`
+picked it at ≥0.95, the step's kind gate is a decisive yes, fewer than three automatic turns have
+run in a row, and for `complete` the invocation was read-only with every todo completed.
+
+### When the post-tool round holds up the next turn
+
+`next_action` is only asked when the judged batch prepared candidates. Without candidates the
+round can do one thing, add a one-line directive, so the runtime (`src/hardened-runtime.ts`)
+starts that round and builds the next model request without waiting for it; the directive, if
+the answer crosses its bar, is added to the first request built after the answer arrives, and an
+answer that a newer round, a steer or a follow-up has superseded is dropped. A round with
+candidates is awaited before the next request, because its answer may replace that request with
+the prepared step. The round's telemetry (`decision.requested`, `decision.completed`,
+`decision.fallback`) is unchanged either way, and a run waits for its last in-flight round before
+`agent.session.completed` so every round in the log is paired. On the 3D-Dodge runs above, where
+the model called `prepare_steps` once in 47 turns, this removes the serial wait (about 0.8 s per
+turn) from every other turn without changing what reaches the model.
+
+## What the service answered
+
+### Preflight (one round, 15 questions)
+
+Two answers were decisive: `needs_delegation` 0.10 (adds the "prefer the current agent" directive)
+and `security_sensitive` 0.03. Everything else sat in the uncertain band:
+`needs_external_docs` 0.43, `architecture_affecting` 0.09 (needs ≤0.05), `verification_tier`
+`targeted` at 0.62, `task_complexity` 0.515 at confidence 0.20, `skill_role` at 0.39.
+
+The meta file's `decisionsApplied: 0` is accurate for what matters (no schema removed, no turn
+skipped) but undercounts: `decision.applied` is only emitted for CodeGraph and memory, so the
+delegation directive that did fire is invisible to the benchmark.
+
+Tool families, against what the agent then used (36 calls):
+
+| Family | Probability | Calls in the run | A ≤0.30 cut would have |
+| --- | --- | --- | --- |
+| filesystem-write | 0.88 | `Write` 1 | kept it |
+| browser | 0.75 | `qa.browser_session` 13 | kept it |
+| artifact | 0.47 | 0 | kept it |
+| shell | 0.42 | 6 (`PowerShell` 1 failed, `NativeExec` 2, `AsyncProcess` 3) | kept it |
+| filesystem-read | 0.30 | 0 | removed it, harmlessly |
+| code-search | 0.30 | 3 (`Glob` on an empty workspace, `Grep` twice for a string in its own file, 1 failed) | removed it; `restore_tools` would have been needed for three marginal calls |
+| codegraph | 0.28 | 0 | removed it, harmlessly |
+| git | 0.25 | 1 (`git_read`, failed: the workspace is not a repository) | removed it, harmlessly |
+
+The task was "create a game in a single HTML file" in an empty folder that is not a git
+repository. A calibrated classifier should put git, codegraph and filesystem-read well under 0.20
+for that task. But the preflight snapshot gives the service no workspace facts (file count,
+whether a repository exists), only the task text, the node instruction and the family list, so
+0.25-0.30 is a hedge the harness partly invited.
+
+### Post-tool (26 rounds, 21 answered, 5 `decision_service_unavailable`)
+
+Each row is one round: the tools whose results were judged, the six probabilities, and what the
+model did next. `!` marks a failed call.
+
+| Turn | Judged results | answered | investigate | retry | change | verify | complete | Model's next call |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| 1 | PowerShell! | 0.06 | 0.57 | 0.31 | 0.87 | 0.06 | 0.02 | Glob |
+| 2 | Glob | 0.45 | 0.49 | 0.35 | 0.83 | 0.08 | 0.03 | update_todos, prepare_steps, Write |
+| 3 | update_todos, prepare_steps, Write | 0.52 | 0.41 | 0.32 | 0.64 | 0.47 | 0.23 | qa.inspect_project (`next_action` at 0.72) |
+| 6 | NativeExec | 0.21 | 0.52 | 0.45 | 0.77 | 0.12 | 0.05 | Grep |
+| 7 | Grep! | 0.06 | 0.70 | 0.39 | 0.90 | 0.07 | 0.02 | NativeExec |
+| 8 | NativeExec | 0.25 | 0.54 | 0.46 | 0.76 | 0.15 | 0.08 | Grep, qa.browser |
+| 10 | update_todos, AsyncProcess | 0.17 | 0.55 | 0.43 | 0.82 | 0.14 | 0.05 | AsyncProcess, qa.browser |
+| 12 | update_todos!, qa.browser_session | 0.19 | 0.60 | 0.45 | 0.86 | 0.17 | 0.10 | update_todos, qa.browser_session |
+| 13 | update_todos, qa.browser_session | 0.33 | 0.57 | 0.43 | 0.82 | 0.28 | 0.20 | qa.browser_session |
+| 14 | qa.browser_session | 0.20 | 0.60 | 0.46 | 0.84 | 0.20 | 0.12 | qa.browser_session |
+| 15 | qa.browser_session | 0.36 | 0.60 | 0.43 | 0.82 | 0.26 | 0.20 | qa.browser_session |
+| 16 | qa.browser_session | 0.20 | 0.60 | 0.45 | 0.85 | 0.19 | 0.12 | qa.browser_session |
+| 17 | qa.browser_session | 0.20 | 0.62 | 0.43 | 0.85 | 0.18 | 0.10 | qa.browser_session |
+| 18 | qa.browser_session! | 0.05 | 0.54 | 0.42 | 0.87 | 0.10 | 0.04 | qa.browser_session |
+| 19 | qa.browser_session | 0.11 | 0.61 | 0.48 | 0.84 | 0.16 | 0.09 | qa.browser_session |
+| 20 | qa.browser_session | 0.23 | 0.61 | 0.48 | 0.84 | 0.18 | 0.14 | qa.browser_session |
+| 21 | qa.browser_session | 0.13 | 0.53 | 0.47 | 0.84 | 0.16 | 0.10 | qa.browser_session |
+| 22 | qa.browser_session | 0.12 | 0.54 | 0.47 | 0.83 | 0.15 | 0.09 | qa.browser_session |
+| 23 | qa.browser_session | 0.24 | 0.54 | 0.46 | 0.83 | 0.20 | 0.14 | qa.browser_session |
+| 24 | qa.browser_session | 0.13 | 0.60 | 0.47 | 0.84 | 0.16 | 0.09 | update_todos, qa.deep |
+| 26 | AsyncProcess, update_todos | 0.12 | 0.56 | 0.43 | 0.86 | 0.11 | 0.07 | completion accepted |
+
+Rounds 4, 5, 9, 11 and 25 got no answer (service unavailable). Round 25 was the one that judged
+the `qa.deep` gate result, so the service never saw the evidence that the task was verified.
+
+Summary statistics over the 21 answered rounds, split by whether a judged call had failed:
+
+| Question | Mean (SD) | After a failure (n=4) | After success (n=17) | Rounds ≥0.80 / ≤0.20 |
+| --- | --- | --- | --- | --- |
+| `result_answered_question` | 0.21 (0.12) | 0.09 | 0.23 | 0 / 10 |
+| `needs_more_investigation` | 0.57 (0.06) | 0.60 | 0.56 | 0 / 0 |
+| `should_retry` | 0.43 (0.05) | 0.39 | 0.44 | 0 / 0 |
+| `strategy_should_change` | 0.83 (0.05) | 0.88 | 0.82 | 18 / 0 |
+| `ready_for_verification` | 0.17 (0.09) | 0.10 | 0.19 | 0 / 16 |
+| `likely_complete` | 0.10 (0.06) | 0.05 | 0.11 | 0 / 18 |
+
+## Thresholds, calibration, or the questions?
+
+**Calibration first, question context second, thresholds not at all.**
+
+- **The answers barely move.** Every post-tool question has a standard deviation of 0.05-0.12
+  across 21 rounds whose evidence ranged from a refused shell to a passing QA gate. The gap
+  between a failed and a successful round is 0.04-0.14 on every question. Whatever threshold is
+  chosen, an answer that does not change with the evidence either fires every round or never.
+- **`strategy_should_change` is biased high.** Mean 0.83, at or above 0.80 in 18 of 21 rounds,
+  including the eleven consecutive rounds where `qa.browser_session` was succeeding and the model
+  correctly kept driving the game. Lowering the bar to 0.80 would have injected "change
+  investigation strategy; do not repeat the same failed approach" into 18 prompts, wrongly in at
+  least 17, into a run that succeeded. At 0.90 it fires once (round 7, after the `Grep` path
+  error, where the model did change course), which is a sample of one.
+- **`should_retry` points the wrong way.** It is lower after failures (0.39) than after
+  successes (0.44). No threshold rescues an inverted signal.
+- **`likely_complete` never approached the bar when it was true.** Its maximum was 0.23; in the
+  round immediately before the model completed it was 0.07. Even a 0.50 bar would not have fired.
+  The decisive **no** answers (six rounds ≤0.05) were correct, but a no on this question does
+  nothing. Part of this is a context gap: the runtime requires every todo to be completed before
+  it allows a completion shortcut, yet the post-tool snapshot never tells the service the todo
+  state, and the round that judged the `qa.deep` verdict failed.
+- **`ready_for_verification` peaked at the right moment** (0.47, right after the `Write`), and
+  the model did go to verification next. That is the only question where a lower bar would have
+  fired once and been right, but 0.47 is below a coin flip and the same bar (≤0.47) would have
+  fired nowhere else, so it is noise that happened to land.
+- **The one `next_action` round agreed with the model.** The model called `prepare_steps` once
+  in 27 turns (turn 3), so candidates existed for one round. Its next call, `qa.inspect_project`,
+  was exactly the prepared `inspect_profile` step. The service answered at confidence 0.72
+  against a 0.95 bar, and the step's kind gate (`needs_more_investigation` ≥0.95) stood at 0.41.
+  A hit there would have saved one model turn (mean 7.5 s in this run). It is the strongest case
+  for a looser bar in the whole run, and it is one round, with the service hedging at 0.72 on a
+  choice it had the evidence to make.
+- **Preflight families were hedges, not confident noes.** Four families sat at 0.25-0.30. A
+  ≤0.30 cut would have removed four families, three of them harmlessly, in this run. It would also
+  have cut `code-search`, which the model used three times, and the answers were not confident
+  enough to defend the cut if the next task is different. With no workspace facts in the
+  snapshot, the service was asked to guess.
+
+The 0.95 / 0.05 bar on "skip the coding model and act" is a safety choice. Nothing above
+justifies loosening it: there is no threshold between the current values and 0.50 that would have
+produced more correct actions than wrong ones in this run. Loosening the 0.80 / 0.20 family bar
+to 0.30 would have been a net win here only because the hedged families happened to be the unused
+ones.
+
+## The second run
+
+A second JEV run of the same prompt (`5ffd22e8`, same day, model and endpoint; yolo mode
+throughout) is under `Benchmarks/3D-Dodge/runs/2026-10-31_andorra-code_jev_glm-5.3-flash_5ffd22e8/`.
+It asked 129 questions over 20 rounds (1 preflight, 19 postflight), got 105 answers, applied none,
+and 4 rounds fell back on `decision_service_unavailable`. Its rounds cost 16.4 s.
+
+Post-tool answers over the 15 answered rounds:
+
+| Question | Mean (SD) | Min–max | Rounds ≥0.95 / ≤0.05 |
+| --- | --- | --- | --- |
+| `result_answered_question` | 0.32 (0.13) | 0.20–0.70 | 0 / 0 |
+| `needs_more_investigation` | 0.37 (0.07) | 0.13–0.43 | 0 / 0 |
+| `should_retry` | 0.52 (0.03) | 0.43–0.56 | 0 / 0 |
+| `strategy_should_change` | 0.50 (0.11) | 0.12–0.59 | 0 / 0 |
+| `ready_for_verification` | 0.29 (0.09) | 0.17–0.54 | 0 / 0 |
+| `likely_complete` | 0.21 (0.10) | 0.09–0.47 | 0 / 0 |
+
+Nothing crossed either bar. The model never called `prepare_steps`, so `next_action` was never
+asked and no turn could have been skipped. The preflight repeated the first run almost exactly
+(`needs_delegation` 0.12, `security_sensitive` 0.03, families at 0.26–0.44 apart from
+filesystem-write 0.87 and browser 0.74).
+
+Two things the second run adds:
+
+- **The answers are not stable between runs.** `strategy_should_change` averaged 0.83 in the
+  first run and 0.50 in the second, and `should_retry` 0.43 against 0.52, on trajectories that
+  differ mostly in which `qa.browser_session` operations were called. A question whose mean moves
+  by 0.33 between two near-identical runs while its standard deviation within a run is 0.05–0.11
+  is not reading the per-round evidence.
+- **The round's cost was paid synchronously.** When these runs were recorded the runtime awaited
+  the postflight round between a turn's tool results and the next model request
+  (`src/hardened-runtime.ts`, `decisionManager.postflight`). The benchmark's "Tool end → next
+  request" metric measures that window: 884 ms and 1,021 ms per turn with Jev, 237 ms and 240 ms
+  without, so about 0.8 s per turn and 16–19 s per run. That was the whole of Jev's direct cost in
+  these runs, and with the answers above it bought no decision. The runtime now awaits the round
+  only when it can act (see "When the post-tool round holds up the next turn" above).
+
+## After the non-blocking post-tool round
+
+`Benchmarks/3D-Dodge-OnePass/` ran the same game with QA forbidden, on a build that has
+[Andorra-Labs/Andorra-Code#354](https://github.com/Andorra-Labs/Andorra-Code/pull/354): one Jev
+run and one without, three requests each.
+
+| | Jev `625d4c10` | No Jev `88797a24` |
+|---|---|---|
+| Wall time | 110.4 s | 102.0 s |
+| Tool end → next request (mean) | 111 ms | 95 ms |
+| Jev rounds (preflight / post-tool) | 1 / 2, all answered | — |
+| Awaited Jev time | 817 ms (preflight) | — |
+| Decisions applied | 0 (one guidance line) | — |
+
+- **The post-tool round no longer costs wall time.** Both rounds ran while the next request was in
+  flight. The 16 ms gap difference is the cost of starting them.
+- **The preflight is now the only awaited round,** at 0.8–0.9 s per run in all three Jev runs,
+  and in none of them did it remove a tool or set a verification tier.
+- **The preflight judged a placeholder.** The prompt was pasted as an attachment, so the task
+  text, which is what the preflight receives, was "See the attached file(s)." The tool-family
+  answers (0.23–0.64) were for a task the service could not read, including the instruction not
+  to use terminals or browsers.
+- **The last post-tool round can never act.** Its answer arrived 1.1 s into the final request and
+  the run ended after it. A decisive answer would have had no request left to reach.
+- The remaining 7.6 s of the gap is the model: 3.2k more output tokens on the turn that wrote the
+  file, partly offset by a shorter final reply.
+
+## The Little Harvest run: why no post-tool directive reached the model
+
+A 25-minute Jev run of the Little Harvest prompt on 2026-11-01 (run `ef9363f2`, `glm-5.3-flash`
+on the GX10 endpoint, `--prompt --yolo --json`, 64 model requests, 60 post-tool rounds, 58 answered)
+was checked for the directives the non-blocking round is supposed to deliver. None of the 64 logged
+requests carried a `Runtime Jev decisions` message. Seven rounds had looked confident:
+`strategy_should_change` reached 0.79–0.85 and `needs_more_investigation` 0.72–0.81. The reason is
+the bar, not the delivery:
+
+- **No answer crossed 0.95.** The six post-tool booleans are decisive at ≥0.95 / ≤0.05
+  (`DecisionPolicy.boolean`). The highest answer in the run was 0.85, so `postToolGuidance` returned
+  an empty directive for all 58 rounds and there was nothing to carry. This is the threshold this
+  page recommends keeping: in the first 3D-Dodge run `strategy_should_change` sat at 0.83 across 18
+  of 21 rounds of a run that succeeded, and a 0.80 bar would have injected "change strategy" into 17
+  prompts wrongly.
+- **The late-arrival path would have carried a decisive answer.** Each of the seven answers arrived
+  0.66–1.01 s into the request built right after its tool batch, and the request after that was
+  built 2.3–12.4 s later, so the answer would have been applied at the top of that iteration and
+  carried by it. The following round answers about a second after that request is built, too late
+  to overwrite the pending answer. The run never called `prepare_steps`, so every round took the
+  non-blocking path. `test/decision-post-tool.test.ts` ("a late post-tool answer is carried by
+  exactly one request…") pins both halves: 0.85 yields no directive, 0.96 yields one directive on
+  the request after the answer and none on the one after that.
+- **The trajectory log is the request.** `src/trajectory.ts` records the `messages` array the
+  runtime passes to the provider (the clone taken at the top of the iteration), and the ephemeral
+  todo snapshot appears in every logged request, so a trailing directive would have been logged too.
+
+What the run did show is the preflight acting against the prompt. The prompt said "Do not create a
+development plan, write a QA report, run automated tests, or produce placeholder implementations."
+The preflight scored `task_complexity` 0.9725 at confidence 0.92, which `decisionExecutionGuidance`
+(`src/decision/toolFamilies.ts`) turns into two system-prompt directives regardless of the chosen
+`verification_tier` (here `targeted` at 0.15): "Run the affected full test suite, build, and
+applicable static checks" and "Break this complex task into explicit milestones and verify
+integration between changed components." The no-Jev sibling run of the same prompt (`bf4dc7b3`) had
+no such block. The block carries the caveat "never overrides explicit user instructions", which
+leaves the contradiction for the coding model to resolve on every request. Both runs still ended with
+a comparable number of browser QA calls (27 with Jev, 33 without), so this run does not show the
+directive changing behavior, but it is the one place where Jev added instructions the user had
+forbidden. Filtering preflight directives against explicit prohibitions in the task before they are
+injected is a harness-side follow-up (see below), not a threshold question.
+
+### A paired A/B on small tasks, same day
+
+To see whether Jev reduces coding-model work at all, three small tasks were run with `jevEnabled`
+on and off from the same signed-in home, on the same provider (GX10 `glm-5.3-flash`), with the
+shipped defaults (no `efficiency` switches), each from a fresh copy of its seed folder, order
+alternating per repetition: a greenfield Node CLI with a `node:test` suite, a seeded bugfix with a
+failing suite, and a read-only investigation of a seeded HTTP service. Every run passed an executable
+check of its result in both arms. One no-Jev bugfix run crashed at startup before any model request
+(a native-module access violation, not Jev) and its pair was replaced.
+
+| Pair | Requests Jev / none | Input tokens Jev / none | Directives reaching the model |
+| --- | --- | --- | --- |
+| greenfield 1 | 7 / 6 | 142,710 / 80,438 | 1 (`result_answered_question`, on the final request) |
+| greenfield 2 | 9 / 7 | 143,793 / 131,237 | 0 |
+| bugfix 1 | 8 / 9 | 125,692 / 154,582 | 0 |
+| bugfix 2 | 7 / 8 | 90,287 / 103,677 | 0 |
+| investigate 1 | 7 / 5 | 107,325 / 67,272 | 4 (`needs_more_investigation`, requests 4–7) |
+| investigate 2 | 7 / 8 | 94,654 / 133,550 | 2 (`needs_more_investigation`) |
+| **Total** | **45 / 43** | **704,461 / 670,756** | 7 of 45 requests |
+
+Three pairs favor each arm; the net is +2 requests and +5 % input tokens with Jev, and the spread
+within one arm (greenfield without Jev: 80k and 131k; investigation without Jev: 67k and 134k) is
+larger than the gap between arms. Jev neither saved nor clearly cost model work on these tasks. What
+the runs show about the mechanism:
+
+- **The lever that skips a turn never engaged.** `prepare_steps` was called 0 times in all seven
+  Jev runs (and 0 times in the 64-turn Little Harvest run), so `next_action` was never asked and
+  `decision.turn_bypassed` is 0 everywhere. The switches that prepare steps without the model's help
+  (`efficiency.preparedContinuations`, `postflightOnlyWithCandidates`) are off by default and were
+  not measured.
+- **The directives that did fire asked for more work.** Six post-tool answers crossed 0.95 in 37
+  rounds, five of them `needs_more_investigation` on the investigation task. "Gather the missing
+  evidence before completing" sat on four consecutive requests of the first pair, and the model
+  answered it with two extra search turns (`Grep`, `Grep`, `git_read`) before completing where the
+  no-Jev run completed two requests earlier. Of the six directives, five add work; the two that
+  could end work early (`result_answered_question`, `likely_complete`) crossed the bar once, on a
+  final request, where nothing was left to save.
+- **The directive stays until a newer answer replaces it.** Consecutive rounds re-answering ≥0.95
+  keep the same line on every request, so a directive that was right once keeps nudging after the
+  model has acted on it.
+- **Fixed cost per request.** The `prepare_steps` schema (about 1,650 serialized characters) and the
+  preflight directive block (about 480) ride every request with Jev on, roughly 2,100 characters,
+  almost all of it cache reads on this provider.
+- **The preflight is consistent.** `task_complexity` repeated within ±0.02 per task across
+  repetitions (0.25 bugfix, 0.26 greenfield, 0.63 investigation), and `verification_tier` was
+  `full-tests` at confidence 1.0 for the bugfix, whose prompt already asked for the suite.
+
+## Jev availability: the run that silently had no Jev
+
+While re-running the A/B of 2026-11-01 on the installed build, 7 of 12 Jev-arm runs had no decision
+round at all, and nothing in the run log or on stderr said why. The run's membership lookup
+(`getMembership`, `src/membership.ts`) had answered "not signed in" on a signed-in account:
+`AuthManager.verify` caught every failure of `jwtVerify` as a rejected credential, including a failed
+fetch of the JSON Web Key Set, which every fresh process performs on its first verification. Twenty-five
+consecutive verifications inside one process and thirty consecutive `/me` calls all succeeded, so the
+failure is transient and per process. Two changes:
+
+- `verify` reports a key-set fetch failure (jose's timeout or non-200 error, a failed fetch, a transport
+  error anywhere in the cause chain) as `network`, which the membership client already treats as
+  `unavailable`. A signature, issuer, audience or claim failure is still `invalid_token`.
+- The CLI retries an unavailable lookup once (the lookup itself has a five-second budget, and a cold
+  account service answered in 5.9 s once and in about 0.5 s afterwards) and, when Jev still cannot run,
+  records `decision.unavailable` (`reason`: `not_signed_in`, `membership_unavailable` or the transport
+  failure reason) in the run log and prints one notice. A run without Jev is now visible as such, and
+  the eval suite (`eval/`) excludes it from a Jev arm instead of averaging it in.
+
+The CLI also takes `--jev on|off` as a session-only policy, so an A/B needs no configuration edits.
+## Recommended changes
+
+### Service side (do these first)
+
+1. **Calibrate the six post-tool booleans against outcome labels.** The benchmark trajectories
+   give labels for free: a judged call that failed and was then retried with different arguments
+   (`should_retry` yes), a successful call followed by the same tool again (`strategy_should_change`
+   no), the round before an accepted completion (`likely_complete` yes). Target: on these six
+   questions, answers after a failed round and after a successful round should differ by more
+   than the current 0.04-0.14, and `strategy_should_change` must not sit at 0.83 while the
+   approach is working.
+2. **Be decisive on `next_action` when the evidence is in the snapshot.** The candidate
+   description said "after index.html is written: get QA profile", and the judged result was the
+   `Write` of `index.html`. 0.72 on that is a hedge.
+3. **Put the tool-family answers for a greenfield task where they belong.** "Create one HTML
+   file" with no repository facts should still put git and codegraph under 0.20.
+4. **Availability.** 5 of 26 post-tool rounds (19%) returned `decision_service_unavailable` in
+   a six-minute run. The lost round included the verification verdict.
+
+### Harness side (no threshold change)
+
+These are follow-ups, not part of this analysis, and the first needs the service to accept the
+fields before the harness sends them:
+
+1. **Send what the gates already require.** The post-tool snapshot should carry a bounded todo
+   summary (counts by status), since the completion shortcut requires every todo to be completed
+   and the service cannot see them. The preflight snapshot should carry workspace facts (whether
+   the folder is empty, whether it is a git repository, a file-count bucket) so the family
+   questions are not guesses.
+2. **Count applied decisions honestly.** Emit a `decision.applied` (or sibling) event when
+   `routeToolFamilies` omits schemas and when `decisionExecutionGuidance` or `postToolGuidance`
+   adds a directive, so a benchmark's "decisions applied" matches what reached the model. Log
+   whether `next_action` chose `llm` (a fixed identifier) so a missed short-circuit can be
+   evaluated after the fact without exposing model-authored step IDs.
+3. **Record the HTTP status class on `decision.provider_error`.** Every non-401, non-429 failure
+   is reported as `decision_service_unavailable`, which hides whether the five failures were 5xx,
+   4xx rejections of the snapshot, or network errors.
+4. **Do not await a postflight round that cannot act.** Done: without prepared-step candidates the
+   round's only possible effect is a directive for the following prompt, which is applied when
+   it arrives. Awaiting it only when candidates exist keeps the short-circuit intact and removes
+   about 0.8 s per model turn from every run where the model does not call `prepare_steps`, which
+   in these two runs was 46 of 47 turns. See "When the post-tool round holds up the next turn"
+   and `tests/jev-postflight-nonblocking-test-plan.md`.
+5. **Give the preflight the attachment text.** When the task text is the attachment-only
+   placeholder, the preflight's task should carry the text of small text attachments (or a
+   bounded excerpt), since that is where the request is.
+6. **Take the preflight off the critical path.** It is the only awaited round left. Starting it
+   before the graph starts would hide it, if its inputs can be assembled that early. Applying a
+   late answer to a later request is not an option for tool routing, because changing the tool
+   list mid-run discards the cached prompt prefix.
+7. **Filter preflight directives against the task's explicit prohibitions.** The Little Harvest run
+   injected "run the full test suite" and "break the task into milestones" into a prompt that
+   forbade automated tests and a development plan. The caveat line leaves the contradiction to the
+   coding model on every request; dropping the optional directive before injection, and recording
+   that it was dropped, keeps Jev advisory.
+8. **Make the "stop" signals able to save a turn, and the "more" signals less sticky.** In the A/B
+   the only decisive post-tool answers asked for more investigation and persisted across requests,
+   while `result_answered_question` and `likely_complete` fired once, too late. A directive that has
+   been carried by one request could be withheld until a later round re-crosses the bar on new
+   evidence, and a decisive `likely_complete` without a prepared `complete` candidate does nothing
+   today.
+
+### When a threshold change would be justified
+
+Revisit the bars only with a labeled set from several runs where, for the question in hand, the
+answers after the labeled-yes rounds and after the labeled-no rounds are separated by at least
+0.30 on average. Until the service's answers discriminate, a lower bar changes how often Jev acts,
+not how often it is right.