@andorra-labs/code 0.0.0-stage → 0.1.5

package/tests/chat-side-panel-test-plan.md

@@ -0,0 +1,309 @@
+# Chat side panel — QA test plan
+
+## Scope and prerequisites
+
+**Purpose.** The interactive terminal draws a status panel to the right of the conversation. The
+panel is a second column of the same frame: the chat keeps its own layout at the reduced width and
+the panel is composed beside it. It reports, top to bottom:
+
+- **Git tools** — the working tree's `+insertions −deletions`, the current branch, and how many
+  tracked files are changed plus how many paths are new (untracked).
+- **Node** — the graph node the run is on (the node the token-speed readout follows), with its
+  loop/copy detail and the provider/model that node actually resolved; when idle it shows the
+  selected graph's entry node.
+- **Progress** — a completion bar and `done · active · waiting` counts for the run's plan.
+- **Plan** — the items in progress, the completed count, and the waiting items. The list fills
+  every row the other sections leave; only a plan longer than that is cut, with a closing line
+  counting what was left out. When there is no room at all, the whole section stands down.
+- **Terminals** — the background terminal sessions the agent opened through `qa.terminal`, with
+  how long each running one has been alive.
+- **Graph** — a zoomed neighbourhood of the current node: what feeds it, the node itself, and what
+  it hands off to, with each node marked visited (`✓`), current (`●`) or not yet reached (`○`), and
+  each hand-off labelled with its edge outcome. This section only exists for a **custom** graph;
+  the built-in Software Engineer harness has nothing to zoom.
+
+While the panel is drawn, the composer's own todo block is not: the plan is reported once, in the
+panel.
+
+**Affected workflows.** Interactive TUI sessions (`andorra` with no `--prompt`) on a terminal that
+is at least 110 columns wide and 14 rows tall. Headless (`--prompt`, `--json`), the graph studio,
+and the desktop client are not affected.
+
+**Not affected.** All keyboard and mouse behavior: Ctrl+C, Ctrl+D, Ctrl+T, Ctrl+O, Ctrl+Y, Ctrl+B
+(dashboard), Esc, Alt+Enter, the wheel (SGR reports), and drag-select at rest work exactly as
+before. The panel takes no input; it is display only.
+
+**Setup.**
+
+1. Build from this branch: `npm ci && npm run build`.
+2. Run the CLI in a real terminal: `node dist/src/cli.js` (or the linked `andorra`). A terminal at
+   least 110 columns wide is required to see the panel; 150–200 columns is comfortable.
+3. For a custom-graph run, open `/graph` → **Create new graph** (web builder), save it, then
+   reopen `/graph` and select it. Any saved custom graph works; a three-or-more node graph makes
+   the zoom section meaningful.
+4. For the Terminals section, the run must spawn a background terminal through `qa.terminal`
+   (a QA-capable graph, e.g. one with the QA tool family enabled and shell permission). A manual
+   smoke alternative: in a scratch checkout, run the QA terminal tool through a graph run.
+5. For Git tools, run in a Git working tree with at least one modified file (`git status` shows
+   a change) — e.g. edit a file in a feature worktree.
+
+**Reset/cleanup.** No persistent state is added. Two panel values deliberately outlive a single
+task and have to be cleared per case:
+
+- **The graph trail** clears itself when the next run starts (`graph.started`) or when the
+  selected graph changes. `/clear` does **not** reset it, and neither does switching sessions.
+  To reset the trail between cases without starting a run, select a different graph and switch
+  back (`/graph`), or restart the CLI.
+- **Terminals** persist until they exit or are stopped; the QA PTY manager keeps a naturally
+  exited session in its list until it is explicitly killed. Stop sessions between cases with the
+  `qa.terminal` tool's `kill` action (or restart the CLI, which stops them at teardown).
+
+`/clear` clears the conversation and the plan, which is what its own documentation promises. The
+panel's Git reading follows the working tree: restore tracked edits with `git checkout -- .` and
+remove the fixture files TC-4 creates explicitly (`rm brand-new.ts`, or `git clean -f -- <path>`
+for the exact path) — `git checkout` leaves untracked files in place, so a later clean-tree case
+would keep reporting an untracked entry.
+
+## How to use the feature
+
+1. Open the CLI in a terminal ≥ 110 columns wide. The panel is already there: `Git tools`, `Node`,
+   `Progress` (and `Plan` once the agent publishes one) on the left of the divider, with
+   `Terminals`/`Graph` at the bottom when they apply.
+2. Submit a task. During the run:
+   - `Node` names the node that is working, with `loop i/n` and `copy i/n` when the graph runs
+     copies, and the provider/model that node resolved (not the session's own selection).
+   - `Plan`/`Progress` follow the todo list the agent publishes (`Ctrl+T` state is irrelevant while
+     the panel is drawn; the panel is the copy on screen).
+   - `Git tools` refreshes at run start, at run end, and after each tool completion (throttled to
+     one `git` read every two seconds).
+   - `Terminals` appears when the run opens a `qa.terminal` session and ages each running session.
+   - `Graph` zooms on the node the run is on; `✓` spreads as nodes complete.
+3. When the run ends, the zoom keeps the trail so the reader can see where the run went. Starting
+   the next run resets the trail.
+4. Narrow the terminal below 110 columns (or below 14 rows) and the panel disappears, giving the
+   full width back to the chat and returning the composer's todo block.
+5. Ctrl+B (dashboard) and any modal screen (a selection list, the setup flow, `/providers`) take
+   the whole frame; the panel returns when the transcript does.
+
+## Test cases and expected results
+
+| # | Case | Steps | Expected |
+|---|------|-------|----------|
+| TC-1 | Panel appears on a wide terminal | Run `andorra` in a 150-column terminal | Panel is drawn right of the chat: `Git tools` heading with `Changes`, a `⎇ <branch>` row and a file count, `Node`, `Progress`. Divider column is unbroken from row 1 to the last row. No row exceeds the terminal width and no row wraps |
+| TC-2 | Panel hides on a narrow terminal | Resize the terminal to 100 columns, then to 109, then back to 150 | At < 110 columns the panel and the divider disappear, the chat and composer expand to the full width, and the composer's `Todo … ctrl+t to expand` block returns. At 150 it returns |
+| TC-3 | Panel hides for other surfaces | Press Ctrl+B (dashboard), then Esc back; open `/providers` (a modal screen), then Esc | Neither the dashboard nor the modal shows the panel or the divider; the dashboard's own cards and the modal use the full width |
+| TC-4 | Git tools reports the tree | In a clean tree, edit a few lines of one tracked file, then create a new file and leave it **untracked** (do not `git add` it); start a run or wait for a tool completion. Clean up by deleting the new file explicitly and `git checkout -- .` | `Changes` shows the tracked file's `+N −M` coloured added/deleted; the branch row matches `git branch --show-current`; the third row reads `1 file · 1 untracked`. A clean tree reads `Changes clean`; a non-repository workspace reads `Not a Git repository`. A *staged* new file is counted under changed files instead (`2 files`), not under `untracked` |
+| TC-5 | Git tools refreshes during a run | Ask for a task that edits a file (e.g. "add a comment to README.md") | Within a few seconds of the tool completing, `Changes` moves without a manual refresh. The count is bounded: repeated edits do not produce a `git` process per edit (one read per two seconds at most) |
+| TC-6 | Node follows the run | Run a custom graph and watch the Node section across node transitions | The section names the current node, updates as nodes change, shows `loop i/n` for a loop node and `copy i/n` for parallel copies, and shows the resolved provider/model of that node (a node-level override appears here, not the session's model) |
+| TC-7 | Node when idle | Let the run finish; then select the default graph and look again | Idle shows `Idle` plus `entry: <entry node name>` of the selected graph |
+| TC-8 | Plan and Progress track the todos | Run a task that publishes a plan | `Progress` shows a bar, a `N%`, and `done · active · waiting`; `Plan` shows `done/total` in its heading, the in-progress item first, the completed count, then the waiting items; `… N waiting` appears only when the plan has more items than rows |
+| TC-9 | One plan, one place | With the panel visible, compare the composer area and the panel | The composer's todo block is absent; the plan text appears exactly once on screen, inside the panel |
+| TC-10 | Terminals section | Let a run open a `qa.terminal` session (or leave one running from a QA smoke) | A `Terminals` section lists each session as `● <id> · <command> · <age>` while running, `○ <id> · <command>` once exited, and `+N more` beyond three. The age advances about once per second while the panel is on screen |
+| TC-11 | Graph zoom for a custom graph | Run a custom graph with ≥ 3 nodes | A `Graph` section sits at the bottom of the panel, headed `Graph n/m`; it shows the current node as `●`, incoming nodes above as `▲ <mark> <name>`, outgoing nodes below as `├─▶`/`└─▶ <mark> <name> · <outcome>`. Marks move from `○` to `✓` as nodes complete |
+| TC-12 | No zoom for the built-in harness | Select the default graph and run | No `Graph` section (the footer's `Graph: Default` row is not part of the panel); `Git tools`, `Node`, `Plan`/`Progress` remain |
+| TC-13 | Trail after the run | Finish a custom-graph run and keep looking at the panel | The zoom still shows the last node as `●` with the visited nodes `✓` and the visited count; starting the next run clears the trail |
+| TC-14 | Panel content never overflows | Resize a live session through 200×60, 160×40, 120×30, 110×24, 109×24, 100×20, 80×24 | Every frame fits its terminal: no row is wider than the terminal and no frame has more rows than the terminal has; the panel appears only at ≥ 110 columns and ≥ 14 rows |
+| TC-15 | Short-terminal ordering | With a plan, a background terminal and a custom graph, shrink the terminal height to 14 rows | The zoom is dropped first, then the terminal list; `Git tools`, `Node` and `Progress` stay. No partially drawn section (no heading without its rows, no rule at the last row) |
+| TC-16 | Interaction regression | With the panel visible: type a draft, press Ctrl+C, Ctrl+O, Ctrl+Y, Page Up/Down, wheel, Ctrl+B, and submit a task | Every key behaves as documented in the existing plans (`tui-text-selection-at-rest-test-plan.md`, `tui-wheel-scrolls-panel-test-plan.md`, `ctrl-c-clear-input-test-plan.md`, `transcript-markdown-test-plan.md`); the panel never intercepts input and the caret stays in the composer |
+| TC-17 | Frame rewrites stay local | During a run, watch a `git diff`-heavy transcript while the panel ages a terminal | Only the changed rows are repainted (the same behavior the frame-diff tests cover); a drag selection over transcript rows is not broken by the panel's per-second age update |
+
+## Edge cases and potential bugs
+
+- **Exactly at the boundary.** 109 columns → no panel; 110 columns → panel with a 76-column chat
+  column (110 minus the panel's 33 columns minus the one-column gutter). 13 rows → no panel; 14 rows → panel. Short terminals must never draw a partial panel.
+- **No Git / no HEAD.** A non-repository workspace shows `Not a Git repository`; a repository with
+  no commits yet counts files but reports `+0 −0` because there is no `HEAD` to diff against. The
+  untracked count is in git's own entries, so a brand-new directory holding ten files is one
+  `1 untracked` entry; untracked content is never read, which is why it contributes no line
+  counts - a `+0` line total for a new file is expected, not a bug.
+- **Existing panels' data races.** The git summary refresh is fire-and-forget and throttled: a
+  request inside the two-second window is scheduled for the rest of it, and one that lands while a
+  read is in flight is queued and starts (or is scheduled) when that read settles. A reading can
+  therefore be up to two seconds old, and the last event before a quiet stretch is still reflected
+  rather than dropped. That is the documented bound.
+- **The read must not lock the workspace.** The summary selects the repository with `git -C` and
+  never makes the workspace the git child's working directory: on Windows a directory that a live
+  process has as its cwd cannot be removed, and a display-only read that outlived a run would make
+  an embedder (a test fixture, an editor session) fail to delete the workspace with `EBUSY`. The
+  interactive CLI suites (`cli-graph-health`, `cli-preflight-cancellation`) drive `main()` against
+  temporary workspaces and are the regression coverage for this.
+- **The read never runs a workspace-controlled program.** Display-only Git resolves `git` to an
+  absolute path from a PATH with every entry inside the workspace **and inside the repository that
+  contains it** removed (`trustedGitEnv`), because a launcher that prepends the repository's
+  `node_modules/.bin` would otherwise turn a file named `git` into code that runs on a panel
+  refresh, with no tool call and no approval - and a workspace nested inside a repository is
+  counted through `git -C`, so that repository's `bin` directories are controlled by the same
+  author. Both sides of the containment check are canonicalized (symlinks and junctions resolved
+  through the longest existing prefix), so a PATH entry or a `git` file that reaches the workspace
+  through a link is excluded too. `gitBranch` reaches the same guarantee by reading Git metadata
+  directly. A workspace that genuinely hosts the user's Git loses the summary rather than running
+  it.
+- **A read that cannot finish keeps the last reading.** Git missing, the timeout, the abort signal,
+  and output past the cap all fail the read (the CLI's `panelReads` controller is aborted at
+  teardown, so closing the terminal is never held up by one); only a directory that genuinely has
+  no repository publishes `isGit: false`. A failure leaves the previous panel reading in place.
+- **Terminals that exit on their own are noticed.** While a displayed session is running the CLI
+  re-lists the QA manager every second, so an exited session stops showing as running and stops
+  its age. The watch stops as soon as no displayed session is running, and also when the panel
+  stops being drawn (a narrowed terminal, the dashboard, a modal) - the `panel` event re-arms it
+  when the panel comes back.
+- **Every refresh is either read now or scheduled.** A tool boundary inside the two-second window
+  queues the read (`scheduleChanges` arms the remainder of the window) and a boundary that lands
+  while a read is in flight re-runs it as soon as that read settles, so the last tool completion
+  of a quiet task still reaches the panel and the run finalizer cannot leave a stale reading.
+- **The panel's git read is invisible to the user's own Git.** It runs with
+  `--no-optional-locks` and `GIT_OPTIONAL_LOCKS=0`, so a status read cannot take the index lock a
+  concurrent `git add` or `git commit` needs - the read starts exactly when a tool is about to
+  write. The executable is resolved to an absolute path from a PATH that excludes the workspace,
+  so neither the filtered PATH nor the child's working directory can reach a repository-supplied
+  `git`.
+- **The panel's rows are anchored.** Git tools, Node, Progress and the plan read from the top; the
+  terminal list and the graph zoom are held against the bottom of the frame with the unclaimed
+  rows left blank between them, so the zoom is at the bottom on a tall terminal instead of
+  sitting under the plan.
+- **The panel is display only for the mouse too.** A left click outside the chat column is
+  ignored, so clicking the panel on a row shared with the reasoning preview does not toggle it.
+- **Panel values are flattened before they are painted.** Node names, terminal commands, plan
+  items and branch names are stripped of control characters and newlines first, so a value cannot
+  escape its row or drive the terminal.
+- **A repository that configures a Git filter driver gets no counts.** `git status` and
+  `git diff` compare the working tree through the filter a `.gitattributes` entry selects, and
+  that filter's `clean` command is a program from the repository's own configuration: refreshing
+  the panel would run it with no tool call and no approval, and no flag turns the conversion off.
+  Git LFS and git-crypt are the legitimate users of that mechanism, so the read stands down
+  (the branch still comes from refs) and the section says `Changes no counts` with
+  `Skipped: a Git filter applies to tracked files` - never `clean`, which is what a zero-count
+  reading would otherwise look like. Both halves of the mechanism have to be in place - a
+  configured `filter.<driver>` entry (any scope, including `include`/`includeIf`) **and** a
+  `.gitattributes` entry selecting it for a tracked path - so a machine that merely has a driver
+  installed (a global `filter.lfs.*`) keeps its counts. Every driver the probe saw is then disarmed
+  in the status and diff commands themselves (`-c filter.<driver>.clean=` and its smudge and
+  process siblings), so an attribute written between the probe and those commands has no command
+  left to select, and a driver whose name contains dots is disarmed under its full name
+  (`filter.foo.bar.clean` belongs to `foo.bar`, not `foo`). The probe runs again on every read,
+  because a tool can add or change that attribute mid-run and a remembered "nothing is filtered"
+  would be the one stale answer that lets the command run; only a refusal is remembered (removing
+  a filter can only make the refusal unnecessary). A probe that cannot be completed (a corrupt
+  index, more tracked paths than the scan limit) refuses the read the same way any other failed
+  read does, and a failure with no earlier success to fall back on renders
+  `Changes no counts` with `Unavailable: the working tree could not be read` instead of leaving the
+  A submodule is the same hazard in a second shape and is handled by not looking inside it: an
+  initialized submodule configures drivers of its own, selected by its own `.gitattributes`, and the
+  probe reads the superproject only. Both comparisons therefore run with `--ignore-submodules=dirty`.
+  The trade is deliberate: a submodule whose recorded commit changed is still reported, while
+  uncommitted content inside a submodule is not counted, because counting it is the inspection that
+  would run that submodule's driver. (Reproduction was attempted on Git 2.55 with `submoduleSummary`
+  and `diff.submodule` amplifiers and executed no driver there, so this is a defensive bound rather
+  than a reproduced fix.)
+- **A PATH entry whose `git` cannot be executed is skipped.** The shell's own lookup skips a file it
+  cannot execute and keeps searching, so the candidate is checked for that before it is accepted: a
+  plain file named `git` earlier in PATH would otherwise fail the spawn and make every refresh report
+  an unreadable tree while the user's own git worked. The check is POSIX's execute bit; on Windows a
+  `.exe` is executable by name, which is what the check reports there too.
+- **A stopped run keeps its position.** `graph.stopped` names the node execution stopped on;
+  that node becomes the zoom's focus (amber `●`) without joining the visited count, because it
+  never completed.
+- **Node with several active nodes.** A parallel run names the followed node and lists the others
+  on one line (`+N more running: …`). The Node section's height is fixed, so the panel cannot grow
+  past the terminal because of it.
+- **A hub node in the zoom.** Predecessors are capped at two and hand-offs at three, with
+  `+N more upstream` and `+N more` for the remainders; a node with many edges must not push the
+  other sections away. Hand-offs are counted by route, not by target, so a node that reaches one
+  End through `pass` and `fail` shows both, and a self-targeting retry edge is drawn with a loop
+  marker (`├─↻` / `└─↻`) rather than dropped.
+- **Very long names.** Node names, branches, commands, plan items and outcomes are clipped to the
+  panel width; nothing wraps onto an unaligned row.
+- **Terminal closed mid-run.** Sessions are tied to the run's signal; a run cancelled with Ctrl+C
+  ends its background terminals, and the section disappears (possibly after one frame showing them
+  as exited).
+- **Suspected (not reproduced).** On a very slow filesystem the throttled `git` read could overlap a
+  run's own `git` commands; both are read-only, and the summary read carries its own timeout.
+- **Not verified on macOS/Linux by automation.** The layout is pure string arithmetic and the tests
+  cover it, but manual QA on a real terminal of each platform is still pending (see Gaps).
+
+- **Nothing is read while the panel cannot be seen.** On a terminal narrower than 110 columns,
+  shorter than 14 rows, or showing the dashboard or a modal screen, the CLI skips the terminal
+  list, the branch read and the git summary entirely - including the filter probe's path scan -
+  and reads them again when a resize or a closed screen makes the panel visible (the terminal
+  announces that transition).
+
+## Gaps and limitations
+
+- Headless runs (`--prompt`, `--json`) and the desktop client never draw the panel; `--json` output
+  is unchanged.
+- The Terminals section lists only `qa.terminal` PTY sessions. A long-running shell command executed
+  through the ordinary Bash/PowerShell tool is not a managed session and does not appear.
+- The Git section is a summary: no per-file list, no staged/unstaged split, no commit or push
+  action. It never opens file contents (unlike the QA change detector).
+- No known defects at the time of writing.
+
+## Validation evidence
+
+Automated (this branch, Windows 11, Node v24.13.1):
+
+- `npm run build` — clean; `npx tsc --noEmit` — clean.
+- `node --test dist/test/side-panel.test.js` — 25/25 pass (section content, colors, packing,
+  duration formatting, running-first terminal ordering, the reserved age, the idle-entry marker,
+  bottom anchoring, values flattened against control characters, and the width bound at six panel
+  widths).
+- `node --test dist/test/git-summary.test.js` — 18/18 pass against real temporary repositories
+  (clean tree, tracked edits, untracked counting, staged+unstaged, rename, non-repository, nested
+  workspace, detached HEAD, unborn HEAD, failed reads rejecting, an unreadable repository
+  rejecting, a workspace-only PATH refused, and the trusted environment).
+- `node --test dist/test/cli-graph-health.test.js` — 11/11 pass;
+  `dist/test/cli-preflight-cancellation.test.js` — 4/4 pass. Both failed 5/11 and 0/4 before the
+  `git -C` fix, with `EBUSY: resource busy or locked, rmdir '…\workspace'` in their teardown; the
+  read is what held the lock. These are the regression coverage for that constraint.
+- `node --test dist/test/terminal-ui.test.js` — 176/176 pass (panel live state, hiding on a narrow
+  terminal/dashboard/modal, the zoom's custom-graph-only rule and trail retention, a frame-bounds
+  sweep, caret navigation and echoed-prompt framing at the chat width, the trail reset on a graph
+  change, a stopped run's node kept as the focus, and a panel click that must not toggle the
+  reasoning preview).
+- End-to-end smoke of the real CLI: a scratch Git repository (one modified file, one untracked
+  file, branch `feature/e2e-panel`) was rendered in a 160x40 terminal-shaped stdout, the composed
+  screen captured, and the CLI driven to exit. The panel painted
+  `Changes +2 -1`, `⎇ feature/e2e-panel`, `1 file · 2 untracked`, `Node Idle / entry: Software Engineer`
+  and `Progress / No plan published`, with no zoom section for the default graph — the numbers
+  match `git diff HEAD --numstat` for that repository.
+- Full suite swept per file through a throwaway `ANDORRACODE_HOME` (the suite-wide run hangs on
+  this machine, so each file runs in its own process): see the PR's Validation section for the
+  final tally. Five files fail identically on the pristine main checkout (`~/andorra-install`,
+  5f6d392) on this machine — `activity-ui`, `anthropic-stream-boundary`, `cli-browser-review`,
+  `cli-context-persistence` and `cli-live-control` — and are unrelated to this branch; none of
+  them imports the panel, the git summary or the terminal manager.
+
+Manual (proposed for QA):
+
+- TC-1 to TC-17 on Windows Terminal (150×40), plus a macOS Terminal.app pass for the box-drawing
+  glyphs (`● ○ ✓ ▲ ├ └ ▶ ─ │ █ ░`) and one Linux (GNOME Terminal) pass.
+- A real custom-graph run with parallel copies to exercise TC-6 and TC-11 together.
+
+## Cases added by the review round on the panel's own details
+
+| ID | Preconditions | Steps | Expected result |
+| --- | --- | --- | --- |
+| TC-18 | Parallel copies of one node resolving different profiles (one falls back to another effort or model) | Watch the footer's model row and the token rate while the readout follows the copy that stayed on the primary | The row names the followed copy's provider and model *and* that copy's own reasoning effort and context window; the rate stays visible, because it was measured under that copy's resolution — a sibling's effort must not be compared against it. Switch focus to the sibling and the row follows it instead |
+| TC-19 | A terminal wide enough for the side panel | Submit a task from the Ctrl+B dashboard, then press Ctrl+B back to the transcript | The echoed prompt box is framed for the transcript column, not the full width the dashboard was using: its border is intact (a complete `╭───╮` row) and exactly as wide as a prompt echoed with the transcript on screen |
+| TC-20 | A plan with three items `in_progress` and three `pending`, in a terminal short enough that the Plan section has three body rows (about 21 rows with a custom graph) | Look at the Plan section with the panel visible | Two active items are listed and the closing line names what the budget left out, active ones included (`… 1 more active · 3 waiting`), so the count the Progress row reports is never contradicted by silence. Grow the terminal and all six items are listed with no closing line |
+| TC-21 | A large plan with a small completed share (1 of 100) | Look at the Progress bar | A nonzero share fills at least one cell: the bar is never empty beside a percentage that says otherwise |
+| TC-22 | A plan of one `in_progress` and six `pending` items; terminal 140x40 with the default graph | Look at the Plan section with the panel visible, then shrink the terminal to 19 rows | At 40 rows all seven items are listed and there is no `… N waiting` line (the list used to stop at two waiting items and print `… 4 waiting` above empty rows). At 19 rows the list fills the rows it has and closes with the count of the rest (`● Inspect…`, three `○` items, `… 3 waiting`) |
+| TC-23 | A plan of one `completed` and four `in_progress` items, in a terminal short enough that the Plan section has four body rows (about 22 rows with a custom graph) | Look at the Plan section, then shrink by two rows | At four body rows: `● First`, `● Second`, `✓ 1 completed`, `… 2 more active` — the completed count never takes the cut line's row. At two body rows: `● First`, `… 3 more active`; the completed row gives way, since the heading still reads `1/5` |
+
+Validation for these four (run on Windows 11, Node v24.13.1):
+
+- `node --test dist/test/terminal-ui.test.js` — 178/178, including the two added cases (the
+  per-copy footer identity with its own effort, and the dashboard echo's frame).
+- `node --test dist/test/side-panel.test.js` — 27/27, including the nonzero-share bar and the
+  omitted-active line (which is clipped like every other row and keeps the width sweep green).
+
+Validation for TC-20 and TC-22 after the plan list stopped capping at two waiting items (Windows 11,
+Node v24.13.1):
+
+- `node scripts/run-tests.mjs dist/test/side-panel.test.js` — 29/29, including the new
+  whole-plan case, the reworked budget cases, and TC-23's completed-count case.
+- `node scripts/run-tests.mjs dist/test/terminal-ui.test.js` — 203/203.
+- Run check: the built CLI (`node dist/src/cli.js`) in a 140-column node-pty session with a
+  throwaway `ANDORRACODE_HOME` and a local fake OpenAI-compatible server on an OS-assigned port
+  that answers the first agent turn with an `update_todos` call of seven items. At 40 rows the
+  panel listed all seven items with no closing line; at 19 rows it listed four and closed with
+  `… 3 waiting`.

package/tests/decision-engine-test-plan.md

@@ -0,0 +1,163 @@
+# Hosted Jev and embedded decision-provider QA plan
+
+The CLI selects hosted Jev (`provider: andorra`, `mode: active`) by default for currently verified signed-in members when no explicit `decisions` configuration is stored. Runtime settings also select Local / self-hosted or Off; follow the [local decision settings QA plan](local-decision-settings-test-plan.md) for URL, model, authentication and popup cases. Both free and paid users can toggle the global enable with `/jev`, `/jev on`, `/jev off`, and `/jev status`; `jevEnabled` persists and defaults to true. Global off disables hosted and local providers; global on resumes the stored provider/mode. Paid custom graphs can additionally set an Agent's `jevEnabled: false`. Global on respects that node setting. Signed-out or unavailable membership retains baseline routing for the hosted selection, while explicitly configured local services do not require hosted sign-in. Follow [the accounts QA plan](accounts-test-plan.md) for account persistence, server quotas and hosted secret configuration. Custom graphs require paid membership; Default remains a single Software Engineer.
+
+## Scope and prerequisites
+
+Verify account-managed CLI defaults, explicit local configuration, global/per-node controls, the embedded decision-provider boundary, batched Agent preflight, safe fallback, cancellation, isolated invocation state and telemetry privacy. Laya active may affect only automatic CodeGraph/Graphiti retrieval; coding-LLM work, security checks, tool exposure and completion requirements remain authoritative. Shadow predictions must never change routing or prompts. The CLI honors explicit global `decisions` selections unless global `jevEnabled: false` disables them; library managers retain explicit config and off/none defaults.
+
+Use Node.js 22 or newer and npm on a supported Andorra host. Windows PowerShell is the implementation environment; macOS/Linux verification remains pending unless recorded below. Automated Node tests use injected providers and need no Android device, ARTEMIS session, Jev account or credentials. Live hosted CLI checks require a coding-model provider, an Andorra account and an administrator-configured hosted secret. Local checks require a structured Laya/Jev decision endpoint and its model ID; an ordinary OpenAI-compatible chat endpoint is insufficient. Never use production credentials as fixture data or include them in reports.
+
+CLI fixtures that are not explicitly testing hosted Jev set `jevEnabled: false` to prevent accidental real decision requests. Embedded decision tests inject their provider/config directly.
+
+Work from the dedicated feature worktree, keep the installed CLI checkout on `main`, and verify the branch before building:
+
+```powershell
+git branch --show-current
+npm ci
+npm run check
+npm run build
+```
+
+Use the current hosted-Jev feature branch in its dedicated worktree. Building produces ignored `dist/` output. Do not switch or rebuild the user's installed checkout on the feature branch. Branch names in historical evidence below describe earlier releases.
+
+## How to use the feature
+
+1. Read [decision providers](../design/decision-engine.md) and the [local settings QA plan](local-decision-settings-test-plan.md). The CLI uses account-managed active hosted Jev by default when no explicit decision configuration is stored; an embedded library manager with no configuration still defaults to off/none.
+2. Run the targeted automated checks described below. Their injected decision and coding-model providers make the checks repeatable without network credentials.
+3. For optional CLI verification, use a disposable directory and set `ANDORRACODE_HOME` to it, remembering the previous environment value. Configure a coding-model provider through Settings > Providers or `/providers` if needed. Use environment references for credentials.
+4. Start the feature CLI in a disposable repository, sign in with `/login`, and inspect `/jev status`. For the hosted-default case, omit `decisions` and leave global Jev enabled, then submit `Explain this repository without modifying files.` Look for public `[Jev]` request/completion or safe fallback activity. Verify `decision.applied` and actual retrieval events before claiming a routing change. Default does not require an Orchestrator intake.
+5. Enter `/jev off` and run another task: no hosted or local decision request should occur. Restart the CLI and confirm the preference persists. `/jev on` lifts this global disable and resumes the stored provider/mode, so explicit off/none still stays off. A paid Agent with **Jev decisions** unchecked remains opted out. Configure local Laya active or Jev shadow through Runtime settings using the same disposable home; verify its configured URL/model is used without hosted sign-in. An explicit local selection must not silently route to hosted Jev. Direct Jev active remains unsupported and safely falls back.
+6. Exit with `/exit` or Ctrl+C. Restore the previous `ANDORRACODE_HOME` value, or remove the environment override if originally absent. Delete only the disposable directory created for this test after verifying its absolute path. Keep credentials out of reports and committed artifacts.
+
+## Test cases and expected results
+
+Each case below is a QA requirement. Execution status is recorded separately under Validation evidence; this table alone does not claim a pass.
+
+### Current hosted CLI cases
+
+| ID | Preconditions and steps | Observable expected result |
+| --- | --- | --- |
+| HJ-01 | For signed-in free and paid accounts, omit both `decisions` and global `jevEnabled`; run Default. | Current membership selects hosted active mode with a 15000ms decision deadline. One batch contains CodeGraph/memory questions only; the coding model still runs. |
+| HJ-02 | While signed out, or with membership unavailable, run Default with no explicit local selection, then repeat with local Laya active or Jev shadow. | Hosted selection makes no request and retains ordinary routing. An explicit local selection can evaluate without hosted sign-in, using its configured API/mode. No stale claim grants hosted service access. |
+| HJ-03 | Test `/jev`, `/jev off`, `/jev on`, and `/jev status` for both tiers with hosted, local and explicit off/none configuration; restart after changing the setting. | Global preference persists; off prevents all decision requests, on resumes the stored selection rather than replacing it. Explicit off/none remains off. Footer's enable indicator reports the global preference; status does not invoke a decision service. |
+| HJ-04 | Paid user disables one Agent's **Jev decisions**; compare enabled/disabled nodes with global on, then off. | Disabled node uses baseline; enabled node can evaluate only with global on. Free users cannot apply paid node overrides. |
+| HJ-05 | Store valid global `decisions` entries for off/none, local Jev shadow, Laya shadow and Laya active; configure local fixtures with distinct URL/model identifiers. Repeat with global `jevEnabled: false`. | CLI honors each stored selection: off/none makes no request, shadow evaluates without applying routes, and Laya active can apply eligible confident CodeGraph/memory decisions. Local requests use the configured URL/model without hosted sign-in and do not call hosted Jev. Global false suppresses every provider/mode. Explicit embedded managers remain independently configurable. |
+| HJ-06 | Return confident yes/no and intermediate answers with eligible CodeGraph/Graphiti fixtures. | Only confident automatic retrieval routes change. Tool permissions, graph edges, verification and coding-LLM execution are unchanged. |
+| HJ-07 | Return 401, a daily quota 429, generic throttling/service error, invalid answers, or stall beyond the hosted deadline; cancel separately. | Fixed public fallback labels distinguish supported causes; baseline remains available, late results are ignored, and secrets/raw responses are absent. No successful inference or savings is inferred from fallback. |
+| HJ-08 | Compare requested/completed/applied events with actual retrieval and the coding-model request. | Public activity is visible, application is attributed per invocation, and actual context matches the applied route. No measured token-saving claim without a separate benchmark. |
+
+### Embedded library and shadow cases
+
+DE/DP cases below exercise explicit library managers, including legacy shadow/direct-provider behavior. Their off/none expectations do not describe the signed-in CLI default.
+
+| ID | Preconditions and steps | Observable expected result |
+| --- | --- | --- |
+| DE-01 | Load a valid model config without `decisions`, then with explicit `off` / `none`. Inject a provider factory that records calls. Execute an Agent. | The factory and evaluator have zero calls; no `decision.*` events; normal Agent completion remains available. |
+| DE-02 | Configure `off` / `jev` with a sentinel environment-variable name. Execute the same fixture. | No decision provider initialization or network request; no credential-dependent behavior or decision telemetry. |
+| DE-03 | In shadow mode inject a recording provider. Enable the supported preflight features; execute one Agent invocation. | Questions sharing its state arrive in one batch, with unique IDs. No request per individual question. |
+| DE-04 | With working mock CodeGraph and memory managers, compare an off run against a shadow run predicting the opposite of the existing baseline. | The same routing/retrieval calls, exposed tools, and prompt context are used. Comparison events report disagreement without applying the prediction. |
+| DE-05 | Run shadow mode with no registered decision backend. | Existing Agent behavior continues; bounded fallback metadata identifies the unavailable provider. No HTTP transport is attempted. |
+| DE-06 | Inject a provider that never settles, using a short finite request timeout. | Execution continues after the decision deadline; timeout fallback is emitted; the supplied provider signal is aborted. A late result cannot alter routing. |
+| DE-07 | Inject providers that throw synchronously and reject asynchronously with a message containing `SENTINEL_SECRET`. | The coding task retains its existing path; provider-error/fallback metadata does not contain the exception message or sentinel. |
+| DE-08 | Return missing/extra record keys, wrong result types, invalid choice labels, invalid choice distributions, NaN, Infinity, and probabilities outside `[0,1]`. | Invalid predictions are rejected rather than trusted; the existing path remains usable. |
+| DE-09 | Return low-confidence and threshold-boundary answers. | Low-confidence answers fall back; accepted answers still affect only comparisons in shadow mode. |
+| DE-10 | Abort before evaluation and while an uncooperative provider is pending. | No delayed coding-task continuation after cancellation; provider signal is canceled; the manager does not wait for the configured timeout to handle parent cancellation. |
+| DE-11 | Run two Agent invocations concurrently with distinct node/task sentinels and reversed provider completion order. | Each batch and comparison stays paired with its own invocation. Neither state nor baseline leaks between invocations. |
+| DE-12 | Execute deterministic Action, Condition, Join, Loop, Approval, and End nodes without Agent invocation. | No semantic decision-provider calls originate from these nodes. |
+| DE-13 | Predict permission-sensitive actions and reduced verification from a shadow provider while graph/tool policy disallows those actions. | Capability policy, user approval, graph completion contracts, and run/tool budgets remain authoritative. |
+| DE-14 | Put distinct sentinels in task, node instruction, provider exception, and credentials; collect decision events. | Events contain only approved metadata and bounded prediction/comparison fields; no raw task, instruction, credentials, or exception text. |
+| DE-15 | Supply invalid modes/providers, non-boolean feature flags, and invalid timeout values. | Configuration validation rejects invalid inputs before execution with actionable errors. |
+| DE-16 | Configure `provider: "jev"`, `mode: "active"` with an injected provider, then run Jev shadow configuration. Separately run Laya active per DP-05. | Jev active emits `decision.fallback` with `active_unsupported` without calling the provider; Jev shadow evaluates without changing routing. Laya active evaluates and can emit `decision.applied` for eligible confident CodeGraph/memory decisions. |
+| DE-17 | Disable optional preflight feature flags and execute an Agent. | Disabled questions are omitted, and an empty batch does not call the provider. Existing harness features are not disabled by decision feature flags. |
+| DE-18 | Describe a permitted tool list containing only the built-in `edit_file`. | The decision capability snapshot includes `filesystem-write`. |
+| DE-19 | Use a disposable config home with shadow/jev and an explicitly missing synthetic `apiKeyEnv` configured. Run an Agent with injected memory and no decision manager, then with an injected off decision manager. Restore the environment afterward. | Both runs succeed. The first emits requested, provider_error and shadow fallback with fixed `provider_error` reason without HTTP; the second emits no decision events, preserving the injected manager. |
+| DE-20 | Run a graph Agent with a two-iteration modifier loop and injected memory returning `LOOP_MEMORY_SENTINEL`. First model output sets `ready: false`; second sets `ready: true`. Record decision snapshots and model prompts. | Snapshots carry `run.iteration` 1 and 2 with the original memory-free task. Both model prompts retain memory; only the second includes the previous result. The graph succeeds after two calls. |
+| DE-21 | Repeat DE-20 with a memory provider whose search throws. Record snapshots, memory comparison events, and `memory.activeFor(result.id)`. | Retrieval trips run memory, but both snapshots retain availability from the pre-retrieval baseline; memory comparisons report `current: true`. Agent execution still succeeds, and no memory payload enters its prompts. |
+
+## Edge cases and potential bugs
+
+- Cancellation and timeout can race with provider resolution. Check that only one terminal result is reported and late rejections do not become unhandled promise rejections.
+- A provider can mutate input objects or return values that it later mutates. Verify invocation isolation rather than relying on fixture politeness.
+- An event observer can throw. Decision telemetry must not make the Agent fail or permit routing changes.
+- Unavailable CodeGraph or memory is a normal baseline, not evidence that a prediction disabled it. Compare against the existing eligibility rules.
+- A low probability of a yes/no answer can be a confident negative prediction; inspect the policy's confidence definition when testing thresholds.
+- No retry or credential persistence behavior should be assumed. HTTP failures and rate limits fall back without retry; no decision circuit breaker is implemented.
+- Cross-platform timer scheduling can vary; use bounded waits in tests and avoid exact millisecond assertions. These risks are test targets, not reproduced defects.
+
+## Gaps and limitations
+
+- Live hosted Jev checks need the configured account service and server secret; record their results separately from local fixtures. Remote cancellation, rate limits and prediction quality are not established by mocked tests.
+- Hosted `andorra` and Laya CodeGraph/memory preflight support active routing. Direct `jev` active, tool filtering, delegation changes, verification reduction and post-tool routing remain unsupported. Shadow comparisons cannot demonstrate savings.
+- Interactive CLI and non-Windows host checks are pending unless explicitly recorded below.
+- No known reproduced defect is asserted by this plan; pending cases and risks are not pass claims.
+
+## Current hosted validation evidence
+
+- Documentation was checked against the hosted policy, graph Agent override, decision manager, provider and graph-builder implementations. `git diff --check` passed for these documentation changes.
+- The integration owner reported live run `70c15b4b-5509-461c-89bc-51e5845d63e4` after `/jev off`: the coding run succeeded with zero decision events. `/jev on` subsequently restored `[Enabled]`. This verifies the global switch separately from active prediction quality.
+- The [active hosted Jev QA plan](default-active-jev-test-plan.md) records its live enabled-run evidence and automated checks. HJ cases above remain requirements rather than blanket pass claims; no benchmark establishes token savings.
+
+## Historical validation evidence
+
+Validated on Windows with Node.js 24.16.0 and npm 11.11.0, from `feat/decision-shadow`:
+
+- `npm run check`: passed.
+- `npm run build`: passed.
+- `node --test dist/test/decision-engine.test.js`: all 12 tests passed. Covers off/none/active fallback, one preflight batch, confidence boundaries, timeout/cancellation and late results, malformed output, provider/sink failures, configuration validation, flags, provider mutation, nonblocking completion, parallel-copy identity, and real CodeGraph/memory manager integration with injected backends.
+- The integration test compares off against opposite shadow predictions: model messages and permitted tools are identical; CodeGraph and memory retrieval still occur; decision state excludes injected memory; Action and End nodes make no semantic requests.
+- Final `npm test`: 956 tests, 917 passed, 34 failed, 5 skipped. Comparison against unchanged `main` at `9fe7f5ddea223c7add3ff8b57eaa52994b0f01cd` reproduced all 34 failures with identical test names. The baseline has 944 tests, 905 passes, 34 failures and 5 skips. Existing failures span interactive CLI, CodeGraph, memory, command screens and runtime tests; this is not a green full-suite claim.
+- `npm pack --dry-run --ignore-scripts --json`: confirmed the framework, usage guide, original plan and QA plan are in the npm package.
+
+CLI smoke check: `node dist/src/cli.js --help` exited successfully and displayed the normal command list. Interactive model-backed checks were not run.
+
+Remaining proposed QA includes distinct-node reversed completion order (parallel copies are covered), explicit pre-aborted requests, exhaustive numeric/schema variants, all deterministic node types beyond Action/End, and detailed approval/completion/budget regressions with shadow enabled. Existing runtime checks are included in the full suite, but its failures prevent a blanket pass claim. This paragraph records the original shadow-release evidence; current adapter and active-routing validation is recorded below.
+
+### PR #34 review follow-up
+
+- `npm run check` and `npm run build`: passed in the dedicated feature worktree.
+- `node --test dist/test/decision-engine.test.js dist/test/graph-runner.test.js dist/test/graph-policy-validation.test.js dist/test/graph-preflight-attribution.test.js`: 68 passed, zero failures, including 16 decision tests and the DE-18 through DE-21 regressions. The latest follow-up verifies modifier iteration metadata, memory-free decision tasks, unchanged model loop context, and consistent availability/baseline snapshots across a real retrieval circuit trip.
+- No full-suite rerun for these narrow fixes; the previously established 34 baseline failures above remain outside this follow-up's scope. No installation-checkout operations or live provider calls were required.
+
+## Embedded dual-provider and active Laya cases
+
+These cases extend the embedded shadow checks. Use explicit `DecisionManager` config objects in a dedicated feature worktree. CLI, desktop and browser runs also honor stored local provider selections, subject to global `jevEnabled`; verify that boundary with HJ-05 and the [local settings QA plan](local-decision-settings-test-plan.md). Automated HTTP fixtures bind ephemeral loopback ports and need no credentials or deployed models. Run `npm ci --ignore-scripts`, `npm run check`, `npm run build`, then `node --test dist/test/decision-http.test.js dist/test/decision-active.test.js dist/test/decision-engine.test.js`. Run `npm test` for broader regression coverage when appropriate.
+
+| ID | Preconditions and steps | Observable expected result |
+| --- | --- | --- |
+| DP-01 | Run HTTP tests with both Jev and Laya fixtures. Inspect request body/path and test bearer header. | One batched request reaches /v1/systemone or /v1/chat/completions, carries configured alias and typed instructions/criteria, and maps noul/choice/ordinal score to validated internal values. No chat-model substitution. |
+| DP-02 | Return Laya answers in top-level laya, then message.laya; return only generated message.content JSON. | Both structured contracts succeed; text-only response rejects. Missing/extra/type-mismatched/invalid probability or score answers reject the whole batch. |
+| DP-03 | Return 401, 307, malformed JSON and >1MiB response. Abort a pending response. | Evaluation rejects; manager preserves baseline. Redirect is not followed, bearer key is not forwarded, response/error text never enters decision telemetry. No retries. |
+| DP-04 | Use off/none, Laya without endpoint, configured missing credential, Jev active. With the same HTTP provider instance, add, rotate, remove and blank its synthetic environment credential. | Off/missing endpoint/missing credential/Jev active make no HTTP request. Off produces no decision events; missing endpoint reports provider_unavailable; missing credential reports provider_error; Jev active reports active_unsupported. Adding and rotating the credential enables the next request with its current value; removing or blanking it blocks subsequent HTTP. Restore the environment afterward. |
+| DP-05 | Active Laya with high/low/mid probabilities for CodeGraph and memory; repeat with feature flags false. | Only confident enabled decisions change eligible automatic retrieval; intermediate/omitted decisions retain baseline. Only two binary questions are sent. |
+| DP-06 | Repeat active checks with unavailable/disallowed CodeGraph, disabled memory and exact-file/literal native tasks. | Existing configuration, capabilities, native routing rules and budgets remain authoritative; no permission or tool exposure changes. |
+| DP-07 | Run concurrent node copies and a modifier loop with differing decisions; timeout or cancel decision requests. | Each invocation uses its own task/result and bounded deadline; memory retrieval follows each attempt's decision; late results do not mutate completed routes. |
+| DP-08 | Send active Laya compact state <=316 UTF-8 bytes then >316, including multibyte text and complete long node instruction. | Small state is sent as a string; oversized state makes no request and preserves baseline. No silent truncation or summarization. Shadow remains observational and may be truncated by server. |
+| DP-09 | In an explicitly embedded manager, select Laya active with the served endpoint/model; later update endpoint to a router alias. | Deployed model yields decision.completed/applied; missing model 404 preserves ordinary Agent behavior. Verify actual retrieval events and model context, not just server reachability. |
+| DP-10 | Configure Jev shadow with TYPESAFE_API_KEY set locally. | Structured answers produce shadow comparisons only; normal retrieval remains identical. Removing key yields provider_error fallback without key/error disclosure or HTTP. |
+| DP-11 | Use a 150ms node limit and two modifier iterations with a mock memory retrieval lasting 250ms. Compare off against active Laya with missing endpoint, throwing/timed-out provider, uncertain answers and memory routing disabled. | Both model calls complete with baseline memory; fallback retrieval suspends only node/modifier clocks. Repeat with a 100ms graph deadline and then caller cancellation: model calls remain zero and the result is exhausted/cancelled. Confident active evaluation and retrieval remain charged to node deadlines. |
+| DP-12 | Recommend CodeGraph and memory confidently. Give a node 20000 context characters; compare empty memory against a 12000-character memory block. Configure only the DecisionManager constructor telemetry sink. | The graph route is initially eligible. Empty memory permits one graph query and applied:true; large memory prevents the query and records applied:false. Exactly one CodeGraph application event is emitted after retrieval; constructor telemetry still receives requested/completed events. |
+
+### Provider limitations and pending manual work
+
+Historical Laya discovery reached http://100.97.4.16:7878/v1 but returned 404 for convaiinnovations/laya; current deployment availability requires a fresh check. DP-09/DP-10 live direct-provider checks, non-Windows validation, server-side cancellation and prediction quality/cost/latency benchmarks remain pending unless separately recorded. The 316-byte active state limit applies specifically to Laya, not hosted Jev. The coding LLM continues to run; no turn-skipping or savings percentage is claimed.
+
+### Historical dual-provider validation evidence
+
+Validated on Windows, Node.js 24.16.0 and npm 11.11.0, from the dedicated `feat/dual-decision-providers` worktree:
+
+- `npm run check` and `npm run build`: passed.
+- Focused decision HTTP, active-routing and original shadow tests: **32 passed, zero failures**. Nine HTTP fixtures cover both contracts, auth, malformed/oversized/redirect/error responses, abort and active context limits. Seven active tests cover actual retrieval/model input, gates, per-loop decisions, concurrent isolation and node deadlines. Sixteen original decision tests still pass.
+- `npm test`: **979 tests, 940 passed, 34 failed, 5 skipped**. Unchanged `origin/main` at `c220e3cdc388dbac599a7a41d272708694429a92` produced 963 tests, 924 passed, the same 34 failures and 5 skips. Exact failure-name sets match; no new failure was observed. This is not a green full-suite claim.
+- `git diff --check`: passed.
+- Live service discovery: endpoint responds, but model request returned 404. Laya deployment is deferred by the user; no successful live inference claimed. No installation config or deployed service was changed.
+
+The historical PR34 evidence above remains historical and must not be read as current provider validation.
+
+### PR35 review follow-up validation (September 19, 2026)
+
+- `npm run check` and `npm run build`: passed in `feat/dual-decision-providers`.
+- `node --test dist/test/decision-active.test.js dist/test/decision-http.test.js dist/test/decision-engine.test.js dist/test/agent-modifiers.test.js dist/test/deadline.test.js dist/test/graph-runner.test.js dist/test/graph-policy-validation.test.js dist/test/graph-preflight-attribution.test.js`: **142 passed, zero failures**. Includes 10 active, 11 HTTP and 16 original decision tests, plus modifier, deadline and graph contract regressions.
+- New fixtures verify request-time credential recovery/rotation/removal, fallback memory across two modifier iterations, preserved graph deadlines/cancellation, post-memory CodeGraph application telemetry, and a constructor-only telemetry sink. All inference is mocked; no live Jev/Laya inference was attempted.
+- The earlier full-suite comparison against unchanged main `c220e3c` remains the evidence for the 34 unrelated baseline failures. The full suite was not rerun for this narrow follow-up.
+- README and DE-16 now distinguish unsupported Jev active mode from supported opt-in Laya active routing. DP-04 and DE-19 record the request-time credential fallback behavior; DP-11/DP-12 describe the new runtime regressions.

package/tests/default-active-jev-test-plan.md

@@ -0,0 +1,85 @@
+# Default active Jev and per-agent settings QA
+
+Superseded in part by [jev-node-opt-in-test-plan.md](jev-node-opt-in-test-plan.md): Jev is now opt-in per Agent node, requires a signed-in account for hosted and local services, and is honored per node on free accounts too. The cases below are updated to match.
+
+## Scope and prerequisites
+
+Signed-in free and paid users receive hosted Jev decision assistance by default when no explicit `decisions` configuration is stored. Runtime settings can instead select a local Laya/Jev decision service or Off; see the [local decision settings QA plan](local-decision-settings-test-plan.md) for setup and UI cases. Paid users can configure individual agent nodes in custom graphs. Jev assists routing; the coding model still reads, implements, and verifies tasks. Existing permissions, hosted authentication and server quota remain authoritative.
+
+Use Node 22+, `npm ci`, `npm run check`, and `npm run build` in the feature worktree. For manual checks run `node dist/src/cli.js --cwd <test-repository>`. A free Cognito test account and a paid account in both base and paid groups are needed to compare entitlement behavior. Hosted verification needs the configured AWS proxy and an administrator-provisioned Jev secret; never put credentials in graphs or QA logs.
+
+## How to use the per-agent setting
+
+1. Sign in with `/login` using a paid test account.
+2. Open `/graph` and select the graph builder. Create or duplicate a graph.
+3. Select an Agent node on the canvas. In its general settings, locate **Jev decisions** below Prompt.
+4. Observe that a new node or an older imported node without a `jevEnabled` property has the checkbox cleared.
+5. Tick the checkbox, save the graph, reopen it, and confirm the checkbox remains selected. Graph JSON contains `jevEnabled: true` for that agent.
+6. Clear the checkbox, save again, and reopen; it remains cleared and the saved JSON has no `jevEnabled` key for that agent.
+7. Exit the builder and restore the original graph selection after testing. Remove only the test graph through the supported graph UI if cleanup is needed.
+
+## Test cases and expected results
+
+- AJ-01: With no explicit `decisions` configuration and global `jevEnabled` omitted or true, a signed-in free user runs the single Software Engineer default graph. Observe configured active hosted Jev decisions and server quota usage; there is no Orchestrator or custom graph access.
+- AJ-02: A paid user sees the same enabled default and can access the graph editor.
+- AJ-03: New Agent and imported legacy Agent definitions with omitted `jevEnabled` display unchecked; the bundled Software Engineer and Andorra presets carry `jevEnabled: true` on their Agents. Explicit true survives validation, saving, immutable revision loading, duplication and JSON import/export; unticking removes the key, and an imported explicit false displays unchecked.
+- AJ-04: Tick one Agent in a graph containing multiple Agents and leave the others unticked. Only the ticked Agent makes decision preflight requests; the others run with ordinary routing. The coding model still executes every agent's task. Free and paid accounts behave the same.
+- AJ-05: Control nodes such as End have no Jev checkbox. Imported `jevEnabled` on a non-Agent node is rejected by server validation, including false.
+- AJ-06: Import `jevEnabled: "false"`, null, zero, array, or object on an Agent. Server validation rejects these values; the editor blocks malformed state before issuing save requests.
+- AJ-07: Sign out or revoke paid membership while the editor is open. Subsequent protected API requests must fail membership validation; a previously open browser does not retain paid access.
+- AJ-08: Exhaust quota or make the hosted service unavailable. Observe safe fallback without elevating tool permissions or stopping otherwise valid coding work. The quota footer reflects authenticated server usage rather than guessed counters.
+- AJ-09: Use `/jev off` as either a free or paid member, run a task, then restart the CLI. The preference remains disabled, the footer shows `[Disabled]`, no hosted or local decision request occurs on any node, and the coding task can finish. `/jev on` lifts the global disable and resumes the stored provider/mode for Agents that turn Jev on; it restores hosted active routing only when no explicit `decisions` override is stored.
+- AJ-10: `/jev` toggles, `/jev status` only reports, and `/jev invalid` reports usage without changing or saving state. Autocompletion after `/jev ` offers only `on`, `off`, `status`.
+- AJ-11: Signed out, `/jev on` can save the preference, but no run makes any decision request, hosted or local: a configured local Laya/Jev service also needs a signed-in account. No fabricated hosted quota usage is reported.
+- AJ-12: With no `decisions` block and no global `jevEnabled` property, signed-in free and paid users get active hosted Jev on the bundled presets and on custom-graph Agents that tick **Jev decisions**; unticked Agents get none. Repeat with explicit `decisions` settings: off/none prevents requests; local Laya active uses its configured endpoint/model for eligible CodeGraph/memory routing on opted-in Agents of a signed-in account; local shadow evaluates without applying routing.
+- AJ-13: Observe a hosted request from any graph node. Public transcript identifies it as `[Jev]`, with safe completion/fallback text, rather than `[Orchestrator]` or `[Software Engineer]`.
+- AJ-14: Inject uncertain decisions, malformed responses, auth/quota errors, timeout or cancellation, and late responses. Preserve baseline retrieval, capability restrictions, abort behavior, and sanitized event fields.
+- AJ-15: Read the root README before signing in or submitting a task. It must disclose the default hosted processing, task/instruction transmission, `/jev off`, both membership tiers, paid node controls and the separate embedded-runtime defaults. Live evidence must not imply measured prediction quality or savings.
+- AJ-16: Run the default workspace-instructions CLI fixture with its disposable config and loopback coding model. Its explicit `jevEnabled: false` must bypass the run's hosted membership/decision path even if the OS keyring has an existing login. Both preferred ANDORRA.md and fallback agents.md tasks must succeed locally.
+- AJ-17: Inject the hosted provider's auth and fetch dependencies. Supply task/instruction text well beyond the former 16,384/8,192-byte limits and a 6000-NUL task whose JSON escaping would previously exceed 32,768 bytes. Expect the hosted request to proceed with a question-scoped snapshot: task <= 4,096 UTF-8 bytes, instruction <= 1,536 bytes, beginning/end preserved for ordinary long prose, irrelevant capability metadata removed, and serialized body <= 32,768 bytes. Only a snapshot that still cannot fit after the tighter second projection may use `decision_input_too_large` fallback.
+- AJ-18: Run a hosted-active graph Agent that invokes `spawn_subagent`, using injected coding and decision providers. The eligible parent can request decisions; the internal child has no routed retrieval capability and makes no hosted request. Repeat with both parent retrieval capabilities unavailable or their decision features disabled: no hosted request, ordinary execution remains available.
+- AJ-19: Inject disabled CodeGraph with eligible memory. The hosted snapshot reports CodeGraph unavailable and omits its question; no graph retrieval or successful graph application is recorded. Enable CodeGraph but return empty context: the retrieval attempt must not produce `decision.applied` with `applied: true`. A nonempty context block retains successful application telemetry.
+
+## Edge cases and potential bugs
+
+Omitted means enabled, so existing saved graphs gain the default without a destructive migration. Explicit false must never be lost through truthiness checks. Agent definition replacement can intentionally replace the node's stored settings; verify the imported definition's value rather than retaining a stale checkbox. Disabling Jev does not disable the coding model. Membership gates are server checked, not granted by graph JSON.
+
+## Validation evidence
+
+Automated schema/editor checks: `npm run build` and `node --test dist/test/graphs.test.js dist/test/graph-ui-client-validation.test.js` passed (38 tests) on Windows. Added tests exercise omitted/true/false persistence, invalid types, non-Agent rejection, checked-default behavior, toggling and editor save blocking.
+
+## Gaps and limitations
+
+Live entitlement revocation, multiple concurrent real graph nodes, browser import/export and definition reset remain proposed QA. Automated editor tests execute the shipped JavaScript with a minimal DOM; they are not browser visual evidence. No production calibration or measured token-savings claim is made. Active Jev selects CodeGraph and automatic Graphiti retrieval only; it does not choose tool permissions, replace the coding LLM, or control graph edges. Local command lessons remain independent of optional Graphiti retrieval.
+
+
+Runtime policy: when no explicit `decisions` block is configured, the CLI selects hosted Jev active for an authenticated free or paid account with a 15-second upper deadline (normal run/node cancellation still applies). An explicit `andorra` selection still requires hosted authentication and honors its stored mode/deadline. Explicit local Laya/Jev selections use their stored endpoint, model, mode and authentication environment-variable reference without requiring hosted sign-in; Laya supports active CodeGraph/memory routing, while direct Jev supports shadow only. Global `/jev off` overrides every selection; `/jev on` resumes the stored configuration rather than replacing it with hosted defaults. Only paid membership permits per-agent overrides. A missing global enable setting means enabled, not that explicit off/none is ignored. Missing hosted authentication or a failed selected service preserves baseline routing. The hosted API key stays in AWS; local service credentials come from the engine's environment.
+
+Live default CLI evidence: run `4f2b0455-6981-4188-ba50-7e7f137a2d32` succeeded with `JEV_DEFAULT_OK`, public `[Jev] Evaluating context needs` and `Decisions received in 1.3s`. JSONL recorded `decision.requested/completed/applied` in active mode, with CodeGraph and memory both not selected for this trivial task. Authenticated quota increased from 0 to 1 of 1000. This proves a real hosted request and applied response, not just a mock.
+
+Live command/terminal evidence: `/jev off` showed `Jev [Disabled]` next to quota. Run `70c15b4b-5509-461c-89bc-51e5845d63e4` then succeeded with zero decision events and zero tool calls. `/jev on` restored `[Enabled]` and saved the enabled machine preference; the CLI exited normally. Free-tier access and failed-save rollback are covered by command/policy fixtures rather than a second real account.
+
+The first broader CLI regression pass used the user's existing keyring login through old fixture configs and consumed additional hosted requests. Those fixtures now explicitly disable Jev; mocked decision tests cover active policy without contacting AWS. Daily quota snapshots may therefore include test requests as well as the live demonstration.
+
+Final automated validation: typecheck/build passed. The focused decision/schema/editor/transcript/terminal suite passed 188 tests; the CLI/commands/config/agent/graph regression group passed 104 tests after isolating CLI fixtures from hosted Jev. These groups have overlapping tests and are not an aggregate unique-test count.
+
+### PR #49 review follow-up (September 19, 2026)
+
+Validated in `feat/default-active-jev` on Windows, Node.js 24.16.0 and npm 11.11.0. The installed CLI was not touched. All new provider checks use injected local fixtures; no live inference, quota consumption, deployment or credential changes were needed.
+
+- `npm run check`, `npm run build` and `git diff --check`: passed.
+- `node --test dist/test/decision-active.test.js dist/test/hosted-decision-policy.test.js`: 21 passed, zero failures. Hosted skip behavior, internal children, paid node overrides and disabled/empty/failed/successful CodeGraph retrieval are covered.
+- The first consolidated run passed 169/175: five override fixtures needed an eligible mocked retrieval route after the new no-capability skip, and the empty-context fixture incorrectly retained structured source content. These fixture issues were corrected without relaxing the assertions.
+- Final consolidated command: `node --test dist/test/decision-active.test.js dist/test/andorra-decision-provider.test.js dist/test/decision-http.test.js dist/test/decision-engine.test.js dist/test/hosted-decision-policy.test.js dist/test/transcript.test.js dist/test/default-workspace-instructions.test.js dist/test/agent-modifiers.test.js dist/test/deadline.test.js dist/test/graph-runner.test.js dist/test/graph-policy-validation.test.js dist/test/graph-preflight-attribution.test.js`: **175 passed, zero failures**. This includes the 21 tests above, UTF-8/request-size boundaries, fallback redaction, CLI guidance fixture, deadlines, cancellation and graph regression checks.
+- AJ-15 was checked against README, hosted policy and account documentation. AJ-16 explicitly sets `jevEnabled: false`; AJ-17 through AJ-19 have injected-provider regression coverage. Broader manual/live cases above remain pending unless previously recorded.
+- The full suite was not rerun. The 34 unrelated baseline failures and unchanged-main comparison in [decision engine QA](decision-engine-test-plan.md) remain historical evidence, not a claim that the full suite passes now.
+
+### Memory application review follow-up
+
+AJ-20: Make hosted Jev recommend memory retrieval, then return a nonempty memory result, an empty result, a provider error, and cancellation. Verify `decision.applied` for `needs_memory` is emitted after retrieval and before model execution, and is true only when the memory block was added to the model task. Empty/error results report false; cancellation must not report successful application or invoke the model. The regression uses the real MemoryManager with injected local Graphiti responses.
+
+Validation on Windows: typecheck/build and `node --test dist/test/decision-active.test.js dist/test/agent-execution.test.js dist/test/hosted-decision-policy.test.js dist/test/agent-modifiers.test.js dist/test/deadline.test.js` passed all 91 tests. A broader run including memory-manager passed 53/60; the same seven memory-manager failures reproduced on installed main `d8019bb` (15/22 passed). These concern existing distillation/supersession and explicit memory-search evidence behavior, not this telemetry fix. No live Jev/Graphiti service was called and no quota was consumed.
+
+### Packaged QA evidence
+
+AJ-21: From the repository root run `npm pack --dry-run --json` and inspect the package file manifest. It must contain `tests/default-active-jev-test-plan.md`, the target of the README QA link. Validation: npm prepack build and postpack cleanup succeeded on Windows, and the generated manifest included this file. No package was published.