@xingular/harness 2.0.0 → 2.0.1-unstable

package/docs/experiments/EXP-0001-terminal-input-latency/plan.md

@@ -0,0 +1,81 @@
+---
+id: EXP-0001
+title: Harness terminal input latency
+status: completed
+created: 2026-10-20
+updated: 2026-10-20
+related: [ADR-0001]
+---
+
+# EXP-0001 — Harness terminal input latency
+
+## Question and falsifiable hypothesis
+
+Does replacing the serialized per-change draft queue with a single-flight latest-state publisher
+reduce draft synchronization backlog while preserving the final dirty state, submit ordering, and
+editor behavior?
+
+The hypothesis is that, when draft replies are slower than incoming keys, the treatment will send
+fewer than one control request per key and reduce queue drain time. Full TUI frame p95 should not
+increase materially. The real local control socket is measured separately from the delayed stress
+workload.
+
+## Baseline, treatment and controls
+
+- **Baseline:** `TerminalApplication.syncDraft()` appends one socket request to a promise chain for
+  every editor change.
+- **Treatment:** one in-flight request plus the latest pending dirty boolean. A pending value is
+  always sent after the in-flight request, including when it has the same boolean, because raw
+  terminal input can change controller state independently.
+- **Fixed controls:** same working tree, Node/Vitest versions, 520-character input sequence, 4 ms
+  inter-key timer, synthetic 8 ms draft reply, 120 by 40 terminal dimensions, and test runtime.
+  The full-frame test calls `renderNow()` for each key and intercepts terminal writes.
+- **Behavior controls:** focused application tests cover submit waiting for the empty-draft reply,
+  coalescing to the latest state, and existing editor callbacks; the controller test covers a
+  multiline raw-input/Ctrl+U correction and the send guard.
+
+## Data, sampling and protocol
+
+Run `npm run profile:terminal-input`. It emits nearest-rank p50/p95 samples for editor input/render,
+full TUI layout frames, and 120 real local draft-socket requests. Its application workload types
+`abcdefghijklmnopqrstuvwxyz` 20 times, waits 4 ms between keys, and injects an 8 ms mock reply for
+each draft update. `XINGULAR_PROFILE_TERMINAL_INPUT_RPC_DELAY_MS` changes that injected delay.
+No model or provider request runs.
+
+The full-frame profile measures Harness layout and diff computation up to `ProcessTerminal.write`;
+the write itself is intercepted, and direct `renderNow()` bypasses the TUI's scheduled frame
+throttle. It does not measure terminal-emulator paint or the user's slower computer.
+
+## Versions and environment
+
+- Harness package: 3.0.0; Pi dependency: 1.2.0.
+- Base commit: `24a9390` on `harness2`; profile and product changes were uncommitted in the shared
+  worktree during the runs.
+- Runtime: Node v26.8.2, Vitest 4.1.11, macOS arm64, Apple M2, 8 CPU cores.
+- No random seed is used; the input string and timing controls are fixed.
+- Profile script SHA-256: `fdcc0d901aae87bbd3e72481471b22dd3d55d1d0d7685ffab4b0d5df0486b509`.
+
+## Metrics and success criteria
+
+Primary metric: draft queue drain after the 520-key sequence. Secondary metrics: draft request
+count, full-frame and combined input/frame p50/p95, and real local socket p50/p95. The change is
+useful if it materially lowers queue drain and request count while keeping the final dirty state,
+submit order and focused behavior tests passing. Frame p95 is checked for regressions, but one run
+per variant is not enough to establish a statistically stable p95 difference.
+
+## Risks and stopping criteria
+
+The injected reply delay is intentionally much slower than the measured local socket. The test can
+show backlog behavior under slow replies, but it cannot establish that socket latency causes the
+reported lag on a different computer. Stop optimization if the exact draft or send guard changes,
+or if repeated slow-machine measurements show no meaningful queue or frame improvement.
+
+## Reproduction
+
+From `Harness/`, run `npm run profile:terminal-input`. For the same synthetic stress profile on
+another computer, use the same Node and dependency versions and leave the default delay unchanged.
+Run the focused suite to verify behavior:
+
+```sh
+npm test -- src/terminal/tui/application.test.ts src/terminal/tui/editor.test.ts src/terminal/terminal.test.ts
+```

package/docs/experiments/EXP-0001-terminal-input-latency/results.md

@@ -0,0 +1,46 @@
+# EXP-0001 — Results
+
+- **Status:** completed on one local machine; slower-computer validation remains open.
+- **Run date:** 2026-10-20.
+- **Code:** base commit `24a9390` plus the shared, uncommitted worktree. Baseline and treatment
+  used the same profiler and workload; only the draft synchronization implementation differed.
+- **Runtime:** Node v26.8.2, Vitest 4.1.11, macOS arm64, Apple M2, 8 CPU cores.
+- **Workload:** 520 keys; 4 ms inter-key timer; 8 ms mocked draft reply; full TUI layout rendered per
+  key at 120 by 40, with terminal output intercepted. Separate local socket sample: 120 requests.
+
+| Variant | Draft RPCs | Draft queue drain | Total sequence | Full TUI frame p50/p95 | Input plus frame p50/p95 |
+|---|---:|---:|---:|---:|---:|
+| Serialized baseline | 520 | 1,101.67 ms | 4,557.45 ms | 1.975 / 3.402 ms | 2.145 / 3.605 ms |
+| Latest-state treatment | 401 | 11.63 ms | 3,440.87 ms | 1.953 / 3.199 ms | 2.155 / 3.410 ms |
+
+The treatment reduced draft requests by 22.9%, queue drain by 98.9%, and total sequence time by
+24.5%. Full-frame p95 was 6.0% lower and combined input/frame p95 was 5.4% lower in these single
+runs; the p50 frame time was effectively unchanged. The small p95 differences are directional only
+because there was one 520-key run per variant.
+
+The separately measured local socket p95 was 0.122 ms for the baseline and 0.118 ms for treatment.
+Editor-only input/render p95 was 0.157 ms and 0.161 ms respectively. These numbers indicate that
+the editor itself was inexpensive on this machine; the treatment mainly prevents request backlog
+when replies take longer than normal.
+
+## Uncertainty, failures and limits
+
+The mocked 8 ms response is about 65 times the measured 0.12 ms local socket p95, so the queue
+improvement is strongest evidence for slow-reply conditions. The profile does not observe physical
+terminal paint and was not run on the reported slower computer. That computer must run the profile
+before claiming the user's lag is fixed there.
+
+The focused Harness terminal/editor suites passed: 52 passed, 4 skipped. The opt-in profile passed
+all three profile tests. The full Harness suite had 611 passing tests and one failure in the existing
+untracked `src/terminal/tui/treeModel.test.ts` search expectation; Harness typecheck also reports
+two errors in the existing untracked `treePicker` files. Neither failure is in the changed files.
+
+Sibling App layer/UI checks, sidecar tests and Rust tests passed. App typecheck and root tests still
+fail on existing Guide fixture/content mismatches in the shared App worktree.
+
+## Conclusion
+
+The hypothesis is supported for backlog under slow draft replies: coalescing cuts queued updates
+and drain time without changing measured frame performance or the focused behavior tests. Whether
+this resolves the reported slow-computer typing lag is inconclusive until the profile is run there
+and the result is checked against interactive use.

package/docs/experiments/EXP-0002-opentui-typing-latency/plan.md

@@ -0,0 +1,70 @@
+---
+id: EXP-0002
+title: OpenTUI terminal typing latency against pure Pi and opencode
+status: completed
+created: 2026-10-21
+updated: 2026-10-21
+related: [EXP-0001, ADR-0006]
+---
+
+# EXP-0002 — OpenTUI terminal typing latency
+
+## Question and falsifiable hypothesis
+
+Why does typing in the Harness terminal feel slow, and do targeted frontend changes bring its
+keystroke-to-screen latency and CPU use in line with pure Pi and opencode while idle, after a long
+history, and while a reply streams?
+
+Hypotheses, from code review before measuring: (1) the renderer's continuous mode makes every key
+wait for a frame tick; (2) every change rebuilds the whole view and the Solid tree, and a streamed
+delta recreates the reply's markdown renderable; (3) each runtime event publishes synchronously, with
+no coalescing; (4) per-request socket connections add steady CPU. EXP-0001 measured the old Pi
+frontend, which ADR-0006 removed, so it says nothing about this frontend.
+
+## Targets and controls
+
+- **Xingular Harness:** `node dist/cli.js` with isolated `--state` and `--data`, an API-key account,
+  and the loopback model in the account's shared `models.json`.
+- **Pure Pi:** `@earendil-works/pi-coding-agent` 1.2.0 `dist/bundle/cli.js` with an isolated
+  `PI_CODING_AGENT_DIR` (never `~/.pi`).
+- **opencode:** v2.1.0, `--standalone`, isolated XDG folders and an `OPENCODE_CONFIG` that points an
+  `@ai-sdk/openai-compatible` provider at the loopback model.
+- **Fixed:** 160 by 45 PTY, `TERM=xterm-256color`, `COLORTERM=truecolor`; four rounds of 60 keys
+  from `zqxjkvwy` (never produced by the replies) at 35 ms; the first 8 keys of a round are not
+  scored; the loopback server answers all three targets identically.
+
+## Scenarios
+
+- **idle:** a fresh session, nothing running.
+- **history:** after the model has looped 80 `read` tool calls (about 240 messages).
+- **stream:** three rounds typed while a reply streams 60 markdown chunks per second for 30 s, with
+  headings and code blocks. In a full run this follows `history`, so it streams over a long session.
+- **approval / question** (Harness only): typing while a tool approval, then an `ask_user` question,
+  waits above the composer.
+- **real:** `--real` runs `stream` with each tool's own login; for Harness, the Codex account on
+  `openai-codex` / `gpt-6-luna`.
+
+## Metric and measurement
+
+Latency is the time from writing a key to the PTY until the typed prefix ending at that key is
+visible on a pyte virtual screen (nearest-rank p50/p95/p99, max). CPU is the tree's CPU time from
+`ps` over the typing window. Bytes per key is PTY output divided by keys. Hot paths were found with
+`bun --cpu-prof --cpu-prof-md` through `XINGULAR_HARNESS_BUN`.
+
+The metric includes the frontend, the PTY and pyte parsing; it does not include a terminal emulator's
+paint. A key that never appears (dropped) is stamped when later text shows, so dropped keys appear
+as very large latencies rather than as misses.
+
+## Success criteria
+
+Harness p95 at or below opencode's and within 1.5 times pure Pi's in every scenario, idle latency
+that does not grow with history, and streaming CPU no higher than opencode's.
+
+## Reproduction
+
+```sh
+npm run build
+python3 scripts/profile-terminal-latency.py --targets xingular,pi,opencode --output <dir>
+python3 scripts/profile-terminal-latency.py --targets xingular --scenarios question --output <dir>
+python3 scripts/profile-terminal-latency.py --real --targets xingular --setup /model '\r' gpt-6-luna '\r' --output <dir>
+```

package/docs/experiments/EXP-0002-opentui-typing-latency/results.md

@@ -0,0 +1,92 @@
+# EXP-0002 — Results
+
+- **Status:** completed on one machine; not yet repeated on a slower computer.
+- **Run date:** 2026-10-21.
+- **Code:** base commit `24a9390` on `harness2` plus the shared, uncommitted worktree (another session
+  was editing the same frontend in parallel; every run used the tree as built at that moment).
+- **Runtime:** macOS arm64, Apple M2, 8 cores; Node v26.8.2; Bun 1.4.2; OpenTUI 0.6.6; Pi 1.2.0;
+  opencode 2.1.0; pyte for the virtual screen.
+
+All latencies in ms, keystroke to visible on screen, p50 / p95 (p99 where it matters). Lower is better.
+
+## Final comparison (loopback model, identical replies)
+
+| Scenario | Xingular before | Xingular after | Pure Pi | opencode `--standalone` |
+|---|---:|---:|---:|---:|
+| idle | 13.1 / 19.2 | 5.1 / 10.1 | 3.9 / 6.2 | 7.5 / 10.9 |
+| history (≈240 messages) | 13.2 / 20.3 | 5.2 / 10.6 | 5.1 / 10.4 | 9.8 / 13.1 |
+| stream over history | 13.3 / 23.7 | 8.1 / 23.5 (p99 26.5) | 3.1 / 22.2 (p99 29.4) | 9.1 / 28.0 (p99 33.2) |
+| CPU while streaming | 73.3% | 15.0% | 14.3% | 36.3% |
+| CPU at rest, nothing typed | — | 2.5% | 0.0% | 3.4% |
+
+Harness-only scenarios:
+
+| Scenario | Before | After |
+|---|---:|---:|
+| typing into a tool-approval dialog during a run | 20.3 / 36.2 | 7.6 / 13.7 |
+| typing while an `ask_user` question waits | 5.7 / 1,154.5 (max 1,402) — keys dropped | 5.0 / 12.9 (max 74) |
+
+Real model (each tool's own login, `stream` scenario, three rounds while the reply streams):
+
+| Target | Model | p50 / p95 / p99 / max | CPU |
+|---|---|---:|---:|
+| Xingular, after | Codex account, `openai-codex` / `gpt-6-luna` | 6.2 / 12.9 / 32.5 / 48.1 | 15.2% |
+| opencode | its own login, MiniMax M3.1 Flash Free | 11.6 / 24.8 / 30.8 / 40.5 | 33.6% |
+
+Pure Pi was not run with real credentials: its only real logins are OAuth accounts whose refresh
+tokens rotate, and copying them into a second client could sign the Harness account out.
+
+## What each change did
+
+Full-run rows stream on top of `history`; the "stream only" rows start fresh and are compared only
+with each other.
+
+| Step | idle | history | stream | stream CPU |
+|---|---:|---:|---:|---:|
+| Baseline | 13.1 / 19.2 | 13.2 / 20.3 | 13.3 / 23.7 | 73.3% |
+| 1. Render on demand (no `renderer.start()`) | 4.9 / 9.0 | 7.2 / 8.5 | 16.5 / 26.3 | 72.2% |
+| 2. Store + `reconcile` by id, `<markdown streaming>`, coalesced `change()` at 16 ms | 4.8 / 8.8 | 4.7 / 9.1 | 18.0 / 28.1 | 30.2% |
+| stream only: 2 + publish at 33 ms | | | 19.4 / 28.1 | |
+| stream only: 2 + `maxFps: 120` | | | 13.6 / 20.0 | |
+| stream only: 2 + both | | | 7.7 / 20.0 | |
+| 3. 33 ms only while running, `maxFps: 120` | 4.8 / 9.7 | 4.8 / 9.0 | 10.0 / 24.0 (p99 34.7, max 45.0) | 25.0% |
+| 4. Socket keep-alive, statistics cache, per-message transcript cache | 4.9 / 6.0 | 4.7 / 6.1 | 6.0 / 19.2 (p99 23.3) | 14.9% |
+| 5. Input publishes at once, draft read at key time, dock reply fix | 5.1 / 10.1 | 5.2 / 10.6 | 8.1 / 23.5 (p99 26.5) | 15.0% |
+
+The CPU profile of the baseline stream (Bun `--cpu-prof`) attributed 34% of samples to `change()` →
+Solid updates, 18% to re-highlighting text (`startHighlight` → `setStyledText`), about 12% to
+destroying and recreating renderables, about 30% to native painting, and 16% to socket requests:
+two 100 ms polls and every draft update each opened a new Unix-socket connection.
+
+Step 5 fixed correctness problems found while measuring, at no material latency cost:
+
+- **Dropped keys:** while a question waited, letters the dock did not use reached nothing, so typing
+  showed only after an `x` happened to collapse the dock. They now move focus to the composer.
+- **Stale draft on Enter:** OpenTUI reports composer text in a microtask after all keys of one input
+  chunk. Keys and Enter arriving together (a busy machine, a paste ending in a newline) sent an empty
+  or truncated draft. The root now reads the composer before each key.
+- **Composer not cleared:** with coalesced publication, a send's cleared draft could be overwritten by
+  the composer's late report. Changes made while handling a key publish synchronously.
+
+An A/B of the per-key composer read gave idle p95 6.7 and 9.4 ms without it and 10.0 and 6.8 ms with
+it. Idle p95 alternates between about 7 and 10 ms from run to run: a key that lands within one
+120 Hz frame (8.3 ms) of the previous frame waits for it.
+
+## Against the success criteria
+
+- p95 at or below opencode's: met in every scenario (10.1 vs 10.9, 10.6 vs 13.1, 23.5 vs 28.0).
+- Within 1.5 times Pi's p95: met for history and stream; idle is 1.6 times in the final run and
+  1.1 times in step 4 and the A/B runs, inside the run-to-run spread above.
+- No growth with history: met (idle 5.1 / 10.1, history 5.2 / 10.6).
+- Streaming CPU no higher than opencode's: met (15.0% vs 36.3%).
+
+## Limits
+
+- One machine, one run per row; the spread of p95 between repeats is about 3–4 ms.
+- The metric includes PTY and pyte parsing but not a terminal emulator's paint.
+- The loopback stream is regular (60 chunks per second); real providers burst. The Luna run agrees
+  with the loopback figures at p50/p95.
+- `scripts/verify-terminal.py` still fails after `/settings` on a `Try "` placeholder from the
+  removed Pi frontend; that check predates this work (`verification.json` recorded `passed: false`
+  for OpenTUI on 2026-10-20). It now matches the rendered screen when pyte is installed, which fixed
+  the earlier false failure on "Keyboard controls".