create-walle 0.9.21 → 0.9.23

package/template/claude-task-manager/docs/codex-terminal-render-controller-handoff.md

@@ -0,0 +1,311 @@
+# Codex Terminal Render Controller Handoff
+
+This document is a handoff package for fixing the recurring CTM Codex terminal rendering failures. It is written for a coding agent that has not seen the previous attempts.
+
+## Problem
+
+CTM has several recurring, related symptoms in Codex sessions:
+
+- Typed characters disappear or appear late in the terminal view.
+- The Codex native skill picker cannot be navigated reliably with `ArrowUp`, `ArrowDown`, or `Tab`.
+- When a skill token is being typed, the input row sometimes jumps up/down, especially when a toast appears or session output refreshes.
+- After a prompt is sent, the terminal or conversation view may not show the latest prompt/output until a reflow or refresh.
+- Completed Codex sessions sometimes stop before the final summary is visible.
+- Phone and CTM web can disagree about the current prompt/output state.
+
+These are not separate UI bugs. They are symptoms of multiple CTM paths mutating the same xterm.js terminal surface while Codex's Ratatui full-screen UI is also drawing synchronized frames.
+
+## Current Ownership Conflict
+
+There are currently too many owners of the same terminal state:
+
+1. Codex/Ratatui owns the live full-screen terminal UI, including the composer, picker, selected row, cursor, and synchronized redraw frames.
+2. CTM server classifies, coalesces, budgets, and snapshots PTY output.
+3. CTM web writes chunks to xterm.js, restores snapshots, repairs cursor/paint, follows scroll, fits geometry, and mirrors draft state.
+4. Conversation and phone views derive prompt/output from durable session records, transcript imports, terminal snapshots, and optimistic client state.
+
+When those paths interleave, any one of them can briefly replace a newer Codex frame with an older snapshot, resize the terminal during native composition, scroll to a stale bottom, or render a partial synchronized frame. Final transcript text can still look correct after reflow, which is why previous tests missed the bug.
+
+## Required Architecture
+
+Implement a single `TerminalRenderController` per live session. It should be the only object allowed to mutate the visible xterm.js terminal for that session.
+
+The controller should own:
+
+- Sequenced terminal writes from websocket output.
+- Snapshot restore and stale-snapshot rejection.
+- Terminal fit/resize/scroll/follow-bottom operations.
+- Paint refresh and cursor/texture repair.
+- Native composer lease state.
+- Synchronized frame buffering.
+- Completion/final-summary flush.
+
+All existing terminal mutation paths should call the controller instead of directly calling xterm.js or global repair helpers.
+
+## Invariants
+
+The implementation is correct only if these invariants hold:
+
+1. A terminal session has exactly one render controller that serializes terminal mutations.
+2. Provider output is applied in sequence order. Older snapshots cannot overwrite newer interactive output.
+3. A DEC synchronized frame (`ESC[?2026h ... ESC[?2026l`) is applied atomically. The user should never see only the clear-screen half of a frame.
+4. While Codex native composition is active, CTM does not mutate the terminal surface except by applying sequenced provider output.
+5. Native composition starts after user input that can alter the provider-owned prompt UI: printable input, `$`, `/`, backspace, delete, arrow keys, `Tab`, paste, and provider input replay.
+6. Native composition ends after provider-confirmed submission/cancel or after a complete synchronized frame plus a quiet period.
+7. Toasts, session status polling, tab switching, mobile sync, and conversation refreshes never call terminal fit/scroll/paint/snapshot restore directly.
+8. Conversation and phone views show a sent prompt from a durable prompt event or optimistic pending prompt immediately, then reconcile to provider output.
+9. Completion status cannot be finalized until the terminal controller has flushed the last sequenced output or installed a final snapshot that includes it.
+10. Once a final-exit snapshot is protected, routine clean snapshots are allowed only if they are equivalent or richer. A shorter clean snapshot must not erase the final answer, exit banner, or settled prompt footer; manual reflow remains the explicit user-authoritative recovery path.
+
+Sequence clocks must stay separate. Provider output `seq`/`frame` values can be compared with terminal mutation sequence state. Browser restore-request `clientRestoreSeq` values are only for attach/reflow supersession and must not be treated as provider output ordering.
+
+## Native Composer Lease
+
+The native composer lease is the key missing abstraction.
+
+Start the lease when CTM forwards any key/paste operation that can change Codex's native input UI:
+
+- printable character
+- `$`
+- `/`
+- `Backspace`
+- `Delete`
+- `ArrowUp`
+- `ArrowDown`
+- `ArrowLeft`
+- `ArrowRight`
+- `Home`
+- `End`
+- `Tab`
+- paste
+- phone/remote submission that is replayed into a live Codex PTY
+
+While the lease is active, these direct mutations are forbidden:
+
+- `fitAddon.fit()`
+- `term.resize()`
+- `term.scrollToLine()`
+- `term.refresh()`
+- `term.clearTextureAtlas()`
+- snapshot restore into xterm
+- stale chunk replay
+- cursor repair writes
+- blank-gap compaction writes
+- scroll-follow writes
+- forced paint/texture repair writes
+
+The controller can queue these operations and apply them after the lease ends if they are still relevant.
+
+End the lease when one of these happens:
+
+- Codex emits a provider frame that confirms submission/cancel and the frame is complete.
+- The user sends `Enter`, `Ctrl-C`, or `Ctrl-D`, and a complete post-submit frame has been applied.
+- A controller-defined quiet period passes after a complete synchronized frame with no interactive key events.
+
+Do not use a fixed timeout alone. It must be tied to sequenced provider output so slow sessions do not flicker or lose input.
+
+### Native Picker Selection Redraws
+
+Codex's `$` and `/` picker is provider-owned terminal UI. Arrow navigation can arrive as a full picker redraw, or as a narrow row-addressed update that only repaints the option rows. The row-only form may omit the `SKILLS` header and `Press enter` hint even though it is still the same active native picker.
+
+Rules:
+
+- Keep the native composer lease active after `ArrowUp`, `ArrowDown`, and `Tab` until a picker-looking provider update has been applied and a short quiet window has passed.
+- Treat option-row redraws as picker output when the native picker is already visible or a provider-navigation lease is active.
+- Do not strip Codex prompt-background ANSI palettes while a native picker is visible. Palette colors such as `48;5;237` and `48;5;238` are also used for selected picker rows.
+- Do not run cursor repair, fit, scroll-follow, snapshot restore, or local echo overlays between the key event and the settled picker redraw.
+- Keep Codex-specific picker detection behind the Codex agent boundary. Claude/Ink redraws share the same "provider owns interactive TUI frames" principle, but should use Claude-specific frame-hold predicates rather than Codex skill-row parsing.
+
+The generic terminal input hold must protect Claude as well as Codex. If a visible terminal has an active draft, CTM should defer follow-bottom and forced repaint even when DOM focus briefly moves away from the helper textarea. Focus is an implementation detail; the visible draft is the ownership signal.
+
+## Synchronized Frame Handling
+
+Codex/Ratatui commonly draws full-screen updates with terminal control sequences. The controller should buffer output once it sees synchronized output mode begin:
+
+- begin: `ESC[?2026h`
+- end: `ESC[?2026l`
+
+Rules:
+
+- Do not write a partial synchronized frame to xterm.
+- If a chunk contains multiple frames, split and apply complete frames in order.
+- If a frame is incomplete, hold it until the closing marker or until the PTY/session ends.
+- If the session ends with an incomplete frame, apply a deliberate recovery snapshot or mark a gap. Do not show a half-cleared terminal.
+- Do not split a complete `2026h`/`2026l` frame merely to satisfy a byte-size chunk target.
+- Treat viewport repaint as a mutation. Defer `term.refresh()` while synchronized output mode is active, otherwise a stale paint-settled or force-paint callback can expose the half-cleared frame that the provider intended to keep atomic.
+
+References:
+
+- xterm.js terminal API: https://xtermjs.org/docs/api/terminal/classes/terminal/
+- xterm.js modes, including synchronized output mode: https://xtermjs.org/docs/api/terminal/interfaces/imodes/
+- Ratatui rendering model: https://ratatui.rs/concepts/rendering/
+- Ratatui under the hood: https://ratatui.rs/concepts/rendering/under-the-hood/
+
+## Server Responsibilities
+
+The server should not drop interactive frames just because output budget is high.
+
+Expected server behavior:
+
+- Assign monotonically increasing sequence numbers to every output event/snapshot/gap.
+- Preserve synchronized frame boundaries or mark chunk metadata so the client can buffer atomically.
+- If budget rules skip output, emit an explicit sequenced gap or fresh snapshot.
+- Never classify provider-native prompt redraws as disposable noise while a session is active.
+- When a process completes, hold completion notification until the final terminal output event is flushed.
+- For hosted PTYs, keep the embedded headless terminal alive after the PTY exits until the parent control plane has serialized the final screen and sent `destroy`. PTY exit is not the same as terminal-state disposal.
+- Final-exit snapshots should use a bounded but longer pending-write drain than routine approval/status checks, because xterm.js writes are asynchronous and the snapshot becomes the durable restore source.
+
+Relevant areas to audit:
+
+- Output classification and budget logic in `claude-task-manager/server.js`.
+- Websocket terminal-output emission.
+- Snapshot persistence and restore metadata.
+- Completion status update path.
+
+## Client Responsibilities
+
+The browser should not call xterm.js directly from scattered code paths. Route all terminal mutations through the controller.
+
+Audit and migrate:
+
+- websocket output writer
+- snapshot restore
+- tab switch/reflow
+- follow-bottom logic
+- fit/resize observer
+- cursor repair
+- terminal dirty/paint repair
+- toast/status update side effects
+- model/status dropdown side effects
+- provider input route
+- remote/mobile replay route
+
+The controller should expose a small API:
+
+```js
+controller.enqueueOutput({ seq, text, source, frameState });
+controller.enqueueSnapshot({ seq, snapshot, reason });
+controller.enqueueGeometry({ reason, width, height });
+controller.noteProviderInput({ key, text, source });
+controller.noteProviderSubmit({ source });
+controller.withMutationLease(reason, fn);
+controller.getState();
+```
+
+Do not expose xterm internals to normal UI components.
+
+## Conversation And Phone Views
+
+The terminal tab is provider-owned raw terminal state. Conversation and phone views should not depend on scraping that state for a prompt the user just sent.
+
+Use this pipeline:
+
+1. Before sending to the PTY, create a durable prompt event with a client id, session id, text, attachments, and timestamp.
+2. Phone and CTM web show the prompt immediately as `pending`.
+3. When the provider transcript/import catches up, reconcile by client id or normalized prompt fingerprint.
+4. If the provider never confirms, keep the pending item with retry/error status instead of hiding it.
+
+This fixes the "I sent it from phone but cannot see it" class of bugs without changing provider copy/paste behavior.
+
+## Implementation Phases
+
+### Phase 1 - Controller Skeleton
+
+- Add a controller object per terminal session.
+- Register it in session state.
+- Keep existing behavior routed through the controller with no behavior change.
+- Add debug state for controller lease, sequence, buffered sync frame bytes, and last applied snapshot sequence.
+
+Acceptance:
+
+- Existing render tests pass.
+- The controller is discoverable from browser tests.
+
+### Phase 2 - Single Writer
+
+- Move websocket terminal writes, snapshot restore, reflow, paint repair, and scroll/follow-bottom into the controller queue.
+- Add stale snapshot rejection.
+- Add mutation logging for forbidden direct calls in tests.
+
+Acceptance:
+
+- No terminal mutation bypasses the controller in normal session view.
+- Tab switching and toasts do not call xterm mutation APIs directly.
+
+### Phase 3 - Native Composer Lease
+
+- Start/extend the lease on provider input.
+- Queue geometry/paint/snapshot operations during lease.
+- End the lease only after a complete provider frame and quiet period or confirmed submit/cancel.
+
+Acceptance:
+
+- Typing `$ctm`, navigating with arrows, pressing `Tab`, and receiving a toast does not move the prompt row or lose chars.
+
+### Phase 4 - Synchronized Frame Atomicity
+
+- Buffer synchronized frames by DEC 2026 markers.
+- Apply complete frames atomically.
+- Add explicit recovery for incomplete final frames.
+
+Acceptance:
+
+- Rapid Codex redraws never show blank/half-cleared terminal frames.
+
+### Phase 5 - Durable Prompt Event Reconciliation
+
+- Ensure CTM web and phone create/show pending prompt events before PTY write.
+- Reconcile with provider transcript/import.
+- Share the same session data source between phone and CTM web conversation views.
+
+Acceptance:
+
+- After sending from phone or CTM web, the prompt appears immediately in conversation view and remains visible after refresh.
+
+### Phase 6 - Completion Flush
+
+- Completion status waits for final terminal output flush.
+- Completed sessions display final summaries without manual reflow.
+
+Acceptance:
+
+- A Codex session that prints final summary then exits shows the summary in terminal and phone/conversation views.
+
+## Render Controller Regression Suite
+
+The file `claude-task-manager/tests/rendering/scenarios/codex-render-controller-contract.spec.js` is part of the normal rendering regression suite. It used to be gated as known-red, but the controller ownership invariants are now expected behavior and should fail loudly when bypassed.
+
+Run it directly when working on this fix:
+
+```bash
+cd claude-task-manager
+npx playwright test --config tests/rendering/playwright.config.js scenarios/codex-render-controller-contract.spec.js --retries=0
+```
+
+Final-summary stability is covered in `codex-durable-final.spec.js`. Keep both the stale busy snapshot case and the shorter clean snapshot case: the first catches stale Running/Working redraws, while the second catches the subtler "looked correct for a minute, then lost the final summary" regression.
+
+Synthetic Codex maintenance writers are also terminal mutations. Stale prompt cleanup, duplicate prompt cleanup, transient status cleanup, blank-gap compaction, and native cursor repair must respect the same ownership boundary:
+
+- They must not run while a final-exit snapshot is protected.
+- They must not run after the session has exited unless the call site explicitly proves it is safe.
+- They must not run while Codex owns a native picker/composer frame. The skill picker can redraw only the current row and leave selection color as background-only SGR; selection detection must parse raw ANSI background cells, not only cleaned text glyphs.
+
+The focused regressions live in `codex-durable-final.spec.js` (`protected final-exit screen ignores delayed synthetic Codex repair writers`) and `codex-skill-typing-visual-stability.spec.js` (`Codex native skill picker clears stale background-only selected rows`).
+
+## Do Not Do
+
+- Do not add arbitrary `setTimeout` redraws to hide flicker.
+- Do not mask missing characters by local-echoing text over the terminal.
+- Do not suppress screenshots or toasts instead of fixing terminal ownership.
+- Do not add special cases for only `$ctm` or only one skill.
+- Do not make phone and CTM web use different session pipelines.
+- Do not treat final transcript correctness as proof that the live terminal UI was correct.
+
+## Completion Checklist
+
+- Contract red suite passes without forbidden direct mutations.
+- Existing render tests still pass.
+- Manual browser smoke: type `$ctm`, arrow through skills, tab-complete, receive a toast, send prompt, and verify no jump/lost chars.
+- Manual browser smoke: switch tabs during active output and verify no stale snapshot replaces newer output.
+- Phone smoke: send prompt, refresh, and verify the pending/sent prompt remains visible.
+- Completed Codex smoke: final summary appears without reflow.