hop2 0.9.1 → 0.11.0

package/INTEGRATION.md

@@ -1,92 +1,84 @@
-# Hop + Termshare Integration Plan
+# Hop + Hay Architecture
 
-This branch integrates [termshare](https://github.com/jzthree/termshare) to replace ttyd/tmux.
+This document describes the current Hop terminal architecture.
 
-## Benefits
+The historical `ttyd`/`tmux` backend has been removed from the active runtime.
+Hop now uses Hay, a local terminal-sharing stack built on WebSocket plus
+`node-pty`, with browser, CLI, and MCP clients all operating on the same
+session model.
 
-- **No ttyd/tmux dependencies** - Pure Node.js solution
-- **Built-in presence** - See who's connected, typing indicators
-- **Control modes** - Collaborative or locked single-controller
-- **No injection hacks** - Direct control over web UI
-- **Mobile-ready** - Responsive design with iOS-style keyboard built-in
-- **CLI client** - Connect from real terminals too
+## Current Stack
 
-## Architecture Change
-
-### Before (ttyd)
 ```
-Browser → Hop Auth Proxy → ttyd (per session) → tmux → shell
-                ↓
-        HTML/WebSocket injection for mobile keyboard
+Browser / Hay CLI / Hop MCP
+            |
+            v
+        Hop daemon
+            |
+            v
+     local hay-host process
+            |
+            v
+      Hay room manager
+            |
+            v
+         node-pty
+            |
+            v
+           shell
 ```
 
-### After (termshare)
-```
-Browser → Hop Auth Proxy → Termshare WebSocket → node-pty → shell
-                ↓
-        Native termshare web UI (mobile keyboard built-in)
-```
-
-## Implementation Steps
-
-### 1. Add Dependencies
-- [x] `node-pty-prebuilt-multiarch` - PTY handling
-- [x] `ws` - WebSocket server
-- [x] Copy termshare room/pty logic into hop (`termshare.js`)
-
-### 2. Remove ttyd/tmux Code
-- [x] Remove `spawn('ttyd', [...])` in `getOrCreateSession()`
-- [ ] Remove `HOP_TMUX_CONF`, `HOP_TMUX_WRAPPER` generation (still present but unused)
-- [ ] Remove `listTmuxSessions()`, `waitForTmuxSessionReady()` (still present but unused)
-- [ ] Remove ttyd dependency check (still present but unused)
-- [x] Remove keyboard HTML injection code (now using termshare's built-in)
-- [x] Remove `getIOSKeyboardInjection()` WebSocket patching (still present but unused)
-
-### 3. Add Termshare Integration
-- [x] Port `Room` and `RoomManager` classes from termshare
-- [x] Port `createPty` function
-- [x] Create termshare WebSocket handler at `/ws` endpoint
-- [x] Map hop sessions to termshare rooms
-
-### 4. Update Session Management
-```javascript
-// Before: activeSessions[name] = { ttyd: Process, port: number }
-// After:  activeSessions[name] = { room: Room }
-```
-✅ Completed
-
-### 5. Serve Termshare Web UI
-- [x] Build termshare web client (with mobile keyboard)
-- [x] Serve static files from hop (`termshare-web/` directory)
-- [x] Inject room parameter for session routing
-
-### 6. Keep Hop Features
-- [x] Authentication (TOTP + cookies)
-- [x] Cloudflare tunneling
-- [x] Session URL routing (`/s/<name>/`)
-- [x] Multi-user support
-
-## Key Files Changed
-
-- `hop` - Main server, updated `getOrCreateSession()` and WebSocket handler
-- `termshare.js` - New module with Room, RoomManager, createPty
-- `termshare-web/` - Built termshare web client with mobile keyboard
-
-## Testing
-
-1. Start hop: `node hop`
-2. Open browser to tunnel URL
-3. Verify:
-   - [ ] Login with TOTP works
-   - [ ] Terminal session works
-   - [ ] Multiple clients see same session
-   - [ ] Presence indicators show
-   - [ ] Control modes work
-   - [ ] Mobile UI works without injection
-
-## Migration Notes
-
-- Sessions are not persistent (by design) - shell exits when all clients disconnect
-- tmux scrollback replaced by termshare's output buffer (200KB)
-- iOS keyboard injection replaced by termshare's built-in mobile keyboard
-- The old ttyd/tmux code paths are still present but no longer used; can be removed in a cleanup pass
+Key properties:
+
+- No `ttyd` dependency
+- No `tmux` dependency
+- Shared session model across browser, CLI, and MCP
+- Session-aware audit logging and history isolation
+- Native mobile UI and virtual keyboard support
+- Local host process survives hop daemon restarts
+
+## Main Components
+
+- `hop`
+  - user-facing CLI and daemon
+  - auth, Cloudflare tunnel management, session metadata, workspace handling
+  - serves the built Hay web client from `hay-web/`
+- `scripts/hay-host.js`
+  - persistent local host process for active Hay rooms
+  - isolates PTY/runtime ownership from the daemon lifecycle
+- `hay/apps/server`
+  - room manager, PTY lifecycle, websocket protocol handling
+- `hay/apps/web`
+  - browser terminal client
+- `hay/apps/cli`
+  - local terminal client used by Hop for direct attach flows
+- `mcp/hop-mcp.js`
+  - MCP server exposing Hop terminal/session primitives
+
+## Session Model
+
+- Terminal sessions are identified by stable Hop session names.
+- MCP attaches to daemon-scoped terminal handles, but can recover across daemon
+  restart by reattaching through the stable session identity.
+- Active PTYs live in Hay rooms managed by the host process.
+- Browser users, local CLI attaches, and agents all talk to the same underlying
+  room runtime.
+
+## Build Outputs
+
+- `hay/apps/web/dist/` is the source web build output.
+- `hay-web/` is the synced bundle served by `hop`.
+- `npm run build` from the repo root rebuilds Hay and syncs `hay-web/`.
+
+## Operational Notes
+
+- Hop always uses the external Hay host runtime.
+- `HOP_TERMINAL_BACKEND` is accepted only for compatibility and ignored.
+- Session history and audit logging are per-session and workspace-aware.
+- Alternate-screen/TUI applications are captured with TUI-aware audit behavior.
+
+## Legacy Notes
+
+The earlier `ttyd`/`tmux` approach is now only historical context.
+If you see older references in commit history or archived notes, treat them as
+obsolete unless the current code still uses them explicitly.

package/MULTI-AGENT-CHECKLIST.md

@@ -12,19 +12,26 @@ Ensure concurrent agent sessions stay isolated, time-bounded, and recoverable.
 ## Execution Loop
 
 1. **Create session** — one `hop_create_terminal` per agent.
-2. **Validate prompt** — call `hop_wait_terminal(until_prompt=true)` to confirm the shell is ready before sending any command.
-3. **Send command** — write via `hop_write_terminal`; never blind-write into mid-execution output.
-4. **Await result** — use `until_regex` or `until_prompt` with `max_wait_ms` to capture output and detect completion.
-5. **Assert state** — verify exit code / expected output before proceeding.
-6. **Repeat** steps 2–5 for each command in the plan.
+2. **Validate prompt** — use `hop_wait_terminal(until_prompt=true, start_from="latest")` before the first command.
+3. **Preferred turn helper** — use `hopx_agent_turn` or `hopx_send_and_wait` for normal multi-turn work.
+4. **Manual fallback** — drop to `hop_write_terminal` + `hop_wait_terminal` only when you need custom wait conditions or lower-level control.
+5. **Assert state** — verify expected output before continuing.
+6. **Repeat** the loop; never write into a terminal that is still busy.
+
+Recommended defaults:
+
+- `capture="readable_raw"`
+- `start_from="latest"` for prompt waits, `start_from="cursor"` for deltas
+- explicit `max_wait_ms`
+- explicit `idle_ms` when using raw wait conditions
 
 ## Recovery
 
 | Step | Action |
 |------|--------|
-| 1 | Send `ctrl_c` via `hop_send_key` |
-| 2 | If unresponsive after 5 s → `hop_close_terminal(killSession=true)` |
-| 3 | On orchestrator crash → kill **all** child terminals (kill switch) |
-| 4 | Verify zero orphaned processes before declaring shutdown complete |
+| 1 | Send `ctrl_c` via `hop_send_key` or `control="interrupt"` via `hopx_agent_turn` |
+| 2 | If still running, continue waiting or poll an async wait job instead of blind-resending |
+| 3 | If unresponsive after deadline, `hop_close_terminal(killSession=true)` |
+| 4 | On orchestrator crash, close all child terminals and verify there are no orphaned sessions |
 
 **Rule:** Never skip escalation. Always tear down — never leave orphaned sessions.

package/README-MCP.md

@@ -1,12 +1,14 @@
 # Hop MCP Server
 
-The Hop MCP server exposes hop’s terminal/session APIs over the Model Context Protocol (MCP), so agents can create terminals, stream output, and send input through Hop.
+The Hop MCP server exposes Hop terminal and session APIs over MCP so agents can
+create terminals, stream output, send input, and recover cleanly across daemon
+restarts.
 
 ## Quick Start
 
-1) Ensure Hop daemon is running locally (creates `~/.hop2/.tunnel-state`).
+1. Ensure the Hop daemon is running locally by running `hop` (interactive) or `hop start` (daemon only).
 
-2) Auto-configure supported MCP clients:
+2. Auto-configure supported MCP clients:
 
 ```bash
 hop-mcp-setup
@@ -27,7 +29,7 @@ Supported auto-detected clients:
 - VS Code / GitHub Copilot (workspace via `.vscode/mcp.json`)
 - Antigravity
 
-3) Or configure your MCP client manually:
+3. Or configure your MCP client manually:
 
 ```json
 {
@@ -39,51 +41,73 @@ Supported auto-detected clients:
 }
 ```
 
-4) Restart your MCP client.
+4. Restart your MCP client.
 
 The server auto-connects to the local Hop daemon via `~/.hop2/.tunnel-state`.
 
-## One Subagent Terminal (Claude/Codex)
+## Recommended Flow: One Agent Terminal
 
-Recommended flow for driving one agent CLI safely:
+Recommended flow for driving one Claude/Codex/Gemini terminal safely:
 
-1) Launch a dedicated terminal:
+1. Launch a dedicated terminal (the returned `id` is the `terminal_id` used by every other tool):
 
 ```
 hop_create_terminal(name="subagent", cwd="/path/to/repo")
 ```
 
-2) Start the agent CLI:
+2. Start the agent CLI:
 
 ```
-hop_write_terminal(data="claude\n")
+hop_write_terminal(terminal_id=<id>, data="claude\n")
 # or
-hop_write_terminal(data="codex\n")
+hop_write_terminal(terminal_id=<id>, data="codex\n")
 ```
 
-3) Wait for a prompt before sending work:
+3. Wait for a prompt before sending work:
 
 ```
-hop_wait_terminal(until_prompt=true, start_from="latest")
+hop_wait_terminal(terminal_id=<id>, until_prompt=true, start_from="latest")
 ```
 
-4) Run a multi-turn loop:
-- preferred: `hopx_send_and_wait(data="<task>", press_enter=true, capture="readable_raw")`
-- helper alternative: `hopx_agent_turn(data="<task>", mode="auto")`
-- manual split (if needed):
-- send one instruction with `hop_write_terminal(data="<task>\n")`
-- wait for output with `hop_wait_terminal(capture="readable_raw")` (defaults to `until_agent_done`)
-- continue with cursor-deltas via `hop_read_terminal(start_from="cursor", cursor=...)`
-- send the next instruction only after output is captured
+4. Run a multi-turn loop:
+- Preferred: `hopx_agent_turn(terminal_id=<id>, data="<task>", mode="auto")`
+- Good lower-level helper: `hopx_send_and_wait(terminal_id=<id>, data="<task>", press_enter=true, capture="readable_raw")`
+- Manual split only when needed:
+- send with `hop_write_terminal(terminal_id=<id>, data="<task>\n")`
+- wait with `hop_wait_terminal(terminal_id=<id>, capture="readable_raw")`
+- continue via cursor deltas from `hop_read_terminal(terminal_id=<id>, start_from="cursor", cursor=...)`
 
-5) Stop/cleanup:
-- interrupt with `hop_send_key(key="ctrl_c")`
-- close with `hop_close_terminal(killSession=false)`
+5. Stop or clean up:
+- interrupt with `hop_send_key(terminal_id=<id>, key="ctrl_c")`
+- close with `hop_close_terminal(terminal_id=<id>, killSession=false)`
 
 Safety notes:
 - Keep one subagent per terminal (no multiplexing multiple agent CLIs in one PTY).
 - For existing user sessions, verify `agentPermitted` (or set with `hop_set_agent_permission`) before writing input.
 
+## Shell Commands: `hopx_exec`
+
+For plain shell work (not interactive TUIs), `hopx_exec` behaves like a Bash
+tool on a persistent terminal: it sends the command, waits for the next shell
+prompt, and returns clean plain-text output.
+
+Worked example:
+
+```
+hop_create_terminal(name="builder", cwd="/path/to/repo")   # returns id
+hopx_exec(terminal_id=<id>, command="npm test")
+# -> { "ok": true, "exit_code": 0, "stdout": "...", "next_cursor": ... }
+hopx_exec(terminal_id=<id>, command="ls /nope")
+# -> { "ok": true, "exit_code": 1, "stdout": "ls: /nope: No such file or directory", ... }
+```
+
+Semantics:
+- `ok` means the shell prompt returned within `timeout_ms` — NOT that the command succeeded.
+- `exit_code` is the command's real exit status, captured by appending a POSIX sentinel (`<cmd>; printf '\n__HOPX_RC_<nonce>=%d\n' "$?"`). It is `null` when the sentinel was never observed (timeout, or a shell without POSIX `$?` semantics) — in that case behavior degrades gracefully to prompt-based capture.
+- The sentinel line, the echoed command, the trailing prompt, and ANSI codes are stripped from `stdout`.
+- On timeout, `timed_out: true` is set and `ok` is `false`.
+- For SSH sessions or unusual prompts, pass `prompt_regex`; for commands with non-standard output endings, pass `idle_ms` as a fallback condition.
+
 ## Remote Hop
 
 To connect to a remote hop instance, set env vars or use `connect_server`:
@@ -107,29 +131,32 @@ connect_server(base_url="https://hop2.example.com", token="<token>", verify=true
 ## Tools
 
 Core tools (`hop_`): stable atomic operations.
-- `connect_server`
-- `hop_server_info`
-- `hop_list_sessions`
-- `hop_list_terminals`
-- `hop_create_terminal`
-- `hop_attach_terminal`
-- `hop_write_terminal`
-- `hop_send_key`
-- `hop_wait_terminal`
-- `hop_wait_start`
-- `hop_wait_poll`
-- `hop_resize_terminal`
-- `hop_read_terminal`
-- `hop_close_terminal`
-- `hop_set_agent_permission`
-- `hop_list_workspaces`
-- `hop_save_workspace`
-- `hop_load_workspace`
-- `hop_use_workspace`
+- `connect_server` — connect to a Hop API base_url (optional token); use for remote hop instances
+- `hop_server_info` — hop-mcp runtime diagnostics (version, script path, read-mode capabilities)
+- `hop_list_sessions` — list Hop sessions and metadata
+- `hop_list_terminals` — list terminal API sessions (created via `hop_create_terminal` / `hop_attach_terminal`)
+- `hop_create_terminal` — create a terminal session and optionally run a startup command; returns the `id` used as `terminal_id` everywhere else
+- `hop_attach_terminal` — attach to an existing terminal session by name or internalName
+- `hop_write_terminal` — write raw input to a terminal session
+- `hop_send_key` — send a named keypress (enter, esc, tab, shift_tab, arrows, f1-f12, ctrl+[a-z], ...)
+- `hop_wait_terminal` — wait for output conditions (regex, prompt, idle, agent_done); `async=true` returns a `wait_id`
+- `hop_wait_start` — deprecated legacy alias for `hop_wait_terminal(async=true)`
+- `hop_wait_poll` — poll or await completion of a background wait job
+- `hop_resize_terminal` — resize the terminal PTY
+- `hop_read_terminal` — read terminal output events (default `mode="readable_raw"`); returns a cursor for incremental reads
+- `hop_close_terminal` — detach the terminal API session; optionally kill the underlying hop session
+- `hop_set_agent_permission` — allow or block agent access for a session
+- `hop_list_workspaces` — list available workspaces
+- `hop_create_workspace` — create an empty workspace by name
+- `hop_show_workspace` — show saved definitions in a workspace
+- `hop_save_workspace` — save a workspace snapshot from live sessions
+- `hop_delete_workspace` — delete a workspace by name
+- `hop_load_workspace` — load a workspace and optionally start sessions
 
 Helper tools (`hopx_`): convenience wrappers built on top of core tools.
-- `hopx_send_and_wait` (single-call send + wait wrapper)
-- `hopx_agent_turn` (single-turn send + wait + mode-aware output, default `mode="auto"`)
+- `hopx_send_and_wait` — single-call send + wait wrapper
+- `hopx_exec` — Bash-tool-style shell execution: command in, clean stdout + `exit_code` out (see section above)
+- `hopx_agent_turn` — single-turn send + wait + mode-aware output, default `mode="auto"`
 
 ## Resources
 
@@ -143,6 +170,18 @@ Helper tools (`hopx_`): convenience wrappers built on top of core tools.
 - `terminal_id` is an ephemeral daemon attachment handle; underlying hop `sessionName` is the stable identity.
 - On daemon restart, terminal-scoped tools auto-reattach once using stored session identity and retry transparently when possible.
 - `hop_read_terminal` returns a cursor for incremental polling.
+- Default fallback terminal size is `140x40` when no explicit `cols`/`rows` are provided.
+- `hop_create_terminal` and `hop_attach_terminal` do a short output warmup before returning, reducing first-command races.
+- Hop isolates shell history per session and writes per-session NDJSON audit logs under `~/.hop2/workspaces/<workspace>/logs/<session>/audit.ndjson`.
+- Audit logging automatically switches to diff-suppressed `tui_keyframe` snapshots for alternate-screen/TUI apps while still capturing all input.
+
+### `hop_read_terminal`
+
+- Defaults are token-thrifty:
+  - `mode` defaults to `"readable_raw"` (pass `"raw"` or `"ui"` explicitly when needed)
+  - `maxEvents` defaults to `200` and `maxBytes` to `65536`; pass `0` explicitly for unlimited (the buffer keeps the last 2000 events)
+  - when these caps truncate a read, the payload includes `truncated: true` plus a `hint`; continue with `start_from="cursor"`, `cursor=<next_cursor>`
+  - `start_from` defaults to `cursor` when `cursor` is provided, otherwise `beginning`
 - `hop_read_terminal` supports `mode: "ui"` for structured terminal snapshots:
   - `ui.lines`: visible screen lines
   - `ui.cursor`: cursor position
@@ -166,6 +205,9 @@ Helper tools (`hopx_`): convenience wrappers built on top of core tools.
   - parses incrementally across chunk boundaries to avoid split ANSI artifacts
   - reconstructs common inline edits (backspace/cursor rewrites) so text is cleaner for agent parsing
   - avoids full ANSI noise while retaining interaction structure
+
+### `hop_wait_terminal`
+
 - `hop_send_key` sends normalized named keys (for example `enter`, `esc`, `up`, `ctrl_c`) to improve interactive TUI control without raw escape-string handling.
 - `hop_wait_terminal` blocks on output conditions without client polling loops:
   - `start_from` controls where matching begins:
@@ -180,15 +222,24 @@ Helper tools (`hopx_`): convenience wrappers built on top of core tools.
   - quiet-period detection via `idle_ms`
   - bounded event capture in `raw` or `readable_raw`
   - for `capture="readable_raw"`, supports `control_level`, `noise_filter`, `coalesce_ms`, `coalesce_max_chars`, and optional `includeMetaEvents`
-- `hop_wait_start` / `hop_wait_poll` provide explicit async wait orchestration for long-running conditions:
-  - start with `hop_wait_start(...)` and poll with `hop_wait_poll(wait_id=..., wait=true)`
+
+### Async wait helpers
+
+- For long-running conditions, start a background wait with `hop_wait_terminal(terminal_id=<id>, async=true, ...)` (returns `wait_id` immediately) and poll with `hop_wait_poll(wait_id=..., wait=true)`
   - use `consume=true` on poll to remove completed jobs
+- `hop_wait_start` is a deprecated legacy alias for `hop_wait_terminal(async=true)`; prefer the latter
+
+### `hopx_send_and_wait`
+
 - `hopx_send_and_wait` combines input send + optional enter/key + wait, and defaults to cursor-based delta capture from just before send.
   - helper defaults are token-thrifty for interactive loops: `capture_max_events=60`, `control_level=none`, `noise_filter=balanced`, `coalesce_ms=350` (unless overridden)
   - for `capture="readable_raw"`, `text_only` defaults to `true`; this condenses wait payloads by returning joined `wait.text`, setting `wait.events=[]`, `wait.eventCount=0`, and preserving prior count in `wait.originalEventCount` (set `text_only=false` to keep full events)
 - `hop_read_terminal` supports deterministic delta reads:
   - `start_from`: `beginning` (default), `cursor`, or `latest`
   - response includes `cursorStart`, `cursorEnd`, and `next_cursor`
+
+### `hopx_agent_turn`
+
 - `hopx_agent_turn` is a helper wrapper (not a core primitive):
   - default `mode="auto"` chooses `ui` when terminal is in alternate-screen, otherwise `readable_raw`
   - in `mode="auto"`, if alternate-screen starts during the same turn, it auto-promotes to `ui` for that response
@@ -202,12 +253,14 @@ Helper tools (`hopx_`): convenience wrappers built on top of core tools.
   - `mode="ui"` defaults wait capture to `capture_max_events=0` (override when you need readable wait-event diagnostics)
   - when using default `until_agent_done` semantics in `mode="ui"`, hopx applies a short UI busy-guard pass to avoid returning while status lines still indicate active work (`esc to interrupt`, `working`, `waiting for process`, etc.)
   - `mode="readable_raw"` / `mode="raw"` returns the same wait payload shape as `hopx_send_and_wait`
-- Tool calls that fail at the Hop API layer now return MCP errors (`isError: true`) with normalized fields:
+
+### Error model
+
+- Tool calls that fail at the Hop API layer return MCP tool errors (`isError: true`) with normalized fields:
   - `ok`
   - `status`
   - `endpoint`
   - `error`
-- Hop now isolates shell history per session and writes per-session NDJSON audit logs under `~/.hop2/workspaces/<workspace>/logs/<session>/audit.ndjson` with truncation metadata for large chunks.
-- Audit logging auto-switches to diff-suppressed `tui_keyframe` snapshots when alternate-screen apps (vim/htop/agent TUIs) are detected, while still capturing all input events.
-- Default fallback terminal size is `140x40` when no explicit `cols`/`rows` are provided.
-- `hop_create_terminal` / `hop_attach_terminal` now do a short output warmup before returning, reducing first-command races in interactive shells.
+  - `hint` (when a remedy is known)
+- Connection-level failures (daemon not running, unreachable host, timeouts) also come back as normalized tool errors with `status: null` and a `hint` explaining how to start the daemon (`hop` or `hop start`) or reconnect via `connect_server(base_url=...)` — not as raw JSON-RPC protocol errors.
+- A `403` "Agent access not permitted" error includes a hint to enable access with `hop_set_agent_permission(name=..., allowed=true)` or `hop session permit <name>`.