agello 0.3.0 → 0.4.1

package/README.md

@@ -5,16 +5,16 @@
 - Chat with the agent from a web page; terminal input, browser input and agent replies are shown as separate bubbles
 - Live session status, tool activity, and prompts queued while the agent is busy (same order as the terminal)
 - Conversation saved per pane in the browser, kept across reloads
+- Interactive terminal view of the existing herdr pane: live cursor, keyboard input, and automatic resize
 - Optional live view of a `terminal-browser` screen, with an on/off switch to control it yourself
-- Annotate: point at an element on that screen and send a request about it, with its CSS selector
 - Embeddable `<agent-bridge>` web component
 
-Currently supports **Claude Code** sessions.
+Chat currently supports **Claude Code** sessions. The terminal view connects directly to the pane, including ordinary shells and other terminal applications.
 
 ## Requirements
 
 - [Bun](https://bun.sh) >= 1.1
-- [herdr](https://herdr.dev) (the agent must run inside a herdr pane)
+- [herdr](https://herdr.dev) (the agent must run inside a herdr pane); interactive terminal view requires `herdr terminal session control`
 - Optional: `terminal-browser` for the screen panel
 
 ## Install
@@ -60,18 +60,6 @@ Messages sent from the page arrive in the agent as:
 
 The approve / reject buttons send `action=approve` / `action=reject`.
 
-### Annotate
-
-The `주석` button on the screen panel turns on annotate mode: hovering tints the element under the pointer (with its tag and size), clicking outlines it and opens an input next to it. The wheel still scrolls the page, and the outline follows the element as the page changes. The request arrives as:
-
-```
-[browser] action=annotate make this red / and bolder
-대상: #app > main > button:nth-of-type(2)
-요소: button.primary "Save" · 위치 120,340 크기 96x32 · http://localhost:3000/
-```
-
-Selector: nearest unique `id` / `data-testid` ancestor, then `tag:nth-of-type` steps (light DOM only). Coordinates are CSS px of the viewport. Off while presenting and while "내 조작" is on.
-
 ### Commands
 
 | Command | Description |
@@ -82,13 +70,25 @@ Selector: nearest unique `id` / `data-testid` ancestor, then `tag:nth-of-type` s
 | `agello open [--port N \| --pane ID]` | Open the page (default: this pane's server) |
 | `agello present [x,y,w,h] [--port N \| --pane ID]` | Presentation mode (see below) |
 | `agello present stop` | End presentation mode |
-| `agello script load <file> \| show` | Load / show a presentation script |
-| `agello present resume \| pause \| goto <n[.m]>` | Play the script from where it stopped / stop it / move to a step |
 
 `start` options: `--pane <id>`, `--port <n>` (default: first free from 8765), `--session <id>`, `--browser <terminal-browser key>`, `--allow-origin <origin>` (repeatable), `--open`, `--foreground`.
 
 State and logs: `~/.local/state/agello/<port>.json`, `<port>.log`.
 
+### Pane picker
+
+Click the status in the header to open a drawer with herdr's workspace -> tab -> pane tree (the current pane's workspace is expanded; a workspace with a single tab lists its panes directly). Search filters by workspace, title, folder, or agent. Picking a pane switches the page to that pane's server, starting it if needed (`?server=` keeps the choice across reloads). Panes without Claude Code open as a terminal, since chat supports Claude Code only.
+
+`+` adds a workspace (top), a tab (workspace row), or splits a pane (pane row); new panes open as a terminal. Nothing can be closed from the page: closing a herdr pane ends its processes.
+
+### Interactive terminal
+
+Click the terminal icon in the header to replace the chat with the pane's live terminal (the screen panel stays). Keyboard input (including Korean text, arrow keys, and Ctrl+C) goes directly to that terminal. The viewport follows the browser size, and the mouse wheel scrolls through Herdr. Click the icon again or close the page to release control; the pane and its process keep running. If disconnected, use **다시 연결** to restore the current screen.
+
+Only one browser connection can control a pane through this server at a time. Agello never forcibly takes over another Herdr controller. While connected, resizing the browser also changes the source terminal size. Terminal input is raw input, without the chat's `[browser]` prefix.
+
+The xterm.js renderer and its styles are served locally; no terminal CDN is required. Herdr sends an initial ANSI screen and subsequent frame updates over the WebSocket, including cursor state. Terminal frames are not stored in browser chat history.
+
 ### Presentation mode
 
 `agello present x,y,w,h` (CSS px of the browser's visible viewport; omit for the whole screen):
@@ -102,23 +102,6 @@ State and logs: `~/.local/state/agello/<port>.json`, `<port>.log`.
 
 Run `present` again to move the crop. User control is off while presenting.
 
-#### Scripts
-
-Write the talk ahead so each line appears the moment its screen does:
-
-```json
-{ "rect": "28,157,688,477",
-  "steps": [
-    { "go": "#1", "say": ["First line", "Second line"] },
-    { "go": "#2", "say": ["..."], "hold": 6 } ] }
-```
-
-- step: `go` brings the screen there (`#n` sets `location.hash`, anything else is JS run in the page; agello knows nothing about the deck), `say` lines are one bubble each, `hold` seconds a line stays on screen (default from length, 10 to 20s; never below 10s). Pacing: screen change, 1s, line, fade out, 0.3s, next line; after a step's last line 0.7s before the next screen
-- `present resume` plays from where it stopped (starts presentation mode with the script's `rect`); it first re-runs the current step's `go`, so the agent may move the screen freely while answering
-- raising a hand pauses at once; the agent gets `action=hand-raise` with the position (`마지막 표시 2.1 · 다음 2.2`)
-- `script load` again after editing keeps the position; `present goto 2.2` then `present resume` replays from a fixed line
-- at the end the agent gets `action=present-done`; while the script is paused and the agent is working, the page shows a small loader
-
 ## Embed
 
 ```html
@@ -142,11 +125,17 @@ See `web/embed-example.html`.
 - **Send**: `herdr agent prompt <pane> "<text>"` (kept to 3 lines so Claude Code does not treat it as a paste)
 - **Status**: `herdr agent get` / `herdr pane get`, polled every 1.5s
 - **Chat**: tails the Claude Code transcript (`$CLAUDE_CONFIG_DIR/projects/*/<session>.jsonl`) and streams it over SSE
+- **Panes**: `GET /panes` (`herdr workspace/tab/pane list`), `POST /panes/connect` (`agello start --pane`), `POST /panes/create` (`herdr workspace create`, `tab create`, `pane split`, always `--no-focus`)
+- **Terminal**: `/terminal` WebSocket relays `herdr terminal session control <pane>` ANSI frames and validated input, resize, and scroll commands to xterm.js
 - **Screen**: finds the terminal-browser CDP port via `terminal-browser ls --json` and relays `Page.startScreencast` frames; user control forwards `Input.*` events only
 
 ## Security
 
-The server listens on `127.0.0.1` only. Data and input endpoints accept requests from the same origin and `localhost` pages only; other origins get `403` unless added with `--allow-origin`. Anything allowed to post to `/send` can type into your agent session, so keep the allow list short.
+The server listens on `127.0.0.1` only. Data and input endpoints accept requests from the same origin and `localhost` pages only; other origins get `403` unless added with `--allow-origin`. Allowed origins can send chat prompts and connect to `/terminal` to type directly into the pane, so keep the allow list short.
+
+## Development
+
+Run `bun install` and `bun test`. Tests cover the pane tree and creation (against a mock herdr), local terminal assets, origin rejection, ANSI frames, keyboard bytes, resize, exclusive control, and reconnect.
 
 ## License
 

package/bin/agello.ts

@@ -27,13 +27,6 @@ Usage:
                          whole screen), with agent replies as bubbles that fade after 10s.
                          Run again to move the crop.
   ${NAME} present stop [--port N|--pane ID]       End presentation mode (bubbles stay in the chat)
-  ${NAME} script load <file.json>   Load a presentation script (keeps the current position):
-                         {"rect": "x,y,w,h", "steps": [{"go": "#1", "say": ["line", ...], "hold": 6}]}
-                         go: "#n" sets location.hash, otherwise JS run in the page; one line = one bubble
-  ${NAME} script show               Script with positions (step.line) and player state
-  ${NAME} present resume            Play the script from where it stopped (starts presentation mode)
-  ${NAME} present pause             Stop the script (raising a hand on the page also pauses)
-  ${NAME} present goto <n[.m]>      Move to step n (line m) and show its screen
   ${NAME} help | --version
 
 start options:
@@ -291,56 +284,6 @@ async function cmdOpen(argv: string[]) {
   console.log(inst.url);
 }
 
-async function api(url: string, path: string, body?: object): Promise<any> {
-  try {
-    const r = await fetch(`${url}${path}`, {
-      method: body ? "POST" : "GET",
-      headers: { "Content-Type": "application/json" },
-      body: body ? JSON.stringify(body) : undefined,
-      signal: AbortSignal.timeout(8000),
-    });
-    return await r.json();
-  } catch {
-    return fail(`server unreachable: ${url}`);
-  }
-}
-
-const playerLine = (p: any) =>
-  `script: ${p.state}${p.last ? ` last=${p.last}` : ""}${p.next ? ` next=${p.next}` : ""} steps=${p.steps}`;
-
-async function cmdScript(argv: string[]) {
-  const { values: o, positionals } = parseArgs({
-    args: argv,
-    options: { port: { type: "string" }, pane: { type: "string" } },
-    allowPositionals: true,
-  });
-  const [sub, file] = positionals;
-  const inst = await resolveTarget(o);
-  if (sub === "load") {
-    if (!file) fail("usage: script load <file.json>");
-    let script: unknown;
-    try {
-      script = await Bun.file(file!).json();
-    } catch (e) {
-      fail(`cannot read ${file}: ${(e as Error).message}`);
-    }
-    const res = await api(inst.url, "/script", { script });
-    if (!res.ok) fail(res.error);
-    console.log(playerLine(res.player));
-  } else if (sub === "show" || !sub) {
-    const res = await api(inst.url, "/script");
-    console.log(playerLine(res.player));
-    for (const [i, st] of (res.script?.steps ?? []).entries()) {
-      console.log(`${i + 1}${st.go ? `  go=${st.go}` : ""}${st.hold ? `  hold=${st.hold}s` : ""}`);
-      st.say.forEach((l: string, j: number) => {
-        const at = `${i + 1}.${j + 1}`;
-        const mark = at === res.player.next ? ">" : at === res.player.last ? "*" : " ";
-        console.log(` ${mark} ${at}  ${l}`);
-      });
-    }
-  } else fail(`unknown script command: ${sub}`);
-}
-
 async function cmdPresent(argv: string[]) {
   const { values: o, positionals } = parseArgs({
     args: argv,
@@ -348,14 +291,6 @@ async function cmdPresent(argv: string[]) {
     allowPositionals: true,
   });
   const inst = await resolveTarget(o);
-  const [sub, at] = positionals;
-  if (sub === "resume" || sub === "pause" || sub === "goto") {
-    if (sub === "goto" && !at) fail("usage: present goto <n[.m]>");
-    const res = await api(inst.url, "/player", { cmd: sub, at });
-    if (!res.ok) fail(`${res.error}${res.player ? ` (${playerLine(res.player)})` : ""}`);
-    console.log(playerLine(res.player));
-    return;
-  }
   const arg = positionals.join(",");
   let body: object;
   if (arg === "stop") body = { stop: true };
@@ -402,9 +337,6 @@ switch (cmd) {
   case "present":
     await cmdPresent(rest);
     break;
-  case "script":
-    await cmdScript(rest);
-    break;
   case "--version":
   case "-v":
     console.log(pkg.version);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agello",
-  "version": "0.3.0",
+  "version": "0.4.1",
   "description": "Talk to a coding agent in a herdr pane from the browser: chat, live status, and a shared browser screen.",
   "keywords": [
     "herdr",
@@ -31,5 +31,9 @@
   "engines": {
     "bun": ">=1.1"
   },
-  "license": "MIT"
+  "license": "MIT",
+  "dependencies": {
+    "@xterm/addon-fit": "0.11.0",
+    "@xterm/xterm": "6.0.0"
+  }
 }

package/src/panes.ts

@@ -0,0 +1,86 @@
+// herdr workspace -> tab -> pane tree, creation, and switching to a pane's own
+// agello server (one server per pane; started on demand through the CLI).
+// Deliberately no close/delete: closing a herdr pane kills its processes.
+
+import { join } from "node:path";
+import { run, runJson } from "./run.ts";
+
+const CLI = join(import.meta.dir, "..", "bin", "agello.ts");
+const ID = /^[A-Za-z0-9]+:[A-Za-z0-9]+$/; // herdr ids: w23, w23:t1, w23:p3
+const WS_ID = /^[A-Za-z0-9]+$/;
+
+// Resolve herdr on the current PATH (Bun.spawn uses the PATH from process start).
+const bin = () => Bun.which("herdr", { PATH: process.env.PATH }) ?? "herdr";
+const herdr = async (...cmd: string[]) => (await runJson([bin(), ...cmd], 5000)) ?? { error: { code: "herdr_failed" } };
+
+export type TreePane = { id: string; agent?: string; status: string; title?: string; cwd?: string; focused: boolean };
+export type TreeTab = { id: string; label: string; number: number; panes: TreePane[] };
+export type TreeWorkspace = { id: string; label: string; number: number; tabs: TreeTab[] };
+
+export async function paneTree(): Promise<TreeWorkspace[] | null> {
+  const [w, t, p] = await Promise.all([herdr("workspace", "list"), herdr("tab", "list"), herdr("pane", "list")]);
+  if (!w.result || !t.result || !p.result) return null;
+  const tabs = new Map<string, TreeTab>();
+  for (const tab of t.result.tabs)
+    tabs.set(tab.tab_id, { id: tab.tab_id, label: tab.label, number: tab.number, panes: [] });
+  for (const pane of p.result.panes)
+    tabs.get(pane.tab_id)?.panes.push({
+      id: pane.pane_id,
+      agent: pane.agent,
+      status: pane.agent_status,
+      title: pane.terminal_title_stripped || undefined,
+      cwd: pane.foreground_cwd || pane.cwd,
+      focused: pane.focused,
+    });
+  return w.result.workspaces.map((ws: any) => ({
+    id: ws.workspace_id,
+    label: ws.label,
+    number: ws.number,
+    tabs: t.result.tabs
+      .filter((tab: any) => tab.workspace_id === ws.workspace_id)
+      .map((tab: any) => tabs.get(tab.tab_id)!),
+  }));
+}
+
+const text = (v: unknown, max = 200) => (typeof v === "string" && v.trim() && v.length <= max ? v.trim() : undefined);
+const fail = (error: string, status = 400) => Response.json({ ok: false, error }, { status });
+
+// POST /panes/connect {pane} -> {ok, url}: the pane's server, started if needed.
+async function connect(pane: unknown): Promise<Response> {
+  if (typeof pane !== "string" || !ID.test(pane)) return fail("invalid_pane");
+  if (!(await herdr("pane", "get", pane)).result) return fail("pane_not_found", 404);
+  const out = await run([process.execPath, CLI, "start", "--pane", pane], 20000);
+  const url = out?.match(/https?:\/\/127\.0\.0\.1:\d+/)?.[0];
+  return url ? Response.json({ ok: true, pane, url }) : fail("server_start_failed", 502);
+}
+
+// POST /panes/create {kind: workspace|tab|pane, ...} -> {ok, pane}: the new root pane.
+async function create(d: any): Promise<Response> {
+  const cwd = text(d.cwd, 1024);
+  if (cwd && !cwd.startsWith("/")) return fail("invalid_cwd");
+  const label = text(d.label, 80);
+  const opt = [...(cwd ? ["--cwd", cwd] : []), ...(label ? ["--label", label] : []), "--no-focus"];
+  let res: any;
+  if (d.kind === "workspace") res = await herdr("workspace", "create", ...opt);
+  else if (d.kind === "tab") {
+    if (typeof d.workspace !== "string" || !WS_ID.test(d.workspace)) return fail("invalid_workspace");
+    res = await herdr("tab", "create", "--workspace", d.workspace, ...opt);
+  } else if (d.kind === "pane") {
+    if (typeof d.pane !== "string" || !ID.test(d.pane)) return fail("invalid_pane");
+    if (d.direction !== "right" && d.direction !== "down") return fail("invalid_direction");
+    res = await herdr("pane", "split", d.pane, "--direction", d.direction, ...(cwd ? ["--cwd", cwd] : []), "--no-focus");
+  } else return fail("invalid_kind");
+  const pane = res.result?.root_pane?.pane_id ?? res.result?.pane?.pane_id;
+  return pane ? Response.json({ ok: true, pane }) : fail(res.error?.code ?? "herdr_failed", 502);
+}
+
+export async function panesRoute(req: Request, path: string, current: string): Promise<Response | null> {
+  if (path === "/panes" && req.method === "GET") {
+    const tree = await paneTree();
+    return tree ? Response.json({ ok: true, current, workspaces: tree }) : fail("herdr_failed", 502);
+  }
+  if (req.method !== "POST" || (path !== "/panes/connect" && path !== "/panes/create")) return null;
+  const d = (await req.json().catch(() => null)) as any;
+  if (!d || typeof d !== "object") return fail("invalid_body");
+  return path === "/panes/connect" ? connect(d.pane) : create(d);
+}

package/src/screencast.ts

@@ -268,75 +268,5 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
     return browser ? { connected: false, browser: browser.key } : { connected: false, reason };
   }
 
-  // Run JS in the relayed tab (presentation script `go`). Needs the relay to be
-  // connected, i.e. at least one viewer. Returns false if it could not run.
-  async function evaluate(expression: string): Promise<boolean> {
-    const r = await request("Runtime.evaluate", { expression, awaitPromise: true });
-    return !!r && !r.exceptionDetails;
-  }
-
-  // Annotation: which element is at a point (hover / click) or where a
-  // selected element is now (by selector). Coordinates are CSS px of the
-  // visible viewport, same as the full screencast frame. Not while cropped.
-  async function inspect(q: InspectQuery): Promise<Inspected | null> {
-    if (clip) return null;
-    const r = await request("Runtime.evaluate", {
-      expression: `(${INSPECT_JS})(${JSON.stringify(q)})`,
-      returnByValue: true,
-    });
-    return r && !r.exceptionDetails ? (r.result?.value ?? null) : null;
-  }
-
-  return { handle, input, describe, setClip, evaluate, inspect };
+  return { handle, input, describe, setClip };
 }
-
-export type InspectQuery = { x?: number; y?: number; selector?: string; full?: boolean };
-export type Inspected = {
-  selector: string;
-  label: string; // tag#id.class, for the hover tooltip
-  rect: { x: number; y: number; w: number; h: number };
-  text?: string; // full only
-  url?: string; // full only
-};
-
-// Runs in the page. Selector: nearest unique id / data-testid ancestor, then
-// tag:nth-of-type steps down to the element (light DOM only).
-const INSPECT_JS = String((q: InspectQuery) => {
-  let el: Element | null = null;
-  if (q.selector) {
-    try {
-      el = document.querySelector(q.selector);
-    } catch {}
-  } else el = document.elementFromPoint(q.x ?? 0, q.y ?? 0);
-  if (!el || el === document.documentElement) return null;
-  const esc = CSS.escape;
-  const unique = (s: string) => document.querySelectorAll(s).length === 1;
-  const step = (e: Element): [string, boolean] => {
-    if (e.id && unique(`#${esc(e.id)}`)) return [`#${esc(e.id)}`, true];
-    for (const a of ["data-testid", "data-test", "data-cy"]) {
-      const v = e.getAttribute(a);
-      if (v && unique(`[${a}="${esc(v)}"]`)) return [`[${a}="${esc(v)}"]`, true];
-    }
-    const same = e.parentElement ? [...e.parentElement.children].filter((c) => c.localName === e.localName) : [];
-    return [same.length > 1 ? `${e.localName}:nth-of-type(${same.indexOf(e) + 1})` : e.localName, false];
-  };
-  const parts: string[] = [];
-  for (let e: Element | null = el; e && e !== document.documentElement; e = e.parentElement) {
-    const [s, anchor] = step(e);
-    parts.unshift(s);
-    if (anchor || e === document.body) break;
-  }
-  const cls = [...el.classList].slice(0, 2).map((c) => `.${c}`).join("");
-  const r = el.getBoundingClientRect();
-  const out: any = {
-    selector: parts.join(" > "),
-    label: `${el.localName}${el.id ? `#${el.id}` : ""}${cls}`,
-    rect: { x: r.x, y: r.y, w: r.width, h: r.height },
-  };
-  if (q.full) {
-    const t = (el as HTMLInputElement).value || el.getAttribute("aria-label") || el.getAttribute("alt") || el.textContent || "";
-    out.text = t.replace(/\s+/g, " ").trim().slice(0, 80);
-    out.url = location.href;
-  }
-  return out;
-});