aipeek 0.2.7 → 0.2.9

package/README.md

@@ -1,10 +1,12 @@
 # aipeek
 
-Gives AI a peek into — and a hand on — your running browser app. Reads the UI tree (React fiber), semantic DOM, console, network, errors, and store state; drives the page (click/fill/press/wait/screenshot). All over plain-text HTTP on your Vite dev server — zero resident context cost, unlike a browser MCP whose tool schemas sit in the model's context whether used or not.
+Gives AI a peek into — and a hand on — your running browser app. Reads the UI tree (React fiber), semantic DOM, console, network, errors, and store state; drives the page (click, fill, press, wait, drag/drop, clipboard, screenshot) and profiles it. All over plain-text HTTP on your Vite dev server — zero resident context cost, unlike a browser MCP whose tool schemas sit in the model's context whether used or not.
 
 **10× faster end-to-end.** What you feel is wall-clock from prompt to done — model thinking, round-trips, all of it. Screenshot agents (Playwright + vision) pay 2–5s of pixel-parsing *every step*; aipeek reads semantic text (instant) and batches a whole interaction into one round-trip with `/chain` — the model thinks once, not N times.
 
-It lives **inside** the open page (injected client + HMR channel), so it reads React/store internals a DOM-only driver can't, and acts on the current tab with no separate browser process. It does **not** open browsers, navigate, run headless, or fire real pointer events — it's the dev inner loop, not E2E. For that, use Playwright.
+It lives **inside** the open page (injected client + HMR channel), so it reads React/store internals a DOM-only driver can't, and acts on the current tab with no separate browser process. It does **not** open browsers, navigate, or run headless — it's the dev inner loop, not E2E. For that, use Playwright. (It *can* fire trusted OS-level pointer events via `realclick` when a synthetic click won't do — see Actions.)
+
+**Self-healing.** The transport rides the Vite HMR socket; the injected client re-announces itself on connect and polls a process-level `BOOT_ID` over HTTP, so a full dev-server restart re-handshakes automatically instead of stranding the page until a human hits ⌘R.
 
 ## Install
 
@@ -29,21 +31,31 @@ export default defineConfig({
 
 ## API Endpoints
 
-All endpoints are available on your Vite dev server:
+All endpoints are available on your Vite dev server. Reads are listed cheapest-first.
 
 | Endpoint | Description |
 |----------|-------------|
-| `GET /__aipeek/screen` | **State-machine projection** — `{view, modal, focus, knobs}`. Start here. |
-| `GET /__aipeek` | Summary of all sections (UI, console, network, errors, state) |
-| `GET /__aipeek/{section}` | Detail for a section (`ui`, `console`, `network`, `errors`, `state`) |
+| `GET /__aipeek/screen` | **State-machine projection** — `{view, modal, focus, knobs, domain}`. Each read prints a `token: tN`. Start here. |
+| `GET /__aipeek/screen?since=tN` | **Delta** — only what moved since token `tN` (view/modal/focus + new errors/failed requests), `(no state change)` if nothing did. The cheap "what changed after I acted" read. |
+| `GET /__aipeek` | Summary of all sections (UI, console, network, errors, state); healthy sections fold to one line, issues expand |
+| `GET /__aipeek?full` | Full dump — UI tree + console + network + errors + state |
+| `GET /__aipeek/{section}` | Detail for a section (`ui`, `console`, `network`, `errors`, `state`, `profile`) |
 | `GET /__aipeek/{section}/{index}` | Detail for a specific item in a section |
 | `GET /__aipeek/{section}?full` | Full detail (no truncation) |
 | `GET /__aipeek/dom[?scope=Name\|?sel=css]` | Semantic DOM — UI as text (see below) |
 | `GET /__aipeek/query?sel=css` | Read-side twin of `sel=`: a selector's live `count` + each match's `text`/`visible`/`attrs` (role, `data-state`, `aria-*`/`data-*`, value, disabled). Per-element assertions without `/eval`. |
+| `GET /__aipeek/check` | Pass/fail health check (no console errors / no uncaught errors / no failed requests / UI rendered). `417` when any assertion fails. Use after a code change. |
+| `GET /__aipeek/console` · `/network` · `/errors` · `/state` | Section shortcuts (same as `/{section}`) |
+| `GET /__aipeek/profile` | Performance profiler — which component/function is burning frames. `/profile/reset` clears the window; `/profile/diff` gives a baseline→after IMPROVED/REGRESSED verdict |
 | `GET /__aipeek/{action}?...` | Drive the page (see Actions) |
+| `GET /__aipeek/tabs` | List live tabs (id, visible/background, title) for `?tab=` addressing |
+| `GET /__aipeek/timeline` | Interleaved action stream across **all** tabs in time order — who clicked what, and the resulting UI change |
 | `POST /__aipeek/chain` | Run a JSON array of actions in one round-trip (see Actions) |
 | `GET\|POST /__aipeek/eval` | Run arbitrary JS in the page (`?code=` or POST body); returns the result. Escape hatch for what typed endpoints can't do — for count/text/state/attr checks reach for `/query` first. |
 
+**Secret redaction.** Password inputs and API-key/token fields render as `‹redacted N chars›`
+across `/dom`, `/query`, and `/screen` — the length stays visible, the value doesn't.
+
 ### Perception layers — UI as text, not pixels
 
 For a model, the UI's optimal representation is its **semantics**, not rendered pixels.
@@ -52,8 +64,10 @@ information is already textual in the DOM. aipeek exposes four layers, cheapest
 
 - **`/screen`** — state-machine projection. The whole UI collapsed to what a human reads
   off a washing-machine panel: `view` (which area), `modal` (is something covering it),
-  `focus`, and `knobs` (the few *reachable* controls now — repeated rows fold to `source ×N`,
-  and when a modal is open only its subtree counts). A handful of lines. *Start here.*
+  `focus`, `knobs` (the few *reachable* controls now — repeated rows fold to `source ×N`,
+  and when a modal is open only its subtree counts), and `domain` (the app's own state
+  variables, if it sets `window.__AIPEEK_SCREEN__`). A handful of lines. *Start here.*
+  Append `?full` for untruncated output; pass `?since=tN` for just the delta.
 - **`/ui`** — React component tree. Full structure. Deep-dive when `/screen` isn't enough.
 - **`/dom`** — semantic DOM: `tag·role·semantic-class·data-*·state` per element, with
   Tailwind/atomic noise stripped and each line tagged with its **source location**
@@ -79,43 +93,103 @@ as the component boundary. Each line's `@File.tsx:line` then tells you exactly w
 | Endpoint | Params | Effect |
 |----------|--------|--------|
 | `/click` | `sel=` (CSS) or `text=` (visible text) | dispatch a real click |
-| `/fill` | `sel=`/`text=` + `value=` | set value on input/textarea/select; **contenteditable** via `execCommand` |
+| `/fill` | `sel=`/`text=` + `value=` | set value on input/textarea/select via React's native value setter (fires onChange on **controlled** inputs); **contenteditable** via `execCommand` |
 | `/press` | `key=` (e.g. `Enter`, `Control+a`) | keydown/keyup on the focused element |
-| `/wait` | `text=`/`sel=`, `timeout=` (ms, default 5000) | poll until it appears; 504 on timeout |
+| `/wait` | `text=`/`sel=`, `timeout=` (ms, default 5000), `gone=1` | poll until it appears (or, with `gone=1`, disappears); 504 on timeout |
+| `/realclick` | `sel=`/`text=`, `button=left\|right` | **trusted** OS-level click via the extension's CDP channel — for popups/context-menus a synthetic click can't open. Electron fires it in-process |
+| `/scrollIntoView` | `sel=`/`text=` | scroll a target into view (off-screen virtualized rows) |
+| `/drag` | `sel=` + `to=` | synthetic pointer drag, source → destination (steps past dnd-kit's activation distance) |
+| `/drop` | `sel=` + `files=a.png,b.pdf` | fire a file-drop (`DataTransfer`) on a target (synthetic Files trigger handlers, no byte content) |
+| `/clipboard` | `mode=read\|write`, `value=` | seed or read the clipboard (needs the tab focused) |
 | `/screenshot` | `sel=`, `out=` | DOM→PNG into `.aipeek/`; skips cross-origin/broken images |
-| `POST /chain` | JSON array of `{type, sel?, text?, value?, key?, timeout?}` | run in sequence, settle between steps, stop on first failure |
+| `POST /chain` | JSON array of steps | run in sequence, settle between steps, stop on first failure |
+
+`click`/`fill`/`press` **settle the DOM and append `--- changed ---`** — only the state-machine
+transition this action caused (`view: a → b`, `modal: opened X`, `focus: …`) plus any new
+errors/failed requests, not a fresh snapshot. `(no state change)` means nothing moved. You read
+the delta and drill into `/ui` or `/dom` for detail only if you need it. On a target miss,
+`/click` and `/fill` return the reachable clickable elements (clipped to the open modal's
+subtree) so you can re-target.
 
-`click`/`fill`/`press` **settle the DOM and append the UI tree after** (`--- ui after ---`)
-to the response — no follow-up read needed. On a target miss, `/click` and `/fill` return the
-reachable clickable elements (clipped to the open modal's subtree) so you can re-target.
+They also append a **`--- recent actions ---` timeline** — the semantic page actions in order
+(`T`=trusted human / `S`=synthetic aipeek), each with its resulting UI change, your own action
+bracketed by `你当前的行为` dividers. If the user manipulates the page concurrently (closes a
+dialog you opened), their action shows up in your next response — conflict surfaces automatically.
 
 A CSS `sel=` with non-ASCII or quotes/brackets must be URL-encoded, or the query parser
 mangles it: `curl -G .../click --data-urlencode 'sel=button[title="知识库"]'`.
 
-**Chain** packs a whole interaction into one round-trip:
+**Multiple tabs.** Every read/drive command takes `?tab=<id>` to address one tab — including
+a **background** one (drive the Chat tab while the user reads a different tab; synthetic events
+and Electron `sendInputEvent` don't need foreground). `GET /tabs` lists the live ids. One tab
+open → omit `?tab=`, it just works. Several open + no `?tab=` → the command returns `409` + the
+tab list instead of randomly hitting one; pick an id and retry with `?tab=`. `GET /timeline`
+interleaves every tab's actions in time order, so an A/B comparison (drive A, watch B react) is
+one read.
+
+**Federation across dev servers.** When several dev servers run at once (a micro-frontend, a
+separate front/back, a teammate's machine), any command takes `?host=<host:port>` to reach a
+*sibling* aipeek. The plugin you curl reverse-proxies the request server-side (no browser, no
+CORS): `…/screen?host=localhost:5174` reads the app on :5174; combine with `?tab=` to point at
+one tab over there. No registry — you name the peer, so list its tabs with `/tabs?host=…` first.
+
+**Trusted input.** A control tagged `{needs-trusted?}` in `/screen` or `/dom` opens a popup
+(`aria-haspopup`) a synthetic click may not trigger — use `/realclick` on it. Right-click menus
+carry no DOM marker; reach for `/realclick` with `button=right` there. (Requires the browser
+extension for a plain tab; Electron fires trusted events in-process.)
+
+**Chain** packs a whole interaction into one round-trip. An `assert` step is a mid-chain judge —
+`{type:"assert", screen, equals}` checks an app domain variable (from `window.__AIPEEK_SCREEN__`),
+or `{type:"assert", sel, equals}` an element's text — and stops the chain with `asserted X=="Y",
+actual "Z"` on mismatch:
 
 ```bash
 curl -X POST localhost:5195/__aipeek/chain -d '[
   {"type":"click","sel":"button[title=\"知识库\"]"},
   {"type":"wait","text":"Done"},
   {"type":"fill","sel":"textarea","value":"hi"},
+  {"type":"assert","screen":"streaming","equals":"false"},
   {"type":"press","key":"Enter"}
 ]'
 ```
 
+**Connection diagnostics.** When a read can't get an answer, aipeek never returns a bare "no
+tab" — it distinguishes, and tells you the fix: *tab connected but the handler hung* (check
+`/console`), *tab closed / mid self-heal* (retry), *pages connected via HMR but the client
+didn't load* (wrong vite server / `vite preview` / production build — check the port), or *no
+browser pointed here at all* (open the app). `/profile` on a backgrounded tab says so too —
+the browser throttles rAF for hidden tabs, so bring it foreground ~2s; it's the only read that
+needs foreground.
+
 ## CLI
 
+A thin wrapper over the HTTP endpoints, with colored `check` output:
+
 ```bash
-npx aipeek                  # fetch from localhost:5195
-npx aipeek --port=3000      # custom port
+npx aipeek                  # full summary (localhost:5173)
+npx aipeek check            # health check (pass/fail)
+npx aipeek console          # console logs
+npx aipeek network/0        # detail for one network request
+npx aipeek errors/1 --full  # untruncated error detail
+npx aipeek --port=5195      # custom port
 ```
 
 ## Store Registration (optional)
 
-Register MobX/other stores for state inspection:
+Register MobX/other stores so their snapshots appear in the `state` section:
 
 ```ts
 window.__AIPEEK_STORES__ = { myStore, anotherStore }
 ```
 
 State is snapshotted on demand (depth-limited, bounded) and included in the `<state>` section.
+
+## Domain projection (optional)
+
+Let aipeek read your app's own state machine — the variables show up in `/screen`'s `domain`
+block, in every `--- changed ---` diff, and are assertable in a chain (`{type:"assert", screen,
+equals}`). Opt in by exposing a snapshot function:
+
+```ts
+window.__AIPEEK_SCREEN__ = () => ({ view, streaming, modal: openDialog })
+```