aipeek 0.1.5 → 0.2.0

package/README.md

@@ -1,6 +1,10 @@
 # aipeek
 
-Gives AI a peek into your running browser app. Captures UI tree (React fiber), console logs, network requests, errors, and store state — served as plain text via Vite dev server.
+Gives AI a peek into — and a hand on — your running browser app. Reads the UI tree (React fiber), semantic DOM, console, network, errors, and store state; drives the page (click/fill/press/wait/screenshot). All over plain-text HTTP on your Vite dev server — zero resident context cost, unlike a browser MCP whose tool schemas sit in the model's context whether used or not.
+
+**10× faster end-to-end.** What you feel is wall-clock from prompt to done — model thinking, round-trips, all of it. Screenshot agents (Playwright + vision) pay 2–5s of pixel-parsing *every step*; aipeek reads semantic text (instant) and batches a whole interaction into one round-trip with `/chain` — the model thinks once, not N times.
+
+It lives **inside** the open page (injected client + HMR channel), so it reads React/store internals a DOM-only driver can't, and acts on the current tab with no separate browser process. It does **not** open browsers, navigate, run headless, or fire real pointer events — it's the dev inner loop, not E2E. For that, use Playwright.
 
 ## Install
 
@@ -14,7 +18,7 @@ bun add aipeek
 
 ## Setup
 
-```tsf
+```ts
 // vite.config.ts
 import { aipeekPlugin } from 'aipeek'
 
@@ -29,10 +33,74 @@ All endpoints are available on your Vite dev server:
 
 | Endpoint | Description |
 |----------|-------------|
+| `GET /__aipeek/screen` | **State-machine projection** — `{view, modal, focus, knobs}`. Start here. |
 | `GET /__aipeek` | Summary of all sections (UI, console, network, errors, state) |
 | `GET /__aipeek/{section}` | Detail for a section (`ui`, `console`, `network`, `errors`, `state`) |
 | `GET /__aipeek/{section}/{index}` | Detail for a specific item in a section |
 | `GET /__aipeek/{section}?full` | Full detail (no truncation) |
+| `GET /__aipeek/dom[?scope=Name\|?sel=css]` | Semantic DOM — UI as text (see below) |
+| `GET /__aipeek/{action}?...` | Drive the page (see Actions) |
+| `POST /__aipeek/chain` | Run a JSON array of actions in one round-trip (see Actions) |
+| `GET\|POST /__aipeek/eval` | Run arbitrary JS in the page (`?code=` or POST body); returns the result. Escape hatch for what typed endpoints can't do. |
+
+### Perception layers — UI as text, not pixels
+
+For a model, the UI's optimal representation is its **semantics**, not rendered pixels.
+A screenshot forces pixel→meaning re-derivation and costs hundreds of tokens; the same
+information is already textual in the DOM. aipeek exposes four layers, cheapest first:
+
+- **`/screen`** — state-machine projection. The whole UI collapsed to what a human reads
+  off a washing-machine panel: `view` (which area), `modal` (is something covering it),
+  `focus`, and `knobs` (the few *reachable* controls now — repeated rows fold to `source ×N`,
+  and when a modal is open only its subtree counts). A handful of lines. *Start here.*
+- **`/ui`** — React component tree. Full structure. Deep-dive when `/screen` isn't enough.
+- **`/dom`** — semantic DOM: `tag·role·semantic-class·data-*·state` per element, with
+  Tailwind/atomic noise stripped and each line tagged with its **source location**
+  (`@File.tsx:line`, via `code-inspector` if present). This is what tells the model
+  *what an element is, its live state, and where to edit it.*
+- **`/screenshot`** — pixels. Lossy DOM→PNG (`html-to-image`). Only for visual checks
+  a human looks at; not the model's primary channel.
+
+**Scoped DOM — work top-down.** The full DOM is huge; a scoped view is accurate. Read
+`/ui` to find a component, then scope the DOM to it:
+
+```bash
+curl localhost:5195/__aipeek/ui                  # find <ChatInput>
+curl 'localhost:5195/__aipeek/dom?scope=ChatInput' # just that subtree (matches source path)
+curl 'localhost:5195/__aipeek/dom?sel=.chat-list'  # or any CSS subtree
+```
+
+`scope=` matches against the `data-insp-path` source path, so directory structure acts
+as the component boundary. Each line's `@File.tsx:line` then tells you exactly where to edit.
+
+### Actions — drive the current tab
+
+| Endpoint | Params | Effect |
+|----------|--------|--------|
+| `/click` | `sel=` (CSS) or `text=` (visible text) | dispatch a real click |
+| `/fill` | `sel=`/`text=` + `value=` | set value on input/textarea/select; **contenteditable** via `execCommand` |
+| `/press` | `key=` (e.g. `Enter`, `Control+a`) | keydown/keyup on the focused element |
+| `/wait` | `text=`/`sel=`, `timeout=` (ms, default 5000) | poll until it appears; 504 on timeout |
+| `/screenshot` | `sel=`, `out=` | DOM→PNG into `.aipeek/`; skips cross-origin/broken images |
+| `POST /chain` | JSON array of `{type, sel?, text?, value?, key?, timeout?}` | run in sequence, settle between steps, stop on first failure |
+
+`click`/`fill`/`press` **settle the DOM and append the UI tree after** (`--- ui after ---`)
+to the response — no follow-up read needed. On a target miss, `/click` and `/fill` return the
+reachable clickable elements (clipped to the open modal's subtree) so you can re-target.
+
+A CSS `sel=` with non-ASCII or quotes/brackets must be URL-encoded, or the query parser
+mangles it: `curl -G .../click --data-urlencode 'sel=button[title="知识库"]'`.
+
+**Chain** packs a whole interaction into one round-trip:
+
+```bash
+curl -X POST localhost:5195/__aipeek/chain -d '[
+  {"type":"click","sel":"button[title=\"知识库\"]"},
+  {"type":"wait","text":"Done"},
+  {"type":"fill","sel":"textarea","value":"hi"},
+  {"type":"press","key":"Enter"}
+]'
+```
 
 ## CLI