@tyouter/dsh-recorder 0.0.0-stage → 0.1.1

package/HANDOFF.md

@@ -0,0 +1,115 @@
+# Handoff — build a DSH recording plugin
+
+You are taking over a prototype whose goal is: **record a DSH plugin's real interface and produce a GIF
+(or stills) from it**, without hand-building a lookalike page.
+
+Read [docs/findings.md](docs/findings.md) first; it contains the measurements. This document is the
+brief: what to build, the seams that already exist, and the traps that already cost time.
+
+---
+
+## The honest starting point
+
+Two instruments were tried against `@tyouter/dsh-notes`:
+
+| Approach | What it produced | Verdict |
+| --- | --- | --- |
+| **Re-render**: a generated HTML page reusing the plugin's renderer + stylesheet, driven by `seek(t)`, captured over CDP | A perfect-looking GIF that was *almost* the app — wrong proportions, invented labels | Rejected by the user: "你的 GIF 不对啊" |
+| **Real capture**: scripted driver inside the real page + window grab from outside | Real pixels, real interactions; the note count went 13 → 14 (proof it ran) | Worked, but the tooling was bolted onto the notes repo and the timing/visibility problems below were never solved |
+
+The second approach is the right one. What is missing is the product around it.
+
+---
+
+## What to build
+
+A plugin, `@tyouter/dsh-recorder` (name is yours), with two surfaces.
+
+### 1. A host tool the agent can call
+
+```
+record_start({ target: "window" | "rect", title?, rect?: [l,t,w,h], fps?, width? })
+record_stop()  →  { gif, frames, seconds, framesCount }
+list_windows() →  [{ hwnd, title, rect }]
+```
+
+Reuse `tools/record-window.py` as the engine: it is a complete, working recorder (Win32 enumerate +
+`ImageGrab` + Pillow GIF). Wrap it as a subprocess, or port the ~120 lines into the plugin's host half.
+Return the GIF path so the agent can present it.
+
+Requirements the prototype implies:
+
+- **start/stop, not fixed duration.** The agent does not know how long a walkthrough will take.
+- **a status line in the UI while recording.** The prototype was silent; the user assumed nothing was
+  happening ("我切过去了，那边没有开始自动录制啊").
+- **explicit rect support.** Titles are not unique; the biggest match is often the wrong pane.
+
+### 2. A client-side driver contract
+
+The half that must live inside the plugin under test. `src/demo-driver.js` is a working reference
+implementation for one plugin; generalise the idea:
+
+- a small **driver API** the plugin under test exposes: `startDemo(steps)`, or a generic "act like a
+  user" helper layered on selectors;
+- a **dev flag** to arm it (the prototype used a `.demo` file in the data folder plus an env var —
+  the file form means arming needs no restart; the *client bundle* still needs a reload);
+- `resetDemo()` so a walk can be replayed without reloading the page;
+- **no network of its own**: the demo should exercise the product's real code paths, never mock them.
+
+### 3. Assembling outputs
+
+- GIF: already in `record-window.py` (128-colour palette, one shared palette per recording).
+- Stills: pick frames by timestamp; `tools/record-window.py` keeps every frame under `frames/`.
+- Optional later: chapter cards, cursor highlight, zoom — none of this exists yet. Do not promise it in
+  a README before it is built; that was the mistake of the re-render approach.
+
+---
+
+## Traps, in the order they cost time
+
+1. **One-shot driver.** A module-level `started` flag means "run once per page load". After the first
+   run the UI is inert, which reads as a bug. Either reload or expose a reset.
+2. **Recording the wrong window.** The panel can be mounted while the chat pane is in front. Sample a
+   frame mid-recording and *look at it*; do not trust the timer.
+3. **Dynamic import does not work in a plugin client half.** The bundle is one factory function; the
+   driver must be concatenated into it (that is why `build.mjs` in dsh-notes has a `parts` list).
+4. **React ignores `value = …`.** Use the native setter + `dispatchEvent(new Event('input'))`.
+5. **Console encoding.** The bundled Python defaults to cp1252; `--list` crashes on a Chinese window
+   title without `sys.stdout.reconfigure(encoding="utf-8")`.
+6. **The GUI is 401-gated and its port changes on restart.** Do not hard-code a port; read it from the
+   app or pass it in.
+7. **The renderer has no debug port** unless the app is started with `--remote-debugging-port`. If you
+   want CDP-based capture (cleaner than screen grabbing: no focus, no occlusion, exact frames), that
+   restart is part of the design, not an accident.
+
+---
+
+## Suggested order of work
+
+1. Wrap `tools/record-window.py` behind a host tool + a `record_start/stop` pair, with a visible status.
+   This alone is useful to any plugin author.
+2. Move the driver into a small, documented module (`driveUI({ steps })`) and use the notes walk as the
+   first fixture; keep selector strings in the fixture, not in the engine.
+3. Add a "record this panel" flow: open the panel, arm the demo, start the recorder, stop on completion.
+4. Only then: overlays (cursor, zoom, chapter cards) — and only as an opt-in post-process on the frames.
+
+## Acceptance for v0.1
+
+- Start a recording of a named window or rect, get a GIF back, and the GIF shows the plugin's real UI.
+- Stop reliably; no orphan process; the frames folder is cleaned on the next start.
+- The recording is visible in the UI while it runs.
+- A replay needs no page reload (reset path exists).
+- Nothing in the plugin under test changes except an opt-in dev flag.
+
+---
+
+## Files
+
+```
+src/demo-driver.js     reference driver (works against dsh-notes 1.1 selectors)
+tools/record-window.py reference recorder (window/rect, fps, GIF, --list)
+docs/findings.md       measurements: ports, 401, window rects, what worked, what did not
+```
+
+The notebook repo (`../dsh-notes`) no longer contains any of this; it was reverted to its product code
+only. Do not add recording concerns back into it.

package/README.md

@@ -1,3 +1,250 @@
-# Temporary Holding Version
+# dsh-recorder
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Records a DSH plugin's **real** interface — the pixels of the running app, not a re-render — and turns
+that into a GIF or stills. It also **drives** that interface like a person: typing with real input
+events, clicking buttons, selecting a range of text, switching panels.
+
+Two ways to use it:
+
+- **The panel**: sidebar → 「录制器」, paste a script, press 「开始录制」. The page executes the script
+  while the real window is recorded, then shows the GIF.
+- **Agent tools**: `recorder_scan` → `recorder_demo` for a one-shot recording from a conversation.
+
+**Status: v0.1, installed and verified on this machine.** A real notes walk produced a 10.8 s,
+65-frame, 2.0 MB GIF plus three stills (`recordings/rec-20261103-142842/`). The plugin is installed as
+`@tyouter/dsh-recorder` in the `desktop` profile; the row is `include:dsh-recorder`.
+
+## The panel
+
+![Recorder panel](docs/panel.png)
+
+Sidebar → 「录制器」 opens a page with two tabs.
+
+**录制** — the whole workflow is one centred column:
+
+| Part | What it does |
+| --- | --- |
+| 录制脚本 | The script, front and centre. JSON steps or one step per line (see below); kept in `localStorage` between visits. |
+| 复制给插件 agent 的 prompt | One click copies a self-contained prompt (the full spec + what to output). Paste it into the agent that owns the plugin, describe the demo, and it returns a script you paste back. `载入示例` restores the notes walk that the verified GIF came from. |
+| 开始录制 | The primary action, a large button under the editor. `排练（不录制）` runs the same steps without recording (real clicks, no video); `停止` ends early. A rehearsal brings the window forward first — a background window throttles the page's timers and would freeze the walk. |
+| 本次录制 | The result of this run: a large GIF preview (click it for a full-size lightbox), its length/frames/size, the path, and the per-step log. The lightbox opens at **100%** — one GIF pixel per device pixel, so the text is exactly the physical size it had on screen; `适应屏幕` fits the whole frame instead (smaller text). |
+
+![Full-size preview](docs/panel-preview.png)
+
+**⚙ 设置** (top right) opens a popover with the parameters — they are no longer mixed into the main
+page:
+
+- 输出工作区 — which registered workspace's `recordings/` receives the output.
+- 录制区域 — `功能区` (crop to the main panel), `整个窗口`, `自定义区域` + a CSS selector.
+- 帧率 / GIF 宽度 / 截图时间点 — 6 fps. The GIF width defaults to the content's **physical** pixel
+  width (≈1470 on a 125% display) so the preview is never upscaled; lower it to shrink files, and
+  remember the encoder cannot add detail beyond the source. Stills are seconds.
+- 铺满当前屏幕 — maximize on the monitor under the cursor.
+
+![Settings](docs/panel-settings.png)
+
+**历史** — a grid of past recordings of the selected workspace (thumbnail, name, length, frames,
+size). Click a card to open it with the GIF, meta, path and copy button. Click the preview (the
+`⤢ 全尺寸` badge) to open the full-size lightbox; click the backdrop or ✕ to close.
+
+The detail view also has **删除这段录制**: the first click arms the button, the second confirms and
+removes the recording directory from disk. Only directories directly under a recordings root whose
+name matches `rec-YYYYMMDD-HHMMSS` can be deleted (the route refuses anything else).
+
+![History](docs/panel-history.png)
+
+![Recorder running](docs/panel-running.png)
+
+While a run is active the status pill counts steps (`录制中 00:01 · 3/3 click`) and the floating REC
+pill below the title bar shows the clock and a stop button. A rehearsal gets the same pill in blue
+(`排练 2/10`) with its own stop, so it can be ended even when the script has switched away from this
+page. The panel is in the main area, so a script that opens another plugin's panel will hide it — the
+pill stays available, and coming back to the recorder panel shows the finished result.
+
+The pill is **hidden while the pointer is away from the top strip** and appears when the pointer enters
+the top 84 px. That keeps it out of the recording (nothing is drawn over the page during a normal
+walk), while moving the mouse to the top is the stop control when the panel is not visible. It sits at
+`top: 46px` because the very top strip is the window's drag region; a click there belongs to the window
+manager. Pressing 停止 hides the pill immediately while the GIF is assembled, so a long splice never
+looks like an ignored click.
+
+### Window size
+
+`铺满屏幕` (and, when `maximizeOnStart` is true in the patch, a moment after the plugin activates)
+moves the app window to the monitor under the cursor and maximizes it, so a freshly launched Harness
+fills the current screen instead of opening at its remembered size. Focusing the window for a rehearsal
+never changes its size: a maximized window stays maximized, at the same rect. Window geometry is the
+plugin's business; **UI scale is not** — the interface still follows the Windows display scale of that
+monitor (120 DPI = 125% here). For a 1:1 UI, set that monitor to 100% or launch the app with Chromium's
+`--force-device-scale-factor=1`; a page plugin cannot change the process's DPI mode.
+
+Keep the Harness window visible while a walk runs: Chromium throttles a background window's timers,
+which freezes every `sleep`/`wait` in the script (and delays a stop). The rehearsal brings the window
+forward before it starts; the capture does the same by design.
+
+### Cursor and click effects
+
+The driver paints the pointer into the page, so the effects are part of the recording — no
+post-processing, the frames are the page:
+
+- a short blue trail follows a move, including a jump across the screen (two breadcrumbs are laid on
+  the straight line when the distance is large);
+- **two expanding blue rings plus a flash** mark the click;
+- the clicked element gets a **temporary blue outline**, so the target is obvious even when the
+  pointer is small.
+
+![Click effect](docs/click-effect.png)
+
+All of it is disabled under `prefers-reduced-motion` (the outline and the pointer are still drawn).
+
+### Recording area
+
+`功能区` is the default and is what most demos want: the page measures the main panel's box and sends
+the **insets in CSS pixels plus `devicePixelRatio`**; the Host maps them through the window's real
+client area (Win32 `GetClientRect` + `ClientToScreen`) and crops every frame. The window is still
+maximized for the shoot — the crop is what removes the sidebar and the app title bar, and
+it stays correct on a mixed-DPI desktop because the mapping is measured, not assumed.
+
+Because the crop is the **main panel**, the GIF's aspect ratio is the panel's, not the window's
+(e.g. 1544×970 for a 1920×1020 window). The image is still scaled proportionally; if the framing has
+to match the full window, pick `整个窗口`.
+
+- `整个窗口` records the maximized window as-is (sidebar included).
+- `自定义区域` takes a CSS selector and crops to that element's box; the element must exist when you
+  press start.
+- If another app takes the foreground mid-recording, those frames show that app; the result then
+  carries a `⚠` warning rather than pretending the footage is clean.
+
+![Cropped recording](docs/crop-example.png)
+
+The same walk in `功能区` mode: 10.6 s / 64 frames / 705 KB, against 2.0 MB for the full-window cut —
+the sidebar and app title bar are simply not in the frame.
+
+### Script syntax
+
+One step per line; `|` introduces the text for that step. Lines starting with `#` are comments.
+
+```
+# 打开记事本，写一段笔记，选中一句
+open notes
+wait .dshn-root
+click .dshn-switch[aria-checked="true"]
+click .dshn-tabs button.dshn-tab:nth-of-type(1)
+set .dshn-textarea |
+click .dshn-textarea
+type .dshn-textarea | 极简笔记\n\n这不是一个新的工具，而是一个关于文字的哲学思考。\n智能不能成为文字的主角。\n功能应当服务于思想。\n书写带来快乐。
+sleep 900
+select .dshn-textarea | 智能不能成为文字的主角。
+sleep 1200
+```
+
+- `open <panel>` — switch to another plugin's main panel (`notes`, `wuche`, `conversation`, …).
+- `click <selector> [| text]` — click a button/tab; `text` picks among matches.
+- `type <selector> | text` — type character by character (60 ms each); `\n` inserts a newline.
+- `set <selector> | text` — set the value in one go (good for clearing a draft).
+- `select <selector> | text` — highlight that text.
+- `wait <selector> [timeoutMs]` — wait for an element (default 8 s).
+- `sleep <ms>`, `key <selector> <Tab|Enter|Escape>`, `scroll <selector>`, `focus <selector>`.
+
+A JSON array of step objects works too — starting the box with `[` switches the parser. Step objects
+use the same `op` names the agent tools use, and can set `ms` on `type`.
+
+A step whose element is missing is reported as failed and the walk continues (unless the JSON sets
+`strict: true`). Selectors come from the plugin's real DOM; the agent tool `recorder_scan` prints them.
+
+The full, versioned grammar is in [docs/script-spec.md](docs/script-spec.md). Before spending a
+recording, press **排练（不录制）**: it runs the same steps for real and lists `✓`/`✗` per step, so a
+wrong selector shows up in a second instead of in a wasted GIF.
+
+### Writing a script with another agent
+
+The plugin's own agent knows its components and class names, but not the recorder's grammar. Hand it
+the prompt:
+
+1. Press **复制给插件 agent 的 prompt**.
+2. Paste it into that plugin's agent conversation, replace the last paragraph with what you want to
+   see (`打开记录页，点 Structure，输入一句话…`), and send.
+3. The agent answers with one code block. Paste it into 录制脚本, press **排练（不录制）**, then **开始录制**.
+
+The prompt carries the whole spec, so the other agent does not need this repository. If you also want
+it to use *real* selectors, send it the output of `recorder_scan` (the agent tool) for the panel you
+mean; the prompt tells it to prefer that list.
+
+## Agent tools
+
+| Tool | What it does |
+| --- | --- |
+| `recorder_windows` | Lists recordable windows (title, screen rect, size) and reports the capture backend. |
+| `recorder_scan` | Reads the live DOM and returns the visible interactive elements with real CSS selectors. |
+| `recorder_start` / `recorder_stop` | Manual start/stop; `start` returns a `probe` frame to check the target. |
+| `recorder_drive` | Runs steps against the live page without recording, or while a recording runs. |
+| `recorder_demo` | start → drive → settle → stop in one call. |
+
+## How it works
+
+- **Capturing** pixels happens in the Host half: screen capture of the real window through
+  `tools/record-window.py` (Win32 + Pillow). It knows nothing about DSH plugins.
+- **Driving** happens in the page that owns the UI. The recorder's **own client half** lives in the
+  same page as every other plugin, so it operates the target's real DOM. The plugin under test does
+  not have to change at all — no dev flag, no injected script, no debug port.
+- **Agent-triggered** runs use a job bridge: the page polls `/recorder/api/poll`, runs the handed-out
+  steps, and posts the report back. The **panel does not use that bridge** — it runs the same driver
+  locally (`POST /start` → steps → `POST /stop`), so a throttled background timer can never stop a
+  demo from starting. `/progress` still feeds the pill's step counter on both paths.
+
+```
+run / recorder_demo (agent path)
+   │  Host: resolve window → activate → maximize → start frames
+   ├────────► /recorder/api/poll ──► page: run steps (type/click/select/openPanel)
+   │  Host: wait for the report ◄── /recorder/api/report (and /progress)
+   │  Host: settle → STOP → GIF assembly (Pillow) → restore window
+   └── { gif, stills, frames, seconds, walk: [step results] }
+
+panel: POST /start → runSteps() in the page → POST /stop   (no poll involved)
+```
+
+Because a screen grab cannot see through another app, a run raises the window (minimize + restore,
+which is the one activation Windows never refuses) and maximizes it for the length of the shoot; the
+previous placement is restored afterwards. The painted cursor and the REC pill are part of the page
+and are recorded in both modes.
+
+## Installing / updating
+
+```
+plugin_manager(install_bundle, target = <this directory>)
+```
+
+A newly installed bundle's **client** half appeared in the running page without a reload on this
+build. Host-side JavaScript changes are **not** picked up by editing files: disable and re-enable the
+row with `plugin_manager(set_plugin)`, or restart the app.
+
+## Limits (v0.1)
+
+- One recording at a time; output goes to `<workspace>/recordings/rec-<stamp>/`.
+- Screen capture means *the screen*: if the window is moved or covered mid-recording, the footage shows
+  that. `recorder_start` returns a `probe` frame for exactly this check.
+- The recorded window is maximized for the shoot and restored afterwards.
+- No zoom, chapter cards, or post-processed cursor: the cursor is a DOM overlay recorded with the page.
+- The panel's output workspace is a registered-workspace picker; the agent tools use the calling
+  session's cwd.
+- The plugin under test is untouched. The notes fixture clears its draft and never saves.
+
+## Files
+
+```
+index.js               Host half: engine wiring, tools, /recorder/api
+lib/engine.js          one recording + one page operation at a time; the job bridge; workspaces
+lib/capture.js         capture backends (Electron capturePage when available, else screen) + stills
+lib/python.js          locating the bundled Python and running the helper
+lib/tools.js           the six model-facing tools
+client.js              page half: poll loop, step driver, DOM scan, REC pill, cursor, recorder panel
+tools/record-window.py window/rect capture, stop-file protocol, probe frames, GIF assembly
+examples/notes-demo.json  the verified walk as JSON steps
+docs/panel.png          the recorder page
+docs/panel-running.png  the page during a run (REC pill + step counter)
+docs/crop-example.png   a 功能区-mode frame (no sidebar, no title bar)
+docs/click-effect.png   the pointer, trail and double click ring over a clicked tab
+docs/script-spec.md     the versioned script grammar (also the source of the copied prompt)
+HANDOFF.md             the original brief (kept as the design record)
+docs/findings.md       measurements: what was proved, what failed, and why
+```