@qontinui/ui-bridge-wrapper 0.12.0 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/README.md +157 -8
  2. package/dist/capture-specs-cli.cjs +301 -53
  3. package/dist/capture-specs-cli.cjs.map +1 -1
  4. package/dist/capture-specs-cli.d.ts.map +1 -1
  5. package/dist/chunk-IKHDNIU4.js +478 -0
  6. package/dist/chunk-IKHDNIU4.js.map +1 -0
  7. package/dist/chunk-Z32CC665.cjs +488 -0
  8. package/dist/chunk-Z32CC665.cjs.map +1 -0
  9. package/dist/create-transport.d.ts.map +1 -1
  10. package/dist/frame-capture.d.ts +3 -28
  11. package/dist/frame-capture.d.ts.map +1 -1
  12. package/dist/frame-page.d.ts +35 -0
  13. package/dist/frame-page.d.ts.map +1 -0
  14. package/dist/index.cjs +145 -375
  15. package/dist/index.cjs.map +1 -1
  16. package/dist/index.d.ts +1 -0
  17. package/dist/index.d.ts.map +1 -1
  18. package/dist/index.js +90 -351
  19. package/dist/index.js.map +1 -1
  20. package/dist/inject-cli.cjs +407 -134
  21. package/dist/inject-cli.cjs.map +1 -1
  22. package/dist/inject-cli.d.ts +1 -1
  23. package/dist/inject-cli.d.ts.map +1 -1
  24. package/dist/live/index.cjs +40 -0
  25. package/dist/live/index.cjs.map +1 -0
  26. package/dist/live/index.d.ts +32 -0
  27. package/dist/live/index.d.ts.map +1 -0
  28. package/dist/live/index.js +3 -0
  29. package/dist/live/index.js.map +1 -0
  30. package/dist/login-web-cli.cjs +312 -61
  31. package/dist/login-web-cli.cjs.map +1 -1
  32. package/dist/login-web-cli.d.ts.map +1 -1
  33. package/dist/network-throttle.d.ts +5 -0
  34. package/dist/network-throttle.d.ts.map +1 -1
  35. package/dist/screenshot-capture.d.ts +82 -0
  36. package/dist/screenshot-capture.d.ts.map +1 -0
  37. package/dist/timeline-capture.d.ts +74 -10
  38. package/dist/timeline-capture.d.ts.map +1 -1
  39. package/dist/transports/headless.d.ts +46 -1
  40. package/dist/transports/headless.d.ts.map +1 -1
  41. package/dist/transports/injected.d.ts +8 -0
  42. package/dist/transports/injected.d.ts.map +1 -1
  43. package/dist/transports/live.d.ts +27 -1
  44. package/dist/transports/live.d.ts.map +1 -1
  45. package/dist/types.d.ts +3 -1
  46. package/dist/types.d.ts.map +1 -1
  47. package/package.json +11 -8
  48. package/dist/cli-args.d.ts +0 -13
  49. package/dist/cli-args.d.ts.map +0 -1
package/README.md CHANGED
@@ -39,6 +39,44 @@ await transport.dispatch('login', { email: 'a@b.c', password: '...' });
39
39
  await transport.close();
40
40
  ```
41
41
 
42
+ ## A live session the runner can capture
43
+
44
+ The runner's vision routes (`POST /ui-bridge/vision/capture` and the analyze,
45
+ OCR and audit routes beside it) frame a WebSocket-transport app by relaying
46
+ `getScreenshot` over its socket — and only to an app whose register frame
47
+ advertises the `screenshot` capability. A Node process that owns a Playwright
48
+ page opts in by handing that page to the live transport:
49
+
50
+ ```ts
51
+ import WebSocket from 'ws';
52
+ import { createTransport } from '@qontinui/ui-bridge-wrapper';
53
+
54
+ const transport = createTransport({
55
+ kind: 'live',
56
+ runnerUrl: 'ws://127.0.0.1:9876/ui-bridge/ws',
57
+ appId: 'my-app',
58
+ options: { webSocketCtor: WebSocket, screenshotPage: page },
59
+ });
60
+ await transport.ready();
61
+ ```
62
+
63
+ `screenshotPage` registers the `getScreenshot` handler and advertises
64
+ `screenshot` together, so the advertisement cannot drift from the behaviour.
65
+ The reply is `{ screenshot: <base64 png>, width, height }` (CSS pixels); a
66
+ capture is bounded at 10 s (`SCREENSHOT_TIMEOUT`, retryable) and a failed one
67
+ answers a typed `SCREENSHOT_FAILED`. Without `screenshotPage` the runner refuses
68
+ the app as a capture target by name (`FRAME_SOURCE_UNSUPPORTED`).
69
+ `registerScreenshotCapture(registrar, page)` registers the same handler on any
70
+ other transport that owns a page.
71
+
72
+ A browser bundle (an extension content script, a page) imports the
73
+ **browser-safe** entry `@qontinui/ui-bridge-wrapper/live` instead of the
74
+ package root: the root re-exports Node-only transports and cannot load where
75
+ there is no `require`. Both entries share one copy of `WrapperTransportError`,
76
+ `HandlerRegistry` and `LiveSessionTransport` within a module format, so they
77
+ may be mixed in one ESM (or one CJS) program — but not a `require` of one with
78
+ an `import` of the other, which load separate copies.
79
+
42
80
  ## Package layout
43
81
 
44
82
  - `src/index.ts` — public surface (transports, helpers, types).
@@ -60,11 +98,11 @@ so **they run from any directory once installed** — there is no "must run from
60
98
  the ui-bridge repo root" coupling. Run via `npx @qontinui/ui-bridge-wrapper`
61
99
  or a global/dev install:
62
100
 
63
- | Bin | What it does |
64
- | --- | --- |
65
- | `ui-bridge-inject` | Drive the injected transport against a UI-Bridge-free page (exec one-shot actions, register as a relay tab, or capture frame-and-snapshot pairs — one, or a viewport × DPR × colour-scheme matrix). |
66
- | `ui-bridge-login-web` | Automated web login via the injected transport — drives the full Cognito OAuth redirect chain in one headless tab. Optionally asserts on-page text (`--expect-text`) and screenshots the authed landing (`--screenshot`/`--scroll-to`). Prints one JSON result line; exit 0 = login confirmed AND all expected text present. |
67
- | `ui-bridge-capture-specs` | Log in (same flow) then `goto` + snapshot a set of authed pages, writing one `<slug>.snapshot.json` per page. |
101
+ | Bin | What it does |
102
+ | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
103
+ | `ui-bridge-inject` | Drive the injected transport against a UI-Bridge-free page (exec one-shot actions, register as a relay tab, or capture frame-and-snapshot pairs — one, or a viewport × DPR × colour-scheme matrix). |
104
+ | `ui-bridge-login-web` | Automated web login via the injected transport — drives the full Cognito OAuth redirect chain in one headless tab. Optionally asserts on-page text (`--expect-text`) and screenshots the authed landing (`--screenshot`/`--scroll-to`). Prints one JSON result line; exit 0 = login confirmed AND all expected text present. |
105
+ | `ui-bridge-capture-specs` | Log in (same flow) then `goto` + snapshot a set of authed pages, writing one `<slug>.snapshot.json` per page. |
68
106
 
69
107
  Each bin supports `--help`.
70
108
 
@@ -129,6 +167,63 @@ ui-bridge-login-web \
129
167
  ui-bridge-capture-specs --out ./spec-capture
130
168
  ```
131
169
 
170
+ ### Reading and acting on one element (`ui-bridge-inject` exec mode)
171
+
172
+ Exec mode (`--exec` / `--exec-stdin`) reads and acts on single elements through
173
+ the injected runtime's in-page dispatcher, the same one the relay drives, with
174
+ no relay call:
175
+
176
+ ```bash
177
+ # read one element, click it, read its state again
178
+ ui-bridge-inject --url http://127.0.0.1:3000/ \
179
+ --exec 'getElement {"id":"open"}' \
180
+ --exec 'executeElementAction {"id":"open","request":{"action":"click"}}' \
181
+ --exec 'getElementState {"id":"open"}'
182
+ ```
183
+
184
+ - `getElement {"id":"<id>"}` answers one element as a snapshot carries it:
185
+ identity (`id`, `type`, `tagName`, `label`, `role`, `accessibleName`),
186
+ `state`, box (`bbox`, `visible`) and `actions`, the same object as the
187
+ matching `getControlSnapshot` element, redaction included.
188
+ `{"elementId":"<id>"}` is accepted too; given both, they must name the same
189
+ element. Nothing settles first, so run `waitForSelector` before it for a
190
+ settled read. Beside the id it tolerates, and ignores, `settleTimeout` and
191
+ `skipSettle` (it does not settle), `targetTabId`, and the WebSocket
192
+ protocol's `includeState` (the state is always included); any other key is
193
+ refused with `INVALID_REQUEST`. An unknown id, or one whose DOM node is gone,
194
+ is a returned `ELEMENT_NOT_FOUND` (an `error` line and exit 1), never a
195
+ `null` result.
196
+ - `getElementState {"id":"<id>"}` answers the state alone. It reads `id` only,
197
+ and an unknown id is a thrown error (`HANDLER_ERROR`).
198
+ - `executeElementAction {"id":"<id>","request":{"action":"click"}}` acts on one
199
+ element. `request` holds the verb and the fields it reads (`value`, `text`,
200
+ `clear`, `params`, `fromSnapshotId`, …). Two other spellings are accepted:
201
+ `elementId` in place of `id`, and a flat payload such as
202
+ `{"id":"<id>","action":"type","text":"hi"}`, which, minus `id` / `elementId`,
203
+ _is_ the request. No key _beside_ a `request` object is silently dropped: a
204
+ top-level `action` or request field there must equal the same field inside
205
+ `request` (an agreeing duplicate is accepted); one that is only at the top
206
+ level, or at both levels with different values, is refused, and so are an
207
+ `id` and an `elementId` that name different elements. The dispatcher keys
208
+ stay allowed at the top level. The request fields themselves (the keys
209
+ inside `request`, or every key of a flat payload but the id) reach the
210
+ action as sent, so a misspelt one there (`txt` for `text`) is not caught.
211
+ Every refused shape is a returned `INVALID_REQUEST` (an `error` line and
212
+ exit 1, never a thrown `TypeError`) whose message names the canonical shape
213
+ `{"id":"<elementId>","request":{"action":"<action>", …}}` and both accepted
214
+ spellings, and names keys and the kinds of values, never a value, with one
215
+ exception: an `"id"` and an `"elementId"` that disagree are both named (they
216
+ are the element ids the caller sent, and naming both is what makes the
217
+ conflict diagnosable); `error.details` adds `expectedPayload` and
218
+ `receivedKeys`. A well-formed payload with an unknown verb is refused with
219
+ `UNSUPPORTED_ACTION`, naming the accepted verbs.
220
+
221
+ These need `@qontinui/ui-bridge` 0.47.0 or later, the first injected runtime
222
+ that serves `getElement` and the payload grammar. Against an older one,
223
+ `getElement` fails with `Unknown command action: getElement`, a flat
224
+ `executeElementAction` throws a `TypeError`, and an `elementId` one answers
225
+ `Element undefined not found`.
226
+
132
227
  ### Frames, device pixel ratio and the capture matrix (`ui-bridge-inject`)
133
228
 
134
229
  `--screenshot <path>` captures a PNG frame **in lockstep** with every
@@ -179,6 +274,39 @@ directory must be empty or not exist yet: a non-empty one is refused before any
179
274
  browser launches, and is never cleared. `--viewport` sides are at most 8192 CSS
180
275
  pixels.
181
276
 
277
+ ### HAR and video recording, service workers and routes (`createTransport` headless/injected)
278
+
279
+ The `headless`, `headed` and `injected` transports pass four options to
280
+ `@qontinui/ui-bridge-headless`'s browser context: `recordHar` (Playwright's
281
+ `recordHar` shape, unchanged), `recordVideo` (`{ dir, size?, … }`; the
282
+ launcher records at the viewport size when `size` is unset),
283
+ `serviceWorkers: 'allow' | 'block'`, and `initRouter(context)`, which runs
284
+ before the first page exists so a `context.route('**/*', …)` it installs
285
+ catches the first document request. The HAR and the video are complete once
286
+ `transport.close()` resolves. A launch that fails inside the launcher writes no
287
+ HAR, but a `ready()` that fails after the launch (an emulation refusal, an
288
+ injected-runtime wait) can still leave a complete archive, so a missing file is
289
+ not how to tell that `ready()` failed. A malformed value fails construction
290
+ with `INVALID_CONFIG`, and a headless peer older than 0.8.0 (ui-bridge#381)
291
+ ignores all four, so the transport then fails `ready()` with
292
+ `CONTEXT_OPTIONS_NOT_APPLIED` and closes the browser it opened.
293
+ `serviceWorkers: 'allow'` alone is exempt, since it is Playwright's default.
294
+ They have no CLI flags: `initRouter` is a function.
295
+
296
+ ```ts
297
+ const transport = createTransport({
298
+ kind: 'injected',
299
+ options: {
300
+ targetUrl: 'https://example.com/',
301
+ recordHar: { path: './capture.zip', content: 'attach', mode: 'minimal' },
302
+ serviceWorkers: 'block',
303
+ },
304
+ });
305
+ await transport.ready();
306
+ // …
307
+ await transport.close(); // the HAR is written
308
+ ```
309
+
182
310
  ### Candidates in situ: the media override, a slow network and a timeline
183
311
 
184
312
  The injected runtime serves the SDK's `mediaOverride` dev control, so a
@@ -194,6 +322,11 @@ ui-bridge-inject --url http://127.0.0.1:3000/ --throttle slow-3g --no-cache \
194
322
  `mediaOverride` answers only after the browser decoded the candidate and its
195
323
  `currentSrc` names it (inside a `<picture>` every `<source srcset>` is replaced
196
324
  too); one it cannot confirm is rolled back and fails the step.
325
+ On Chromium 158 a candidate URL the document already decoded is served with no
326
+ request, even under `--no-cache` (Chromium 147 refetched it); only a new URL or
327
+ a reload reaches the network. A candidate a timeline (or `captureTimeline`'s
328
+ `during`) should observe ARRIVING therefore needs a URL that document has not
329
+ decoded, such as a per-candidate query string (`cand.png?2`).
197
330
  A `loading="lazy"` slot is loaded eagerly while overridden, so an offscreen one
198
331
  still decodes. `mediaOverrideRestore` puts every attribute back and verifies
199
332
  what the browser then RENDERS (the slot's `currentSrc` back on an original URL,
@@ -203,12 +336,28 @@ rewrote a value meanwhile) or `verified: false` is an unknown one.
203
336
  `mediaOverrideList` returning `active: []` proves nothing is still overridden.
204
337
  `--throttle <slow-3g|fast-3g|latency=<ms>,down=<kbps>,up=<kbps>>` applies CDP
205
338
  network emulation once the page is ready (it persists across a reload) and
206
- `--no-cache` disables the HTTP cache; what Chromium accepted is stamped as
207
- `network` on timeline results, matrix cells and the relay info line.
339
+ `--no-cache` disables the HTTP cache (not the document's in-memory image cache:
340
+ on Chromium 158 a URL the page already decoded is answered at once, throttle or
341
+ not, so give a repeated `mediaOverride` candidate its own URL, e.g.
342
+ `cand.png?2`); what Chromium accepted is stamped as `network` on timeline
343
+ results, matrix cells and the relay info line.
208
344
  `captureTimeline` screenshots the page `frames` times, `intervalMs` apart, each
209
345
  frame stamped with its offset; `"reload":true` starts the clock at a fresh
210
346
  navigation commit (the loading-to-loaded timeline) and waits for the runtime to
211
- settle again. `"during":{"action":"mediaOverride","payload":{...}}` runs the
347
+ settle again. Frame 1 of a reload is the first frame the page produced after
348
+ the commit. Right after it the new document can have no compositor frame yet,
349
+ and Chromium refuses the screenshot (`Unable to capture screenshot`), so frame 1
350
+ retries that refusal every 16 ms until `readyTimeoutMs` (default 30000) after
351
+ the commit. A frame's `offsetMs` is when the screenshot that produced it was
352
+ requested, so a frame 1 that had to wait reads tens of milliseconds, not 0, and
353
+ a frame that took more than one attempt carries `attempts`. Under `--throttle`
354
+ that first frame can still be the previous document's held frame; `offsetMs`
355
+ dates it either way. Every other frame retries the same refusal only while the
356
+ retry falls due before the next frame's due time (`"intervalMs":0` is one
357
+ attempt), so retries never take a later frame's slot; an attempt still under
358
+ way when the next frame falls due delays it by what is left of that attempt, as
359
+ any slow screenshot does. No other error is retried; a frame still refused at
360
+ its deadline carries `error` and fails the step. `"during":{"action":"mediaOverride","payload":{...}}` runs the
212
361
  override concurrently with the frames, so the timeline shows the candidate
213
362
  arriving (under `--throttle`, its slow-network load) while the override still
214
363
  confirms decode before it answers. In `--capture-matrix` a relative `dir` lands