@qontinui/ui-bridge-wrapper 0.12.0 → 0.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +157 -8
- package/dist/capture-specs-cli.cjs +301 -53
- package/dist/capture-specs-cli.cjs.map +1 -1
- package/dist/capture-specs-cli.d.ts.map +1 -1
- package/dist/chunk-IKHDNIU4.js +478 -0
- package/dist/chunk-IKHDNIU4.js.map +1 -0
- package/dist/chunk-Z32CC665.cjs +488 -0
- package/dist/chunk-Z32CC665.cjs.map +1 -0
- package/dist/create-transport.d.ts.map +1 -1
- package/dist/frame-capture.d.ts +3 -28
- package/dist/frame-capture.d.ts.map +1 -1
- package/dist/frame-page.d.ts +35 -0
- package/dist/frame-page.d.ts.map +1 -0
- package/dist/index.cjs +145 -375
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +90 -351
- package/dist/index.js.map +1 -1
- package/dist/inject-cli.cjs +407 -134
- package/dist/inject-cli.cjs.map +1 -1
- package/dist/inject-cli.d.ts +1 -1
- package/dist/inject-cli.d.ts.map +1 -1
- package/dist/live/index.cjs +40 -0
- package/dist/live/index.cjs.map +1 -0
- package/dist/live/index.d.ts +32 -0
- package/dist/live/index.d.ts.map +1 -0
- package/dist/live/index.js +3 -0
- package/dist/live/index.js.map +1 -0
- package/dist/login-web-cli.cjs +312 -61
- package/dist/login-web-cli.cjs.map +1 -1
- package/dist/login-web-cli.d.ts.map +1 -1
- package/dist/network-throttle.d.ts +5 -0
- package/dist/network-throttle.d.ts.map +1 -1
- package/dist/screenshot-capture.d.ts +82 -0
- package/dist/screenshot-capture.d.ts.map +1 -0
- package/dist/timeline-capture.d.ts +74 -10
- package/dist/timeline-capture.d.ts.map +1 -1
- package/dist/transports/headless.d.ts +46 -1
- package/dist/transports/headless.d.ts.map +1 -1
- package/dist/transports/injected.d.ts +8 -0
- package/dist/transports/injected.d.ts.map +1 -1
- package/dist/transports/live.d.ts +27 -1
- package/dist/transports/live.d.ts.map +1 -1
- package/dist/types.d.ts +3 -1
- package/dist/types.d.ts.map +1 -1
- package/package.json +11 -8
- package/dist/cli-args.d.ts +0 -13
- package/dist/cli-args.d.ts.map +0 -1
package/README.md
CHANGED
|
@@ -39,6 +39,44 @@ await transport.dispatch('login', { email: 'a@b.c', password: '...' });
|
|
|
39
39
|
await transport.close();
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
+
## A live session the runner can capture
|
|
43
|
+
|
|
44
|
+
The runner's vision routes (`POST /ui-bridge/vision/capture` and the analyze,
|
|
45
|
+
OCR and audit routes beside it) frame a WebSocket-transport app by relaying
|
|
46
|
+
`getScreenshot` over its socket — and only to an app whose register frame
|
|
47
|
+
advertises the `screenshot` capability. A Node process that owns a Playwright
|
|
48
|
+
page opts in by handing that page to the live transport:
|
|
49
|
+
|
|
50
|
+
```ts
|
|
51
|
+
import WebSocket from 'ws';
|
|
52
|
+
import { createTransport } from '@qontinui/ui-bridge-wrapper';
|
|
53
|
+
|
|
54
|
+
const transport = createTransport({
|
|
55
|
+
kind: 'live',
|
|
56
|
+
runnerUrl: 'ws://127.0.0.1:9876/ui-bridge/ws',
|
|
57
|
+
appId: 'my-app',
|
|
58
|
+
options: { webSocketCtor: WebSocket, screenshotPage: page },
|
|
59
|
+
});
|
|
60
|
+
await transport.ready();
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
`screenshotPage` registers the `getScreenshot` handler and advertises
|
|
64
|
+
`screenshot` together, so the advertisement cannot drift from the behaviour.
|
|
65
|
+
The reply is `{ screenshot: <base64 png>, width, height }` (CSS pixels); a
|
|
66
|
+
capture is bounded at 10 s (`SCREENSHOT_TIMEOUT`, retryable) and a failed one
|
|
67
|
+
answers a typed `SCREENSHOT_FAILED`. Without `screenshotPage` the runner refuses
|
|
68
|
+
the app as a capture target by name (`FRAME_SOURCE_UNSUPPORTED`).
|
|
69
|
+
`registerScreenshotCapture(registrar, page)` registers the same handler on any
|
|
70
|
+
other transport that owns a page.
|
|
71
|
+
|
|
72
|
+
A browser bundle (an extension content script, a page) imports the
|
|
73
|
+
**browser-safe** entry `@qontinui/ui-bridge-wrapper/live` instead of the
|
|
74
|
+
package root: the root re-exports Node-only transports and cannot load where
|
|
75
|
+
there is no `require`. Both entries share one copy of `WrapperTransportError`,
|
|
76
|
+
`HandlerRegistry` and `LiveSessionTransport` within a module format, so they
|
|
77
|
+
may be mixed in one ESM (or one CJS) program — but not a `require` of one with
|
|
78
|
+
an `import` of the other, which load separate copies.
|
|
79
|
+
|
|
42
80
|
## Package layout
|
|
43
81
|
|
|
44
82
|
- `src/index.ts` — public surface (transports, helpers, types).
|
|
@@ -60,11 +98,11 @@ so **they run from any directory once installed** — there is no "must run from
|
|
|
60
98
|
the ui-bridge repo root" coupling. Run via `npx @qontinui/ui-bridge-wrapper`
|
|
61
99
|
or a global/dev install:
|
|
62
100
|
|
|
63
|
-
| Bin
|
|
64
|
-
|
|
|
65
|
-
| `ui-bridge-inject`
|
|
66
|
-
| `ui-bridge-login-web`
|
|
67
|
-
| `ui-bridge-capture-specs` | Log in (same flow) then `goto` + snapshot a set of authed pages, writing one `<slug>.snapshot.json` per page.
|
|
101
|
+
| Bin | What it does |
|
|
102
|
+
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
103
|
+
| `ui-bridge-inject` | Drive the injected transport against a UI-Bridge-free page (exec one-shot actions, register as a relay tab, or capture frame-and-snapshot pairs — one, or a viewport × DPR × colour-scheme matrix). |
|
|
104
|
+
| `ui-bridge-login-web` | Automated web login via the injected transport — drives the full Cognito OAuth redirect chain in one headless tab. Optionally asserts on-page text (`--expect-text`) and screenshots the authed landing (`--screenshot`/`--scroll-to`). Prints one JSON result line; exit 0 = login confirmed AND all expected text present. |
|
|
105
|
+
| `ui-bridge-capture-specs` | Log in (same flow) then `goto` + snapshot a set of authed pages, writing one `<slug>.snapshot.json` per page. |
|
|
68
106
|
|
|
69
107
|
Each bin supports `--help`.
|
|
70
108
|
|
|
@@ -129,6 +167,63 @@ ui-bridge-login-web \
|
|
|
129
167
|
ui-bridge-capture-specs --out ./spec-capture
|
|
130
168
|
```
|
|
131
169
|
|
|
170
|
+
### Reading and acting on one element (`ui-bridge-inject` exec mode)
|
|
171
|
+
|
|
172
|
+
Exec mode (`--exec` / `--exec-stdin`) reads and acts on single elements through
|
|
173
|
+
the injected runtime's in-page dispatcher, the same one the relay drives, with
|
|
174
|
+
no relay call:
|
|
175
|
+
|
|
176
|
+
```bash
|
|
177
|
+
# read one element, click it, read its state again
|
|
178
|
+
ui-bridge-inject --url http://127.0.0.1:3000/ \
|
|
179
|
+
--exec 'getElement {"id":"open"}' \
|
|
180
|
+
--exec 'executeElementAction {"id":"open","request":{"action":"click"}}' \
|
|
181
|
+
--exec 'getElementState {"id":"open"}'
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
- `getElement {"id":"<id>"}` answers one element as a snapshot carries it:
|
|
185
|
+
identity (`id`, `type`, `tagName`, `label`, `role`, `accessibleName`),
|
|
186
|
+
`state`, box (`bbox`, `visible`) and `actions`, the same object as the
|
|
187
|
+
matching `getControlSnapshot` element, redaction included.
|
|
188
|
+
`{"elementId":"<id>"}` is accepted too; given both, they must name the same
|
|
189
|
+
element. Nothing settles first, so run `waitForSelector` before it for a
|
|
190
|
+
settled read. Beside the id it tolerates, and ignores, `settleTimeout` and
|
|
191
|
+
`skipSettle` (it does not settle), `targetTabId`, and the WebSocket
|
|
192
|
+
protocol's `includeState` (the state is always included); any other key is
|
|
193
|
+
refused with `INVALID_REQUEST`. An unknown id, or one whose DOM node is gone,
|
|
194
|
+
is a returned `ELEMENT_NOT_FOUND` (an `error` line and exit 1), never a
|
|
195
|
+
`null` result.
|
|
196
|
+
- `getElementState {"id":"<id>"}` answers the state alone. It reads `id` only,
|
|
197
|
+
and an unknown id is a thrown error (`HANDLER_ERROR`).
|
|
198
|
+
- `executeElementAction {"id":"<id>","request":{"action":"click"}}` acts on one
|
|
199
|
+
element. `request` holds the verb and the fields it reads (`value`, `text`,
|
|
200
|
+
`clear`, `params`, `fromSnapshotId`, …). Two other spellings are accepted:
|
|
201
|
+
`elementId` in place of `id`, and a flat payload such as
|
|
202
|
+
`{"id":"<id>","action":"type","text":"hi"}`, which, minus `id` / `elementId`,
|
|
203
|
+
_is_ the request. No key _beside_ a `request` object is silently dropped: a
|
|
204
|
+
top-level `action` or request field there must equal the same field inside
|
|
205
|
+
`request` (an agreeing duplicate is accepted); one that is only at the top
|
|
206
|
+
level, or at both levels with different values, is refused, and so are an
|
|
207
|
+
`id` and an `elementId` that name different elements. The dispatcher keys
|
|
208
|
+
stay allowed at the top level. The request fields themselves (the keys
|
|
209
|
+
inside `request`, or every key of a flat payload but the id) reach the
|
|
210
|
+
action as sent, so a misspelt one there (`txt` for `text`) is not caught.
|
|
211
|
+
Every refused shape is a returned `INVALID_REQUEST` (an `error` line and
|
|
212
|
+
exit 1, never a thrown `TypeError`) whose message names the canonical shape
|
|
213
|
+
`{"id":"<elementId>","request":{"action":"<action>", …}}` and both accepted
|
|
214
|
+
spellings, and names keys and the kinds of values, never a value, with one
|
|
215
|
+
exception: an `"id"` and an `"elementId"` that disagree are both named (they
|
|
216
|
+
are the element ids the caller sent, and naming both is what makes the
|
|
217
|
+
conflict diagnosable); `error.details` adds `expectedPayload` and
|
|
218
|
+
`receivedKeys`. A well-formed payload with an unknown verb is refused with
|
|
219
|
+
`UNSUPPORTED_ACTION`, naming the accepted verbs.
|
|
220
|
+
|
|
221
|
+
These need `@qontinui/ui-bridge` 0.47.0 or later, the first injected runtime
|
|
222
|
+
that serves `getElement` and the payload grammar. Against an older one,
|
|
223
|
+
`getElement` fails with `Unknown command action: getElement`, a flat
|
|
224
|
+
`executeElementAction` throws a `TypeError`, and an `elementId` one answers
|
|
225
|
+
`Element undefined not found`.
|
|
226
|
+
|
|
132
227
|
### Frames, device pixel ratio and the capture matrix (`ui-bridge-inject`)
|
|
133
228
|
|
|
134
229
|
`--screenshot <path>` captures a PNG frame **in lockstep** with every
|
|
@@ -179,6 +274,39 @@ directory must be empty or not exist yet: a non-empty one is refused before any
|
|
|
179
274
|
browser launches, and is never cleared. `--viewport` sides are at most 8192 CSS
|
|
180
275
|
pixels.
|
|
181
276
|
|
|
277
|
+
### HAR and video recording, service workers and routes (`createTransport` headless/injected)
|
|
278
|
+
|
|
279
|
+
The `headless`, `headed` and `injected` transports pass four options to
|
|
280
|
+
`@qontinui/ui-bridge-headless`'s browser context: `recordHar` (Playwright's
|
|
281
|
+
`recordHar` shape, unchanged), `recordVideo` (`{ dir, size?, … }`; the
|
|
282
|
+
launcher records at the viewport size when `size` is unset),
|
|
283
|
+
`serviceWorkers: 'allow' | 'block'`, and `initRouter(context)`, which runs
|
|
284
|
+
before the first page exists so a `context.route('**/*', …)` it installs
|
|
285
|
+
catches the first document request. The HAR and the video are complete once
|
|
286
|
+
`transport.close()` resolves. A launch that fails inside the launcher writes no
|
|
287
|
+
HAR, but a `ready()` that fails after the launch (an emulation refusal, an
|
|
288
|
+
injected-runtime wait) can still leave a complete archive, so a missing file is
|
|
289
|
+
not how to tell that `ready()` failed. A malformed value fails construction
|
|
290
|
+
with `INVALID_CONFIG`, and a headless peer older than 0.8.0 (ui-bridge#381)
|
|
291
|
+
ignores all four, so the transport then fails `ready()` with
|
|
292
|
+
`CONTEXT_OPTIONS_NOT_APPLIED` and closes the browser it opened.
|
|
293
|
+
`serviceWorkers: 'allow'` alone is exempt, since it is Playwright's default.
|
|
294
|
+
They have no CLI flags: `initRouter` is a function.
|
|
295
|
+
|
|
296
|
+
```ts
|
|
297
|
+
const transport = createTransport({
|
|
298
|
+
kind: 'injected',
|
|
299
|
+
options: {
|
|
300
|
+
targetUrl: 'https://example.com/',
|
|
301
|
+
recordHar: { path: './capture.zip', content: 'attach', mode: 'minimal' },
|
|
302
|
+
serviceWorkers: 'block',
|
|
303
|
+
},
|
|
304
|
+
});
|
|
305
|
+
await transport.ready();
|
|
306
|
+
// …
|
|
307
|
+
await transport.close(); // the HAR is written
|
|
308
|
+
```
|
|
309
|
+
|
|
182
310
|
### Candidates in situ: the media override, a slow network and a timeline
|
|
183
311
|
|
|
184
312
|
The injected runtime serves the SDK's `mediaOverride` dev control, so a
|
|
@@ -194,6 +322,11 @@ ui-bridge-inject --url http://127.0.0.1:3000/ --throttle slow-3g --no-cache \
|
|
|
194
322
|
`mediaOverride` answers only after the browser decoded the candidate and its
|
|
195
323
|
`currentSrc` names it (inside a `<picture>` every `<source srcset>` is replaced
|
|
196
324
|
too); one it cannot confirm is rolled back and fails the step.
|
|
325
|
+
On Chromium 158 a candidate URL the document already decoded is served with no
|
|
326
|
+
request, even under `--no-cache` (Chromium 147 refetched it); only a new URL or
|
|
327
|
+
a reload reaches the network. A candidate a timeline (or `captureTimeline`'s
|
|
328
|
+
`during`) should observe ARRIVING therefore needs a URL that document has not
|
|
329
|
+
decoded, such as a per-candidate query string (`cand.png?2`).
|
|
197
330
|
A `loading="lazy"` slot is loaded eagerly while overridden, so an offscreen one
|
|
198
331
|
still decodes. `mediaOverrideRestore` puts every attribute back and verifies
|
|
199
332
|
what the browser then RENDERS (the slot's `currentSrc` back on an original URL,
|
|
@@ -203,12 +336,28 @@ rewrote a value meanwhile) or `verified: false` is an unknown one.
|
|
|
203
336
|
`mediaOverrideList` returning `active: []` proves nothing is still overridden.
|
|
204
337
|
`--throttle <slow-3g|fast-3g|latency=<ms>,down=<kbps>,up=<kbps>>` applies CDP
|
|
205
338
|
network emulation once the page is ready (it persists across a reload) and
|
|
206
|
-
`--no-cache` disables the HTTP cache
|
|
207
|
-
|
|
339
|
+
`--no-cache` disables the HTTP cache (not the document's in-memory image cache:
|
|
340
|
+
on Chromium 158 a URL the page already decoded is answered at once, throttle or
|
|
341
|
+
not, so give a repeated `mediaOverride` candidate its own URL, e.g.
|
|
342
|
+
`cand.png?2`); what Chromium accepted is stamped as `network` on timeline
|
|
343
|
+
results, matrix cells and the relay info line.
|
|
208
344
|
`captureTimeline` screenshots the page `frames` times, `intervalMs` apart, each
|
|
209
345
|
frame stamped with its offset; `"reload":true` starts the clock at a fresh
|
|
210
346
|
navigation commit (the loading-to-loaded timeline) and waits for the runtime to
|
|
211
|
-
settle again.
|
|
347
|
+
settle again. Frame 1 of a reload is the first frame the page produced after
|
|
348
|
+
the commit. Right after it the new document can have no compositor frame yet,
|
|
349
|
+
and Chromium refuses the screenshot (`Unable to capture screenshot`), so frame 1
|
|
350
|
+
retries that refusal every 16 ms until `readyTimeoutMs` (default 30000) after
|
|
351
|
+
the commit. A frame's `offsetMs` is when the screenshot that produced it was
|
|
352
|
+
requested, so a frame 1 that had to wait reads tens of milliseconds, not 0, and
|
|
353
|
+
a frame that took more than one attempt carries `attempts`. Under `--throttle`
|
|
354
|
+
that first frame can still be the previous document's held frame; `offsetMs`
|
|
355
|
+
dates it either way. Every other frame retries the same refusal only while the
|
|
356
|
+
retry falls due before the next frame's due time (`"intervalMs":0` is one
|
|
357
|
+
attempt), so retries never take a later frame's slot; an attempt still under
|
|
358
|
+
way when the next frame falls due delays it by what is left of that attempt, as
|
|
359
|
+
any slow screenshot does. No other error is retried; a frame still refused at
|
|
360
|
+
its deadline carries `error` and fails the step. `"during":{"action":"mediaOverride","payload":{...}}` runs the
|
|
212
361
|
override concurrently with the frames, so the timeline shows the candidate
|
|
213
362
|
arriving (under `--throttle`, its slow-network load) while the override still
|
|
214
363
|
confirms decode before it answers. In `--capture-matrix` a relative `dir` lands
|