@kana-ai/edgar-ext-screenshots 0.1.2 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,20 +1,21 @@
1
1
  # @kana-ai/edgar-ext-screenshots
2
2
 
3
- The screenshots extension of EDGAR. It captures a web page as a PNG file in the run's sandbox with the machine's Chrome or Chromium, and compares two PNG files pixel by pixel. Workflows call its tools as `screenshots__capture`, `screenshots__compare` and `screenshots__preview`. It also puts a picture of each service's preview page in the body of the pull request that `remediate` opens.
3
+ The screenshots extension of EDGAR. It captures a web page as a PNG file in the run's sandbox with the machine's Chrome or Chromium, and compares two PNG files pixel by pixel. It also records a captioned flow through a service's pages. Workflows call its tools as `screenshots__capture`, `screenshots__compare`, `screenshots__preview` and `screenshots__record`. It also puts a picture of each service's preview page, or a flow through the screen a change touches, in the body of the pull request that `remediate` opens.
4
4
 
5
5
  ## Tools
6
6
 
7
- | Tool | Input | Output | Notes |
8
- | --------- | ---------------------------------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
9
- | `capture` | `url`, `path`, `viewport` (`width`, `height`), `full_page`, `wait_for` | `path`, `width`, `height`, `bytes` | `url` is `http` or `https`. `path` is a relative path in the work folder that ends in `.png`. `viewport` is 1280 by 800 by default. `wait_for` is a CSS selector, or a number of milliseconds after the page loads. |
10
- | `compare` | `before`, `after`, `threshold` | `differing_pixels`, `share`, `same` | `before` and `after` are PNG files of the work folder. A pixel differs when one channel differs. `same` is `share <= threshold`; `threshold` is 0 by default. Two sizes differ everywhere. |
11
- | `preview` | `service` | `path`, `bytes`, `caption`; or `missing` | Captures the page that `[extensions.screenshots] previews` names for `service` into `/tmp/edgar-preview.png`, the run's temporary folder, which no commit holds. A service with no page is `skipped`. A page that does not load is `done` with `missing` and the reason. |
7
+ | Tool | Input | Output | Notes |
8
+ | --------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
9
+ | `capture` | `url`, `path`, `viewport` (`width`, `height`), `full_page`, `wait_for` | `path`, `width`, `height`, `bytes` | `url` is `http` or `https`. `path` is a relative path in the work folder that ends in `.png`. `viewport` is 1280 by 800 by default. `wait_for` is a CSS selector, or a number of milliseconds after the page loads. |
10
+ | `compare` | `before`, `after`, `threshold` | `differing_pixels`, `share`, `same` | `before` and `after` are PNG files of the work folder. A pixel differs when one channel differs. `same` is `share <= threshold`; `threshold` is 0 by default. Two sizes differ everywhere. |
11
+ | `preview` | `service` | `path`, `bytes`, `caption`, and `url` when the picture is in the project's storage; or `missing` | Captures the page that `[extensions.screenshots] previews` names for `service` into `/tmp/edgar-preview.png`, the run's temporary folder, which no commit holds. When EDGAR keeps pictures in `[storage]`, the picture goes there too, and `url` is the address that shows it. A service with no page is `skipped`. A page that does not load is `done` with `missing` and the reason. |
12
+ | `record` | `service`, `steps` (1 to 10 of `action`, `url` or `selector`, `caption`) | `path`, `bytes`, `caption`, `url` when the sheet is in the project's storage, and `video` (`path`, `bytes`) with `ffmpeg`; or `missing` | Starts at the page `previews` names for `service`. `open` takes `url`, a path from that page or a url on its site; `click` and `wait` take a CSS `selector`. Each frame carries its step's caption at its foot. The frames make one contact sheet, `/tmp/edgar-flow.png`, and with `ffmpeg` in the sandbox an mp4, `/tmp/edgar-flow.mp4`, each frame held two seconds. Each file steps down in size until it is at most 1 MB, what a pull request body shows. With no `ffmpeg`, or when it fails, there is no `video`, and the log says why. A service with no page is `skipped`; an `open` off the page's site or a step that fails is `done` with `missing`. |
12
13
 
13
14
  ## How it works
14
15
 
15
16
  1. The tool sends the source of `src/browser/session.ts` to the sandbox as `node --input-type=module-typescript -e <source> -- <job>`. The session imports only `node:` modules and types, which Node erases.
16
17
  2. Inside the sandbox, the session starts the first browser that exists, headless, with `--remote-debugging-pipe`, and drives it over the DevTools protocol on file descriptors 3 and 4. It needs no package, and it downloads no browser.
17
- 3. The session writes the PNG file and prints one JSON line, which the tool reads.
18
+ 3. The session writes the PNG file and prints one JSON line, which the tool reads. For `record`, it draws the contact sheet in the same browser, and runs `ffmpeg` from `PATH` for the video.
18
19
 
19
20
  The browser is the first that exists of:
20
21
 
@@ -22,7 +23,7 @@ The browser is the first that exists of:
22
23
  2. `google-chrome`, `google-chrome-stable`, `chromium`, `chromium-browser` and `chrome` in each `PATH` folder.
23
24
  3. The usual install paths: `/Applications/Google Chrome.app`, `/Applications/Chromium.app`, `/opt/google/chrome/chrome`, `/usr/lib/chromium/chromium` and `/snap/bin/chromium`.
24
25
 
25
- `edgar check` refuses the project with `X008` when the extension is enabled and no browser is found. The extension also declares `node` in `executables.requires`, because `capture` runs the browser session on Node in the sandbox, so a machine with no `node` on `PATH` is `X008` too. The sandbox must see the browser: the `worktree` kind binds `/usr`, `/bin`, `/lib` and each `PATH` folder, so a browser under `/opt` must have its folder on `PATH`. The `container` kind needs the browser in `[sandbox] image`. On Debian, `/usr/bin/chromium` is a script that reads `/etc/chromium.d`, which the sandbox does not bind, so set `browser = "/usr/lib/chromium/chromium"`.
26
+ `edgar check` refuses the project with `X008` when the extension is enabled and no browser is found. The extension also declares `node` in `executables.requires`, because `capture` runs the browser session on Node in the sandbox, so a machine with no `node` on `PATH` is `X008` too. `ffmpeg` is optional, so the manifest does not require it: `record` makes its video only when the sandbox finds it on `PATH`. The sandbox must see the browser: the `worktree` kind binds `/usr`, `/bin`, `/lib` and each `PATH` folder, so a browser under `/opt` must have its folder on `PATH`. The `container` kind needs the browser in `[sandbox] image`. On Debian, `/usr/bin/chromium` is a script that reads `/etc/chromium.d`, which the sandbox does not bind, so set `browser = "/usr/lib/chromium/chromium"`.
26
27
 
27
28
  ## Enable and approve
28
29
 
@@ -37,7 +38,7 @@ The browser is the first that exists of:
37
38
  previews = { api = "http://127.0.0.1:8080/" } # optional: the page of each service's stack
38
39
  ```
39
40
 
40
- 2. Approve its capabilities. The manifest declares `sandbox:read` and `sandbox:write`, no secret and no network:
41
+ 2. Approve its capabilities. The manifest declares `sandbox:read` and `sandbox:write`, no secret and no network. A new version that adds a tool or changes a schema, such as `record`, needs the approval again:
41
42
 
42
43
  ```sh
43
44
  edgar extensions approve screenshots
@@ -50,7 +51,8 @@ The browser is the first that exists of:
50
51
  The extension adds the layer `workflows/remediate.toml` onto the built-in `remediate`:
51
52
 
52
53
  1. `preview` runs `screenshots__preview` after `final_checks` passes, with `stack = true`, so the service's environment starts its stack first.
53
- 2. `attach_preview` runs `github__attach_screenshot` after `open_pr`. It stores the picture on the branch `edgar/screenshots` and puts it in the body. See the [GitHub extension](/packages/extensions/ext-github/README.md).
54
+ 2. `flow` runs `screenshots__record` in place of `preview`, the same way, when the change sets `touches_ui = true` and gives `demo`, the steps of the changed screen that the `implement` role returns. The flow shows the change only, not the page before it.
55
+ 3. `attach_preview` runs `github__attach_screenshot` after `open_pr`, with the output of `flow`, else that of `preview`, and links a flow's video under its picture. It puts the picture in the body: by its `url` when EDGAR keeps pictures in `[storage]` (see [Screenshots](/docs/guides/config.md#screenshots)), else stored on the branch `edgar/screenshots`. See the [GitHub extension](/packages/extensions/ext-github/README.md).
54
56
 
55
57
  A service with no entry in `previews` gets no picture and no line in its body. The layer needs the `github` extension, as `remediate` does.
56
58
 
@@ -104,7 +106,7 @@ Each tool returns a failure as its outcome and throws only for a bug:
104
106
 
105
107
  ```sh
106
108
  pnpm build # writes edgar-extension.json and schemas/
107
- pnpm test # unit tests: the schemas, the browser lookup, the session's arguments, the decoder and the comparison, with no browser
109
+ pnpm test # unit tests: the schemas, the browser lookup, the session's arguments, the decoder, the comparison and the flow's steps, sheet and video ladder, with no browser
108
110
  ```
109
111
 
110
112
  `edgar-extension.json` and `schemas/` are committed: a change to them in a diff is a change to what a person approves.
@@ -7,6 +7,21 @@ export interface Image {
7
7
  readonly height: number;
8
8
  readonly rgba: Uint8Array;
9
9
  }
10
+ /** One step of a recorded flow: open a page, click an element, or wait for one to show. */
11
+ export type Step = {
12
+ readonly action: "open";
13
+ readonly url: string;
14
+ readonly caption: string;
15
+ } | {
16
+ readonly action: "click" | "wait";
17
+ readonly selector: string;
18
+ readonly caption: string;
19
+ };
20
+ /** One rung of the video's ladder: the width of its frames and its x264 quality, higher is smaller. */
21
+ export interface Rung {
22
+ readonly width: number;
23
+ readonly crf: number;
24
+ }
10
25
  /** What the tools ask the session to do. */
11
26
  export type Job = {
12
27
  readonly kind: "capture";
@@ -22,6 +37,15 @@ export type Job = {
22
37
  readonly before: string;
23
38
  readonly after: string;
24
39
  readonly threshold: number;
40
+ } | {
41
+ readonly kind: "record";
42
+ readonly browsers: readonly string[];
43
+ readonly steps: readonly Step[];
44
+ readonly width: number;
45
+ readonly height: number;
46
+ readonly sheet: string;
47
+ readonly video: string;
48
+ readonly max_bytes: number;
25
49
  };
26
50
  /** The file of `path` on this machine: a path under `/tmp/` is in the run's temporary folder, `TMPDIR`, as the sandbox's ports read it, and any other path is in the work folder. */
27
51
  export declare function placed(path: string): string;
@@ -42,6 +66,14 @@ export declare function compare(before: Image, after: Image, threshold: number):
42
66
  readonly share: number;
43
67
  readonly same: boolean;
44
68
  };
69
+ /** The script that puts `caption` at the foot of the page as text, for the next frame. */
70
+ export declare function overlay(caption: string): string;
71
+ /** The page of the contact sheet: the frames in order, as PNG data in base64, two to a row across `width`. */
72
+ export declare function sheetHtml(frames: readonly string[], width: number): string;
73
+ /** The `ffmpeg` arguments that make an mp4 of the frames in `folder`, each held two seconds, at the rung's width and quality. */
74
+ export declare function ffmpegArgs(folder: string, out: string, rung: Rung): readonly string[];
75
+ /** The size of the first rung of `ladder` whose file is at most `max` bytes; each rung writes the file again, smaller than the one before. */
76
+ export declare function fitted<T>(ladder: readonly T[], make: (rung: T) => Promise<Result<number, string>>, max: number): Promise<Result<number, string>>;
45
77
  /** Runs one job. */
46
78
  export declare function run(job: Job): Promise<Answer>;
47
79
  /**
@@ -1,13 +1,23 @@
1
1
  // The browser session that runs inside the sandbox as `node -e`. It imports only `node:` modules and erased types,
2
2
  // since the tools pass its source text and nothing else crosses into the sandbox.
3
- import { spawn } from "node:child_process";
4
- import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
3
+ import { spawn, spawnSync } from "node:child_process";
4
+ import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync, } from "node:fs";
5
5
  import { tmpdir } from "node:os";
6
6
  import { dirname, join } from "node:path";
7
7
  import { Readable, Writable } from "node:stream";
8
8
  import { inflateSync } from "node:zlib";
9
9
  const LOAD_MS = 30_000;
10
10
  const POLL_MS = 100;
11
+ /** How long a click has to show its effect before its frame. */
12
+ const SETTLE_MS = 500;
13
+ const SHEET_WIDTHS = [1280, 960, 640];
14
+ const VIDEO_LADDER = [
15
+ { width: 1280, crf: 28 },
16
+ { width: 960, crf: 32 },
17
+ { width: 640, crf: 36 },
18
+ ];
19
+ const CAPTION_ID = "edgar-caption";
20
+ const CAPTION_STYLE = "position:fixed;left:0;right:0;bottom:0;z-index:2147483647;margin:0;padding:12px 16px;background:rgba(0,0,0,0.8);color:#fff;font:20px/1.4 sans-serif";
11
21
  function isRecord(value) {
12
22
  return typeof value === "object" && value !== null && !Array.isArray(value);
13
23
  }
@@ -141,6 +151,54 @@ export function compare(before, after, threshold) {
141
151
  const share = total === 0 ? 0 : differing / total;
142
152
  return { differing_pixels: differing, share, same: share <= threshold };
143
153
  }
154
+ /** The script that puts `caption` at the foot of the page as text, for the next frame. */
155
+ export function overlay(caption) {
156
+ return `(() => { const note = document.createElement("div"); note.id = ${JSON.stringify(CAPTION_ID)}; note.style.cssText = ${JSON.stringify(CAPTION_STYLE)}; note.textContent = ${JSON.stringify(caption)}; document.documentElement.append(note); })()`;
157
+ }
158
+ /** The page of the contact sheet: the frames in order, as PNG data in base64, two to a row across `width`. */
159
+ export function sheetHtml(frames, width) {
160
+ const images = frames.map((frame) => `<img src="data:image/png;base64,${frame}">`).join("");
161
+ const columns = Math.min(2, frames.length);
162
+ const style = `body{margin:0;padding:8px;width:${String(width - 16)}px;display:grid;grid-template-columns:repeat(${String(columns)},1fr);gap:8px;background:#fff}img{display:block;width:100%}`;
163
+ return `<!doctype html><html><head><style>${style}</style></head><body>${images}</body></html>`;
164
+ }
165
+ /** The `ffmpeg` arguments that make an mp4 of the frames in `folder`, each held two seconds, at the rung's width and quality. */
166
+ export function ffmpegArgs(folder, out, rung) {
167
+ return [
168
+ "-y",
169
+ "-loglevel",
170
+ "error",
171
+ "-framerate",
172
+ "1/2",
173
+ "-i",
174
+ join(folder, "frame-%03d.png"),
175
+ "-vf",
176
+ `scale=${String(rung.width)}:-2,format=yuv420p`,
177
+ "-r",
178
+ "10",
179
+ "-c:v",
180
+ "libx264",
181
+ "-crf",
182
+ String(rung.crf),
183
+ "-movflags",
184
+ "+faststart",
185
+ out,
186
+ ];
187
+ }
188
+ /** The size of the first rung of `ladder` whose file is at most `max` bytes; each rung writes the file again, smaller than the one before. */
189
+ export async function fitted(ladder, make, max) {
190
+ let smallest = 0;
191
+ for (const rung of ladder) {
192
+ const made = await make(rung);
193
+ if (!made.ok || made.value <= max)
194
+ return made;
195
+ smallest = made.value;
196
+ }
197
+ return {
198
+ ok: false,
199
+ error: `it is ${String(smallest)} bytes at its smallest, over the ${String(max)} bytes it may be`,
200
+ };
201
+ }
144
202
  /** A DevTools protocol connection over Chrome's pipe: JSON messages, each ended by a NUL byte. */
145
203
  class Pipe {
146
204
  #to;
@@ -230,7 +288,7 @@ async function waitFor(call, selector) {
230
288
  }
231
289
  return false;
232
290
  }
233
- async function shoot(pipe, job) {
291
+ async function tab(pipe, width, height) {
234
292
  const target = await pipe.send("Target.createTarget", { url: "about:blank" });
235
293
  if (!target.ok)
236
294
  return target;
@@ -243,48 +301,181 @@ async function shoot(pipe, job) {
243
301
  const sessionId = String(attached.value["sessionId"]);
244
302
  const call = (method, params) => pipe.send(method, params, sessionId);
245
303
  await call("Page.enable", {});
246
- const metrics = { width: job.width, height: job.height, deviceScaleFactor: 1, mobile: false };
247
- await call("Emulation.setDeviceMetricsOverride", metrics);
248
- const loaded = pipe.event("Page.loadEventFired", sessionId, LOAD_MS);
249
- const navigated = await call("Page.navigate", { url: job.url });
304
+ await call("Emulation.setDeviceMetricsOverride", {
305
+ width,
306
+ height,
307
+ deviceScaleFactor: 1,
308
+ mobile: false,
309
+ });
310
+ return { ok: true, value: { sessionId, call } };
311
+ }
312
+ async function go(pipe, page, url) {
313
+ const loaded = pipe.event("Page.loadEventFired", page.sessionId, LOAD_MS);
314
+ const navigated = await page.call("Page.navigate", { url });
250
315
  if (!navigated.ok)
251
316
  return navigated;
252
317
  const failed = navigated.value["errorText"];
253
318
  if (typeof failed === "string")
254
- return { ok: false, error: `${job.url} did not load: ${failed}` };
319
+ return { ok: false, error: `${url} did not load: ${failed}` };
255
320
  if (!(await loaded))
256
- return { ok: false, error: `${job.url} did not finish loading in 30 s` };
257
- if (typeof job.wait_for === "number")
258
- await sleep(job.wait_for);
259
- if (typeof job.wait_for === "string" && !(await waitFor(call, job.wait_for))) {
260
- return { ok: false, error: `no element matched \`${job.wait_for}\` within 30 s` };
261
- }
321
+ return { ok: false, error: `${url} did not finish loading in 30 s` };
322
+ return { ok: true, value: undefined };
323
+ }
324
+ async function screenshot(page, fullPage) {
262
325
  let params = { format: "png" };
263
- if (job.full_page) {
264
- const layout = await call("Page.getLayoutMetrics", {});
326
+ if (fullPage) {
327
+ const layout = await page.call("Page.getLayoutMetrics", {});
265
328
  const size = layout.ok ? layout.value["cssContentSize"] : undefined;
266
329
  if (isRecord(size)) {
267
330
  const clip = { x: 0, y: 0, width: size["width"], height: size["height"], scale: 1 };
268
331
  params = { format: "png", captureBeyondViewport: true, clip };
269
332
  }
270
333
  }
271
- const shot = await call("Page.captureScreenshot", params);
334
+ const shot = await page.call("Page.captureScreenshot", params);
272
335
  if (!shot.ok)
273
336
  return shot;
274
- const png = Buffer.from(String(shot.value["data"]), "base64");
275
- const file = placed(job.path);
337
+ return { ok: true, value: Buffer.from(String(shot.value["data"]), "base64") };
338
+ }
339
+ function write(path, data) {
340
+ const file = placed(path);
276
341
  mkdirSync(dirname(file), { recursive: true });
277
- writeFileSync(file, png);
278
- return { ok: true, value: { path: job.path, ...dimensions(png), bytes: png.byteLength } };
342
+ writeFileSync(file, data);
343
+ }
344
+ async function shoot(pipe, job) {
345
+ const page = await tab(pipe, job.width, job.height);
346
+ if (!page.ok)
347
+ return page;
348
+ const loaded = await go(pipe, page.value, job.url);
349
+ if (!loaded.ok)
350
+ return loaded;
351
+ if (typeof job.wait_for === "number")
352
+ await sleep(job.wait_for);
353
+ if (typeof job.wait_for === "string" && !(await waitFor(page.value.call, job.wait_for))) {
354
+ return { ok: false, error: `no element matched \`${job.wait_for}\` within 30 s` };
355
+ }
356
+ const png = await screenshot(page.value, job.full_page);
357
+ if (!png.ok)
358
+ return png;
359
+ write(job.path, png.value);
360
+ return {
361
+ ok: true,
362
+ value: { path: job.path, ...dimensions(png.value), bytes: png.value.byteLength },
363
+ };
364
+ }
365
+ async function act(pipe, page, step) {
366
+ switch (step.action) {
367
+ case "open":
368
+ return await go(pipe, page, step.url);
369
+ case "click": {
370
+ const selector = JSON.stringify(step.selector);
371
+ const expression = `(() => { const found = document.querySelector(${selector}); if (found === null) return false; found.scrollIntoView({ block: "center" }); found.click(); return true; })()`;
372
+ const clicked = await page.call("Runtime.evaluate", { expression, returnByValue: true });
373
+ const result = clicked.ok ? clicked.value["result"] : undefined;
374
+ if (!isRecord(result) || result["value"] !== true) {
375
+ return { ok: false, error: `no element matched \`${step.selector}\`` };
376
+ }
377
+ await sleep(SETTLE_MS);
378
+ return { ok: true, value: undefined };
379
+ }
380
+ case "wait":
381
+ return (await waitFor(page.call, step.selector))
382
+ ? { ok: true, value: undefined }
383
+ : { ok: false, error: `no element matched \`${step.selector}\` within 30 s` };
384
+ default:
385
+ step;
386
+ throw new Error("unreachable");
387
+ }
388
+ }
389
+ /** Draws the contact sheet in `page`, a blank tab of its own, so no script of the service's page runs beside it. */
390
+ async function sheet(page, frames, path, width) {
391
+ await page.call("Emulation.setDeviceMetricsOverride", {
392
+ width,
393
+ height: 600,
394
+ deviceScaleFactor: 1,
395
+ mobile: false,
396
+ });
397
+ const html = sheetHtml(frames.map((frame) => frame.toString("base64")), width);
398
+ const expression = `document.open(); document.write(${JSON.stringify(html)}); document.close(); Promise.all([...document.images].map((image) => image.decode())).then(() => true)`;
399
+ const drawn = await page.call("Runtime.evaluate", { expression, awaitPromise: true });
400
+ if (!drawn.ok)
401
+ return drawn;
402
+ const png = await screenshot(page, true);
403
+ if (!png.ok)
404
+ return png;
405
+ write(path, png.value);
406
+ return { ok: true, value: png.value.byteLength };
407
+ }
408
+ /** The mp4 of `frames` at `path`, stepped down until it is at most `max` bytes; `ffmpeg` is optional, so its absence is a reason, not a failure of the flow. */
409
+ async function video(frames, path, max) {
410
+ const folder = mkdtempSync(join(tmpdir(), "edgar-frames-"));
411
+ try {
412
+ frames.forEach((frame, index) => {
413
+ writeFileSync(join(folder, `frame-${String(index).padStart(3, "0")}.png`), frame);
414
+ });
415
+ const file = placed(path);
416
+ mkdirSync(dirname(file), { recursive: true });
417
+ const encode = (rung) => {
418
+ const ran = spawnSync("ffmpeg", ffmpegArgs(folder, file, rung), {
419
+ stdio: ["ignore", "ignore", "pipe"],
420
+ timeout: 120_000,
421
+ });
422
+ if (ran.error !== undefined) {
423
+ return Promise.resolve({ ok: false, error: `ffmpeg did not run: ${ran.error.message}` });
424
+ }
425
+ if (ran.status !== 0) {
426
+ const said = ran.stderr.toString("utf8").trim().slice(-300);
427
+ return Promise.resolve({
428
+ ok: false,
429
+ error: `ffmpeg exited ${String(ran.status)}: ${said}`,
430
+ });
431
+ }
432
+ return Promise.resolve({ ok: true, value: statSync(file).size });
433
+ };
434
+ return await fitted(VIDEO_LADDER, encode, max);
435
+ }
436
+ finally {
437
+ rmSync(folder, { recursive: true, force: true });
438
+ }
279
439
  }
280
- async function capture(job) {
281
- const browser = pick(job.browsers, existsSync);
440
+ async function record(pipe, job) {
441
+ const page = await tab(pipe, job.width, job.height);
442
+ if (!page.ok)
443
+ return page;
444
+ const frames = [];
445
+ for (const [index, step] of job.steps.entries()) {
446
+ const label = `${String(index + 1)}. ${step.caption}`;
447
+ const acted = await act(pipe, page.value, step);
448
+ if (!acted.ok)
449
+ return { ok: false, error: `step ${label}: ${acted.error}` };
450
+ await page.value.call("Runtime.evaluate", { expression: overlay(label) });
451
+ const frame = await screenshot(page.value, false);
452
+ await page.value.call("Runtime.evaluate", {
453
+ expression: `document.getElementById(${JSON.stringify(CAPTION_ID)})?.remove()`,
454
+ });
455
+ if (!frame.ok)
456
+ return frame;
457
+ frames.push(frame.value);
458
+ }
459
+ const board = await tab(pipe, job.width, job.height);
460
+ if (!board.ok)
461
+ return board;
462
+ const drawn = await fitted(SHEET_WIDTHS, (width) => sheet(board.value, frames, job.sheet, width), job.max_bytes);
463
+ if (!drawn.ok)
464
+ return { ok: false, error: `the contact sheet: ${drawn.error}` };
465
+ const made = await video(frames, job.video, job.max_bytes);
466
+ const clip = made.ok
467
+ ? { video: { path: job.video, bytes: made.value } }
468
+ : { video_missing: made.error };
469
+ return { ok: true, value: { path: job.sheet, bytes: drawn.value, ...clip } };
470
+ }
471
+ async function browse(browsers, width, height, use) {
472
+ const browser = pick(browsers, existsSync);
282
473
  if (browser === undefined) {
283
- return { ok: false, error: `no Chrome or Chromium at any of: ${job.browsers.join(", ")}` };
474
+ return { ok: false, error: `no Chrome or Chromium at any of: ${browsers.join(", ")}` };
284
475
  }
285
476
  const profile = mkdtempSync(join(tmpdir(), "edgar-chrome-"));
286
477
  const proxy = process.env["HTTPS_PROXY"] ?? process.env["https_proxy"];
287
- const args = chromeArgs(profile, proxy, job.width, job.height);
478
+ const args = chromeArgs(profile, proxy, width, height);
288
479
  const child = spawn(browser, args, { stdio: ["ignore", "ignore", "pipe", "pipe", "pipe"] });
289
480
  let said = "";
290
481
  child.stderr?.on("data", (chunk) => {
@@ -295,7 +486,7 @@ async function capture(job) {
295
486
  if (!(to instanceof Writable) || !(from instanceof Readable)) {
296
487
  return { ok: false, error: "Chrome opened no DevTools pipe" };
297
488
  }
298
- return await shoot(new Pipe(to, from, () => said.trim()), job);
489
+ return await use(new Pipe(to, from, () => said.trim()));
299
490
  }
300
491
  finally {
301
492
  child.kill();
@@ -312,7 +503,9 @@ function read(path) {
312
503
  export async function run(job) {
313
504
  switch (job.kind) {
314
505
  case "capture":
315
- return await capture(job);
506
+ return await browse(job.browsers, job.width, job.height, (pipe) => shoot(pipe, job));
507
+ case "record":
508
+ return await browse(job.browsers, job.width, job.height, (pipe) => record(pipe, job));
316
509
  case "compare": {
317
510
  const [before, after] = [read(job.before), read(job.after)];
318
511
  if (!before.ok)
@@ -326,6 +519,22 @@ export async function run(job) {
326
519
  throw new Error("unreachable");
327
520
  }
328
521
  }
522
+ function parseStep(value) {
523
+ if (!isRecord(value))
524
+ return undefined;
525
+ const { action, url, selector, caption } = value;
526
+ if (typeof caption !== "string")
527
+ return undefined;
528
+ if (action === "open" && typeof url === "string" && selector === undefined) {
529
+ return { action, url, caption };
530
+ }
531
+ if ((action === "click" || action === "wait") &&
532
+ typeof selector === "string" &&
533
+ url === undefined) {
534
+ return { action, selector, caption };
535
+ }
536
+ return undefined;
537
+ }
329
538
  /**
330
539
  * The job the tools wrote as the one argument; `undefined` when it is not one. Its `try` stays: the session imports
331
540
  * only `node:` modules and types, so `attempt` cannot reach it in the sandbox.
@@ -340,7 +549,7 @@ export function parseJob(text) {
340
549
  }
341
550
  if (!isRecord(value))
342
551
  return undefined;
343
- const { kind, browsers, url, path, width, height, full_page, wait_for, before, after, threshold, } = value;
552
+ const { kind, browsers, url, path, width, height, full_page, wait_for, before, after, threshold, steps, sheet, video, max_bytes, } = value;
344
553
  if (kind === "capture" &&
345
554
  Array.isArray(browsers) &&
346
555
  browsers.every((b) => typeof b === "string") &&
@@ -358,6 +567,20 @@ export function parseJob(text) {
358
567
  typeof threshold === "number") {
359
568
  return { kind, before, after, threshold };
360
569
  }
570
+ const parsed = Array.isArray(steps) ? steps.map(parseStep) : [];
571
+ const flow = parsed.filter((step) => step !== undefined);
572
+ if (kind === "record" &&
573
+ Array.isArray(browsers) &&
574
+ browsers.every((b) => typeof b === "string") &&
575
+ flow.length > 0 &&
576
+ flow.length === parsed.length &&
577
+ typeof width === "number" &&
578
+ typeof height === "number" &&
579
+ typeof sheet === "string" &&
580
+ typeof video === "string" &&
581
+ typeof max_bytes === "number") {
582
+ return { kind, browsers, steps: flow, width, height, sheet, video, max_bytes };
583
+ }
361
584
  return undefined;
362
585
  }
363
586
  if (import.meta.main) {
@@ -30,10 +30,38 @@ export declare const tools: {
30
30
  path: string;
31
31
  bytes: number;
32
32
  caption: string;
33
+ url?: string | undefined;
33
34
  } | {
34
35
  missing: string;
35
36
  }, never>;
37
+ record: import("@kana-ai/edgar-sdk").Tool<{
38
+ service: string;
39
+ steps: ({
40
+ action: "open";
41
+ url: string;
42
+ caption: string;
43
+ } | {
44
+ action: "click";
45
+ selector: string;
46
+ caption: string;
47
+ } | {
48
+ action: "wait";
49
+ selector: string;
50
+ caption: string;
51
+ })[];
52
+ }, {
53
+ missing: string;
54
+ } | {
55
+ path: string;
56
+ bytes: number;
57
+ caption: string;
58
+ url?: string | undefined;
59
+ video?: {
60
+ path: string;
61
+ bytes: number;
62
+ } | undefined;
63
+ }, never>;
36
64
  };
37
- /** The screenshots extension: page captures and image comparisons inside the run's sandbox, and a picture of each service's preview page in its pull request. The network is the stage's `network` list, so a stage names the url's host. */
65
+ /** The screenshots extension: page captures and image comparisons inside the run's sandbox, and a picture of each service's preview page in its pull request, or a captioned flow through the screen a change touches. The network is the stage's `network` list, so a stage names the url's host. */
38
66
  declare const _default: import("@kana-ai/edgar-sdk").Extension;
39
67
  export default _default;
package/dist/src/index.js CHANGED
@@ -3,9 +3,10 @@ import { config } from "#schemas/config";
3
3
  import { access, capture } from "#tools/capture";
4
4
  import { compare } from "#tools/compare";
5
5
  import { preview } from "#tools/preview";
6
+ import { record } from "#tools/record";
6
7
  /** Every tool of the screenshots extension, by the name the manifest gives it. */
7
- export const tools = { capture, compare, preview };
8
- /** The screenshots extension: page captures and image comparisons inside the run's sandbox, and a picture of each service's preview page in its pull request. The network is the stage's `network` list, so a stage names the url's host. */
8
+ export const tools = { capture, compare, preview, record };
9
+ /** The screenshots extension: page captures and image comparisons inside the run's sandbox, and a picture of each service's preview page in its pull request, or a captioned flow through the screen a change touches. The network is the stage's `network` list, so a stage names the url's host. */
9
10
  export default defineExtension({
10
11
  name: "screenshots",
11
12
  config,
@@ -17,7 +18,7 @@ export default defineExtension({
17
18
  requires: [
18
19
  {
19
20
  name: "node",
20
- reason: "The tool capture runs the browser session on Node in the sandbox of the run.",
21
+ reason: "The tools run the browser session on Node in the sandbox of the run.",
21
22
  },
22
23
  ],
23
24
  },
@@ -1,6 +1,13 @@
1
1
  import { z } from "zod";
2
2
  /** The extension's `[extensions.screenshots]` table: `browser` is the path of Chrome or Chromium when it is not found on its own, and `previews` gives each service the page of its stack that a pull request shows. */
3
3
  export const config = z.strictObject({
4
- browser: z.string().min(1).optional(),
5
- previews: z.record(z.string().min(1), z.url({ protocol: /^https?$/ })).optional(),
4
+ browser: z
5
+ .string()
6
+ .min(1)
7
+ .optional()
8
+ .describe("The path of Chrome or Chromium, when it is not on PATH or in a usual install folder. When set, only this path is tried."),
9
+ previews: z
10
+ .record(z.string().min(1), z.url({ protocol: /^https?$/ }))
11
+ .optional()
12
+ .describe("The page of each service's stack that its pull request shows, by service name. A service with no page gets no picture."),
6
13
  });
@@ -27,10 +27,66 @@ export declare const previewTool: {
27
27
  path: z.ZodString;
28
28
  bytes: z.ZodInt;
29
29
  caption: z.ZodString;
30
+ url: z.ZodOptional<z.ZodString>;
30
31
  }, z.core.$strip>, z.ZodObject<{
31
32
  missing: z.ZodString;
32
33
  }, z.core.$strip>]>;
33
34
  };
35
+ /** One step of a flow: `open` a page, a path from the service's preview page or a url on its site, `click` the element a CSS selector finds, or `wait` until one shows. Its `caption` says what the frame after it shows. */
36
+ export declare const flowStep: z.ZodDiscriminatedUnion<[z.ZodObject<{
37
+ action: z.ZodLiteral<"open">;
38
+ url: z.ZodString;
39
+ caption: z.ZodString;
40
+ }, z.core.$strict>, z.ZodObject<{
41
+ action: z.ZodLiteral<"click">;
42
+ selector: z.ZodString;
43
+ caption: z.ZodString;
44
+ }, z.core.$strict>, z.ZodObject<{
45
+ action: z.ZodLiteral<"wait">;
46
+ selector: z.ZodString;
47
+ caption: z.ZodString;
48
+ }, z.core.$strict>], "action">;
49
+ /** The input and output of `record`: the captioned contact sheet of a flow through a service's pages, with its mp4 when the sandbox has `ffmpeg`, or why there is none. */
50
+ export declare const recordTool: {
51
+ input: z.ZodObject<{
52
+ service: z.ZodString;
53
+ steps: z.ZodArray<z.ZodDiscriminatedUnion<[z.ZodObject<{
54
+ action: z.ZodLiteral<"open">;
55
+ url: z.ZodString;
56
+ caption: z.ZodString;
57
+ }, z.core.$strict>, z.ZodObject<{
58
+ action: z.ZodLiteral<"click">;
59
+ selector: z.ZodString;
60
+ caption: z.ZodString;
61
+ }, z.core.$strict>, z.ZodObject<{
62
+ action: z.ZodLiteral<"wait">;
63
+ selector: z.ZodString;
64
+ caption: z.ZodString;
65
+ }, z.core.$strict>], "action">>;
66
+ }, z.core.$strict>;
67
+ output: z.ZodUnion<readonly [z.ZodObject<{
68
+ path: z.ZodString;
69
+ bytes: z.ZodInt;
70
+ caption: z.ZodString;
71
+ url: z.ZodOptional<z.ZodString>;
72
+ video: z.ZodOptional<z.ZodObject<{
73
+ path: z.ZodString;
74
+ bytes: z.ZodInt;
75
+ }, z.core.$strip>>;
76
+ }, z.core.$strip>, z.ZodObject<{
77
+ missing: z.ZodString;
78
+ }, z.core.$strip>]>;
79
+ };
80
+ /** What the session answers for a flow: the contact sheet, and the video or why there is none. */
81
+ export declare const recordAnswer: z.ZodObject<{
82
+ path: z.ZodString;
83
+ bytes: z.ZodInt;
84
+ video: z.ZodOptional<z.ZodObject<{
85
+ path: z.ZodString;
86
+ bytes: z.ZodInt;
87
+ }, z.core.$strip>>;
88
+ video_missing: z.ZodOptional<z.ZodString>;
89
+ }, z.core.$strip>;
34
90
  /** The input and output of `compare`. `threshold` is the share of pixels that may differ while the images stay `same`. */
35
91
  export declare const compareTool: {
36
92
  input: z.ZodObject<{
@@ -27,10 +27,38 @@ export const captureTool = {
27
27
  export const previewTool = {
28
28
  input: z.strictObject({ service: z.string().min(1) }),
29
29
  output: z.union([
30
- z.object({ path: z.string(), bytes: z.int(), caption: z.string() }),
30
+ z.object({
31
+ path: z.string(),
32
+ bytes: z.int(),
33
+ caption: z.string(),
34
+ url: z.string().optional(),
35
+ }),
31
36
  z.object({ missing: z.string() }),
32
37
  ]),
33
38
  };
39
+ const caption = z.string().min(1).max(200);
40
+ const selector = z.string().min(1).max(500);
41
+ /** One step of a flow: `open` a page, a path from the service's preview page or a url on its site, `click` the element a CSS selector finds, or `wait` until one shows. Its `caption` says what the frame after it shows. */
42
+ export const flowStep = z.discriminatedUnion("action", [
43
+ z.strictObject({ action: z.literal("open"), url: z.string().min(1).max(2000), caption }),
44
+ z.strictObject({ action: z.literal("click"), selector, caption }),
45
+ z.strictObject({ action: z.literal("wait"), selector, caption }),
46
+ ]);
47
+ /** A file the session wrote: its path and its size. */
48
+ const written = z.object({ path: z.string(), bytes: z.int() });
49
+ /** The input and output of `record`: the captioned contact sheet of a flow through a service's pages, with its mp4 when the sandbox has `ffmpeg`, or why there is none. */
50
+ export const recordTool = {
51
+ input: z.strictObject({ service: z.string().min(1), steps: z.array(flowStep).min(1).max(10) }),
52
+ output: z.union([
53
+ written.extend({ caption: z.string(), url: z.string().optional(), video: written.optional() }),
54
+ z.object({ missing: z.string() }),
55
+ ]),
56
+ };
57
+ /** What the session answers for a flow: the contact sheet, and the video or why there is none. */
58
+ export const recordAnswer = written.extend({
59
+ video: written.optional(),
60
+ video_missing: z.string().optional(),
61
+ });
34
62
  /** The input and output of `compare`. `threshold` is the share of pixels that may differ while the images stay `same`. */
35
63
  export const compareTool = {
36
64
  input: z.strictObject({
@@ -1,9 +1,12 @@
1
1
  import { type z } from "zod";
2
2
  import { type Capability, type Ctx, type Failure, type Result } from "@kana-ai/edgar-sdk";
3
3
  import type { Job } from "#browser/session";
4
+ import { config } from "#schemas/config";
4
5
  import { captureTool } from "#schemas/tools";
5
6
  /** The tools' sandbox access: `exec` runs the session, which reads and writes the images. */
6
7
  export declare const access: readonly Capability[];
8
+ /** The extension's `[extensions.screenshots]` table, or the failure that says how it does not match its schema. */
9
+ export declare function settingsOf<S extends string>(ctx: Ctx<S>): Result<z.output<typeof config>, Failure>;
7
10
  /** The `node` arguments that run the session's source in the sandbox with `job`; a `.ts` source runs with its types stripped. */
8
11
  export declare function sessionArgs(source: string, file: string, job: Job): readonly string[];
9
12
  /** Runs `job` in the sandbox's browser session; its answer, or the failure. */
@@ -11,6 +11,15 @@ export const access = ["sandbox:read", "sandbox:write"];
11
11
  function json(line) {
12
12
  return unwrapOr(attempt(() => JSON.parse(line), () => undefined), undefined);
13
13
  }
14
+ /** The extension's `[extensions.screenshots]` table, or the failure that says how it does not match its schema. */
15
+ export function settingsOf(ctx) {
16
+ const settings = config.safeParse(ctx.config ?? {});
17
+ if (settings.success)
18
+ return ok(settings.data);
19
+ return err(permanentFailure("invalid", "the [extensions.screenshots] table does not match its schema", {
20
+ cause: settings.error.issues.map((issue) => issue.message).join("; "),
21
+ }));
22
+ }
14
23
  /** The `node` arguments that run the session's source in the sandbox with `job`; a `.ts` source runs with its types stripped. */
15
24
  export function sessionArgs(source, file, job) {
16
25
  const type = extname(file) === ".ts" ? "module-typescript" : "module";
@@ -55,11 +64,10 @@ export const capture = defineTool({
55
64
  const input = captureTool.input.safeParse(raw);
56
65
  if (!input.success)
57
66
  return permanentFailure("invalid_input", input.error.message);
58
- const settings = config.safeParse(ctx.config ?? {});
59
- if (!settings.success) {
60
- return permanentFailure("invalid", "the [extensions.screenshots] table does not match its schema", { cause: settings.error.issues.map((issue) => issue.message).join("; ") });
61
- }
62
- const job = captureJob(input.data, candidates(settings.data.browser, process.env["PATH"] ?? ""));
67
+ const settings = settingsOf(ctx);
68
+ if (!settings.ok)
69
+ return settings.error;
70
+ const job = captureJob(input.data, candidates(settings.value.browser, process.env["PATH"] ?? ""));
63
71
  const ran = await drive(ctx, job);
64
72
  if (!ran.ok)
65
73
  return ran.error;
@@ -1,12 +1,18 @@
1
+ import { type Ctx } from "@kana-ai/edgar-sdk";
1
2
  /** Where the picture of a preview goes: the run's temporary folder, which no commit of the change holds. */
2
3
  export declare const PREVIEW = "/tmp/edgar-preview.png";
3
- /** Captures the page that `[extensions.screenshots] previews` names for a service, for its pull request. A page that does not load is `missing` with why, since a picture is no reason to stop a run; a service with no page is `skipped`. */
4
+ /** The address of the picture at `path` in the project's storage, when EDGAR keeps pictures there; a refusal for another reason goes to the log, and the picture stays in the sandbox. */
5
+ export declare function stored<S extends string>(ctx: Ctx<S>, path: string): Promise<{
6
+ readonly url: string;
7
+ } | Record<string, never>>;
8
+ /** Captures the page that `[extensions.screenshots] previews` names for a service, for its pull request, and stores it in the project's storage when EDGAR keeps pictures there, with the `url` that shows it. A page that does not load is `missing` with why, since a picture is no reason to stop a run; a service with no page is `skipped`. */
4
9
  export declare const preview: import("@kana-ai/edgar-sdk").Tool<{
5
10
  service: string;
6
11
  }, {
7
12
  path: string;
8
13
  bytes: number;
9
14
  caption: string;
15
+ url?: string | undefined;
10
16
  } | {
11
17
  missing: string;
12
18
  }, never>;
@@ -1,26 +1,34 @@
1
- import { defineTool, done, permanentFailure, skipped } from "@kana-ai/edgar-sdk";
1
+ import { defineTool, done, skipped } from "@kana-ai/edgar-sdk";
2
2
  import { candidates } from "#browser/find";
3
- import { config } from "#schemas/config";
4
3
  import { captureTool, previewTool } from "#schemas/tools";
5
- import { access, captureJob, drive } from "#tools/capture";
4
+ import { access, captureJob, drive, settingsOf } from "#tools/capture";
6
5
  /** Where the picture of a preview goes: the run's temporary folder, which no commit of the change holds. */
7
6
  export const PREVIEW = "/tmp/edgar-preview.png";
8
- /** Captures the page that `[extensions.screenshots] previews` names for a service, for its pull request. A page that does not load is `missing` with why, since a picture is no reason to stop a run; a service with no page is `skipped`. */
7
+ /** The address of the picture at `path` in the project's storage, when EDGAR keeps pictures there; a refusal for another reason goes to the log, and the picture stays in the sandbox. */
8
+ export async function stored(ctx, path) {
9
+ const published = await ctx.pictures.publish(path);
10
+ if (published.ok)
11
+ return { url: published.value.url };
12
+ if (published.error.code !== "capability_refused") {
13
+ ctx.log(`the picture stays in the sandbox: ${published.error.message}`);
14
+ }
15
+ return {};
16
+ }
17
+ /** Captures the page that `[extensions.screenshots] previews` names for a service, for its pull request, and stores it in the project's storage when EDGAR keeps pictures there, with the `url` that shows it. A page that does not load is `missing` with why, since a picture is no reason to stop a run; a service with no page is `skipped`. */
9
18
  export const preview = defineTool({
10
19
  description: "Capture the preview page that [extensions.screenshots] previews names for a service, or say why there is no picture",
11
20
  input: previewTool.input,
12
21
  output: previewTool.output,
13
22
  capabilities: access,
14
23
  execute: async ({ service }, ctx) => {
15
- const settings = config.safeParse(ctx.config ?? {});
16
- if (!settings.success) {
17
- return permanentFailure("invalid", "the [extensions.screenshots] table does not match its schema", { cause: settings.error.issues.map((issue) => issue.message).join("; ") });
18
- }
19
- const url = settings.data.previews?.[service];
24
+ const settings = settingsOf(ctx);
25
+ if (!settings.ok)
26
+ return settings.error;
27
+ const url = settings.value.previews?.[service];
20
28
  if (url === undefined)
21
29
  return skipped();
22
30
  const input = { url, path: PREVIEW, viewport: { width: 1280, height: 800 }, full_page: false };
23
- const browsers = candidates(settings.data.browser, process.env["PATH"] ?? "");
31
+ const browsers = candidates(settings.value.browser, process.env["PATH"] ?? "");
24
32
  const ran = await drive(ctx, captureJob(input, browsers));
25
33
  if (!ran.ok && ran.error.kind === "cancelled")
26
34
  return ran.error;
@@ -34,6 +42,7 @@ export const preview = defineTool({
34
42
  return done({ missing: `${url} was captured, but the session answered no image` });
35
43
  }
36
44
  const caption = `The page ${new URL(url).pathname} of ${service} with this change`;
37
- return done({ path: shot.data.path, bytes: shot.data.bytes, caption });
45
+ const picture = { path: shot.data.path, bytes: shot.data.bytes, caption };
46
+ return done({ ...picture, ...(await stored(ctx, shot.data.path)) });
38
47
  },
39
48
  });
@@ -0,0 +1,39 @@
1
+ import { type z } from "zod";
2
+ import { type Result } from "@kana-ai/edgar-sdk";
3
+ import type { Step } from "#browser/session";
4
+ import { type flowStep } from "#schemas/tools";
5
+ /** Where the contact sheet and the video of a flow go: the run's temporary folder, which no commit holds. */
6
+ export declare const FLOW_SHEET = "/tmp/edgar-flow.png";
7
+ export declare const FLOW_VIDEO = "/tmp/edgar-flow.mp4";
8
+ /** The largest contact sheet or video: what a pull request body shows from the GitHub extension's screenshots branch. */
9
+ export declare const MAX_FLOW_BYTES = 1048576;
10
+ /** The steps with each `open` url made whole from the preview page `base`; a url off its site is refused, since the steps come from the change's role. */
11
+ export declare function resolved(steps: readonly z.output<typeof flowStep>[], base: string, service: string): Result<readonly Step[], string>;
12
+ /** Records a flow through the pages of a service from the page `[extensions.screenshots] previews` names for it: each step's frame carries its caption, the frames make one contact sheet, and an mp4 too when the sandbox has `ffmpeg`. A flow that fails is `missing` with why, since a picture is no reason to stop a run; a service with no page is `skipped`. */
13
+ export declare const record: import("@kana-ai/edgar-sdk").Tool<{
14
+ service: string;
15
+ steps: ({
16
+ action: "open";
17
+ url: string;
18
+ caption: string;
19
+ } | {
20
+ action: "click";
21
+ selector: string;
22
+ caption: string;
23
+ } | {
24
+ action: "wait";
25
+ selector: string;
26
+ caption: string;
27
+ })[];
28
+ }, {
29
+ missing: string;
30
+ } | {
31
+ path: string;
32
+ bytes: number;
33
+ caption: string;
34
+ url?: string | undefined;
35
+ video?: {
36
+ path: string;
37
+ bytes: number;
38
+ } | undefined;
39
+ }, never>;
@@ -0,0 +1,80 @@
1
+ import {} from "zod";
2
+ import { defineTool, done, err, ok, skipped } from "@kana-ai/edgar-sdk";
3
+ import { candidates } from "#browser/find";
4
+ import { recordAnswer, recordTool } from "#schemas/tools";
5
+ import { access, drive, settingsOf } from "#tools/capture";
6
+ import { stored } from "#tools/preview";
7
+ /** Where the contact sheet and the video of a flow go: the run's temporary folder, which no commit holds. */
8
+ export const FLOW_SHEET = "/tmp/edgar-flow.png";
9
+ export const FLOW_VIDEO = "/tmp/edgar-flow.mp4";
10
+ /** The largest contact sheet or video: what a pull request body shows from the GitHub extension's screenshots branch. */
11
+ export const MAX_FLOW_BYTES = 1_048_576;
12
+ /** The steps with each `open` url made whole from the preview page `base`; a url off its site is refused, since the steps come from the change's role. */
13
+ export function resolved(steps, base, service) {
14
+ const site = new URL(base).origin;
15
+ const whole = [];
16
+ for (const [index, step] of steps.entries()) {
17
+ if (step.action !== "open") {
18
+ whole.push(step);
19
+ continue;
20
+ }
21
+ const url = URL.parse(step.url, base);
22
+ if (url === null || url.origin !== site) {
23
+ const at = `step ${String(index + 1)} opens ${step.url}`;
24
+ return err(`${at}, which is not on ${site}, the site of the preview page of ${service}`);
25
+ }
26
+ whole.push({ ...step, url: url.href });
27
+ }
28
+ return ok(whole);
29
+ }
30
+ /** Records a flow through the pages of a service from the page `[extensions.screenshots] previews` names for it: each step's frame carries its caption, the frames make one contact sheet, and an mp4 too when the sandbox has `ffmpeg`. A flow that fails is `missing` with why, since a picture is no reason to stop a run; a service with no page is `skipped`. */
31
+ export const record = defineTool({
32
+ description: "Record a captioned flow through a service's pages from its preview page, as a contact sheet and, with ffmpeg, an mp4, or say why there is none",
33
+ input: recordTool.input,
34
+ output: recordTool.output,
35
+ capabilities: access,
36
+ execute: async ({ service, steps }, ctx) => {
37
+ const settings = settingsOf(ctx);
38
+ if (!settings.ok)
39
+ return settings.error;
40
+ const base = settings.value.previews?.[service];
41
+ if (base === undefined)
42
+ return skipped();
43
+ const flow = resolved(steps, base, service);
44
+ if (!flow.ok)
45
+ return done({ missing: flow.error });
46
+ const ran = await drive(ctx, {
47
+ kind: "record",
48
+ browsers: candidates(settings.value.browser, process.env["PATH"] ?? ""),
49
+ steps: flow.value,
50
+ width: 1280,
51
+ height: 800,
52
+ sheet: FLOW_SHEET,
53
+ video: FLOW_VIDEO,
54
+ max_bytes: MAX_FLOW_BYTES,
55
+ });
56
+ if (!ran.ok && ran.error.kind === "cancelled")
57
+ return ran.error;
58
+ if (!ran.ok) {
59
+ const why = ran.error.cause === undefined ? "" : `: ${ran.error.cause}`;
60
+ const text = `the flow through ${service} was not recorded: ${ran.error.message}${why}`;
61
+ return done({ missing: text.replaceAll(/\s+/g, " ").trim().slice(0, 300) });
62
+ }
63
+ const answer = recordAnswer.safeParse(ran.value);
64
+ if (!answer.success) {
65
+ return done({ missing: `the flow through ${service} was recorded, but with no sheet` });
66
+ }
67
+ const { path, bytes, video, video_missing } = answer.data;
68
+ if (video_missing !== undefined)
69
+ ctx.log(`the flow has no video: ${video_missing}`);
70
+ const count = `${String(steps.length)} ${steps.length === 1 ? "step" : "steps"}`;
71
+ const caption = `The flow through ${service} with this change, in ${count}`;
72
+ return done({
73
+ path,
74
+ bytes,
75
+ caption,
76
+ ...(await stored(ctx, path)),
77
+ ...(video === undefined ? {} : { video }),
78
+ });
79
+ },
80
+ });
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "contract": 1,
3
3
  "name": "screenshots",
4
- "version": "0.1.2",
4
+ "version": "0.1.3",
5
5
  "config_schema": "schemas/config.json",
6
6
  "secrets": [],
7
7
  "capabilities": ["sandbox:read", "sandbox:write"],
@@ -38,6 +38,17 @@
38
38
  "side_effect": false,
39
39
  "approval": "never",
40
40
  "disabled": false
41
+ },
42
+ {
43
+ "name": "record",
44
+ "description": "Record a captioned flow through a service's pages from its preview page, as a contact sheet and, with ffmpeg, an mp4, or say why there is none",
45
+ "input_schema": "schemas/record.in.json",
46
+ "output_schema": "schemas/record.out.json",
47
+ "secrets": [],
48
+ "capabilities": ["sandbox:read", "sandbox:write"],
49
+ "side_effect": false,
50
+ "approval": "never",
51
+ "disabled": false
41
52
  }
42
53
  ],
43
54
  "webhooks": [],
@@ -51,7 +62,7 @@
51
62
  "requires": [
52
63
  {
53
64
  "name": "node",
54
- "reason": "The tool capture runs the browser session on Node in the sandbox of the run."
65
+ "reason": "The tools run the browser session on Node in the sandbox of the run."
55
66
  }
56
67
  ],
57
68
  "provides": []
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@kana-ai/edgar-ext-screenshots",
3
- "version": "0.1.2",
3
+ "version": "0.1.3",
4
4
  "description": "EDGAR extension for screenshots: capture a page with the machine's Chrome or Chromium, and compare two images.",
5
5
  "license": "Apache-2.0",
6
6
  "repository": {
@@ -22,11 +22,11 @@
22
22
  ".": "./dist/src/index.js"
23
23
  },
24
24
  "dependencies": {
25
- "@kana-ai/edgar-sdk": "0.1.2",
25
+ "@kana-ai/edgar-sdk": "0.1.3",
26
26
  "zod": "4.6.7"
27
27
  },
28
28
  "devDependencies": {
29
- "@kana-ai/edgar-contracts": "0.1.2",
29
+ "@kana-ai/edgar-contracts": "0.1.3",
30
30
  "@types/node": "24.19.2",
31
31
  "ajv": "8.20.0",
32
32
  "oxfmt": "0.76.0",
@@ -3,10 +3,12 @@
3
3
  "type": "object",
4
4
  "properties": {
5
5
  "browser": {
6
+ "description": "The path of Chrome or Chromium, when it is not on PATH or in a usual install folder. When set, only this path is tried.",
6
7
  "type": "string",
7
8
  "minLength": 1
8
9
  },
9
10
  "previews": {
11
+ "description": "The page of each service's stack that its pull request shows, by service name. A service with no page gets no picture.",
10
12
  "type": "object",
11
13
  "propertyNames": {
12
14
  "type": "string",
@@ -14,6 +14,9 @@
14
14
  },
15
15
  "caption": {
16
16
  "type": "string"
17
+ },
18
+ "url": {
19
+ "type": "string"
17
20
  }
18
21
  },
19
22
  "required": ["path", "bytes", "caption"]
@@ -0,0 +1,84 @@
1
+ {
2
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3
+ "type": "object",
4
+ "properties": {
5
+ "service": {
6
+ "type": "string",
7
+ "minLength": 1
8
+ },
9
+ "steps": {
10
+ "minItems": 1,
11
+ "maxItems": 10,
12
+ "type": "array",
13
+ "items": {
14
+ "oneOf": [
15
+ {
16
+ "type": "object",
17
+ "properties": {
18
+ "action": {
19
+ "type": "string",
20
+ "const": "open"
21
+ },
22
+ "url": {
23
+ "type": "string",
24
+ "minLength": 1,
25
+ "maxLength": 2000
26
+ },
27
+ "caption": {
28
+ "type": "string",
29
+ "minLength": 1,
30
+ "maxLength": 200
31
+ }
32
+ },
33
+ "required": ["action", "url", "caption"],
34
+ "additionalProperties": false
35
+ },
36
+ {
37
+ "type": "object",
38
+ "properties": {
39
+ "action": {
40
+ "type": "string",
41
+ "const": "click"
42
+ },
43
+ "selector": {
44
+ "type": "string",
45
+ "minLength": 1,
46
+ "maxLength": 500
47
+ },
48
+ "caption": {
49
+ "type": "string",
50
+ "minLength": 1,
51
+ "maxLength": 200
52
+ }
53
+ },
54
+ "required": ["action", "selector", "caption"],
55
+ "additionalProperties": false
56
+ },
57
+ {
58
+ "type": "object",
59
+ "properties": {
60
+ "action": {
61
+ "type": "string",
62
+ "const": "wait"
63
+ },
64
+ "selector": {
65
+ "type": "string",
66
+ "minLength": 1,
67
+ "maxLength": 500
68
+ },
69
+ "caption": {
70
+ "type": "string",
71
+ "minLength": 1,
72
+ "maxLength": 200
73
+ }
74
+ },
75
+ "required": ["action", "selector", "caption"],
76
+ "additionalProperties": false
77
+ }
78
+ ]
79
+ }
80
+ }
81
+ },
82
+ "required": ["service", "steps"],
83
+ "additionalProperties": false
84
+ }
@@ -0,0 +1,48 @@
1
+ {
2
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3
+ "anyOf": [
4
+ {
5
+ "type": "object",
6
+ "properties": {
7
+ "path": {
8
+ "type": "string"
9
+ },
10
+ "bytes": {
11
+ "type": "integer",
12
+ "minimum": -9007199254740991,
13
+ "maximum": 9007199254740991
14
+ },
15
+ "caption": {
16
+ "type": "string"
17
+ },
18
+ "url": {
19
+ "type": "string"
20
+ },
21
+ "video": {
22
+ "type": "object",
23
+ "properties": {
24
+ "path": {
25
+ "type": "string"
26
+ },
27
+ "bytes": {
28
+ "type": "integer",
29
+ "minimum": -9007199254740991,
30
+ "maximum": 9007199254740991
31
+ }
32
+ },
33
+ "required": ["path", "bytes"]
34
+ }
35
+ },
36
+ "required": ["path", "bytes", "caption"]
37
+ },
38
+ {
39
+ "type": "object",
40
+ "properties": {
41
+ "missing": {
42
+ "type": "string"
43
+ }
44
+ },
45
+ "required": ["missing"]
46
+ }
47
+ ]
48
+ }
@@ -1,22 +1,50 @@
1
- # Onto the built-in `remediate`: once the last checks pass, a picture of the service's preview page,
2
- # taken with the service's stack up, goes into the body of the pull request. A service with no
3
- # page in `[extensions.screenshots] previews` gets no picture and no line.
1
+ # Onto the built-in `remediate`: once the last checks pass, the service's stack comes up and a
2
+ # picture goes into the body of the pull request. A change that shows on a screen and gives the
3
+ # steps of its `demo` gets a captioned flow through them; any other change gets the service's
4
+ # preview page. A service with no page in `[extensions.screenshots] previews` gets no picture.
4
5
  version = 1
5
6
 
6
7
  [stages.preview]
7
8
  needs = ["final_checks"]
8
- when = { path = "stages.final_checks.output.passed", equals = true }
9
9
  tool = "screenshots__preview"
10
10
  stack = true
11
11
  input = { service = "trigger.card.service.name" }
12
12
 
13
+ [stages.preview.when]
14
+ all = [
15
+ { path = "stages.final_checks.output.passed", equals = true },
16
+ { any = [
17
+ { path = "stages.implement.output.touches_ui", not_equals = true },
18
+ { path = "stages.implement.output.demo", exists = false },
19
+ ] },
20
+ ]
21
+
22
+ [stages.flow]
23
+ needs = ["final_checks"]
24
+ tool = "screenshots__record"
25
+ stack = true
26
+ input = { service = "trigger.card.service.name", steps = "stages.implement.output.demo" }
27
+
28
+ [stages.flow.when]
29
+ all = [
30
+ { path = "stages.final_checks.output.passed", equals = true },
31
+ { path = "stages.implement.output.touches_ui", equals = true },
32
+ { path = "stages.implement.output.demo", exists = true },
33
+ ]
34
+
13
35
  [stages.attach_preview]
14
- needs = ["open_pr", "preview"]
36
+ needs = ["open_pr", "preview", "flow"]
15
37
  tool = "github__attach_screenshot"
16
- input = { url = "stages.open_pr.output.url", shot = "stages.preview.output" }
38
+
39
+ [stages.attach_preview.input]
40
+ url = "stages.open_pr.output.url"
41
+ shot = ["stages.flow.output", "stages.preview.output"]
17
42
 
18
43
  [stages.attach_preview.when]
19
44
  all = [
20
45
  { path = "stages.open_pr.output.url", exists = true },
21
- { path = "stages.preview.output", exists = true },
46
+ { any = [
47
+ { path = "stages.flow.output", exists = true },
48
+ { path = "stages.preview.output", exists = true },
49
+ ] },
22
50
  ]