@elyracode/browser-tools 0.9.26
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/README.md +52 -0
- package/extensions/index.ts +445 -0
- package/package.json +40 -0
- package/skills/browser-tools/SKILL.md +65 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.9.26] - 2026-08-07
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
- Initial release: browser automation via Playwright. One persistent headless tab per session with `browser_navigate`, `browser_snapshot` (accessibility-tree first — token-cheap page state), `browser_interact` (click/fill/press/select/check/hover targeted by role+name, label, text, or css), `browser_screenshot` (element/viewport/full-page with responsive viewport resize), `browser_run_tests` (Playwright suite runner), and `browser_close`. Console errors ride along with every result. Falls back to installed Google Chrome when the bundled Chromium is missing, and every failure names its fix. Ships a `browser-tools` skill and a `/browser <url>` command.
|
package/README.md
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# @elyracode/browser-tools
|
|
2
|
+
|
|
3
|
+
Browser automation for Elyra, built on [Playwright](https://playwright.dev). The agent gets hands in a real browser: navigate, click, fill, verify — with token-cheap accessibility snapshots as the primary way of seeing, and pixel screenshots when looks matter.
|
|
4
|
+
|
|
5
|
+
This completes Elyra's verification ladder: LSP auto-diagnostics check the *code*, design-tools check the *looks*, browser-tools check the *behavior*.
|
|
6
|
+
|
|
7
|
+
## Install
|
|
8
|
+
|
|
9
|
+
```
|
|
10
|
+
elyra install npm:@elyracode/browser-tools
|
|
11
|
+
npx playwright install chromium # browser binaries (once)
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
If the bundled Chromium is missing, installed Google Chrome is used as a fallback. Every failure names its fix — nothing degrades silently.
|
|
15
|
+
|
|
16
|
+
## Tools
|
|
17
|
+
|
|
18
|
+
| Tool | Description |
|
|
19
|
+
|------|-------------|
|
|
20
|
+
| `browser_navigate` | Open a URL; returns title, accessibility snapshot, and console errors |
|
|
21
|
+
| `browser_snapshot` | Re-read the current page (optionally scoped to a css selector) |
|
|
22
|
+
| `browser_interact` | click / fill / press / select / check / uncheck / hover, targeted by role+name, label, text, or css; returns the resulting page state |
|
|
23
|
+
| `browser_screenshot` | Element, viewport, or full-page screenshot; optional viewport resize for responsive checks |
|
|
24
|
+
| `browser_run_tests` | Run `npx playwright test` (a spec path or `--grep` filter) and summarize |
|
|
25
|
+
| `browser_close` | Close the session's tab |
|
|
26
|
+
|
|
27
|
+
One persistent headless tab per session: login state and SPA state survive across tool calls, so flows chain naturally. The tab closes with the session.
|
|
28
|
+
|
|
29
|
+
## Commands
|
|
30
|
+
|
|
31
|
+
| Command | Description |
|
|
32
|
+
|---------|-------------|
|
|
33
|
+
| `/browser <url>` | Open a URL and describe the page + console state |
|
|
34
|
+
|
|
35
|
+
## Design notes
|
|
36
|
+
|
|
37
|
+
- **Snapshot-first**: accessibility snapshots (Playwright's aria snapshot) are the primary page representation — a fraction of a screenshot's token cost, and they name every interactive element precisely for targeting.
|
|
38
|
+
- **Console errors ride along** with every result, drained since the last report.
|
|
39
|
+
- **Screenshots respect provider limits**: oversized captures are downscaled by Elyra core before reaching the model.
|
|
40
|
+
- Interactions auto-wait (Playwright semantics) with a 10s timeout and actionable failure messages.
|
|
41
|
+
|
|
42
|
+
## Example
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
> Verify note capture works end to end on https://freddy.test
|
|
46
|
+
|
|
47
|
+
browser_navigate https://freddy.test
|
|
48
|
+
browser_interact fill label="Note" value="Buy more RAM"
|
|
49
|
+
browser_interact press label="Note" value="Enter"
|
|
50
|
+
browser_snapshot -> list now contains "Buy more RAM"
|
|
51
|
+
Console: clean
|
|
52
|
+
```
|
|
@@ -0,0 +1,445 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Browser automation for Elyra, built on Playwright.
|
|
3
|
+
*
|
|
4
|
+
* Design principles:
|
|
5
|
+
* - Snapshot-first: interactions read the page as an accessibility tree
|
|
6
|
+
* (token-cheap, precise) and reach for pixel screenshots only when looks
|
|
7
|
+
* matter. Screenshots pass through core's provider image-limit safety net.
|
|
8
|
+
* - One live browser tab per session: navigate, interact, and screenshot
|
|
9
|
+
* share state, so flows chain naturally across tool calls.
|
|
10
|
+
* - Fail loudly: Playwright is a real dependency, but its browser binaries
|
|
11
|
+
* are a separate install step that can be missing. Every failure names the
|
|
12
|
+
* fix (npx playwright install chromium) instead of degrading silently.
|
|
13
|
+
*/
|
|
14
|
+
|
|
15
|
+
import { randomUUID } from "node:crypto";
|
|
16
|
+
import { mkdirSync, writeFileSync } from "node:fs";
|
|
17
|
+
import { createRequire } from "node:module";
|
|
18
|
+
import { tmpdir } from "node:os";
|
|
19
|
+
import { join } from "node:path";
|
|
20
|
+
import type { ExtensionAPI } from "@elyracode/coding-agent";
|
|
21
|
+
import { Type } from "typebox";
|
|
22
|
+
|
|
23
|
+
// ── Structural types for the Playwright surface we use ─────────────────────
|
|
24
|
+
// (Kept minimal and structural so this file type-checks without Playwright's
|
|
25
|
+
// own type packages being resolvable at extension load time.)
|
|
26
|
+
|
|
27
|
+
interface PWLocator {
|
|
28
|
+
click(options?: { timeout?: number }): Promise<void>;
|
|
29
|
+
fill(value: string, options?: { timeout?: number }): Promise<void>;
|
|
30
|
+
press(key: string, options?: { timeout?: number }): Promise<void>;
|
|
31
|
+
selectOption(value: string, options?: { timeout?: number }): Promise<unknown>;
|
|
32
|
+
check(options?: { timeout?: number }): Promise<void>;
|
|
33
|
+
uncheck(options?: { timeout?: number }): Promise<void>;
|
|
34
|
+
hover(options?: { timeout?: number }): Promise<void>;
|
|
35
|
+
ariaSnapshot(): Promise<string>;
|
|
36
|
+
screenshot(options?: { timeout?: number }): Promise<Buffer>;
|
|
37
|
+
first(): PWLocator;
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
interface PWConsoleMessage {
|
|
41
|
+
type(): string;
|
|
42
|
+
text(): string;
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
interface PWPage {
|
|
46
|
+
goto(url: string, options?: { waitUntil?: string; timeout?: number }): Promise<unknown>;
|
|
47
|
+
title(): Promise<string>;
|
|
48
|
+
url(): string;
|
|
49
|
+
locator(selector: string): PWLocator;
|
|
50
|
+
getByRole(role: string, options?: { name?: string; exact?: boolean }): PWLocator;
|
|
51
|
+
getByText(text: string, options?: { exact?: boolean }): PWLocator;
|
|
52
|
+
getByLabel(text: string, options?: { exact?: boolean }): PWLocator;
|
|
53
|
+
screenshot(options?: { fullPage?: boolean; timeout?: number }): Promise<Buffer>;
|
|
54
|
+
setViewportSize(size: { width: number; height: number }): Promise<void>;
|
|
55
|
+
waitForLoadState(state?: string, options?: { timeout?: number }): Promise<void>;
|
|
56
|
+
on(event: "console", handler: (message: PWConsoleMessage) => void): void;
|
|
57
|
+
on(event: "pageerror", handler: (error: Error) => void): void;
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
interface PWContext {
|
|
61
|
+
newPage(): Promise<PWPage>;
|
|
62
|
+
close(): Promise<void>;
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
interface PWBrowser {
|
|
66
|
+
newContext(options?: { viewport?: { width: number; height: number } }): Promise<PWContext>;
|
|
67
|
+
close(): Promise<void>;
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
interface PWBrowserType {
|
|
71
|
+
launch(options?: { headless?: boolean; channel?: string }): Promise<PWBrowser>;
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
interface PlaywrightModule {
|
|
75
|
+
chromium: PWBrowserType;
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
// ── Module state ────────────────────────────────────────────────────────────
|
|
79
|
+
|
|
80
|
+
const require = createRequire(import.meta.url);
|
|
81
|
+
|
|
82
|
+
let playwright: PlaywrightModule | null | undefined;
|
|
83
|
+
let browser: PWBrowser | null = null;
|
|
84
|
+
let context: PWContext | null = null;
|
|
85
|
+
let page: PWPage | null = null;
|
|
86
|
+
let consoleBuffer: string[] = [];
|
|
87
|
+
|
|
88
|
+
const SNAPSHOT_MAX_CHARS = 6000;
|
|
89
|
+
const DEFAULT_ACTION_TIMEOUT = 10_000;
|
|
90
|
+
|
|
91
|
+
function loadPlaywright(): PlaywrightModule | null {
|
|
92
|
+
if (playwright !== undefined) return playwright;
|
|
93
|
+
try {
|
|
94
|
+
playwright = require("playwright") as PlaywrightModule;
|
|
95
|
+
} catch {
|
|
96
|
+
playwright = null;
|
|
97
|
+
}
|
|
98
|
+
return playwright;
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
const MISSING_MODULE =
|
|
102
|
+
"Playwright is not installed alongside @elyracode/browser-tools. Reinstall the extension: elyra install npm:@elyracode/browser-tools";
|
|
103
|
+
|
|
104
|
+
const MISSING_BROWSER =
|
|
105
|
+
"Playwright is installed but no browser binary is available. Run: npx playwright install chromium " +
|
|
106
|
+
"(or install Google Chrome, which is used as a fallback).";
|
|
107
|
+
|
|
108
|
+
async function ensurePage(): Promise<{ page: PWPage } | { error: string }> {
|
|
109
|
+
const pw = loadPlaywright();
|
|
110
|
+
if (!pw) return { error: MISSING_MODULE };
|
|
111
|
+
|
|
112
|
+
if (page) return { page };
|
|
113
|
+
|
|
114
|
+
let launched: PWBrowser | null = null;
|
|
115
|
+
try {
|
|
116
|
+
launched = await pw.chromium.launch({ headless: true });
|
|
117
|
+
} catch {
|
|
118
|
+
// Bundled Chromium missing - try the user's Google Chrome.
|
|
119
|
+
try {
|
|
120
|
+
launched = await pw.chromium.launch({ headless: true, channel: "chrome" });
|
|
121
|
+
} catch {
|
|
122
|
+
return { error: MISSING_BROWSER };
|
|
123
|
+
}
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
browser = launched;
|
|
127
|
+
context = await launched.newContext({ viewport: { width: 1280, height: 900 } });
|
|
128
|
+
const newPage = await context.newPage();
|
|
129
|
+
newPage.on("console", (message) => {
|
|
130
|
+
const kind = message.type();
|
|
131
|
+
if (kind === "error" || kind === "warning") consoleBuffer.push(`${kind}: ${message.text()}`);
|
|
132
|
+
});
|
|
133
|
+
newPage.on("pageerror", (error) => consoleBuffer.push(`pageerror: ${error.message}`));
|
|
134
|
+
page = newPage;
|
|
135
|
+
return { page: newPage };
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
async function closeBrowser(): Promise<void> {
|
|
139
|
+
try {
|
|
140
|
+
await context?.close();
|
|
141
|
+
await browser?.close();
|
|
142
|
+
} catch {
|
|
143
|
+
// Best-effort cleanup.
|
|
144
|
+
}
|
|
145
|
+
page = null;
|
|
146
|
+
context = null;
|
|
147
|
+
browser = null;
|
|
148
|
+
consoleBuffer = [];
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
/** Drain console errors/warnings collected since the last report. */
|
|
152
|
+
function drainConsole(): string {
|
|
153
|
+
if (consoleBuffer.length === 0) return "Console: clean";
|
|
154
|
+
const lines = consoleBuffer.slice(-20);
|
|
155
|
+
const overflow = consoleBuffer.length > 20 ? ` (+${consoleBuffer.length - 20} earlier)` : "";
|
|
156
|
+
consoleBuffer = [];
|
|
157
|
+
return `Console${overflow}:\n${lines.map((l) => ` ${l}`).join("\n")}`;
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
async function snapshotOf(target: PWPage): Promise<string> {
|
|
161
|
+
try {
|
|
162
|
+
const snapshot = await target.locator("body").ariaSnapshot();
|
|
163
|
+
if (snapshot.length > SNAPSHOT_MAX_CHARS) {
|
|
164
|
+
return `${snapshot.slice(0, SNAPSHOT_MAX_CHARS)}\n... (snapshot truncated at ${SNAPSHOT_MAX_CHARS} chars; use browser_snapshot with a css scope to narrow)`;
|
|
165
|
+
}
|
|
166
|
+
return snapshot || "(empty page)";
|
|
167
|
+
} catch (error) {
|
|
168
|
+
return `(could not snapshot: ${error instanceof Error ? error.message : String(error)})`;
|
|
169
|
+
}
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
function ok(text: string) {
|
|
173
|
+
return { content: [{ type: "text" as const, text }], details: {} };
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
function err(text: string) {
|
|
177
|
+
return { content: [{ type: "text" as const, text }], details: {}, isError: true as const };
|
|
178
|
+
}
|
|
179
|
+
|
|
180
|
+
interface Target {
|
|
181
|
+
role?: string;
|
|
182
|
+
name?: string;
|
|
183
|
+
text?: string;
|
|
184
|
+
label?: string;
|
|
185
|
+
css?: string;
|
|
186
|
+
}
|
|
187
|
+
|
|
188
|
+
function resolveTarget(current: PWPage, target: Target): PWLocator | null {
|
|
189
|
+
if (target.css) return current.locator(target.css).first();
|
|
190
|
+
if (target.role) return current.getByRole(target.role, target.name ? { name: target.name } : undefined).first();
|
|
191
|
+
if (target.label) return current.getByLabel(target.label).first();
|
|
192
|
+
if (target.text) return current.getByText(target.text).first();
|
|
193
|
+
return null;
|
|
194
|
+
}
|
|
195
|
+
|
|
196
|
+
function describeTarget(target: Target): string {
|
|
197
|
+
if (target.css) return `css=${target.css}`;
|
|
198
|
+
if (target.role) return `role=${target.role}${target.name ? ` name="${target.name}"` : ""}`;
|
|
199
|
+
if (target.label) return `label="${target.label}"`;
|
|
200
|
+
if (target.text) return `text="${target.text}"`;
|
|
201
|
+
return "(no target)";
|
|
202
|
+
}
|
|
203
|
+
|
|
204
|
+
// ── Extension ───────────────────────────────────────────────────────────────
|
|
205
|
+
|
|
206
|
+
export default function (elyra: ExtensionAPI): void {
|
|
207
|
+
// ── Tool: browser_navigate ──
|
|
208
|
+
elyra.registerTool({
|
|
209
|
+
name: "browser_navigate",
|
|
210
|
+
label: "Browser: Navigate",
|
|
211
|
+
description:
|
|
212
|
+
"Open a URL in the session's headless browser and return the page as an accessibility snapshot " +
|
|
213
|
+
"(roles, names, states - far cheaper than a screenshot and precise enough to interact against), " +
|
|
214
|
+
"plus any console errors. The tab persists: later browser_interact / browser_screenshot calls " +
|
|
215
|
+
"continue from here. Works with localhost and .test dev URLs.",
|
|
216
|
+
parameters: Type.Object({
|
|
217
|
+
url: Type.String({ description: "The URL to open" }),
|
|
218
|
+
wait_until: Type.Optional(
|
|
219
|
+
Type.Union([Type.Literal("load"), Type.Literal("domcontentloaded"), Type.Literal("networkidle")], {
|
|
220
|
+
description: "When navigation counts as done (default: load)",
|
|
221
|
+
}),
|
|
222
|
+
),
|
|
223
|
+
}),
|
|
224
|
+
execute: async (_id, params) => {
|
|
225
|
+
const result = await ensurePage();
|
|
226
|
+
if ("error" in result) return err(result.error);
|
|
227
|
+
try {
|
|
228
|
+
await result.page.goto(params.url, { waitUntil: params.wait_until ?? "load", timeout: 30_000 });
|
|
229
|
+
} catch (error) {
|
|
230
|
+
return err(`Navigation failed: ${error instanceof Error ? error.message : String(error)}`);
|
|
231
|
+
}
|
|
232
|
+
const title = await result.page.title();
|
|
233
|
+
const snapshot = await snapshotOf(result.page);
|
|
234
|
+
return ok(`# ${title}\nURL: ${result.page.url()}\n\n${snapshot}\n\n${drainConsole()}`);
|
|
235
|
+
},
|
|
236
|
+
});
|
|
237
|
+
|
|
238
|
+
// ── Tool: browser_snapshot ──
|
|
239
|
+
elyra.registerTool({
|
|
240
|
+
name: "browser_snapshot",
|
|
241
|
+
label: "Browser: Snapshot",
|
|
242
|
+
description:
|
|
243
|
+
"Re-read the current page as an accessibility snapshot without navigating - use after the page " +
|
|
244
|
+
"changed on its own (SPA updates, timers) or to scope into part of a large page with a css selector.",
|
|
245
|
+
parameters: Type.Object({
|
|
246
|
+
css: Type.Optional(Type.String({ description: "Scope the snapshot to this selector (e.g. 'main', '#results')" })),
|
|
247
|
+
}),
|
|
248
|
+
execute: async (_id, params) => {
|
|
249
|
+
if (!page) return err("No page is open. Use browser_navigate first.");
|
|
250
|
+
try {
|
|
251
|
+
if (params.css) {
|
|
252
|
+
const scoped = await page.locator(params.css).first().ariaSnapshot();
|
|
253
|
+
const clipped =
|
|
254
|
+
scoped.length > SNAPSHOT_MAX_CHARS ? `${scoped.slice(0, SNAPSHOT_MAX_CHARS)}\n... (truncated)` : scoped;
|
|
255
|
+
return ok(`${clipped || "(empty)"}\n\n${drainConsole()}`);
|
|
256
|
+
}
|
|
257
|
+
const snapshot = await snapshotOf(page);
|
|
258
|
+
return ok(`URL: ${page.url()}\n\n${snapshot}\n\n${drainConsole()}`);
|
|
259
|
+
} catch (error) {
|
|
260
|
+
return err(`Snapshot failed: ${error instanceof Error ? error.message : String(error)}`);
|
|
261
|
+
}
|
|
262
|
+
},
|
|
263
|
+
});
|
|
264
|
+
|
|
265
|
+
// ── Tool: browser_interact ──
|
|
266
|
+
elyra.registerTool({
|
|
267
|
+
name: "browser_interact",
|
|
268
|
+
label: "Browser: Interact",
|
|
269
|
+
description:
|
|
270
|
+
"Act on the current page: click, fill, press, select, check, uncheck, or hover. Target elements " +
|
|
271
|
+
"the way the accessibility snapshot names them - role+name is the most robust (role='button', " +
|
|
272
|
+
"name='Save'), or label for form fields, text for links, css as the escape hatch. Returns the " +
|
|
273
|
+
"page state after the action so the effect is visible immediately.",
|
|
274
|
+
parameters: Type.Object({
|
|
275
|
+
action: Type.Union([
|
|
276
|
+
Type.Literal("click"),
|
|
277
|
+
Type.Literal("fill"),
|
|
278
|
+
Type.Literal("press"),
|
|
279
|
+
Type.Literal("select"),
|
|
280
|
+
Type.Literal("check"),
|
|
281
|
+
Type.Literal("uncheck"),
|
|
282
|
+
Type.Literal("hover"),
|
|
283
|
+
]),
|
|
284
|
+
role: Type.Optional(Type.String({ description: "ARIA role, e.g. 'button', 'textbox', 'link'" })),
|
|
285
|
+
name: Type.Optional(Type.String({ description: "Accessible name to match with role" })),
|
|
286
|
+
label: Type.Optional(Type.String({ description: "Form field label" })),
|
|
287
|
+
text: Type.Optional(Type.String({ description: "Visible text (links, buttons)" })),
|
|
288
|
+
css: Type.Optional(Type.String({ description: "CSS selector (escape hatch)" })),
|
|
289
|
+
value: Type.Optional(Type.String({ description: "For fill/select: the value. For press: the key (e.g. 'Enter')" })),
|
|
290
|
+
}),
|
|
291
|
+
execute: async (_id, params) => {
|
|
292
|
+
if (!page) return err("No page is open. Use browser_navigate first.");
|
|
293
|
+
const locator = resolveTarget(page, params);
|
|
294
|
+
if (!locator) return err("No target given. Provide role(+name), label, text, or css.");
|
|
295
|
+
const timeout = DEFAULT_ACTION_TIMEOUT;
|
|
296
|
+
try {
|
|
297
|
+
switch (params.action) {
|
|
298
|
+
case "click":
|
|
299
|
+
await locator.click({ timeout });
|
|
300
|
+
break;
|
|
301
|
+
case "fill":
|
|
302
|
+
if (params.value == null) return err("'fill' needs a value.");
|
|
303
|
+
await locator.fill(params.value, { timeout });
|
|
304
|
+
break;
|
|
305
|
+
case "press":
|
|
306
|
+
if (!params.value) return err("'press' needs a key in value (e.g. 'Enter').");
|
|
307
|
+
await locator.press(params.value, { timeout });
|
|
308
|
+
break;
|
|
309
|
+
case "select":
|
|
310
|
+
if (params.value == null) return err("'select' needs a value.");
|
|
311
|
+
await locator.selectOption(params.value, { timeout });
|
|
312
|
+
break;
|
|
313
|
+
case "check":
|
|
314
|
+
await locator.check({ timeout });
|
|
315
|
+
break;
|
|
316
|
+
case "uncheck":
|
|
317
|
+
await locator.uncheck({ timeout });
|
|
318
|
+
break;
|
|
319
|
+
case "hover":
|
|
320
|
+
await locator.hover({ timeout });
|
|
321
|
+
break;
|
|
322
|
+
}
|
|
323
|
+
} catch (error) {
|
|
324
|
+
const message = error instanceof Error ? error.message.split("\n")[0] : String(error);
|
|
325
|
+
return err(
|
|
326
|
+
`${params.action} on ${describeTarget(params)} failed: ${message} ` +
|
|
327
|
+
"(take a browser_snapshot to see what the page actually offers)",
|
|
328
|
+
);
|
|
329
|
+
}
|
|
330
|
+
try {
|
|
331
|
+
await page.waitForLoadState("load", { timeout: 5_000 });
|
|
332
|
+
} catch {
|
|
333
|
+
// SPAs may never fire another load; the snapshot below shows reality.
|
|
334
|
+
}
|
|
335
|
+
const snapshot = await snapshotOf(page);
|
|
336
|
+
return ok(
|
|
337
|
+
`Did ${params.action} on ${describeTarget(params)}.\nURL now: ${page.url()}\n\n${snapshot}\n\n${drainConsole()}`,
|
|
338
|
+
);
|
|
339
|
+
},
|
|
340
|
+
});
|
|
341
|
+
|
|
342
|
+
// ── Tool: browser_screenshot ──
|
|
343
|
+
elyra.registerTool({
|
|
344
|
+
name: "browser_screenshot",
|
|
345
|
+
label: "Browser: Screenshot",
|
|
346
|
+
description:
|
|
347
|
+
"Take a pixel screenshot of the current page (or one element) for visual judgement - layout, " +
|
|
348
|
+
"spacing, contrast. Use snapshots for behavior and structure; screenshots for looks. Optionally " +
|
|
349
|
+
"resize the viewport first to test responsive layouts (e.g. 390x844 for phone).",
|
|
350
|
+
parameters: Type.Object({
|
|
351
|
+
css: Type.Optional(Type.String({ description: "Screenshot only this element" })),
|
|
352
|
+
full_page: Type.Optional(Type.Boolean({ description: "Capture the full scroll height" })),
|
|
353
|
+
width: Type.Optional(Type.Number({ description: "Viewport width to set first" })),
|
|
354
|
+
height: Type.Optional(Type.Number({ description: "Viewport height to set first" })),
|
|
355
|
+
}),
|
|
356
|
+
execute: async (_id, params) => {
|
|
357
|
+
if (!page) return err("No page is open. Use browser_navigate first.");
|
|
358
|
+
try {
|
|
359
|
+
if (params.width && params.height) {
|
|
360
|
+
await page.setViewportSize({ width: params.width, height: params.height });
|
|
361
|
+
await page.waitForLoadState("load", { timeout: 3_000 }).catch(() => undefined);
|
|
362
|
+
}
|
|
363
|
+
const buffer = params.css
|
|
364
|
+
? await page.locator(params.css).first().screenshot({ timeout: DEFAULT_ACTION_TIMEOUT })
|
|
365
|
+
: await page.screenshot({ fullPage: params.full_page ?? false, timeout: DEFAULT_ACTION_TIMEOUT });
|
|
366
|
+
|
|
367
|
+
const dir = join(tmpdir(), "elyra-browser-tools");
|
|
368
|
+
mkdirSync(dir, { recursive: true });
|
|
369
|
+
const filePath = join(dir, `shot-${randomUUID()}.png`);
|
|
370
|
+
writeFileSync(filePath, buffer);
|
|
371
|
+
|
|
372
|
+
return {
|
|
373
|
+
content: [
|
|
374
|
+
{
|
|
375
|
+
type: "text" as const,
|
|
376
|
+
text: `Screenshot captured (${params.css ? `element ${params.css}` : params.full_page ? "full page" : "viewport"}): ${filePath}\n${drainConsole()}`,
|
|
377
|
+
},
|
|
378
|
+
{ type: "image" as const, data: buffer.toString("base64"), mimeType: "image/png" },
|
|
379
|
+
],
|
|
380
|
+
details: {},
|
|
381
|
+
};
|
|
382
|
+
} catch (error) {
|
|
383
|
+
return err(`Screenshot failed: ${error instanceof Error ? error.message : String(error)}`);
|
|
384
|
+
}
|
|
385
|
+
},
|
|
386
|
+
});
|
|
387
|
+
|
|
388
|
+
// ── Tool: browser_run_tests ──
|
|
389
|
+
elyra.registerTool({
|
|
390
|
+
name: "browser_run_tests",
|
|
391
|
+
label: "Browser: Run Playwright Tests",
|
|
392
|
+
description:
|
|
393
|
+
"Run the project's Playwright test suite (npx playwright test) and return a concise result. " +
|
|
394
|
+
"Pass a path to run one spec, or grep to filter by title. Failures include the error lines; " +
|
|
395
|
+
"traces are kept where Playwright's config puts them.",
|
|
396
|
+
parameters: Type.Object({
|
|
397
|
+
path: Type.Optional(Type.String({ description: "A spec file or directory to run" })),
|
|
398
|
+
grep: Type.Optional(Type.String({ description: "Only run tests matching this title" })),
|
|
399
|
+
}),
|
|
400
|
+
execute: async (_id, params, _signal, _onUpdate, ctx) => {
|
|
401
|
+
const args = ["playwright", "test", "--reporter=line"];
|
|
402
|
+
if (params.path) args.push(params.path);
|
|
403
|
+
if (params.grep) args.push("--grep", params.grep);
|
|
404
|
+
const result = await elyra.exec("npx", args, { cwd: ctx.cwd, timeout: 300_000 });
|
|
405
|
+
const output = `${result.stdout}\n${result.stderr}`.trim();
|
|
406
|
+
const lines = output.split("\n");
|
|
407
|
+
const tail = lines.length > 60 ? `...\n${lines.slice(-60).join("\n")}` : output;
|
|
408
|
+
if (result.code === 0) {
|
|
409
|
+
return ok(`Playwright tests passed.\n\n${tail}`);
|
|
410
|
+
}
|
|
411
|
+
return err(`Playwright tests failed (exit ${result.code}).\n\n${tail}`);
|
|
412
|
+
},
|
|
413
|
+
});
|
|
414
|
+
|
|
415
|
+
// ── Tool: browser_close ──
|
|
416
|
+
elyra.registerTool({
|
|
417
|
+
name: "browser_close",
|
|
418
|
+
label: "Browser: Close",
|
|
419
|
+
description: "Close the session's browser tab and free the browser. A later browser_navigate starts fresh.",
|
|
420
|
+
parameters: Type.Object({}),
|
|
421
|
+
execute: async () => {
|
|
422
|
+
if (!browser) return ok("No browser was open.");
|
|
423
|
+
await closeBrowser();
|
|
424
|
+
return ok("Browser closed.");
|
|
425
|
+
},
|
|
426
|
+
});
|
|
427
|
+
|
|
428
|
+
// ── Command: /browser ──
|
|
429
|
+
elyra.registerCommand("browser", {
|
|
430
|
+
description: "Open a URL in the agent's browser: /browser <url>",
|
|
431
|
+
handler: async (args, ctx) => {
|
|
432
|
+
const url = args.trim();
|
|
433
|
+
if (!url) {
|
|
434
|
+
ctx.ui.notify("Usage: /browser <url> \u2014 e.g. /browser https://freddy.test", "error");
|
|
435
|
+
return;
|
|
436
|
+
}
|
|
437
|
+
elyra.sendUserMessage(`Open ${url} with browser_navigate and describe what the page shows and whether the console is clean.`);
|
|
438
|
+
},
|
|
439
|
+
});
|
|
440
|
+
|
|
441
|
+
// ── Cleanup ──
|
|
442
|
+
elyra.on("session_shutdown", async () => {
|
|
443
|
+
await closeBrowser();
|
|
444
|
+
});
|
|
445
|
+
}
|
package/package.json
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@elyracode/browser-tools",
|
|
3
|
+
"version": "0.9.26",
|
|
4
|
+
"description": "Browser automation for Elyra via Playwright - the agent navigates, clicks, fills, and verifies behavior, with token-cheap accessibility snapshots and screenshots",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"keywords": [
|
|
7
|
+
"elyra-package",
|
|
8
|
+
"playwright",
|
|
9
|
+
"browser",
|
|
10
|
+
"e2e",
|
|
11
|
+
"testing"
|
|
12
|
+
],
|
|
13
|
+
"license": "MIT",
|
|
14
|
+
"author": "Knut W. Horne",
|
|
15
|
+
"repository": {
|
|
16
|
+
"type": "git",
|
|
17
|
+
"url": "git+https://github.com/kwhorne/elyra.git",
|
|
18
|
+
"directory": "packages/browser-tools"
|
|
19
|
+
},
|
|
20
|
+
"elyra": {
|
|
21
|
+
"skills": [
|
|
22
|
+
"./skills"
|
|
23
|
+
],
|
|
24
|
+
"extensions": [
|
|
25
|
+
"./extensions/index.ts"
|
|
26
|
+
]
|
|
27
|
+
},
|
|
28
|
+
"dependencies": {
|
|
29
|
+
"playwright": "^1.62.1"
|
|
30
|
+
},
|
|
31
|
+
"peerDependencies": {
|
|
32
|
+
"@elyracode/coding-agent": "*",
|
|
33
|
+
"typebox": "*"
|
|
34
|
+
},
|
|
35
|
+
"scripts": {
|
|
36
|
+
"clean": "echo 'nothing to clean'",
|
|
37
|
+
"build": "echo 'nothing to build'",
|
|
38
|
+
"check": "echo 'nothing to check'"
|
|
39
|
+
}
|
|
40
|
+
}
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: browser-tools
|
|
3
|
+
description: Browser automation via Playwright. Use when the user asks to test UI behavior in a real browser, verify a flow works (forms, navigation, login), reproduce a frontend bug, check responsive layouts, or run Playwright e2e tests.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Browser Tools
|
|
7
|
+
|
|
8
|
+
The agent's hands in a real browser: one persistent headless tab per session, driven through accessibility snapshots.
|
|
9
|
+
|
|
10
|
+
## When to Use
|
|
11
|
+
|
|
12
|
+
- Verifying behavior after building UI: does the form actually submit? Does the search return results?
|
|
13
|
+
- Reproducing a reported frontend bug step by step
|
|
14
|
+
- Checking responsive layouts at phone/tablet widths
|
|
15
|
+
- Running or debugging Playwright e2e specs
|
|
16
|
+
- Reading console errors that only appear on interaction
|
|
17
|
+
|
|
18
|
+
## Available Tools
|
|
19
|
+
|
|
20
|
+
| Tool | Use for |
|
|
21
|
+
|------|---------|
|
|
22
|
+
| `browser_navigate` | Open a URL; returns title + accessibility snapshot + console errors |
|
|
23
|
+
| `browser_snapshot` | Re-read the page (or a css-scoped part) without navigating |
|
|
24
|
+
| `browser_interact` | click / fill / press / select / check / uncheck / hover |
|
|
25
|
+
| `browser_screenshot` | Pixel screenshot (element, viewport, or full page; optional viewport resize) |
|
|
26
|
+
| `browser_run_tests` | Run `npx playwright test`, optionally one spec or a title grep |
|
|
27
|
+
| `browser_close` | Close the tab; next navigate starts fresh |
|
|
28
|
+
|
|
29
|
+
## Snapshot-First Discipline
|
|
30
|
+
|
|
31
|
+
**Snapshots for behavior, screenshots for looks.** An accessibility snapshot costs a fraction of a screenshot's tokens and names every interactive element precisely. Only take a screenshot when visual judgement matters (spacing, contrast, layout) — and consider `design_diff` from design-tools for before/after comparisons.
|
|
32
|
+
|
|
33
|
+
## Targeting Elements
|
|
34
|
+
|
|
35
|
+
Prefer, in order:
|
|
36
|
+
1. `role` + `name` — `role=button, name=Save` (most robust, matches what the snapshot shows)
|
|
37
|
+
2. `label` — form fields by their label
|
|
38
|
+
3. `text` — links and buttons by visible text
|
|
39
|
+
4. `css` — escape hatch when the page lacks semantics (and consider flagging the missing semantics as an accessibility finding)
|
|
40
|
+
|
|
41
|
+
If an interaction fails, take a `browser_snapshot` first — the page may not offer what you assumed.
|
|
42
|
+
|
|
43
|
+
## A Verification Flow
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
browser_navigate https://freddy.test
|
|
47
|
+
browser_interact fill label="Note" value="Try the swarm on search"
|
|
48
|
+
browser_interact press label="Note" value="Enter"
|
|
49
|
+
browser_snapshot # is the note in the list?
|
|
50
|
+
browser_screenshot # only if looks matter
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Responsive Checks
|
|
54
|
+
|
|
55
|
+
`browser_screenshot` with `width`/`height` resizes the viewport first: 390x844 (phone), 768x1024 (tablet), 1440x900 (desktop). Snapshot again after resizing if behavior (menus collapsing) is the question.
|
|
56
|
+
|
|
57
|
+
## Setup and Failure Modes
|
|
58
|
+
|
|
59
|
+
- The package depends on Playwright; browser binaries are a separate step: `npx playwright install chromium`. Google Chrome is used as a fallback when the bundled Chromium is missing.
|
|
60
|
+
- The session keeps one tab: state (login, SPA state) persists across tool calls until `browser_close`.
|
|
61
|
+
- The dev server must be running — the agent must NOT start long-running dev servers itself; ask the user.
|
|
62
|
+
|
|
63
|
+
## Console Errors Ride Along
|
|
64
|
+
|
|
65
|
+
Every navigate/interact/snapshot reports console errors and warnings collected since the last report. A visible bug is often a JS error — read them.
|