opencode-browser-kit 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +89 -2
- package/dist/plugin.js +423 -0
- package/package.json +28 -4
- package/src/plugin.ts +482 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Rishikesh Joshi
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,90 @@
|
|
|
1
|
-
#
|
|
1
|
+
# opencode-browser-kit
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
A headless browser + visual element picker for [OpenCode](https://opencode.ai). Gives any session seven `browser_*` tools — including **`browser_inspect`**, which opens a *visible* browser window where **you click an element and type what you want changed**; the agent receives a precise selector, the element's text/HTML/styles, a React source hint, and your note, then makes the change and verifies with screenshots.
|
|
4
|
+
|
|
5
|
+
## Install
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
opencode plugin opencode-browser-kit -g
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
That's it — deps are installed automatically on next startup. Or add it manually to `opencode.json`:
|
|
12
|
+
|
|
13
|
+
```json
|
|
14
|
+
{
|
|
15
|
+
"plugin": ["opencode-browser-kit"]
|
|
16
|
+
}
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Restart OpenCode so the tools load.
|
|
20
|
+
|
|
21
|
+
### Browser binary
|
|
22
|
+
|
|
23
|
+
On first use the plugin tries the bundled Playwright Chromium. If missing, install it once:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
bunx playwright install chromium
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
If that's not present, it automatically falls back to your system **Chrome**, then **Edge** — no download needed in that case.
|
|
30
|
+
|
|
31
|
+
## Tools
|
|
32
|
+
|
|
33
|
+
| Tool | What it does |
|
|
34
|
+
| --- | --- |
|
|
35
|
+
| `browser_navigate` | Go to a URL; returns title + screenshot path |
|
|
36
|
+
| `browser_snapshot` | Read page text (optionally scoped by selector) |
|
|
37
|
+
| `browser_click` | Click by CSS / `text=` / xpath target |
|
|
38
|
+
| `browser_type` | Fill a field, optional Enter to submit |
|
|
39
|
+
| `browser_screenshot` | Viewport or full-page screenshot |
|
|
40
|
+
| `browser_close` | Close the browser, release resources |
|
|
41
|
+
| `browser_inspect` | **Interactive picker**: visible window, you hover/click an element and type the desired change → returns selector + styles + React source + your note |
|
|
42
|
+
|
|
43
|
+
## The `/pick` command (recommended)
|
|
44
|
+
|
|
45
|
+
Add this to `~/.config/opencode/commands/pick.md` (create the folder if needed):
|
|
46
|
+
|
|
47
|
+
```markdown
|
|
48
|
+
---
|
|
49
|
+
description: Open the visual element picker (browser_inspect) so you can click an element and describe the change
|
|
50
|
+
agent: build
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
Call the `browser_inspect` tool NOW with exactly these args:
|
|
54
|
+
|
|
55
|
+
- url: $ARGUMENTS
|
|
56
|
+
- timeoutSeconds: 300
|
|
57
|
+
|
|
58
|
+
If $ARGUMENTS is empty, use `http://localhost:3000` instead.
|
|
59
|
+
|
|
60
|
+
A visible browser window will open for the user to hover, click an element, and type what they want changed. WAIT for the tool to return (it can take several minutes — do not call other tools meanwhile).
|
|
61
|
+
|
|
62
|
+
When the result arrives, use the returned selector, text, classes, styles, React source hint and the user's note to:
|
|
63
|
+
1. Locate the responsible component in this codebase (grep by text, class names or the React source hint).
|
|
64
|
+
2. Make the requested change.
|
|
65
|
+
3. Verify by calling `browser_navigate` on the same URL, then `browser_screenshot`, and view the screenshot with the read tool.
|
|
66
|
+
4. Summarize what you changed, with file paths.
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Then in any session:
|
|
70
|
+
|
|
71
|
+
```
|
|
72
|
+
/pick http://localhost:3000
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## The visual-edit workflow
|
|
76
|
+
|
|
77
|
+
1. Run your dev server (`npm run dev`)
|
|
78
|
+
2. `/pick http://localhost:3000`
|
|
79
|
+
3. A browser window opens — hover to outline elements, **click** the one you want changed
|
|
80
|
+
4. A panel appears — **type what you want changed** (e.g. "make this heading bigger and dark") → **Apply**
|
|
81
|
+
5. The agent locates the component, edits it, and shows you a verification screenshot — all inside the OpenCode session
|
|
82
|
+
|
|
83
|
+
## Notes
|
|
84
|
+
|
|
85
|
+
- Screenshots are saved to `/tmp/opencode-browser-screenshots/` and their paths are returned so the agent can view them inline.
|
|
86
|
+
- Works best in projects with `data-testid` attributes; the picker prefers them for stable selectors.
|
|
87
|
+
|
|
88
|
+
## License
|
|
89
|
+
|
|
90
|
+
MIT
|
package/dist/plugin.js
ADDED
|
@@ -0,0 +1,423 @@
|
|
|
1
|
+
// src/plugin.ts
|
|
2
|
+
import { tool } from "@opencode-ai/plugin";
|
|
3
|
+
import { chromium } from "playwright";
|
|
4
|
+
import fs from "fs";
|
|
5
|
+
import os from "os";
|
|
6
|
+
import path from "path";
|
|
7
|
+
var SCREENSHOT_DIR = path.join(os.tmpdir(), "opencode-browser-screenshots");
|
|
8
|
+
var VIEWPORT = { width: 1280, height: 800 };
|
|
9
|
+
var browser = null;
|
|
10
|
+
var context = null;
|
|
11
|
+
var page = null;
|
|
12
|
+
async function launchBrowser(headless = true) {
|
|
13
|
+
const attempts = [
|
|
14
|
+
{},
|
|
15
|
+
{ channel: "chrome" },
|
|
16
|
+
{ channel: "msedge" }
|
|
17
|
+
];
|
|
18
|
+
let lastError;
|
|
19
|
+
for (const opts of attempts) {
|
|
20
|
+
try {
|
|
21
|
+
return await chromium.launch({ headless, ...opts });
|
|
22
|
+
} catch (e) {
|
|
23
|
+
lastError = e;
|
|
24
|
+
}
|
|
25
|
+
}
|
|
26
|
+
throw new Error(`Could not launch any Chromium. Tried default install and system Chrome/Edge. ${String(lastError)}`);
|
|
27
|
+
}
|
|
28
|
+
async function ensurePage() {
|
|
29
|
+
if (!browser || !browser.isConnected()) {
|
|
30
|
+
browser = await launchBrowser();
|
|
31
|
+
context = await browser.newContext({
|
|
32
|
+
viewport: VIEWPORT,
|
|
33
|
+
deviceScaleFactor: 2
|
|
34
|
+
});
|
|
35
|
+
context.setDefaultTimeout(20000);
|
|
36
|
+
page = await context.newPage();
|
|
37
|
+
}
|
|
38
|
+
if (!context || !page || page.isClosed()) {
|
|
39
|
+
page = await context.newPage();
|
|
40
|
+
}
|
|
41
|
+
return page;
|
|
42
|
+
}
|
|
43
|
+
function resolveSelector(target) {
|
|
44
|
+
const t = target.trim();
|
|
45
|
+
if (/^(text=|css=|xpath=|role=|label=|id=|\/{1,2}|\.\.)/i.test(t) || /^[.#*[\]]/.test(t)) {
|
|
46
|
+
return t;
|
|
47
|
+
}
|
|
48
|
+
const looksCss = /^(html|head|body|div|span|a|button|input|textarea|select|form|img|svg|main|nav|header|footer|section|article|ul|ol|li|table|tr|td|th|h[1-6]|p|label|iframe|dialog)([.#\[:\s>]|$)/i.test(t) || /^[\w-]+$/i.test(t);
|
|
49
|
+
if (looksCss)
|
|
50
|
+
return t;
|
|
51
|
+
return `text=${t}`;
|
|
52
|
+
}
|
|
53
|
+
async function saveScreenshot(p, label, fullPage = false) {
|
|
54
|
+
fs.mkdirSync(SCREENSHOT_DIR, { recursive: true });
|
|
55
|
+
const file = path.join(SCREENSHOT_DIR, `${label}-${Date.now()}.png`);
|
|
56
|
+
await p.screenshot({ path: file, fullPage });
|
|
57
|
+
return file;
|
|
58
|
+
}
|
|
59
|
+
function formatSnap(file) {
|
|
60
|
+
return `
|
|
61
|
+
|
|
62
|
+
Screenshot saved: ${file}
|
|
63
|
+
(Use the read tool on that path to view it.)`;
|
|
64
|
+
}
|
|
65
|
+
var PICKER_JS = `(function () {
|
|
66
|
+
if (window.__ocPickerActive) return
|
|
67
|
+
window.__ocPickerActive = true
|
|
68
|
+
window.__ocPicked = undefined
|
|
69
|
+
var CSS_TXT =
|
|
70
|
+
'.oc-ph { outline: 2px solid #e11d48 !important; outline-offset: -2px !important; cursor: crosshair !important; }' +
|
|
71
|
+
'.oc-panel { position: fixed; z-index: 2147483647; right: 16px; top: 16px; width: 340px; box-sizing: border-box; font: 13px/1.45 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; background: #fff; color: #111; border-radius: 10px; box-shadow: 0 8px 30px rgba(0,0,0,.28); padding: 12px 14px; }' +
|
|
72
|
+
'.oc-panel .oc-tag { font-weight: 600; font-size: 13px; }' +
|
|
73
|
+
'.oc-panel .oc-sel { color: #888; font: 11px ui-monospace, SFMono-Regular, Menlo, monospace; word-break: break-all; margin-top: 2px; }' +
|
|
74
|
+
'.oc-panel .oc-txt { color: #444; font-size: 12px; margin-top: 4px; max-height: 48px; overflow: hidden; }' +
|
|
75
|
+
'.oc-panel textarea { width: 100%; height: 70px; box-sizing: border-box; margin: 10px 0 8px; padding: 8px; border: 1px solid #ddd; border-radius: 6px; font: inherit; resize: vertical; }' +
|
|
76
|
+
'.oc-btns { display: flex; gap: 8px; }' +
|
|
77
|
+
'.oc-btns button { flex: 1; padding: 7px 10px; border-radius: 6px; border: 1px solid #ddd; background: #fff; color: #111; font: inherit; cursor: pointer; }' +
|
|
78
|
+
'.oc-btns button[data-a="apply"] { background: #111; color: #fff; border-color: #111; }'
|
|
79
|
+
var style = document.createElement('style')
|
|
80
|
+
style.textContent = CSS_TXT
|
|
81
|
+
style.setAttribute('data-oc', '1')
|
|
82
|
+
document.head.appendChild(style)
|
|
83
|
+
|
|
84
|
+
var banner = document.createElement('div')
|
|
85
|
+
banner.className = 'oc-panel'
|
|
86
|
+
banner.textContent = 'Click any element to select it · Esc to cancel'
|
|
87
|
+
document.documentElement.appendChild(banner)
|
|
88
|
+
|
|
89
|
+
var picking = true
|
|
90
|
+
var last = null
|
|
91
|
+
|
|
92
|
+
function hover(e) {
|
|
93
|
+
if (!picking) return
|
|
94
|
+
if (last) last.classList.remove('oc-ph')
|
|
95
|
+
last = null
|
|
96
|
+
var el = document.elementFromPoint(e.clientX, e.clientY)
|
|
97
|
+
if (el && el !== document.documentElement && el !== document.body && !(el.closest && el.closest('.oc-panel'))) {
|
|
98
|
+
last = el
|
|
99
|
+
el.classList.add('oc-ph')
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
function selFor(el) {
|
|
104
|
+
if (el.dataset && el.dataset.testid) return '[data-testid="' + el.dataset.testid + '"]'
|
|
105
|
+
var parts = []
|
|
106
|
+
var node = el
|
|
107
|
+
while (node && node.nodeType === 1 && node !== document.body) {
|
|
108
|
+
var p = node.nodeName.toLowerCase()
|
|
109
|
+
if (node.id) { parts.unshift('#' + CSS.escape(node.id)); break }
|
|
110
|
+
var cl = Array.prototype.filter
|
|
111
|
+
.call(node.classList, function (c) { return !c.startsWith('oc-') })
|
|
112
|
+
.slice(0, 2)
|
|
113
|
+
.map(function (c) { return '.' + CSS.escape(c) })
|
|
114
|
+
.join('')
|
|
115
|
+
p += cl
|
|
116
|
+
if (node.parentElement) {
|
|
117
|
+
var sib = Array.prototype.filter.call(node.parentElement.children, function (c) { return c.nodeName === node.nodeName })
|
|
118
|
+
if (sib.length > 1) p += ':nth-of-type(' + (sib.indexOf(node) + 1) + ')'
|
|
119
|
+
}
|
|
120
|
+
parts.unshift(p)
|
|
121
|
+
node = node.parentElement
|
|
122
|
+
}
|
|
123
|
+
return parts.join(' > ') || 'body'
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
function reactInfo(el) {
|
|
127
|
+
try {
|
|
128
|
+
for (var k in el) {
|
|
129
|
+
if (k.indexOf('__reactFiber$') === 0) {
|
|
130
|
+
var f = el[k]
|
|
131
|
+
var name = null
|
|
132
|
+
while (f) {
|
|
133
|
+
if (f._debugSource) {
|
|
134
|
+
return { file: f._debugSource.fileName, line: f._debugSource.lineNumber, comp: name || '' }
|
|
135
|
+
}
|
|
136
|
+
var t = f.type || f.elementType
|
|
137
|
+
if (typeof t === 'function' && !name) name = t.displayName || t.name
|
|
138
|
+
f = f.return
|
|
139
|
+
}
|
|
140
|
+
return { comp: name || '' }
|
|
141
|
+
}
|
|
142
|
+
}
|
|
143
|
+
} catch (err) {}
|
|
144
|
+
return null
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
function collect(el) {
|
|
148
|
+
var cs = getComputedStyle(el)
|
|
149
|
+
var r = el.getBoundingClientRect()
|
|
150
|
+
return {
|
|
151
|
+
selector: selFor(el),
|
|
152
|
+
tag: el.tagName.toLowerCase(),
|
|
153
|
+
id: el.id || '',
|
|
154
|
+
classes: Array.prototype.filter.call(el.classList, function (c) { return !c.startsWith('oc-') }).join(' '),
|
|
155
|
+
text: (el.innerText || '')
|
|
156
|
+
.split('\\n')
|
|
157
|
+
.map(function (t) { return t.trim() })
|
|
158
|
+
.filter(Boolean)
|
|
159
|
+
.slice(0, 3)
|
|
160
|
+
.join(' | ')
|
|
161
|
+
.slice(0, 150),
|
|
162
|
+
html: el.outerHTML.slice(0, 400),
|
|
163
|
+
rect: { x: Math.round(r.x), y: Math.round(r.y), width: Math.round(r.width), height: Math.round(r.height) },
|
|
164
|
+
styles: {
|
|
165
|
+
fontSize: cs.fontSize,
|
|
166
|
+
fontWeight: cs.fontWeight,
|
|
167
|
+
color: cs.color,
|
|
168
|
+
backgroundColor: cs.backgroundColor,
|
|
169
|
+
borderRadius: cs.borderRadius,
|
|
170
|
+
fontFamily: cs.fontFamily,
|
|
171
|
+
},
|
|
172
|
+
react: reactInfo(el),
|
|
173
|
+
}
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
function mk(a, label) {
|
|
177
|
+
var b = document.createElement('button')
|
|
178
|
+
b.setAttribute('data-a', a)
|
|
179
|
+
b.textContent = label
|
|
180
|
+
return b
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
function openPanel(el, info) {
|
|
184
|
+
if (last) last.classList.remove('oc-ph')
|
|
185
|
+
last = el
|
|
186
|
+
el.classList.add('oc-ph')
|
|
187
|
+
banner.remove()
|
|
188
|
+
var panel = document.createElement('div')
|
|
189
|
+
panel.className = 'oc-panel'
|
|
190
|
+
var tag = document.createElement('div')
|
|
191
|
+
tag.className = 'oc-tag'
|
|
192
|
+
tag.textContent = '<' + info.tag + '>' + (info.id ? ' #' + info.id : info.classes ? ' ' + info.classes : '')
|
|
193
|
+
var path = document.createElement('div')
|
|
194
|
+
path.className = 'oc-sel'
|
|
195
|
+
path.textContent = info.selector
|
|
196
|
+
var txt = document.createElement('div')
|
|
197
|
+
txt.className = 'oc-txt'
|
|
198
|
+
txt.textContent = info.text || ''
|
|
199
|
+
var ta = document.createElement('textarea')
|
|
200
|
+
ta.placeholder = 'Describe what you want changed…'
|
|
201
|
+
ta.setAttribute('data-oc-ta', '1')
|
|
202
|
+
var btns = document.createElement('div')
|
|
203
|
+
btns.className = 'oc-btns'
|
|
204
|
+
btns.append(mk('apply', 'Apply'), mk('reselect', 'Reselect'), mk('cancel', 'Cancel'))
|
|
205
|
+
panel.append(tag, path, txt, ta, btns)
|
|
206
|
+
document.documentElement.appendChild(panel)
|
|
207
|
+
ta.focus()
|
|
208
|
+
panel.addEventListener('mousedown', function (e) { e.stopPropagation() })
|
|
209
|
+
panel.addEventListener('click', function (e) {
|
|
210
|
+
var a = e.target.getAttribute && e.target.getAttribute('data-a')
|
|
211
|
+
if (a === 'apply') finish(Object.assign({}, info, { note: ta.value.trim() }))
|
|
212
|
+
else if (a === 'cancel') finish(null)
|
|
213
|
+
else if (a === 'reselect') {
|
|
214
|
+
panel.remove()
|
|
215
|
+
picking = true
|
|
216
|
+
document.documentElement.appendChild(banner)
|
|
217
|
+
document.addEventListener('mousemove', hover, true)
|
|
218
|
+
}
|
|
219
|
+
})
|
|
220
|
+
}
|
|
221
|
+
|
|
222
|
+
function down(e) {
|
|
223
|
+
if (!picking || e.button !== 0) return
|
|
224
|
+
var el = document.elementFromPoint(e.clientX, e.clientY)
|
|
225
|
+
if (!el || (el.closest && el.closest('.oc-panel'))) return
|
|
226
|
+
e.preventDefault()
|
|
227
|
+
e.stopPropagation()
|
|
228
|
+
picking = false
|
|
229
|
+
document.removeEventListener('mousemove', hover, true)
|
|
230
|
+
openPanel(el, collect(el))
|
|
231
|
+
}
|
|
232
|
+
|
|
233
|
+
function esc(e) {
|
|
234
|
+
if (e.key === 'Escape') { e.preventDefault(); finish(null) }
|
|
235
|
+
}
|
|
236
|
+
|
|
237
|
+
function blockClick(e) {
|
|
238
|
+
if (e.target.closest && e.target.closest('.oc-panel')) return
|
|
239
|
+
e.preventDefault()
|
|
240
|
+
e.stopPropagation()
|
|
241
|
+
}
|
|
242
|
+
|
|
243
|
+
function finish(val) { cleanup(); window.__ocPicked = val }
|
|
244
|
+
|
|
245
|
+
function cleanup() {
|
|
246
|
+
picking = false
|
|
247
|
+
document.removeEventListener('mousemove', hover, true)
|
|
248
|
+
document.removeEventListener('mousedown', down, true)
|
|
249
|
+
document.removeEventListener('keydown', esc, true)
|
|
250
|
+
document.removeEventListener('click', blockClick, true)
|
|
251
|
+
Array.prototype.forEach.call(document.querySelectorAll('.oc-ph'), function (n) { n.classList.remove('oc-ph') })
|
|
252
|
+
Array.prototype.forEach.call(document.querySelectorAll('.oc-panel'), function (n) { n.remove() })
|
|
253
|
+
var st = document.querySelector('style[data-oc]')
|
|
254
|
+
if (st) st.remove()
|
|
255
|
+
window.__ocPickerActive = false
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
document.addEventListener('mousemove', hover, true)
|
|
259
|
+
document.addEventListener('mousedown', down, true)
|
|
260
|
+
document.addEventListener('keydown', esc, true)
|
|
261
|
+
document.addEventListener('click', blockClick, true)
|
|
262
|
+
})()`;
|
|
263
|
+
var BrowserPlugin = async (ctx) => {
|
|
264
|
+
return {
|
|
265
|
+
tool: {
|
|
266
|
+
browser_navigate: tool({
|
|
267
|
+
description: "Navigate the embedded headless browser to a URL. Returns page title plus a screenshot path you can view with the read tool. The browser session persists across tool calls.",
|
|
268
|
+
args: {
|
|
269
|
+
url: tool.schema.string().describe("Full URL, e.g. https://example.com or http://localhost:3000")
|
|
270
|
+
},
|
|
271
|
+
async execute(args) {
|
|
272
|
+
const p = await ensurePage();
|
|
273
|
+
try {
|
|
274
|
+
await p.goto(args.url, { waitUntil: "networkidle", timeout: 20000 });
|
|
275
|
+
} catch {
|
|
276
|
+
await p.goto(args.url, { waitUntil: "domcontentloaded", timeout: 20000 });
|
|
277
|
+
}
|
|
278
|
+
const title = await p.title();
|
|
279
|
+
const file = await saveScreenshot(p, "navigate");
|
|
280
|
+
return `Navigated to ${p.url()}
|
|
281
|
+
Title: ${title}${formatSnap(file)}`;
|
|
282
|
+
}
|
|
283
|
+
}),
|
|
284
|
+
browser_snapshot: tool({
|
|
285
|
+
description: "Read the currently open page's content as text (title, url, visible text). Optionally scope to a CSS/text selector.",
|
|
286
|
+
args: {
|
|
287
|
+
selector: tool.schema.string().optional().describe("Optional CSS or text= selector to scope the snapshot to a region, e.g. main or text=Pricing"),
|
|
288
|
+
maxLength: tool.schema.number().optional().describe("Max characters of text to return (default 8000)")
|
|
289
|
+
},
|
|
290
|
+
async execute(args) {
|
|
291
|
+
const p = await ensurePage();
|
|
292
|
+
const scope = args.selector ? resolveSelector(args.selector) : "body";
|
|
293
|
+
const text = await p.evaluate(({ sel }) => {
|
|
294
|
+
const el = document.querySelector(sel);
|
|
295
|
+
if (!el)
|
|
296
|
+
return null;
|
|
297
|
+
return el.innerText;
|
|
298
|
+
}, { sel: scope });
|
|
299
|
+
if (text === null) {
|
|
300
|
+
return `No element matched "${scope}" on ${p.url()}`;
|
|
301
|
+
}
|
|
302
|
+
const max = args.maxLength ?? 8000;
|
|
303
|
+
const trimmed = text.length > max ? text.slice(0, max) + `
|
|
304
|
+
…[truncated]` : text;
|
|
305
|
+
return `URL: ${p.url()}
|
|
306
|
+
Title: ${await p.title()}
|
|
307
|
+
|
|
308
|
+
---
|
|
309
|
+
${trimmed}`;
|
|
310
|
+
}
|
|
311
|
+
}),
|
|
312
|
+
browser_click: tool({
|
|
313
|
+
description: "Click an element in the browser. Accepts a CSS selector (#id, .class, button), xpath (//button[2]), or plain text (wrapped as text= automatically). A screenshot is saved after clicking.",
|
|
314
|
+
args: {
|
|
315
|
+
target: tool.schema.string().describe('e.g. "#submit", "a", //button[2], "Sign in"')
|
|
316
|
+
},
|
|
317
|
+
async execute(args) {
|
|
318
|
+
const p = await ensurePage();
|
|
319
|
+
const selector = resolveSelector(args.target);
|
|
320
|
+
await p.click(selector, { timeout: 15000 });
|
|
321
|
+
await p.waitForLoadState("domcontentloaded").catch(() => {});
|
|
322
|
+
const file = await saveScreenshot(p, "click");
|
|
323
|
+
return `Clicked "${selector}" on ${p.url()}${formatSnap(file)}`;
|
|
324
|
+
}
|
|
325
|
+
}),
|
|
326
|
+
browser_type: tool({
|
|
327
|
+
description: "Type text into a form field. Clears existing value, types the text, and optionally presses Enter to submit.",
|
|
328
|
+
args: {
|
|
329
|
+
target: tool.schema.string().describe('Field selector, e.g. "input[name=q]" or "Search"'),
|
|
330
|
+
text: tool.schema.string().describe("Text to type"),
|
|
331
|
+
submit: tool.schema.boolean().optional().describe("Press Enter after typing (default false)")
|
|
332
|
+
},
|
|
333
|
+
async execute(args) {
|
|
334
|
+
const p = await ensurePage();
|
|
335
|
+
const selector = resolveSelector(args.target);
|
|
336
|
+
await p.fill(selector, args.text, { timeout: 15000 });
|
|
337
|
+
if (args.submit) {
|
|
338
|
+
await p.press(selector, "Enter");
|
|
339
|
+
await p.waitForLoadState("domcontentloaded").catch(() => {});
|
|
340
|
+
}
|
|
341
|
+
const file = await saveScreenshot(p, "type");
|
|
342
|
+
return `Typed ${JSON.stringify(args.text)} into "${selector}" (submit=${!!args.submit})${formatSnap(file)}`;
|
|
343
|
+
}
|
|
344
|
+
}),
|
|
345
|
+
browser_screenshot: tool({
|
|
346
|
+
description: "Take a screenshot of the current browser page. Returns the file path; use the read tool to view it.",
|
|
347
|
+
args: {
|
|
348
|
+
fullPage: tool.schema.boolean().optional().describe("Capture the full scrollable page (default: viewport only)")
|
|
349
|
+
},
|
|
350
|
+
async execute(args) {
|
|
351
|
+
const p = await ensurePage();
|
|
352
|
+
const file = await saveScreenshot(p, "shot", !!args.fullPage);
|
|
353
|
+
return `Screenshot of ${p.url()} saved: ${file}`;
|
|
354
|
+
}
|
|
355
|
+
}),
|
|
356
|
+
browser_close: tool({
|
|
357
|
+
description: "Close the embedded browser and release its resources.",
|
|
358
|
+
args: {},
|
|
359
|
+
async execute() {
|
|
360
|
+
if (browser) {
|
|
361
|
+
await browser.close().catch(() => {});
|
|
362
|
+
browser = null;
|
|
363
|
+
context = null;
|
|
364
|
+
page = null;
|
|
365
|
+
}
|
|
366
|
+
return "Browser closed.";
|
|
367
|
+
}
|
|
368
|
+
}),
|
|
369
|
+
browser_inspect: tool({
|
|
370
|
+
description: "Interactive element picker for design changes. Opens a VISIBLE browser window on the URL, waits for the user to hover and CLICK an element, then the user TYPES what they want changed and clicks Apply. Returns the element's unique selector, text, HTML, styles, React component source (when available) and the user's change note. Use whenever the user wants to point at something on the page to modify it.",
|
|
371
|
+
args: {
|
|
372
|
+
url: tool.schema.string().describe("URL to open, e.g. http://localhost:3000"),
|
|
373
|
+
timeoutSeconds: tool.schema.number().optional().describe("How long to wait for the user's selection (default 300)")
|
|
374
|
+
},
|
|
375
|
+
async execute(args) {
|
|
376
|
+
let inspectBrowser = null;
|
|
377
|
+
try {
|
|
378
|
+
inspectBrowser = await launchBrowser(false);
|
|
379
|
+
const pageCtx = await inspectBrowser.newContext({
|
|
380
|
+
viewport: VIEWPORT,
|
|
381
|
+
deviceScaleFactor: 1
|
|
382
|
+
});
|
|
383
|
+
pageCtx.setDefaultTimeout(30000);
|
|
384
|
+
const p = await pageCtx.newPage();
|
|
385
|
+
await p.goto(args.url, { waitUntil: "domcontentloaded", timeout: 30000 });
|
|
386
|
+
await p.evaluate(PICKER_JS);
|
|
387
|
+
try {
|
|
388
|
+
await p.waitForFunction("window.__ocPicked !== undefined", null, {
|
|
389
|
+
timeout: (args.timeoutSeconds ?? 300) * 1000,
|
|
390
|
+
polling: 200
|
|
391
|
+
});
|
|
392
|
+
} catch {
|
|
393
|
+
return "No selection made within the timeout — visible browser closed.";
|
|
394
|
+
}
|
|
395
|
+
const outcome = await p.evaluate("window.__ocPicked");
|
|
396
|
+
if (outcome === null)
|
|
397
|
+
return "Selection cancelled by user.";
|
|
398
|
+
const lines = [
|
|
399
|
+
`User picked an element on ${p.url()}`,
|
|
400
|
+
`Selector: ${outcome.selector}`,
|
|
401
|
+
`Tag: ${outcome.tag}${outcome.id ? ` #${outcome.id}` : ""}${outcome.classes ? ` (${outcome.classes})` : ""}`,
|
|
402
|
+
`Text: ${outcome.text || "(none)"}`,
|
|
403
|
+
`Box: ${outcome.rect.width}x${outcome.rect.height} at (${outcome.rect.x}, ${outcome.rect.y})`,
|
|
404
|
+
`Styles: fontSize=${outcome.styles.fontSize} fontWeight=${outcome.styles.fontWeight} color=${outcome.styles.color} background=${outcome.styles.backgroundColor} radius=${outcome.styles.borderRadius} font=${(outcome.styles.fontFamily || "").split(",")[0]}`,
|
|
405
|
+
`HTML: ${outcome.html}`,
|
|
406
|
+
outcome.react?.file ? `React source: ${outcome.react.file}${outcome.react.line ? `:${outcome.react.line}` : ""}${outcome.react.comp ? ` (${outcome.react.comp})` : ""}` : null,
|
|
407
|
+
`User note: ${outcome.note || "(nothing typed — ask the user what to change)"}`
|
|
408
|
+
].filter(Boolean);
|
|
409
|
+
return lines.join(`
|
|
410
|
+
`);
|
|
411
|
+
} finally {
|
|
412
|
+
if (inspectBrowser)
|
|
413
|
+
await inspectBrowser.close().catch(() => {});
|
|
414
|
+
}
|
|
415
|
+
}
|
|
416
|
+
})
|
|
417
|
+
}
|
|
418
|
+
};
|
|
419
|
+
};
|
|
420
|
+
export {
|
|
421
|
+
PICKER_JS,
|
|
422
|
+
BrowserPlugin
|
|
423
|
+
};
|
package/package.json
CHANGED
|
@@ -1,6 +1,30 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "opencode-browser-kit",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Headless browser + visual element picker for OpenCode. Tools: browser_navigate, browser_snapshop, browser_click, browser_type, browser_screenshot, browser_close and browser_inspect (opens a visible browser so you click an element and type what to change).",
|
|
5
|
+
"main": "dist/plugin.js",
|
|
6
|
+
"type": "module",
|
|
7
|
+
"files": [
|
|
8
|
+
"dist",
|
|
9
|
+
"src",
|
|
10
|
+
"README.md"
|
|
11
|
+
],
|
|
12
|
+
"scripts": {
|
|
13
|
+
"build": "bun build src/plugin.ts --outdir dist --target node --external @opencode-ai/plugin --external playwright",
|
|
14
|
+
"prepublishOnly": "bun run build"
|
|
15
|
+
},
|
|
16
|
+
"keywords": [
|
|
17
|
+
"opencode",
|
|
18
|
+
"opencode-plugin",
|
|
19
|
+
"browser",
|
|
20
|
+
"playwright",
|
|
21
|
+
"automation",
|
|
22
|
+
"screenshots",
|
|
23
|
+
"element-picker"
|
|
24
|
+
],
|
|
25
|
+
"license": "MIT",
|
|
26
|
+
"dependencies": {
|
|
27
|
+
"@opencode-ai/plugin": "^1.19.0",
|
|
28
|
+
"playwright": "^1.64.0"
|
|
29
|
+
}
|
|
30
|
+
}
|
package/src/plugin.ts
ADDED
|
@@ -0,0 +1,482 @@
|
|
|
1
|
+
import { type Plugin, tool } from "@opencode-ai/plugin"
|
|
2
|
+
import type { Browser, BrowserContext, Page } from "playwright"
|
|
3
|
+
import { chromium } from "playwright"
|
|
4
|
+
import fs from "fs"
|
|
5
|
+
import os from "os"
|
|
6
|
+
import path from "path"
|
|
7
|
+
|
|
8
|
+
const SCREENSHOT_DIR = path.join(os.tmpdir(), "opencode-browser-screenshots")
|
|
9
|
+
const VIEWPORT = { width: 1280, height: 800 }
|
|
10
|
+
|
|
11
|
+
let browser: Browser | null = null
|
|
12
|
+
let context: BrowserContext | null = null
|
|
13
|
+
let page: Page | null = null
|
|
14
|
+
|
|
15
|
+
async function launchBrowser(headless = true): Promise<Browser> {
|
|
16
|
+
const attempts: { channel?: string }[] = [
|
|
17
|
+
{},
|
|
18
|
+
{ channel: "chrome" },
|
|
19
|
+
{ channel: "msedge" },
|
|
20
|
+
]
|
|
21
|
+
let lastError: unknown
|
|
22
|
+
for (const opts of attempts) {
|
|
23
|
+
try {
|
|
24
|
+
return await chromium.launch({ headless, ...opts })
|
|
25
|
+
} catch (e) {
|
|
26
|
+
lastError = e
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
throw new Error(
|
|
30
|
+
`Could not launch any Chromium. Tried default install and system Chrome/Edge. ${String(lastError)}`,
|
|
31
|
+
)
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
async function ensurePage(): Promise<Page> {
|
|
35
|
+
if (!browser || !browser.isConnected()) {
|
|
36
|
+
browser = await launchBrowser()
|
|
37
|
+
context = await browser.newContext({
|
|
38
|
+
viewport: VIEWPORT,
|
|
39
|
+
deviceScaleFactor: 2,
|
|
40
|
+
})
|
|
41
|
+
context.setDefaultTimeout(20000)
|
|
42
|
+
page = await context.newPage()
|
|
43
|
+
}
|
|
44
|
+
if (!context || !page || page.isClosed()) {
|
|
45
|
+
page = await context!.newPage()
|
|
46
|
+
}
|
|
47
|
+
return page!
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
function resolveSelector(target: string): string {
|
|
51
|
+
const t = target.trim()
|
|
52
|
+
if (
|
|
53
|
+
/^(text=|css=|xpath=|role=|label=|id=|\/{1,2}|\.\.)/i.test(t) ||
|
|
54
|
+
/^[.#*[\]]/.test(t)
|
|
55
|
+
) {
|
|
56
|
+
return t
|
|
57
|
+
}
|
|
58
|
+
const looksCss =
|
|
59
|
+
/^(html|head|body|div|span|a|button|input|textarea|select|form|img|svg|main|nav|header|footer|section|article|ul|ol|li|table|tr|td|th|h[1-6]|p|label|iframe|dialog)([.#\[:\s>]|$)/i.test(
|
|
60
|
+
t,
|
|
61
|
+
) || /^[\w-]+$/i.test(t)
|
|
62
|
+
if (looksCss) return t
|
|
63
|
+
return `text=${t}`
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
async function saveScreenshot(
|
|
67
|
+
p: Page,
|
|
68
|
+
label: string,
|
|
69
|
+
fullPage = false,
|
|
70
|
+
): Promise<string> {
|
|
71
|
+
fs.mkdirSync(SCREENSHOT_DIR, { recursive: true })
|
|
72
|
+
const file = path.join(SCREENSHOT_DIR, `${label}-${Date.now()}.png`)
|
|
73
|
+
await p.screenshot({ path: file, fullPage })
|
|
74
|
+
return file
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
function formatSnap(file: string): string {
|
|
78
|
+
return `\n\nScreenshot saved: ${file}\n(Use the read tool on that path to view it.)`
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
export const PICKER_JS = `(function () {
|
|
82
|
+
if (window.__ocPickerActive) return
|
|
83
|
+
window.__ocPickerActive = true
|
|
84
|
+
window.__ocPicked = undefined
|
|
85
|
+
var CSS_TXT =
|
|
86
|
+
'.oc-ph { outline: 2px solid #e11d48 !important; outline-offset: -2px !important; cursor: crosshair !important; }' +
|
|
87
|
+
'.oc-panel { position: fixed; z-index: 2147483647; right: 16px; top: 16px; width: 340px; box-sizing: border-box; font: 13px/1.45 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; background: #fff; color: #111; border-radius: 10px; box-shadow: 0 8px 30px rgba(0,0,0,.28); padding: 12px 14px; }' +
|
|
88
|
+
'.oc-panel .oc-tag { font-weight: 600; font-size: 13px; }' +
|
|
89
|
+
'.oc-panel .oc-sel { color: #888; font: 11px ui-monospace, SFMono-Regular, Menlo, monospace; word-break: break-all; margin-top: 2px; }' +
|
|
90
|
+
'.oc-panel .oc-txt { color: #444; font-size: 12px; margin-top: 4px; max-height: 48px; overflow: hidden; }' +
|
|
91
|
+
'.oc-panel textarea { width: 100%; height: 70px; box-sizing: border-box; margin: 10px 0 8px; padding: 8px; border: 1px solid #ddd; border-radius: 6px; font: inherit; resize: vertical; }' +
|
|
92
|
+
'.oc-btns { display: flex; gap: 8px; }' +
|
|
93
|
+
'.oc-btns button { flex: 1; padding: 7px 10px; border-radius: 6px; border: 1px solid #ddd; background: #fff; color: #111; font: inherit; cursor: pointer; }' +
|
|
94
|
+
'.oc-btns button[data-a="apply"] { background: #111; color: #fff; border-color: #111; }'
|
|
95
|
+
var style = document.createElement('style')
|
|
96
|
+
style.textContent = CSS_TXT
|
|
97
|
+
style.setAttribute('data-oc', '1')
|
|
98
|
+
document.head.appendChild(style)
|
|
99
|
+
|
|
100
|
+
var banner = document.createElement('div')
|
|
101
|
+
banner.className = 'oc-panel'
|
|
102
|
+
banner.textContent = 'Click any element to select it · Esc to cancel'
|
|
103
|
+
document.documentElement.appendChild(banner)
|
|
104
|
+
|
|
105
|
+
var picking = true
|
|
106
|
+
var last = null
|
|
107
|
+
|
|
108
|
+
function hover(e) {
|
|
109
|
+
if (!picking) return
|
|
110
|
+
if (last) last.classList.remove('oc-ph')
|
|
111
|
+
last = null
|
|
112
|
+
var el = document.elementFromPoint(e.clientX, e.clientY)
|
|
113
|
+
if (el && el !== document.documentElement && el !== document.body && !(el.closest && el.closest('.oc-panel'))) {
|
|
114
|
+
last = el
|
|
115
|
+
el.classList.add('oc-ph')
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
function selFor(el) {
|
|
120
|
+
if (el.dataset && el.dataset.testid) return '[data-testid="' + el.dataset.testid + '"]'
|
|
121
|
+
var parts = []
|
|
122
|
+
var node = el
|
|
123
|
+
while (node && node.nodeType === 1 && node !== document.body) {
|
|
124
|
+
var p = node.nodeName.toLowerCase()
|
|
125
|
+
if (node.id) { parts.unshift('#' + CSS.escape(node.id)); break }
|
|
126
|
+
var cl = Array.prototype.filter
|
|
127
|
+
.call(node.classList, function (c) { return !c.startsWith('oc-') })
|
|
128
|
+
.slice(0, 2)
|
|
129
|
+
.map(function (c) { return '.' + CSS.escape(c) })
|
|
130
|
+
.join('')
|
|
131
|
+
p += cl
|
|
132
|
+
if (node.parentElement) {
|
|
133
|
+
var sib = Array.prototype.filter.call(node.parentElement.children, function (c) { return c.nodeName === node.nodeName })
|
|
134
|
+
if (sib.length > 1) p += ':nth-of-type(' + (sib.indexOf(node) + 1) + ')'
|
|
135
|
+
}
|
|
136
|
+
parts.unshift(p)
|
|
137
|
+
node = node.parentElement
|
|
138
|
+
}
|
|
139
|
+
return parts.join(' > ') || 'body'
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
function reactInfo(el) {
|
|
143
|
+
try {
|
|
144
|
+
for (var k in el) {
|
|
145
|
+
if (k.indexOf('__reactFiber$') === 0) {
|
|
146
|
+
var f = el[k]
|
|
147
|
+
var name = null
|
|
148
|
+
while (f) {
|
|
149
|
+
if (f._debugSource) {
|
|
150
|
+
return { file: f._debugSource.fileName, line: f._debugSource.lineNumber, comp: name || '' }
|
|
151
|
+
}
|
|
152
|
+
var t = f.type || f.elementType
|
|
153
|
+
if (typeof t === 'function' && !name) name = t.displayName || t.name
|
|
154
|
+
f = f.return
|
|
155
|
+
}
|
|
156
|
+
return { comp: name || '' }
|
|
157
|
+
}
|
|
158
|
+
}
|
|
159
|
+
} catch (err) {}
|
|
160
|
+
return null
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
function collect(el) {
|
|
164
|
+
var cs = getComputedStyle(el)
|
|
165
|
+
var r = el.getBoundingClientRect()
|
|
166
|
+
return {
|
|
167
|
+
selector: selFor(el),
|
|
168
|
+
tag: el.tagName.toLowerCase(),
|
|
169
|
+
id: el.id || '',
|
|
170
|
+
classes: Array.prototype.filter.call(el.classList, function (c) { return !c.startsWith('oc-') }).join(' '),
|
|
171
|
+
text: (el.innerText || '')
|
|
172
|
+
.split('\\n')
|
|
173
|
+
.map(function (t) { return t.trim() })
|
|
174
|
+
.filter(Boolean)
|
|
175
|
+
.slice(0, 3)
|
|
176
|
+
.join(' | ')
|
|
177
|
+
.slice(0, 150),
|
|
178
|
+
html: el.outerHTML.slice(0, 400),
|
|
179
|
+
rect: { x: Math.round(r.x), y: Math.round(r.y), width: Math.round(r.width), height: Math.round(r.height) },
|
|
180
|
+
styles: {
|
|
181
|
+
fontSize: cs.fontSize,
|
|
182
|
+
fontWeight: cs.fontWeight,
|
|
183
|
+
color: cs.color,
|
|
184
|
+
backgroundColor: cs.backgroundColor,
|
|
185
|
+
borderRadius: cs.borderRadius,
|
|
186
|
+
fontFamily: cs.fontFamily,
|
|
187
|
+
},
|
|
188
|
+
react: reactInfo(el),
|
|
189
|
+
}
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
function mk(a, label) {
|
|
193
|
+
var b = document.createElement('button')
|
|
194
|
+
b.setAttribute('data-a', a)
|
|
195
|
+
b.textContent = label
|
|
196
|
+
return b
|
|
197
|
+
}
|
|
198
|
+
|
|
199
|
+
function openPanel(el, info) {
|
|
200
|
+
if (last) last.classList.remove('oc-ph')
|
|
201
|
+
last = el
|
|
202
|
+
el.classList.add('oc-ph')
|
|
203
|
+
banner.remove()
|
|
204
|
+
var panel = document.createElement('div')
|
|
205
|
+
panel.className = 'oc-panel'
|
|
206
|
+
var tag = document.createElement('div')
|
|
207
|
+
tag.className = 'oc-tag'
|
|
208
|
+
tag.textContent = '<' + info.tag + '>' + (info.id ? ' #' + info.id : info.classes ? ' ' + info.classes : '')
|
|
209
|
+
var path = document.createElement('div')
|
|
210
|
+
path.className = 'oc-sel'
|
|
211
|
+
path.textContent = info.selector
|
|
212
|
+
var txt = document.createElement('div')
|
|
213
|
+
txt.className = 'oc-txt'
|
|
214
|
+
txt.textContent = info.text || ''
|
|
215
|
+
var ta = document.createElement('textarea')
|
|
216
|
+
ta.placeholder = 'Describe what you want changed…'
|
|
217
|
+
ta.setAttribute('data-oc-ta', '1')
|
|
218
|
+
var btns = document.createElement('div')
|
|
219
|
+
btns.className = 'oc-btns'
|
|
220
|
+
btns.append(mk('apply', 'Apply'), mk('reselect', 'Reselect'), mk('cancel', 'Cancel'))
|
|
221
|
+
panel.append(tag, path, txt, ta, btns)
|
|
222
|
+
document.documentElement.appendChild(panel)
|
|
223
|
+
ta.focus()
|
|
224
|
+
panel.addEventListener('mousedown', function (e) { e.stopPropagation() })
|
|
225
|
+
panel.addEventListener('click', function (e) {
|
|
226
|
+
var a = e.target.getAttribute && e.target.getAttribute('data-a')
|
|
227
|
+
if (a === 'apply') finish(Object.assign({}, info, { note: ta.value.trim() }))
|
|
228
|
+
else if (a === 'cancel') finish(null)
|
|
229
|
+
else if (a === 'reselect') {
|
|
230
|
+
panel.remove()
|
|
231
|
+
picking = true
|
|
232
|
+
document.documentElement.appendChild(banner)
|
|
233
|
+
document.addEventListener('mousemove', hover, true)
|
|
234
|
+
}
|
|
235
|
+
})
|
|
236
|
+
}
|
|
237
|
+
|
|
238
|
+
function down(e) {
|
|
239
|
+
if (!picking || e.button !== 0) return
|
|
240
|
+
var el = document.elementFromPoint(e.clientX, e.clientY)
|
|
241
|
+
if (!el || (el.closest && el.closest('.oc-panel'))) return
|
|
242
|
+
e.preventDefault()
|
|
243
|
+
e.stopPropagation()
|
|
244
|
+
picking = false
|
|
245
|
+
document.removeEventListener('mousemove', hover, true)
|
|
246
|
+
openPanel(el, collect(el))
|
|
247
|
+
}
|
|
248
|
+
|
|
249
|
+
function esc(e) {
|
|
250
|
+
if (e.key === 'Escape') { e.preventDefault(); finish(null) }
|
|
251
|
+
}
|
|
252
|
+
|
|
253
|
+
function blockClick(e) {
|
|
254
|
+
if (e.target.closest && e.target.closest('.oc-panel')) return
|
|
255
|
+
e.preventDefault()
|
|
256
|
+
e.stopPropagation()
|
|
257
|
+
}
|
|
258
|
+
|
|
259
|
+
function finish(val) { cleanup(); window.__ocPicked = val }
|
|
260
|
+
|
|
261
|
+
function cleanup() {
|
|
262
|
+
picking = false
|
|
263
|
+
document.removeEventListener('mousemove', hover, true)
|
|
264
|
+
document.removeEventListener('mousedown', down, true)
|
|
265
|
+
document.removeEventListener('keydown', esc, true)
|
|
266
|
+
document.removeEventListener('click', blockClick, true)
|
|
267
|
+
Array.prototype.forEach.call(document.querySelectorAll('.oc-ph'), function (n) { n.classList.remove('oc-ph') })
|
|
268
|
+
Array.prototype.forEach.call(document.querySelectorAll('.oc-panel'), function (n) { n.remove() })
|
|
269
|
+
var st = document.querySelector('style[data-oc]')
|
|
270
|
+
if (st) st.remove()
|
|
271
|
+
window.__ocPickerActive = false
|
|
272
|
+
}
|
|
273
|
+
|
|
274
|
+
document.addEventListener('mousemove', hover, true)
|
|
275
|
+
document.addEventListener('mousedown', down, true)
|
|
276
|
+
document.addEventListener('keydown', esc, true)
|
|
277
|
+
document.addEventListener('click', blockClick, true)
|
|
278
|
+
})()`
|
|
279
|
+
|
|
280
|
+
export const BrowserPlugin: Plugin = async (ctx) => {
|
|
281
|
+
return {
|
|
282
|
+
tool: {
|
|
283
|
+
browser_navigate: tool({
|
|
284
|
+
description:
|
|
285
|
+
"Navigate the embedded headless browser to a URL. Returns page title plus a screenshot path you can view with the read tool. The browser session persists across tool calls.",
|
|
286
|
+
args: {
|
|
287
|
+
url: tool.schema
|
|
288
|
+
.string()
|
|
289
|
+
.describe("Full URL, e.g. https://example.com or http://localhost:3000"),
|
|
290
|
+
},
|
|
291
|
+
async execute(args) {
|
|
292
|
+
const p = await ensurePage()
|
|
293
|
+
try {
|
|
294
|
+
await p.goto(args.url, { waitUntil: "networkidle", timeout: 20000 })
|
|
295
|
+
} catch {
|
|
296
|
+
await p.goto(args.url, { waitUntil: "domcontentloaded", timeout: 20000 })
|
|
297
|
+
}
|
|
298
|
+
const title = await p.title()
|
|
299
|
+
const file = await saveScreenshot(p, "navigate")
|
|
300
|
+
return `Navigated to ${p.url()}\nTitle: ${title}${formatSnap(file)}`
|
|
301
|
+
},
|
|
302
|
+
}),
|
|
303
|
+
|
|
304
|
+
browser_snapshot: tool({
|
|
305
|
+
description:
|
|
306
|
+
"Read the currently open page's content as text (title, url, visible text). Optionally scope to a CSS/text selector.",
|
|
307
|
+
args: {
|
|
308
|
+
selector: tool.schema
|
|
309
|
+
.string()
|
|
310
|
+
.optional()
|
|
311
|
+
.describe(
|
|
312
|
+
"Optional CSS or text= selector to scope the snapshot to a region, e.g. main or text=Pricing",
|
|
313
|
+
),
|
|
314
|
+
maxLength: tool.schema
|
|
315
|
+
.number()
|
|
316
|
+
.optional()
|
|
317
|
+
.describe("Max characters of text to return (default 8000)"),
|
|
318
|
+
},
|
|
319
|
+
async execute(args) {
|
|
320
|
+
const p = await ensurePage()
|
|
321
|
+
const scope = args.selector ? resolveSelector(args.selector) : "body"
|
|
322
|
+
const text = await p.evaluate(
|
|
323
|
+
({ sel }) => {
|
|
324
|
+
const el = document.querySelector(sel)
|
|
325
|
+
if (!el) return null
|
|
326
|
+
return (el as HTMLElement).innerText
|
|
327
|
+
},
|
|
328
|
+
{ sel: scope },
|
|
329
|
+
)
|
|
330
|
+
if (text === null) {
|
|
331
|
+
return `No element matched "${scope}" on ${p.url()}`
|
|
332
|
+
}
|
|
333
|
+
const max = args.maxLength ?? 8000
|
|
334
|
+
const trimmed =
|
|
335
|
+
text.length > max ? text.slice(0, max) + "\n…[truncated]" : text
|
|
336
|
+
return `URL: ${p.url()}\nTitle: ${await p.title()}\n\n---\n${trimmed}`
|
|
337
|
+
},
|
|
338
|
+
}),
|
|
339
|
+
|
|
340
|
+
browser_click: tool({
|
|
341
|
+
description:
|
|
342
|
+
"Click an element in the browser. Accepts a CSS selector (#id, .class, button), xpath (//button[2]), or plain text (wrapped as text= automatically). A screenshot is saved after clicking.",
|
|
343
|
+
args: {
|
|
344
|
+
target: tool.schema
|
|
345
|
+
.string()
|
|
346
|
+
.describe('e.g. "#submit", "a", //button[2], "Sign in"'),
|
|
347
|
+
},
|
|
348
|
+
async execute(args) {
|
|
349
|
+
const p = await ensurePage()
|
|
350
|
+
const selector = resolveSelector(args.target)
|
|
351
|
+
await p.click(selector, { timeout: 15000 })
|
|
352
|
+
await p.waitForLoadState("domcontentloaded").catch(() => {})
|
|
353
|
+
const file = await saveScreenshot(p, "click")
|
|
354
|
+
return `Clicked "${selector}" on ${p.url()}${formatSnap(file)}`
|
|
355
|
+
},
|
|
356
|
+
}),
|
|
357
|
+
|
|
358
|
+
browser_type: tool({
|
|
359
|
+
description:
|
|
360
|
+
"Type text into a form field. Clears existing value, types the text, and optionally presses Enter to submit.",
|
|
361
|
+
args: {
|
|
362
|
+
target: tool.schema
|
|
363
|
+
.string()
|
|
364
|
+
.describe('Field selector, e.g. "input[name=q]" or "Search"'),
|
|
365
|
+
text: tool.schema.string().describe("Text to type"),
|
|
366
|
+
submit: tool.schema
|
|
367
|
+
.boolean()
|
|
368
|
+
.optional()
|
|
369
|
+
.describe("Press Enter after typing (default false)"),
|
|
370
|
+
},
|
|
371
|
+
async execute(args) {
|
|
372
|
+
const p = await ensurePage()
|
|
373
|
+
const selector = resolveSelector(args.target)
|
|
374
|
+
await p.fill(selector, args.text, { timeout: 15000 })
|
|
375
|
+
if (args.submit) {
|
|
376
|
+
await p.press(selector, "Enter")
|
|
377
|
+
await p.waitForLoadState("domcontentloaded").catch(() => {})
|
|
378
|
+
}
|
|
379
|
+
const file = await saveScreenshot(p, "type")
|
|
380
|
+
return `Typed ${JSON.stringify(args.text)} into "${selector}" (submit=${!!args.submit})${formatSnap(file)}`
|
|
381
|
+
},
|
|
382
|
+
}),
|
|
383
|
+
|
|
384
|
+
browser_screenshot: tool({
|
|
385
|
+
description:
|
|
386
|
+
"Take a screenshot of the current browser page. Returns the file path; use the read tool to view it.",
|
|
387
|
+
args: {
|
|
388
|
+
fullPage: tool.schema
|
|
389
|
+
.boolean()
|
|
390
|
+
.optional()
|
|
391
|
+
.describe("Capture the full scrollable page (default: viewport only)"),
|
|
392
|
+
},
|
|
393
|
+
async execute(args) {
|
|
394
|
+
const p = await ensurePage()
|
|
395
|
+
const file = await saveScreenshot(p, "shot", !!args.fullPage)
|
|
396
|
+
return `Screenshot of ${p.url()} saved: ${file}`
|
|
397
|
+
},
|
|
398
|
+
}),
|
|
399
|
+
|
|
400
|
+
browser_close: tool({
|
|
401
|
+
description: "Close the embedded browser and release its resources.",
|
|
402
|
+
args: {},
|
|
403
|
+
async execute() {
|
|
404
|
+
if (browser) {
|
|
405
|
+
await browser.close().catch(() => {})
|
|
406
|
+
browser = null
|
|
407
|
+
context = null
|
|
408
|
+
page = null
|
|
409
|
+
}
|
|
410
|
+
return "Browser closed."
|
|
411
|
+
},
|
|
412
|
+
}),
|
|
413
|
+
|
|
414
|
+
browser_inspect: tool({
|
|
415
|
+
description:
|
|
416
|
+
"Interactive element picker for design changes. Opens a VISIBLE browser window on the URL, waits for the user to hover and CLICK an element, then the user TYPES what they want changed and clicks Apply. Returns the element's unique selector, text, HTML, styles, React component source (when available) and the user's change note. Use whenever the user wants to point at something on the page to modify it.",
|
|
417
|
+
args: {
|
|
418
|
+
url: tool.schema
|
|
419
|
+
.string()
|
|
420
|
+
.describe("URL to open, e.g. http://localhost:3000"),
|
|
421
|
+
timeoutSeconds: tool.schema
|
|
422
|
+
.number()
|
|
423
|
+
.optional()
|
|
424
|
+
.describe("How long to wait for the user's selection (default 300)"),
|
|
425
|
+
},
|
|
426
|
+
async execute(args) {
|
|
427
|
+
let inspectBrowser: Browser | null = null
|
|
428
|
+
try {
|
|
429
|
+
inspectBrowser = await launchBrowser(false)
|
|
430
|
+
const pageCtx = await inspectBrowser.newContext({
|
|
431
|
+
viewport: VIEWPORT,
|
|
432
|
+
deviceScaleFactor: 1,
|
|
433
|
+
})
|
|
434
|
+
pageCtx.setDefaultTimeout(30000)
|
|
435
|
+
const p = await pageCtx.newPage()
|
|
436
|
+
await p.goto(args.url, { waitUntil: "domcontentloaded", timeout: 30000 })
|
|
437
|
+
await p.evaluate(PICKER_JS)
|
|
438
|
+
try {
|
|
439
|
+
await p.waitForFunction("window.__ocPicked !== undefined", null, {
|
|
440
|
+
timeout: (args.timeoutSeconds ?? 300) * 1000,
|
|
441
|
+
polling: 200,
|
|
442
|
+
})
|
|
443
|
+
} catch {
|
|
444
|
+
return "No selection made within the timeout — visible browser closed."
|
|
445
|
+
}
|
|
446
|
+
const outcome = (await p.evaluate("window.__ocPicked")) as
|
|
447
|
+
| null
|
|
448
|
+
| {
|
|
449
|
+
note: string
|
|
450
|
+
selector: string
|
|
451
|
+
tag: string
|
|
452
|
+
id: string
|
|
453
|
+
classes: string
|
|
454
|
+
text: string
|
|
455
|
+
html: string
|
|
456
|
+
rect: { x: number; y: number; width: number; height: number }
|
|
457
|
+
styles: Record<string, string>
|
|
458
|
+
react: { file?: string; line?: number; comp?: string } | null
|
|
459
|
+
}
|
|
460
|
+
if (outcome === null) return "Selection cancelled by user."
|
|
461
|
+
const lines = [
|
|
462
|
+
`User picked an element on ${p.url()}`,
|
|
463
|
+
`Selector: ${outcome.selector}`,
|
|
464
|
+
`Tag: ${outcome.tag}${outcome.id ? ` #${outcome.id}` : ""}${outcome.classes ? ` (${outcome.classes})` : ""}`,
|
|
465
|
+
`Text: ${outcome.text || "(none)"}`,
|
|
466
|
+
`Box: ${outcome.rect.width}x${outcome.rect.height} at (${outcome.rect.x}, ${outcome.rect.y})`,
|
|
467
|
+
`Styles: fontSize=${outcome.styles.fontSize} fontWeight=${outcome.styles.fontWeight} color=${outcome.styles.color} background=${outcome.styles.backgroundColor} radius=${outcome.styles.borderRadius} font=${(outcome.styles.fontFamily || "").split(",")[0]}`,
|
|
468
|
+
`HTML: ${outcome.html}`,
|
|
469
|
+
outcome.react?.file
|
|
470
|
+
? `React source: ${outcome.react.file}${outcome.react.line ? `:${outcome.react.line}` : ""}${outcome.react.comp ? ` (${outcome.react.comp})` : ""}`
|
|
471
|
+
: null,
|
|
472
|
+
`User note: ${outcome.note || "(nothing typed — ask the user what to change)"}`,
|
|
473
|
+
].filter(Boolean)
|
|
474
|
+
return lines.join("\n")
|
|
475
|
+
} finally {
|
|
476
|
+
if (inspectBrowser) await inspectBrowser.close().catch(() => {})
|
|
477
|
+
}
|
|
478
|
+
},
|
|
479
|
+
}),
|
|
480
|
+
},
|
|
481
|
+
}
|
|
482
|
+
}
|