aieye 1.2.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +92 -92
- package/dist/index.js +9 -9
- package/dist/index.js.map +3 -3
- package/package.json +7 -6
package/README.md
CHANGED
|
@@ -1,35 +1,35 @@
|
|
|
1
|
-
#
|
|
1
|
+
# aieye
|
|
2
2
|
|
|
3
|
-
MCP (Model Context Protocol) server for
|
|
3
|
+
MCP (Model Context Protocol) server for AI Eye. Exposes browser automation and inspection tools to AI agents like Claude.
|
|
4
4
|
|
|
5
5
|
## Features
|
|
6
6
|
|
|
7
7
|
The MCP server provides 13 tools for AI-driven visual development:
|
|
8
8
|
|
|
9
9
|
### 🔍 Inspection Tools
|
|
10
|
-
- **
|
|
11
|
-
- **
|
|
12
|
-
- **
|
|
13
|
-
- **
|
|
10
|
+
- **aieye_inspect_element** - Get detailed element information with source mapping
|
|
11
|
+
- **aieye_get_source** - Get source file location for any element
|
|
12
|
+
- **aieye_get_structure** - Get page DOM structure with source mapping
|
|
13
|
+
- **aieye_find_elements** - Find elements by text, role, or CSS selector
|
|
14
14
|
|
|
15
15
|
### 🎯 Interaction Tools
|
|
16
|
-
- **
|
|
17
|
-
- **
|
|
18
|
-
- **
|
|
16
|
+
- **aieye_click** - Click elements
|
|
17
|
+
- **aieye_type** - Type text into inputs
|
|
18
|
+
- **aieye_scroll** - Scroll the page
|
|
19
19
|
|
|
20
20
|
### 📸 Debugging Tools
|
|
21
|
-
- **
|
|
22
|
-
- **
|
|
23
|
-
- **
|
|
24
|
-
- **
|
|
21
|
+
- **aieye_screenshot** - Capture page screenshots
|
|
22
|
+
- **aieye_get_console_logs** - Retrieve console logs
|
|
23
|
+
- **aieye_clear_console_logs** - Clear captured logs
|
|
24
|
+
- **aieye_get_hmr_status** - Check HMR status
|
|
25
25
|
|
|
26
26
|
### 🌐 Navigation Tools
|
|
27
|
-
- **
|
|
28
|
-
- **
|
|
27
|
+
- **aieye_navigate** - Navigate to URLs
|
|
28
|
+
- **aieye_get_current_url** - Get current page URL
|
|
29
29
|
|
|
30
30
|
## Installation
|
|
31
31
|
|
|
32
|
-
The MCP server is automatically built as part of the
|
|
32
|
+
The MCP server is automatically built as part of the AI Eye workspace:
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
35
|
# Build all packages
|
|
@@ -49,7 +49,7 @@ Add to your Claude Desktop config at `~/Library/Application Support/Claude/claud
|
|
|
49
49
|
```json
|
|
50
50
|
{
|
|
51
51
|
"mcpServers": {
|
|
52
|
-
"
|
|
52
|
+
"aieye": {
|
|
53
53
|
"command": "node",
|
|
54
54
|
"args": ["/Users/yourname/path/to/vscode-eye/packages/mcp-server/dist/index.js"]
|
|
55
55
|
}
|
|
@@ -57,7 +57,7 @@ Add to your Claude Desktop config at `~/Library/Application Support/Claude/claud
|
|
|
57
57
|
}
|
|
58
58
|
```
|
|
59
59
|
|
|
60
|
-
**Important**: Replace `/Users/yourname/path/to/vscode-eye` with the actual path to your
|
|
60
|
+
**Important**: Replace `/Users/yourname/path/to/vscode-eye` with the actual path to your AI Eye project.
|
|
61
61
|
|
|
62
62
|
### Starting the Preview
|
|
63
63
|
|
|
@@ -77,30 +77,30 @@ Once configured, you can ask Claude to interact with your UI:
|
|
|
77
77
|
**Example prompts:**
|
|
78
78
|
|
|
79
79
|
1. **"Take a screenshot of the current page"**
|
|
80
|
-
- Claude will call `
|
|
80
|
+
- Claude will call `aieye_screenshot`
|
|
81
81
|
- Returns base64-encoded image
|
|
82
82
|
|
|
83
83
|
2. **"What button elements are on the page?"**
|
|
84
|
-
- Claude will call `
|
|
84
|
+
- Claude will call `aieye_find_elements` with query "button"
|
|
85
85
|
- Returns array of buttons with selectors and source locations
|
|
86
86
|
|
|
87
87
|
3. **"Click the increment button"**
|
|
88
|
-
- Claude will call `
|
|
89
|
-
- Then `
|
|
88
|
+
- Claude will call `aieye_find_elements` to find button
|
|
89
|
+
- Then `aieye_click` to click it
|
|
90
90
|
|
|
91
91
|
4. **"Where is the h1 element defined in the source code?"**
|
|
92
|
-
- Claude will call `
|
|
92
|
+
- Claude will call `aieye_get_source` with selector "h1"
|
|
93
93
|
- Returns file path, line, and column
|
|
94
94
|
|
|
95
95
|
5. **"Show me the page structure"**
|
|
96
|
-
- Claude will call `
|
|
96
|
+
- Claude will call `aieye_get_structure`
|
|
97
97
|
- Returns DOM tree with source mapping
|
|
98
98
|
|
|
99
99
|
## How It Works
|
|
100
100
|
|
|
101
101
|
1. **STDIO Transport**: MCP server communicates with Claude via standard input/output
|
|
102
102
|
2. **Playwright Connection**: Connects to browser using Playwright
|
|
103
|
-
3. **Bridge API**: Calls `window.
|
|
103
|
+
3. **Bridge API**: Calls `window.__AIEYE__` APIs injected by Vite plugin
|
|
104
104
|
4. **Source Mapping**: All elements include source file locations
|
|
105
105
|
|
|
106
106
|
## Architecture
|
|
@@ -111,7 +111,7 @@ Claude Desktop
|
|
|
111
111
|
MCP Server
|
|
112
112
|
↕ (Playwright/CDP)
|
|
113
113
|
Browser (Chromium)
|
|
114
|
-
↕ (window.
|
|
114
|
+
↕ (window.__AIEYE__)
|
|
115
115
|
Your React App
|
|
116
116
|
```
|
|
117
117
|
|
|
@@ -144,7 +144,7 @@ The server will wait for STDIO input in MCP format.
|
|
|
144
144
|
|
|
145
145
|
## Tool Reference
|
|
146
146
|
|
|
147
|
-
###
|
|
147
|
+
### aieye_screenshot
|
|
148
148
|
|
|
149
149
|
Capture a screenshot of the current page.
|
|
150
150
|
|
|
@@ -154,7 +154,7 @@ Capture a screenshot of the current page.
|
|
|
154
154
|
|
|
155
155
|
**Returns:** Base64-encoded image data
|
|
156
156
|
|
|
157
|
-
###
|
|
157
|
+
### aieye_inspect_element
|
|
158
158
|
|
|
159
159
|
Inspect an element and get detailed information.
|
|
160
160
|
|
|
@@ -174,7 +174,7 @@ Inspect an element and get detailed information.
|
|
|
174
174
|
}
|
|
175
175
|
```
|
|
176
176
|
|
|
177
|
-
###
|
|
177
|
+
### aieye_get_source
|
|
178
178
|
|
|
179
179
|
Get source location for an element.
|
|
180
180
|
|
|
@@ -190,7 +190,7 @@ Get source location for an element.
|
|
|
190
190
|
}
|
|
191
191
|
```
|
|
192
192
|
|
|
193
|
-
###
|
|
193
|
+
### aieye_click
|
|
194
194
|
|
|
195
195
|
Click an element.
|
|
196
196
|
|
|
@@ -204,7 +204,7 @@ Click an element.
|
|
|
204
204
|
}
|
|
205
205
|
```
|
|
206
206
|
|
|
207
|
-
###
|
|
207
|
+
### aieye_type
|
|
208
208
|
|
|
209
209
|
Type text into an input element.
|
|
210
210
|
|
|
@@ -219,7 +219,7 @@ Type text into an input element.
|
|
|
219
219
|
}
|
|
220
220
|
```
|
|
221
221
|
|
|
222
|
-
###
|
|
222
|
+
### aieye_find_elements
|
|
223
223
|
|
|
224
224
|
Find elements by text, role, or CSS selector.
|
|
225
225
|
|
|
@@ -229,7 +229,7 @@ Find elements by text, role, or CSS selector.
|
|
|
229
229
|
|
|
230
230
|
**Returns:** Array of matching elements with selectors and source locations
|
|
231
231
|
|
|
232
|
-
###
|
|
232
|
+
### aieye_get_structure
|
|
233
233
|
|
|
234
234
|
Get page DOM structure.
|
|
235
235
|
|
|
@@ -238,7 +238,7 @@ Get page DOM structure.
|
|
|
238
238
|
|
|
239
239
|
**Returns:** Tree structure with source mapping
|
|
240
240
|
|
|
241
|
-
###
|
|
241
|
+
### aieye_get_console_logs
|
|
242
242
|
|
|
243
243
|
Retrieve captured console logs.
|
|
244
244
|
|
|
@@ -271,7 +271,7 @@ Retrieve captured console logs.
|
|
|
271
271
|
|
|
272
272
|
### Overview
|
|
273
273
|
|
|
274
|
-
This MCP server is **your primary interface** to
|
|
274
|
+
This MCP server is **your primary interface** to AI Eye. As an AI agent, you communicate with this server through the Model Context Protocol (MCP) to interact with the user's browser and application.
|
|
275
275
|
|
|
276
276
|
**You have access to 14 tools** (see list at top of this README) that allow you to:
|
|
277
277
|
- See the UI (screenshots)
|
|
@@ -287,14 +287,14 @@ You (AI Agent)
|
|
|
287
287
|
This MCP Server
|
|
288
288
|
↕ Playwright/CDP
|
|
289
289
|
Browser (Running user's app)
|
|
290
|
-
↕ window.
|
|
291
|
-
User's Application (with
|
|
290
|
+
↕ window.__AIEYE__
|
|
291
|
+
User's Application (with AI Eye plugin)
|
|
292
292
|
```
|
|
293
293
|
|
|
294
294
|
**Key points:**
|
|
295
295
|
1. You don't talk to the browser directly - you talk to this MCP server
|
|
296
296
|
2. This server handles browser automation using Playwright
|
|
297
|
-
3. The browser has
|
|
297
|
+
3. The browser has AI Eye bridge injected for source mapping
|
|
298
298
|
4. All responses are JSON (tool results)
|
|
299
299
|
|
|
300
300
|
### Connection Modes
|
|
@@ -314,7 +314,7 @@ The MCP server supports three connection modes:
|
|
|
314
314
|
3. **CDP-Only (fallback)**
|
|
315
315
|
- No bridge available
|
|
316
316
|
- Limited functionality (screenshots, clicking)
|
|
317
|
-
- Falls back when
|
|
317
|
+
- Falls back when AI Eye plugin not configured
|
|
318
318
|
|
|
319
319
|
### Tool Usage Patterns
|
|
320
320
|
|
|
@@ -323,13 +323,13 @@ The MCP server supports three connection modes:
|
|
|
323
323
|
User: "The button looks weird"
|
|
324
324
|
|
|
325
325
|
You:
|
|
326
|
-
1.
|
|
327
|
-
2.
|
|
328
|
-
3.
|
|
326
|
+
1. aieye_screenshot → See the button
|
|
327
|
+
2. aieye_find_elements → Find button selector
|
|
328
|
+
3. aieye_inspect_element → Get styles & source location
|
|
329
329
|
4. [Read source file at src/Button.tsx:45]
|
|
330
330
|
5. [Identify CSS issue]
|
|
331
331
|
6. [Fix code]
|
|
332
|
-
7.
|
|
332
|
+
7. aieye_screenshot → Verify fix
|
|
333
333
|
```
|
|
334
334
|
|
|
335
335
|
**Pattern 2: Feature Request**
|
|
@@ -337,11 +337,11 @@ You:
|
|
|
337
337
|
User: "Add a dark mode toggle"
|
|
338
338
|
|
|
339
339
|
You:
|
|
340
|
-
1.
|
|
341
|
-
2.
|
|
340
|
+
1. aieye_screenshot → See current UI
|
|
341
|
+
2. aieye_get_structure → Understand layout
|
|
342
342
|
3. [Create new component]
|
|
343
|
-
4.
|
|
344
|
-
5.
|
|
343
|
+
4. aieye_get_console_logs → Check for errors
|
|
344
|
+
5. aieye_screenshot → Show new feature
|
|
345
345
|
```
|
|
346
346
|
|
|
347
347
|
**Pattern 3: Bug Investigation**
|
|
@@ -349,37 +349,37 @@ You:
|
|
|
349
349
|
User: "Something's broken but I don't know what"
|
|
350
350
|
|
|
351
351
|
You:
|
|
352
|
-
1.
|
|
353
|
-
2.
|
|
354
|
-
3.
|
|
355
|
-
4.
|
|
352
|
+
1. aieye_get_console_logs → Check for errors
|
|
353
|
+
2. aieye_screenshot → See error state
|
|
354
|
+
3. aieye_inspect_element → Find problematic element
|
|
355
|
+
4. aieye_get_source → Get code location
|
|
356
356
|
5. [Analyze & fix]
|
|
357
|
-
6.
|
|
357
|
+
6. aieye_get_console_logs → Verify no errors
|
|
358
358
|
```
|
|
359
359
|
|
|
360
360
|
### Tool Selection Guide
|
|
361
361
|
|
|
362
362
|
**For seeing UI:**
|
|
363
|
-
- `
|
|
363
|
+
- `aieye_screenshot` - Takes ~1-2 seconds, use sparingly
|
|
364
364
|
|
|
365
365
|
**For finding elements:**
|
|
366
|
-
- `
|
|
366
|
+
- `aieye_find_elements` - Search by text, role, or CSS
|
|
367
367
|
- Use this when you don't know the exact selector
|
|
368
368
|
|
|
369
369
|
**For inspecting:**
|
|
370
|
-
- `
|
|
371
|
-
- `
|
|
370
|
+
- `aieye_get_source` - Fast, just returns file location
|
|
371
|
+
- `aieye_inspect_element` - Slower, returns everything (styles, attributes, location)
|
|
372
372
|
- **Tip:** Use `get_source` if you only need the file location
|
|
373
373
|
|
|
374
374
|
**For interacting:**
|
|
375
|
-
- `
|
|
376
|
-
- `
|
|
377
|
-
- `
|
|
375
|
+
- `aieye_click` - Click buttons, links, etc.
|
|
376
|
+
- `aieye_type` - Fill forms
|
|
377
|
+
- `aieye_scroll` - Access elements below fold
|
|
378
378
|
|
|
379
379
|
**For debugging:**
|
|
380
|
-
- `
|
|
381
|
-
- `
|
|
382
|
-
- `
|
|
380
|
+
- `aieye_get_console_logs` - Check for errors (use filter: "error")
|
|
381
|
+
- `aieye_get_hmr_status` - Check if HMR is working
|
|
382
|
+
- `aieye_clear_console_logs` - Clear logs before testing
|
|
383
383
|
|
|
384
384
|
### Common Errors and Solutions
|
|
385
385
|
|
|
@@ -399,7 +399,7 @@ You:
|
|
|
399
399
|
|
|
400
400
|
**AI should:**
|
|
401
401
|
1. Take screenshot to see what's actually there
|
|
402
|
-
2. Use `
|
|
402
|
+
2. Use `aieye_find_elements` with text search:
|
|
403
403
|
```json
|
|
404
404
|
{"query": "Submit", "mode": "text"}
|
|
405
405
|
```
|
|
@@ -408,11 +408,11 @@ You:
|
|
|
408
408
|
|
|
409
409
|
**Error:** "Bridge not available"
|
|
410
410
|
|
|
411
|
-
**What it means:**
|
|
411
|
+
**What it means:** AI Eye Vite/Babel plugin not configured
|
|
412
412
|
|
|
413
413
|
**AI should:**
|
|
414
414
|
1. Explain plugin is required for source mapping
|
|
415
|
-
2. Guide user to install: `npm install @
|
|
415
|
+
2. Guide user to install: `npm install @ai-eye/vite-plugin --save-dev`
|
|
416
416
|
3. Guide user to configure vite.config.ts
|
|
417
417
|
4. Tell user to restart dev server
|
|
418
418
|
|
|
@@ -427,18 +427,18 @@ You:
|
|
|
427
427
|
|
|
428
428
|
### Configuration for Users
|
|
429
429
|
|
|
430
|
-
When setting up
|
|
430
|
+
When setting up AI Eye with users, guide them to add this to their Claude Desktop config:
|
|
431
431
|
|
|
432
432
|
**macOS:**
|
|
433
433
|
```json
|
|
434
434
|
{
|
|
435
435
|
"mcpServers": {
|
|
436
|
-
"
|
|
436
|
+
"aieye": {
|
|
437
437
|
"command": "node",
|
|
438
|
-
"args": ["/absolute/path/to/
|
|
438
|
+
"args": ["/absolute/path/to/ai-eye/packages/mcp-server/dist/index.js"],
|
|
439
439
|
"env": {
|
|
440
|
-
"
|
|
441
|
-
"
|
|
440
|
+
"AIEYE_URL": "http://localhost:5173",
|
|
441
|
+
"AIEYE_MODE": "playwright-launch"
|
|
442
442
|
}
|
|
443
443
|
}
|
|
444
444
|
}
|
|
@@ -446,8 +446,8 @@ When setting up VisionCraft with users, guide them to add this to their Claude D
|
|
|
446
446
|
```
|
|
447
447
|
|
|
448
448
|
**Environment variables:**
|
|
449
|
-
- `
|
|
450
|
-
- `
|
|
449
|
+
- `AIEYE_URL` - Dev server URL (default: http://localhost:5173)
|
|
450
|
+
- `AIEYE_MODE` - Connection mode:
|
|
451
451
|
- `playwright-launch` (default) - Launch own browser
|
|
452
452
|
- `cdp-connect` - Connect to existing Chrome
|
|
453
453
|
- `cdp-only` - CDP without bridge (fallback)
|
|
@@ -455,11 +455,11 @@ When setting up VisionCraft with users, guide them to add this to their Claude D
|
|
|
455
455
|
### Best Practices for AI Agents
|
|
456
456
|
|
|
457
457
|
**DO:**
|
|
458
|
-
- ✅ Always start with `
|
|
459
|
-
- ✅ Use `
|
|
460
|
-
- ✅ Check `
|
|
461
|
-
- ✅ Clear console logs before testing: `
|
|
462
|
-
- ✅ Use `
|
|
458
|
+
- ✅ Always start with `aieye_screenshot` to see current state
|
|
459
|
+
- ✅ Use `aieye_get_source` for quick source lookups
|
|
460
|
+
- ✅ Check `aieye_get_console_logs` after interactions
|
|
461
|
+
- ✅ Clear console logs before testing: `aieye_clear_console_logs`
|
|
462
|
+
- ✅ Use `aieye_find_elements` to search by description
|
|
463
463
|
- ✅ Take before/after screenshots to verify changes
|
|
464
464
|
|
|
465
465
|
**DON'T:**
|
|
@@ -477,37 +477,37 @@ When setting up VisionCraft with users, guide them to add this to their Claude D
|
|
|
477
477
|
|
|
478
478
|
### Verification Checklist
|
|
479
479
|
|
|
480
|
-
Before telling user "
|
|
480
|
+
Before telling user "AI Eye is set up correctly", verify:
|
|
481
481
|
|
|
482
482
|
```
|
|
483
483
|
□ 1. Dev server is running
|
|
484
484
|
[User confirms or you check with them]
|
|
485
485
|
|
|
486
486
|
□ 2. Navigate to dev server
|
|
487
|
-
|
|
487
|
+
aieye_navigate: "http://localhost:5173"
|
|
488
488
|
|
|
489
489
|
□ 3. Page loads successfully
|
|
490
|
-
|
|
490
|
+
aieye_get_current_url
|
|
491
491
|
Should match the navigate URL
|
|
492
492
|
|
|
493
493
|
□ 4. Can take screenshot
|
|
494
|
-
|
|
494
|
+
aieye_screenshot
|
|
495
495
|
Should return image data
|
|
496
496
|
|
|
497
497
|
□ 5. Bridge is available (source mapping works)
|
|
498
|
-
|
|
498
|
+
aieye_inspect_element: "body"
|
|
499
499
|
Check if sourceFile/sourceLine exist
|
|
500
500
|
|
|
501
501
|
□ 6. Console logs work
|
|
502
|
-
|
|
502
|
+
aieye_get_console_logs
|
|
503
503
|
Should return array (may be empty)
|
|
504
504
|
|
|
505
505
|
□ 7. HMR tracking works
|
|
506
|
-
|
|
506
|
+
aieye_get_hmr_status
|
|
507
507
|
Should show connected: true
|
|
508
508
|
```
|
|
509
509
|
|
|
510
|
-
If all checks pass →
|
|
510
|
+
If all checks pass → AI Eye is fully operational ✅
|
|
511
511
|
|
|
512
512
|
### Debugging MCP Server Issues
|
|
513
513
|
|
|
@@ -519,7 +519,7 @@ When things aren't working:
|
|
|
519
519
|
|
|
520
520
|
**Check 2: Can it connect to browser?**
|
|
521
521
|
```
|
|
522
|
-
|
|
522
|
+
aieye_get_current_url
|
|
523
523
|
```
|
|
524
524
|
- Success → Connected ✅
|
|
525
525
|
- Error → Not connected ❌
|
|
@@ -530,7 +530,7 @@ visioncraft_get_current_url
|
|
|
530
530
|
|
|
531
531
|
**Check 4: Is bridge available?**
|
|
532
532
|
```
|
|
533
|
-
|
|
533
|
+
aieye_inspect_element: "body"
|
|
534
534
|
```
|
|
535
535
|
- Has sourceFile → Bridge working ✅
|
|
536
536
|
- No sourceFile → Bridge not available ❌
|
|
@@ -541,15 +541,15 @@ visioncraft_inspect_element: "body"
|
|
|
541
541
|
|
|
542
542
|
**AI Response:**
|
|
543
543
|
```
|
|
544
|
-
Let me investigate using
|
|
544
|
+
Let me investigate using AI Eye:
|
|
545
545
|
|
|
546
546
|
1. Taking screenshot of current state...
|
|
547
|
-
[calls
|
|
547
|
+
[calls aieye_screenshot]
|
|
548
548
|
|
|
549
549
|
I can see a blank page with an error overlay.
|
|
550
550
|
|
|
551
551
|
2. Checking console for errors...
|
|
552
|
-
[calls
|
|
552
|
+
[calls aieye_get_console_logs with level: "error"]
|
|
553
553
|
|
|
554
554
|
Found error: "Uncaught TypeError: Cannot read property 'map' of undefined"
|
|
555
555
|
at App.tsx:45
|
|
@@ -564,12 +564,12 @@ Let me investigate using VisionCraft:
|
|
|
564
564
|
{items?.map(...)} or {items && items.map(...)}
|
|
565
565
|
|
|
566
566
|
5. Verifying fix...
|
|
567
|
-
[calls
|
|
567
|
+
[calls aieye_get_console_logs again]
|
|
568
568
|
|
|
569
569
|
No errors! ✅
|
|
570
570
|
|
|
571
571
|
6. Taking screenshot to confirm...
|
|
572
|
-
[calls
|
|
572
|
+
[calls aieye_screenshot]
|
|
573
573
|
|
|
574
574
|
App is now loading correctly!
|
|
575
575
|
```
|