@memtensor/memos-local-openclaw-plugin 0.3.18 → 0.3.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -11
- package/dist/capture/index.d.ts +1 -1
- package/dist/capture/index.d.ts.map +1 -1
- package/dist/capture/index.js +7 -2
- package/dist/capture/index.js.map +1 -1
- package/dist/index.d.ts +1 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +2 -2
- package/dist/index.js.map +1 -1
- package/dist/ingest/dedup.d.ts +2 -2
- package/dist/ingest/dedup.d.ts.map +1 -1
- package/dist/ingest/dedup.js +4 -4
- package/dist/ingest/dedup.js.map +1 -1
- package/dist/ingest/task-processor.d.ts +1 -1
- package/dist/ingest/task-processor.d.ts.map +1 -1
- package/dist/ingest/task-processor.js +14 -13
- package/dist/ingest/task-processor.js.map +1 -1
- package/dist/ingest/worker.d.ts.map +1 -1
- package/dist/ingest/worker.js +7 -3
- package/dist/ingest/worker.js.map +1 -1
- package/dist/recall/engine.d.ts +5 -1
- package/dist/recall/engine.d.ts.map +1 -1
- package/dist/recall/engine.js +77 -2
- package/dist/recall/engine.js.map +1 -1
- package/dist/skill/evolver.d.ts +2 -1
- package/dist/skill/evolver.d.ts.map +1 -1
- package/dist/skill/evolver.js +2 -2
- package/dist/skill/evolver.js.map +1 -1
- package/dist/skill/generator.d.ts +3 -1
- package/dist/skill/generator.d.ts.map +1 -1
- package/dist/skill/generator.js +15 -1
- package/dist/skill/generator.js.map +1 -1
- package/dist/storage/sqlite.d.ts +24 -8
- package/dist/storage/sqlite.d.ts.map +1 -1
- package/dist/storage/sqlite.js +233 -28
- package/dist/storage/sqlite.js.map +1 -1
- package/dist/storage/vector.d.ts +1 -1
- package/dist/storage/vector.d.ts.map +1 -1
- package/dist/storage/vector.js +3 -3
- package/dist/storage/vector.js.map +1 -1
- package/dist/types.d.ts +16 -0
- package/dist/types.d.ts.map +1 -1
- package/dist/types.js.map +1 -1
- package/dist/viewer/html.d.ts +1 -1
- package/dist/viewer/html.d.ts.map +1 -1
- package/dist/viewer/html.js +107 -1
- package/dist/viewer/html.js.map +1 -1
- package/dist/viewer/server.d.ts +1 -0
- package/dist/viewer/server.d.ts.map +1 -1
- package/dist/viewer/server.js +52 -3
- package/dist/viewer/server.js.map +1 -1
- package/index.ts +171 -7
- package/package.json +1 -1
- package/skill/browserwing-admin/SKILL.md +521 -0
- package/skill/browserwing-executor/SKILL.md +510 -0
- package/skill/memos-memory-guide/SKILL.md +62 -36
- package/src/capture/index.ts +7 -1
- package/src/index.ts +3 -2
- package/src/ingest/dedup.ts +4 -2
- package/src/ingest/task-processor.ts +14 -13
- package/src/ingest/worker.ts +7 -3
- package/src/recall/engine.ts +94 -4
- package/src/skill/evolver.ts +3 -1
- package/src/skill/generator.ts +15 -0
- package/src/storage/sqlite.ts +262 -34
- package/src/storage/vector.ts +3 -2
- package/src/types.ts +18 -0
- package/src/viewer/html.ts +107 -1
- package/src/viewer/server.ts +48 -3
|
@@ -0,0 +1,510 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: browserwing-executor
|
|
3
|
+
description: Control browser automation through HTTP API. Supports page navigation, element interaction (click, type, select), data extraction, accessibility snapshot analysis, screenshot, JavaScript execution, and batch operations.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# BrowserWing Executor API
|
|
7
|
+
|
|
8
|
+
## Overview
|
|
9
|
+
|
|
10
|
+
BrowserWing Executor provides comprehensive browser automation capabilities through HTTP APIs. You can control browser navigation, interact with page elements, extract data, and analyze page structure.
|
|
11
|
+
|
|
12
|
+
**API Base URL:** `http://localhost:8080/api/v1/executor`
|
|
13
|
+
|
|
14
|
+
**Authentication:** Use `X-BrowserWing-Key: <api-key>` header or `Authorization: Bearer <token>`
|
|
15
|
+
|
|
16
|
+
## Core Capabilities
|
|
17
|
+
|
|
18
|
+
- **Page Navigation:** Navigate to URLs, go back/forward, reload
|
|
19
|
+
- **Element Interaction:** Click, type, select, hover on page elements
|
|
20
|
+
- **Data Extraction:** Extract text, attributes, values from elements
|
|
21
|
+
- **Accessibility Analysis:** Get accessibility snapshot to understand page structure
|
|
22
|
+
- **Advanced Operations:** Screenshot, JavaScript execution, keyboard input
|
|
23
|
+
- **Batch Processing:** Execute multiple operations in sequence
|
|
24
|
+
|
|
25
|
+
## API Endpoints
|
|
26
|
+
|
|
27
|
+
### 1. Discover Available Commands
|
|
28
|
+
|
|
29
|
+
**IMPORTANT:** Always call this endpoint first to see all available commands and their parameters.
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
curl -X GET 'http://localhost:8080/api/v1/executor/help'
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
**Response:** Returns complete list of all commands with parameters, examples, and usage guidelines.
|
|
36
|
+
|
|
37
|
+
**Query specific command:**
|
|
38
|
+
```bash
|
|
39
|
+
curl -X GET 'http://localhost:8080/api/v1/executor/help?command=extract'
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
### 2. Get Accessibility Snapshot
|
|
43
|
+
|
|
44
|
+
**CRITICAL:** Always call this after navigation to understand page structure and get element RefIDs.
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
curl -X GET 'http://localhost:8080/api/v1/executor/snapshot'
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
**Response Example:**
|
|
51
|
+
```json
|
|
52
|
+
{
|
|
53
|
+
"success": true,
|
|
54
|
+
"snapshot_text": "Clickable Elements:\n @e1 Login (role: button)\n @e2 Sign Up (role: link)\n\nInput Elements:\n @e3 Email (role: textbox) [placeholder: your@email.com]\n @e4 Password (role: textbox)"
|
|
55
|
+
}
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
**Use Cases:**
|
|
59
|
+
- Understand what interactive elements are on the page
|
|
60
|
+
- Get element RefIDs (@e1, @e2, etc.) for precise identification
|
|
61
|
+
- See element labels, roles, and attributes
|
|
62
|
+
- The accessibility tree is cleaner than raw DOM and better for LLMs
|
|
63
|
+
- RefIDs are stable references that work reliably across page changes
|
|
64
|
+
|
|
65
|
+
### 3. Common Operations
|
|
66
|
+
|
|
67
|
+
#### Navigate to URL
|
|
68
|
+
```bash
|
|
69
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/navigate' \
|
|
70
|
+
-H 'Content-Type: application/json' \
|
|
71
|
+
-d '{"url": "https://example.com"}'
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
#### Click Element
|
|
75
|
+
```bash
|
|
76
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/click' \
|
|
77
|
+
-H 'Content-Type: application/json' \
|
|
78
|
+
-d '{"identifier": "@e1"}'
|
|
79
|
+
```
|
|
80
|
+
**Identifier formats:**
|
|
81
|
+
- **RefID (Recommended):** `@e1`, `@e2` (from snapshot)
|
|
82
|
+
- **CSS Selector:** `#button-id`, `.class-name`
|
|
83
|
+
- **XPath:** `//button[@type='submit']`
|
|
84
|
+
- **Text:** `Login` (text content)
|
|
85
|
+
|
|
86
|
+
#### Type Text
|
|
87
|
+
```bash
|
|
88
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/type' \
|
|
89
|
+
-H 'Content-Type: application/json' \
|
|
90
|
+
-d '{"identifier": "@e3", "text": "user@example.com"}'
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
#### Extract Data
|
|
94
|
+
```bash
|
|
95
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/extract' \
|
|
96
|
+
-H 'Content-Type: application/json' \
|
|
97
|
+
-d '{
|
|
98
|
+
"selector": ".product-item",
|
|
99
|
+
"fields": ["text", "href"],
|
|
100
|
+
"multiple": true
|
|
101
|
+
}'
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
#### Wait for Element
|
|
105
|
+
```bash
|
|
106
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/wait' \
|
|
107
|
+
-H 'Content-Type: application/json' \
|
|
108
|
+
-d '{"identifier": ".loading", "state": "hidden", "timeout": 10}'
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
#### Batch Operations
|
|
112
|
+
```bash
|
|
113
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/batch' \
|
|
114
|
+
-H 'Content-Type: application/json' \
|
|
115
|
+
-d '{
|
|
116
|
+
"operations": [
|
|
117
|
+
{"type": "navigate", "params": {"url": "https://example.com"}, "stop_on_error": true},
|
|
118
|
+
{"type": "click", "params": {"identifier": "@e1"}, "stop_on_error": true},
|
|
119
|
+
{"type": "type", "params": {"identifier": "@e3", "text": "query"}, "stop_on_error": true}
|
|
120
|
+
]
|
|
121
|
+
}'
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
## Instructions
|
|
125
|
+
|
|
126
|
+
**Step-by-step workflow:**
|
|
127
|
+
|
|
128
|
+
1. **Discover commands:** Call `GET /help` to see all available operations and their parameters (do this first if unsure).
|
|
129
|
+
|
|
130
|
+
2. **Navigate:** Use `POST /navigate` to open the target webpage.
|
|
131
|
+
|
|
132
|
+
3. **Analyze page:** Call `GET /snapshot` to understand page structure and get element RefIDs.
|
|
133
|
+
|
|
134
|
+
4. **Interact:** Use element RefIDs (like `@e1`, `@e2`) or CSS selectors to:
|
|
135
|
+
- Click elements: `POST /click`
|
|
136
|
+
- Input text: `POST /type`
|
|
137
|
+
- Select options: `POST /select`
|
|
138
|
+
- Wait for elements: `POST /wait`
|
|
139
|
+
|
|
140
|
+
5. **Extract data:** Use `POST /extract` to get information from the page.
|
|
141
|
+
|
|
142
|
+
6. **Present results:** Format and show extracted data to the user.
|
|
143
|
+
|
|
144
|
+
## Complete Example
|
|
145
|
+
|
|
146
|
+
**User Request:** "Search for 'laptop' on example.com and get the first 5 results"
|
|
147
|
+
|
|
148
|
+
**Your Actions:**
|
|
149
|
+
|
|
150
|
+
1. Navigate to search page:
|
|
151
|
+
```bash
|
|
152
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/navigate' \
|
|
153
|
+
-H 'Content-Type: application/json' \
|
|
154
|
+
-d '{"url": "https://example.com/search"}'
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
2. Get page structure to find search input:
|
|
158
|
+
```bash
|
|
159
|
+
curl -X GET 'http://localhost:8080/api/v1/executor/snapshot'
|
|
160
|
+
```
|
|
161
|
+
Response shows: `@e3 Search (role: textbox) [placeholder: Search...]`
|
|
162
|
+
|
|
163
|
+
3. Type search query:
|
|
164
|
+
```bash
|
|
165
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/type' \
|
|
166
|
+
-H 'Content-Type: application/json' \
|
|
167
|
+
-d '{"identifier": "@e3", "text": "laptop"}'
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
4. Press Enter to submit:
|
|
171
|
+
```bash
|
|
172
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/press-key' \
|
|
173
|
+
-H 'Content-Type: application/json' \
|
|
174
|
+
-d '{"key": "Enter"}'
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
5. Wait for results to load:
|
|
178
|
+
```bash
|
|
179
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/wait' \
|
|
180
|
+
-H 'Content-Type: application/json' \
|
|
181
|
+
-d '{"identifier": ".search-results", "state": "visible", "timeout": 10}'
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
6. Extract search results:
|
|
185
|
+
```bash
|
|
186
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/extract' \
|
|
187
|
+
-H 'Content-Type: application/json' \
|
|
188
|
+
-d '{
|
|
189
|
+
"selector": ".result-item",
|
|
190
|
+
"fields": ["text", "href"],
|
|
191
|
+
"multiple": true
|
|
192
|
+
}'
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
7. Present the extracted data:
|
|
196
|
+
```
|
|
197
|
+
Found 15 results for 'laptop':
|
|
198
|
+
1. Gaming Laptop - $1299 (https://...)
|
|
199
|
+
2. Business Laptop - $899 (https://...)
|
|
200
|
+
...
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
## Key Commands Reference
|
|
204
|
+
|
|
205
|
+
### Navigation
|
|
206
|
+
- `POST /navigate` - Navigate to URL
|
|
207
|
+
- `POST /go-back` - Go back in history
|
|
208
|
+
- `POST /go-forward` - Go forward in history
|
|
209
|
+
- `POST /reload` - Reload current page
|
|
210
|
+
|
|
211
|
+
### Element Interaction
|
|
212
|
+
- `POST /click` - Click element (supports: RefID `@e1`, CSS selector, XPath, text content)
|
|
213
|
+
- `POST /type` - Type text into input (supports: RefID `@e3`, CSS selector, XPath)
|
|
214
|
+
- `POST /select` - Select dropdown option
|
|
215
|
+
- `POST /hover` - Hover over element
|
|
216
|
+
- `POST /wait` - Wait for element state (visible, hidden, enabled)
|
|
217
|
+
- `POST /press-key` - Press keyboard key (Enter, Tab, Ctrl+S, etc.)
|
|
218
|
+
|
|
219
|
+
### Data Extraction
|
|
220
|
+
- `POST /extract` - Extract data from elements (supports multiple elements, custom fields)
|
|
221
|
+
- `POST /get-text` - Get element text content
|
|
222
|
+
- `POST /get-value` - Get input element value
|
|
223
|
+
- `GET /page-info` - Get page URL and title
|
|
224
|
+
- `GET /page-text` - Get all page text
|
|
225
|
+
- `GET /page-content` - Get full HTML
|
|
226
|
+
|
|
227
|
+
### Page Analysis
|
|
228
|
+
- `GET /snapshot` - Get accessibility snapshot (⭐ **ALWAYS call after navigation**)
|
|
229
|
+
- `GET /clickable-elements` - Get all clickable elements
|
|
230
|
+
- `GET /input-elements` - Get all input elements
|
|
231
|
+
|
|
232
|
+
### Advanced
|
|
233
|
+
- `POST /screenshot` - Take page screenshot (base64 encoded)
|
|
234
|
+
- `POST /evaluate` - Execute JavaScript code
|
|
235
|
+
- `POST /batch` - Execute multiple operations in sequence
|
|
236
|
+
- `POST /scroll-to-bottom` - Scroll to page bottom
|
|
237
|
+
- `POST /resize` - Resize browser window
|
|
238
|
+
- `POST /tabs` - Manage browser tabs (list, new, switch, close)
|
|
239
|
+
- `POST /fill-form` - Intelligently fill multiple form fields at once
|
|
240
|
+
|
|
241
|
+
### Debug & Monitoring
|
|
242
|
+
- `GET /console-messages` - Get browser console messages (logs, warnings, errors)
|
|
243
|
+
- `GET /network-requests` - Get network requests made by the page
|
|
244
|
+
- `POST /handle-dialog` - Configure JavaScript dialog (alert, confirm, prompt) handling
|
|
245
|
+
- `POST /file-upload` - Upload files to input elements
|
|
246
|
+
- `POST /drag` - Drag and drop elements
|
|
247
|
+
- `POST /close-page` - Close the current page/tab
|
|
248
|
+
|
|
249
|
+
## Element Identification
|
|
250
|
+
|
|
251
|
+
You can identify elements using:
|
|
252
|
+
|
|
253
|
+
1. **RefID (Recommended):** `@e1`, `@e2`, `@e3`
|
|
254
|
+
- Most reliable method - stable across page changes
|
|
255
|
+
- Get RefIDs from `/snapshot` endpoint
|
|
256
|
+
- Valid for 5 minutes after snapshot
|
|
257
|
+
- Example: `"identifier": "@e1"`
|
|
258
|
+
- Works with multi-strategy fallback for robustness
|
|
259
|
+
|
|
260
|
+
2. **CSS Selector:** `#id`, `.class`, `button[type="submit"]`
|
|
261
|
+
- Standard CSS selectors
|
|
262
|
+
- Example: `"identifier": "#login-button"`
|
|
263
|
+
|
|
264
|
+
3. **XPath:** `//button[@id='login']`, `//a[contains(text(), 'Submit')]`
|
|
265
|
+
- XPath expressions for complex queries
|
|
266
|
+
- Example: `"identifier": "//button[@id='login']"`
|
|
267
|
+
|
|
268
|
+
4. **Text Content:** `Login`, `Sign Up`, `Submit`
|
|
269
|
+
- Searches buttons and links with matching text
|
|
270
|
+
- Example: `"identifier": "Login"`
|
|
271
|
+
|
|
272
|
+
5. **ARIA Label:** Elements with `aria-label` attribute
|
|
273
|
+
- Automatically searched
|
|
274
|
+
|
|
275
|
+
## Guidelines
|
|
276
|
+
|
|
277
|
+
**Before starting:**
|
|
278
|
+
- Call `GET /help` if you're unsure about available commands or their parameters
|
|
279
|
+
- Ensure browser is started (if not, it will auto-start on first operation)
|
|
280
|
+
|
|
281
|
+
**During automation:**
|
|
282
|
+
- **Always call `/snapshot` after navigation** to get page structure and RefIDs
|
|
283
|
+
- **Prefer RefIDs** (like `@e1`) over CSS selectors for reliability and stability
|
|
284
|
+
- **Re-snapshot after page changes** to get updated RefIDs
|
|
285
|
+
- **Use `/wait`** for dynamic content that loads asynchronously
|
|
286
|
+
- **Check element states** before interaction (visible, enabled)
|
|
287
|
+
- **Use `/batch`** for multiple sequential operations to improve efficiency
|
|
288
|
+
|
|
289
|
+
**Error handling:**
|
|
290
|
+
- If operation fails, check element identifier and try different format
|
|
291
|
+
- For timeout errors, increase timeout value
|
|
292
|
+
- If element not found, call `/snapshot` again to refresh page structure
|
|
293
|
+
- Explain errors clearly to user with suggested solutions
|
|
294
|
+
|
|
295
|
+
**Data extraction:**
|
|
296
|
+
- Use `fields` parameter to specify what to extract: `["text", "href", "src"]`
|
|
297
|
+
- Set `multiple: true` to extract from multiple elements
|
|
298
|
+
- Format extracted data in a readable way for user
|
|
299
|
+
|
|
300
|
+
## Complete Workflow Example
|
|
301
|
+
|
|
302
|
+
**Scenario:** User wants to login to a website
|
|
303
|
+
|
|
304
|
+
```
|
|
305
|
+
User: "Please log in to example.com with username 'john' and password 'secret123'"
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
**Your Actions:**
|
|
309
|
+
|
|
310
|
+
**Step 1:** Navigate to login page
|
|
311
|
+
```bash
|
|
312
|
+
POST http://localhost:8080/api/v1/executor/navigate
|
|
313
|
+
{"url": "https://example.com/login"}
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
**Step 2:** Get page structure
|
|
317
|
+
```bash
|
|
318
|
+
GET http://localhost:8080/api/v1/executor/snapshot
|
|
319
|
+
```
|
|
320
|
+
Response:
|
|
321
|
+
```
|
|
322
|
+
Clickable Elements:
|
|
323
|
+
@e1 Login (role: button)
|
|
324
|
+
|
|
325
|
+
Input Elements:
|
|
326
|
+
@e2 Username (role: textbox)
|
|
327
|
+
@e3 Password (role: textbox)
|
|
328
|
+
```
|
|
329
|
+
|
|
330
|
+
**Step 3:** Enter username
|
|
331
|
+
```bash
|
|
332
|
+
POST http://localhost:8080/api/v1/executor/type
|
|
333
|
+
{"identifier": "@e2", "text": "john"}
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
**Step 4:** Enter password
|
|
337
|
+
```bash
|
|
338
|
+
POST http://localhost:8080/api/v1/executor/type
|
|
339
|
+
{"identifier": "@e3", "text": "secret123"}
|
|
340
|
+
```
|
|
341
|
+
|
|
342
|
+
**Step 5:** Click login button
|
|
343
|
+
```bash
|
|
344
|
+
POST http://localhost:8080/api/v1/executor/click
|
|
345
|
+
{"identifier": "@e1"}
|
|
346
|
+
```
|
|
347
|
+
|
|
348
|
+
**Step 6:** Wait for login success (optional)
|
|
349
|
+
```bash
|
|
350
|
+
POST http://localhost:8080/api/v1/executor/wait
|
|
351
|
+
{"identifier": ".welcome-message", "state": "visible", "timeout": 10}
|
|
352
|
+
```
|
|
353
|
+
|
|
354
|
+
**Step 7:** Inform user
|
|
355
|
+
```
|
|
356
|
+
"Successfully logged in to example.com!"
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
## Batch Operation Example
|
|
360
|
+
|
|
361
|
+
**Scenario:** Fill out a form with multiple fields
|
|
362
|
+
|
|
363
|
+
Instead of making 5 separate API calls, use one batch operation:
|
|
364
|
+
|
|
365
|
+
```bash
|
|
366
|
+
curl -X POST 'http://localhost:8080/api/v1/executor/batch' \
|
|
367
|
+
-H 'Content-Type: application/json' \
|
|
368
|
+
-d '{
|
|
369
|
+
"operations": [
|
|
370
|
+
{
|
|
371
|
+
"type": "navigate",
|
|
372
|
+
"params": {"url": "https://example.com/form"},
|
|
373
|
+
"stop_on_error": true
|
|
374
|
+
},
|
|
375
|
+
{
|
|
376
|
+
"type": "type",
|
|
377
|
+
"params": {"identifier": "#name", "text": "John Doe"},
|
|
378
|
+
"stop_on_error": true
|
|
379
|
+
},
|
|
380
|
+
{
|
|
381
|
+
"type": "type",
|
|
382
|
+
"params": {"identifier": "#email", "text": "john@example.com"},
|
|
383
|
+
"stop_on_error": true
|
|
384
|
+
},
|
|
385
|
+
{
|
|
386
|
+
"type": "select",
|
|
387
|
+
"params": {"identifier": "#country", "value": "United States"},
|
|
388
|
+
"stop_on_error": true
|
|
389
|
+
},
|
|
390
|
+
{
|
|
391
|
+
"type": "click",
|
|
392
|
+
"params": {"identifier": "#submit"},
|
|
393
|
+
"stop_on_error": true
|
|
394
|
+
}
|
|
395
|
+
]
|
|
396
|
+
}'
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
## Best Practices
|
|
400
|
+
|
|
401
|
+
1. **Discovery first:** If unsure, call `/help` or `/help?command=<name>` to learn about commands
|
|
402
|
+
2. **Structure first:** Always call `/snapshot` after navigation to understand the page
|
|
403
|
+
3. **Use accessibility indices:** They're more reliable than CSS selectors (elements might have dynamic classes)
|
|
404
|
+
4. **Wait for dynamic content:** Use `/wait` before interacting with elements that load asynchronously
|
|
405
|
+
5. **Batch when possible:** Use `/batch` for multiple sequential operations
|
|
406
|
+
6. **Handle errors gracefully:** Provide clear explanations and suggestions when operations fail
|
|
407
|
+
7. **Verify results:** After operations, check if desired outcome was achieved
|
|
408
|
+
|
|
409
|
+
## Common Scenarios
|
|
410
|
+
|
|
411
|
+
### Form Filling
|
|
412
|
+
1. Navigate to form page
|
|
413
|
+
2. Get accessibility snapshot to find input elements and their RefIDs
|
|
414
|
+
3. Use `/type` for each field: `@e1`, `@e2`, etc.
|
|
415
|
+
4. Use `/select` for dropdowns
|
|
416
|
+
5. Click submit button using its RefID
|
|
417
|
+
|
|
418
|
+
### Data Scraping
|
|
419
|
+
1. Navigate to target page
|
|
420
|
+
2. Wait for content to load with `/wait`
|
|
421
|
+
3. Use `/extract` with CSS selector and `multiple: true`
|
|
422
|
+
4. Specify fields to extract: `["text", "href", "src"]`
|
|
423
|
+
|
|
424
|
+
### Search Operations
|
|
425
|
+
1. Navigate to search page
|
|
426
|
+
2. Get accessibility snapshot to locate search input
|
|
427
|
+
3. Type search query into input
|
|
428
|
+
4. Press Enter or click search button
|
|
429
|
+
5. Wait for results
|
|
430
|
+
6. Extract results data
|
|
431
|
+
|
|
432
|
+
### Login Automation
|
|
433
|
+
1. Navigate to login page
|
|
434
|
+
2. Get accessibility snapshot to find RefIDs
|
|
435
|
+
3. Type username: `@e2`
|
|
436
|
+
4. Type password: `@e3`
|
|
437
|
+
5. Click login button: `@e1`
|
|
438
|
+
6. Wait for success indicator
|
|
439
|
+
|
|
440
|
+
## Important Notes
|
|
441
|
+
|
|
442
|
+
- Browser must be running (it will auto-start on first operation if needed)
|
|
443
|
+
- Operations are executed on the **currently active browser tab**
|
|
444
|
+
- Accessibility snapshot updates after each navigation and click operation
|
|
445
|
+
- All timeouts are in seconds
|
|
446
|
+
- Use `wait_visible: true` (default) for reliable element interaction
|
|
447
|
+
- Replace `localhost:8080` with actual API host address
|
|
448
|
+
- Authentication required: use `X-BrowserWing-Key` header or JWT token
|
|
449
|
+
|
|
450
|
+
## Troubleshooting
|
|
451
|
+
|
|
452
|
+
**Element not found:**
|
|
453
|
+
- Call `/snapshot` to see available elements
|
|
454
|
+
- Try different identifier format (accessibility index, CSS selector, text)
|
|
455
|
+
- Check if page has finished loading
|
|
456
|
+
|
|
457
|
+
**Timeout errors:**
|
|
458
|
+
- Increase timeout value in request
|
|
459
|
+
- Check if element actually appears on page
|
|
460
|
+
- Use `/wait` with appropriate state before interaction
|
|
461
|
+
|
|
462
|
+
**Extraction returns empty:**
|
|
463
|
+
- Verify CSS selector matches target elements
|
|
464
|
+
- Check if content has loaded (use `/wait` first)
|
|
465
|
+
- Try different extraction fields or type
|
|
466
|
+
|
|
467
|
+
## Quick Reference
|
|
468
|
+
|
|
469
|
+
```bash
|
|
470
|
+
# Discover commands
|
|
471
|
+
GET localhost:8080/api/v1/executor/help
|
|
472
|
+
|
|
473
|
+
# Navigate
|
|
474
|
+
POST localhost:8080/api/v1/executor/navigate {"url": "..."}
|
|
475
|
+
|
|
476
|
+
# Get page structure
|
|
477
|
+
GET localhost:8080/api/v1/executor/snapshot
|
|
478
|
+
|
|
479
|
+
# Click element
|
|
480
|
+
POST localhost:8080/api/v1/executor/click {"identifier": "@e1"}
|
|
481
|
+
|
|
482
|
+
# Type text
|
|
483
|
+
POST localhost:8080/api/v1/executor/type {"identifier": "@e3", "text": "..."}
|
|
484
|
+
|
|
485
|
+
# Extract data
|
|
486
|
+
POST localhost:8080/api/v1/executor/extract {"selector": "...", "fields": [...], "multiple": true}
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
## Response Format
|
|
490
|
+
|
|
491
|
+
All operations return:
|
|
492
|
+
```json
|
|
493
|
+
{
|
|
494
|
+
"success": true,
|
|
495
|
+
"message": "Operation description",
|
|
496
|
+
"timestamp": "2026-01-15T10:30:00Z",
|
|
497
|
+
"data": {
|
|
498
|
+
// Operation-specific data
|
|
499
|
+
}
|
|
500
|
+
}
|
|
501
|
+
```
|
|
502
|
+
|
|
503
|
+
**Error response:**
|
|
504
|
+
```json
|
|
505
|
+
{
|
|
506
|
+
"error": "error.operationFailed",
|
|
507
|
+
"detail": "Detailed error message"
|
|
508
|
+
}
|
|
509
|
+
```
|
|
510
|
+
|
|
@@ -1,86 +1,112 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: memos-memory-guide
|
|
3
|
-
description: Use the MemOS Local memory system to search and use the user's past conversations. Use this skill whenever the user refers to past chats, their own preferences or history, or when you need to answer from prior context. When auto-recall returns nothing (long or unclear user query), generate your own short search query and call memory_search. Use task_summary when you need full task context, skill_get for experience guides, and memory_timeline to expand around a memory hit.
|
|
3
|
+
description: Use the MemOS Local memory system to search and use the user's past conversations. Use this skill whenever the user refers to past chats, their own preferences or history, or when you need to answer from prior context. When auto-recall returns nothing (long or unclear user query), generate your own short search query and call memory_search. Use task_summary when you need full task context, skill_get for experience guides, skill_search to discover public skills, memory_write_public for shared knowledge, and memory_timeline to expand around a memory hit.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# MemOS Local Memory — Agent Guide
|
|
7
7
|
|
|
8
|
-
This skill describes how to use the MemOS memory tools so you can reliably search and use the user's long-term conversation history.
|
|
8
|
+
This skill describes how to use the MemOS memory tools so you can reliably search and use the user's long-term conversation history, share knowledge across agents, and discover public skills.
|
|
9
9
|
|
|
10
10
|
## How memory is provided each turn
|
|
11
11
|
|
|
12
12
|
- **Automatic recall (hook):** At the start of each turn, the system runs a memory search using the user's current message and injects relevant past memories into your context. You do not need to call any tool for that.
|
|
13
|
-
- **When that is not enough:** If the user's message is very long, vague, or the automatic search returns **no memories**, you should **generate your own short, focused query** and call `memory_search` yourself.
|
|
14
|
-
|
|
15
|
-
- Auto-recall said "no memories" or you see no memory block → call `memory_search` with a query you derive (e.g. the user's name, a topic they often mention, or a rephrased question).
|
|
16
|
-
- **When you need more detail:** Search results only give excerpts and IDs. Use the tools below to fetch full task context, skill content, or surrounding messages.
|
|
13
|
+
- **When that is not enough:** If the user's message is very long, vague, or the automatic search returns **no memories**, you should **generate your own short, focused query** and call `memory_search` yourself.
|
|
14
|
+
- **Memory isolation:** Each agent can only see its own memories and memories marked as `public`. Other agents' private memories are invisible to you.
|
|
17
15
|
|
|
18
16
|
## Tools — what they do and when to call
|
|
19
17
|
|
|
20
18
|
### memory_search
|
|
21
19
|
|
|
22
|
-
- **What it does:** Searches the user's stored conversation memory by a natural-language query. Returns a list of relevant excerpts with `chunkId` and optionally `task_id`.
|
|
20
|
+
- **What it does:** Searches the user's stored conversation memory by a natural-language query. Returns a list of relevant excerpts with `chunkId` and optionally `task_id`. Only returns memories belonging to the current agent or marked as public.
|
|
23
21
|
- **When to call:**
|
|
24
|
-
- The automatic recall did not run or returned nothing
|
|
25
|
-
- The user's query is long or unclear — **generate a short query yourself**
|
|
26
|
-
- You need to search with a different angle (e.g. filter by `role='user'`
|
|
27
|
-
- **Parameters:** `query` (required), optional `minScore`, `role
|
|
28
|
-
|
|
22
|
+
- The automatic recall did not run or returned nothing.
|
|
23
|
+
- The user's query is long or unclear — **generate a short query yourself** and call `memory_search(query="...")`.
|
|
24
|
+
- You need to search with a different angle (e.g. filter by `role='user'`).
|
|
25
|
+
- **Parameters:** `query` (required), optional `minScore`, `role`.
|
|
26
|
+
|
|
27
|
+
### memory_write_public
|
|
28
|
+
|
|
29
|
+
- **What it does:** Writes a piece of information to **public memory**. Public memory is visible to all agents — any agent doing `memory_search` can find it.
|
|
30
|
+
- **When to call:** In multi-agent or collaborative scenarios, when you have **persistent information useful to everyone** (e.g. shared decisions, conventions, configurations, workflows). Do not write session-only or purely private content.
|
|
31
|
+
- **Parameters:** `content` (required), `summary` (optional).
|
|
29
32
|
|
|
30
33
|
### task_summary
|
|
31
34
|
|
|
32
|
-
- **What it does:** Returns the full task summary for a given `task_id`: title, status, and the complete narrative summary
|
|
33
|
-
- **When to call:** A `memory_search` hit included a `task_id` and you need the full story of that task
|
|
35
|
+
- **What it does:** Returns the full task summary for a given `task_id`: title, status, and the complete narrative summary.
|
|
36
|
+
- **When to call:** A `memory_search` hit included a `task_id` and you need the full story of that task.
|
|
34
37
|
- **Parameters:** `taskId` (from a search hit).
|
|
35
|
-
- **Effect:** You get one coherent summary of the whole task instead of isolated excerpts.
|
|
36
38
|
|
|
37
39
|
### skill_get
|
|
38
40
|
|
|
39
|
-
- **What it does:** Returns the content of a learned skill (experience guide) by `skillId` or by `taskId`.
|
|
40
|
-
- **When to call:** A search hit has a `task_id` and the task
|
|
41
|
+
- **What it does:** Returns the content of a learned skill (experience guide) by `skillId` or by `taskId`.
|
|
42
|
+
- **When to call:** A search hit has a `task_id` and the task has a "how to do this again" guide. Use this to follow the same approach or reuse steps.
|
|
41
43
|
- **Parameters:** `skillId` (direct) or `taskId` (lookup).
|
|
42
|
-
|
|
44
|
+
|
|
45
|
+
### skill_search
|
|
46
|
+
|
|
47
|
+
- **What it does:** Searches available **skills** (capabilities/guides) by natural language. Can search your own skills, other agents' public skills, or both — controlled by the `scope` parameter.
|
|
48
|
+
- **When to call:** The current task requires a capability or guide you don't have. Use `skill_search` to find one first; after finding it, use `skill_get` to read it, then `skill_install` to load it for future turns. Set `scope` to `public` to only see others' public skills, `self` for only your own, or leave as default `mix` for both.
|
|
49
|
+
- **Parameters:** `query` (required, natural language description of the need), `scope` (optional, default `mix`: self + public; `self`: own only; `public`: public only).
|
|
43
50
|
|
|
44
51
|
### skill_install
|
|
45
52
|
|
|
46
|
-
- **What it does:** Installs a skill (by `skillId`) into the workspace
|
|
47
|
-
- **When to call:** After `skill_get` when the skill is useful for ongoing use
|
|
53
|
+
- **What it does:** Installs a skill (by `skillId`) into the workspace for future sessions.
|
|
54
|
+
- **When to call:** After `skill_get` when the skill is useful for ongoing use.
|
|
55
|
+
- **Parameters:** `skillId`.
|
|
56
|
+
|
|
57
|
+
### skill_publish
|
|
58
|
+
|
|
59
|
+
- **What it does:** Makes a skill **public** so other agents can discover and install it via `skill_search`.
|
|
60
|
+
- **When to call:** You have a useful skill that other agents could benefit from, and you want to share it.
|
|
61
|
+
- **Parameters:** `skillId`.
|
|
62
|
+
|
|
63
|
+
### skill_unpublish
|
|
64
|
+
|
|
65
|
+
- **What it does:** Makes a skill **private** again. Other agents will no longer discover it.
|
|
66
|
+
- **When to call:** You want to stop sharing a previously published skill.
|
|
48
67
|
- **Parameters:** `skillId`.
|
|
49
68
|
|
|
50
69
|
### memory_timeline
|
|
51
70
|
|
|
52
|
-
- **What it does:** Expands context around a single memory chunk: returns the surrounding conversation messages
|
|
53
|
-
- **When to call:** A `memory_search` hit is relevant but you need the surrounding dialogue
|
|
71
|
+
- **What it does:** Expands context around a single memory chunk: returns the surrounding conversation messages.
|
|
72
|
+
- **When to call:** A `memory_search` hit is relevant but you need the surrounding dialogue.
|
|
54
73
|
- **Parameters:** `chunkId` (from a search hit), optional `window` (default 2).
|
|
55
|
-
- **Effect:** You get a short, linear slice of the conversation around that chunk.
|
|
56
74
|
|
|
57
75
|
### memory_viewer
|
|
58
76
|
|
|
59
|
-
- **What it does:** Returns the URL of the MemOS Memory Viewer
|
|
60
|
-
- **When to call:** The user asks how to view their memories
|
|
77
|
+
- **What it does:** Returns the URL of the MemOS Memory Viewer web dashboard.
|
|
78
|
+
- **When to call:** The user asks how to view their memories or open the memory dashboard.
|
|
61
79
|
- **Parameters:** None.
|
|
62
|
-
- **Effect:** You can tell the user to open that URL in a browser.
|
|
63
80
|
|
|
64
81
|
## Quick decision flow
|
|
65
82
|
|
|
66
|
-
1. **No memories in context or auto-recall reported nothing**
|
|
67
|
-
→ Call `memory_search` with a **self-generated short query
|
|
83
|
+
1. **No memories in context or auto-recall reported nothing**
|
|
84
|
+
→ Call `memory_search` with a **self-generated short query**.
|
|
68
85
|
|
|
69
|
-
2. **Search returned hits with `task_id` and you need full context**
|
|
86
|
+
2. **Search returned hits with `task_id` and you need full context**
|
|
70
87
|
→ Call `task_summary(taskId)`.
|
|
71
88
|
|
|
72
|
-
3. **Task has an experience guide you want to follow**
|
|
73
|
-
→ Call `skill_get(taskId=...)`
|
|
89
|
+
3. **Task has an experience guide you want to follow**
|
|
90
|
+
→ Call `skill_get(taskId=...)` or `skill_get(skillId=...)`. Optionally `skill_install(skillId)` for future use.
|
|
74
91
|
|
|
75
|
-
4. **You need the exact surrounding conversation of a hit**
|
|
92
|
+
4. **You need the exact surrounding conversation of a hit**
|
|
76
93
|
→ Call `memory_timeline(chunkId=...)`.
|
|
77
94
|
|
|
78
|
-
5. **
|
|
95
|
+
5. **You need a capability/guide that you don't have**
|
|
96
|
+
→ Call `skill_search(query="...", scope="mix")` to discover available skills.
|
|
97
|
+
|
|
98
|
+
6. **You have shared knowledge useful to all agents**
|
|
99
|
+
→ Call `memory_write_public(content="...")` to persist it in public memory.
|
|
100
|
+
|
|
101
|
+
7. **You want to share a useful skill with other agents**
|
|
102
|
+
→ Call `skill_publish(skillId=...)`.
|
|
103
|
+
|
|
104
|
+
8. **User asks where to see or manage their memories**
|
|
79
105
|
→ Call `memory_viewer()` and share the URL.
|
|
80
106
|
|
|
81
107
|
## Writing good search queries
|
|
82
108
|
|
|
83
109
|
- Prefer **short, focused** queries (a few words or one clear question).
|
|
84
|
-
- Use **concrete terms**: names, topics, tools, or decisions
|
|
85
|
-
- If the user
|
|
86
|
-
- Use `role='user'` when you specifically want to find what the user said
|
|
110
|
+
- Use **concrete terms**: names, topics, tools, or decisions.
|
|
111
|
+
- If the user's message is long, **derive one or two sub-queries** rather than pasting the whole message.
|
|
112
|
+
- Use `role='user'` when you specifically want to find what the user said.
|