natureco-cli 5.64.6 → 5.65.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/README.md +2 -0
- package/THIRD_PARTY_NOTICES.md +12 -0
- package/package.json +4 -1
- package/scripts/e2e-browser-smoke.js +21 -0
- package/src/tools/browser.js +139 -92
- package/src/tools/computer_use_loop.js +39 -13
- package/src/tools/workflow.js +3 -3
- package/src/utils/tools.js +4 -4
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,31 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to NatureCo CLI will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [5.65.0] - 2026-07-13 — Persistent browser agent
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
- Replaced the one-shot headless browser with a persistent Chrome/Chromium agent using the installed system browser and a dedicated NatureCo profile.
|
|
9
|
+
- Added a structured `open → snapshot → @ref click/fill → snapshot` workflow with visible mode by default, persistent login/storage, keyboard actions, screenshots, text extraction, and explicit session close.
|
|
10
|
+
- Added cross-platform system-browser discovery without downloading a bundled Chromium; `playwright-core` supplies the automation protocol.
|
|
11
|
+
- Added an end-to-end browser smoke test that opens a real page, produces a stable interactive reference, and closes the session.
|
|
12
|
+
|
|
13
|
+
### Safety
|
|
14
|
+
- Navigation accepts only HTTP(S), interactions require fresh snapshot references instead of guessed selectors, and the prompt distinguishes the NatureCo browser profile from the user's already-open Chrome window.
|
|
15
|
+
- Added attribution for the MIT-licensed gstack browser architecture patterns reviewed during development.
|
|
16
|
+
|
|
17
|
+
### Verification
|
|
18
|
+
- Live system-Chrome smoke test completed `open → snapshot → close` and exposed the expected `@e1` reference.
|
|
19
|
+
|
|
20
|
+
## [5.64.7] - 2026-07-13 — Resilient GUI vision decisions
|
|
21
|
+
|
|
22
|
+
### Fixed
|
|
23
|
+
- GUI actions now validate required coordinates, text, and keys before generating macOS AppleScript, preventing `undefined değişkeni tanımlanmamış` failures.
|
|
24
|
+
- Truncated, malformed, empty, or Markdown-wrapped vision JSON is handled safely; invalid decisions trigger another visual-analysis step instead of crashing the GUI loop.
|
|
25
|
+
- The independent completion verifier now treats malformed JSON as failed evidence rather than throwing.
|
|
26
|
+
|
|
27
|
+
### Tests
|
|
28
|
+
- Added regressions for missing GUI parameters, Markdown JSON responses, and the exact unterminated-string response reported with MiniMax vision.
|
|
29
|
+
|
|
5
30
|
## [5.64.6] - 2026-07-13 — Typed MiniMax tool arguments
|
|
6
31
|
|
|
7
32
|
### Fixed
|
package/README.md
CHANGED
|
@@ -51,6 +51,8 @@ natureco code
|
|
|
51
51
|
|
|
52
52
|
| Version | Highlights |
|
|
53
53
|
|---------|-----------|
|
|
54
|
+
| **v5.65.0** | **Persistent browser agent:** visible system Chrome/Chromium, persistent login/storage, and reliable `snapshot → @ref click/fill → verify` automation replace one-shot headless browsing. |
|
|
55
|
+
| **v5.64.7** | **Resilient GUI vision:** invalid action parameters and truncated vision JSON are rejected safely and retried instead of crashing macOS automation. |
|
|
54
56
|
| **v5.64.6** | **Reliable MiniMax tool calls:** XML parameters are converted to their declared schema types, fixing `computer_use_loop` failures such as `maxSteps: expected number, got string`. |
|
|
55
57
|
| **v5.64.5** | **Deterministic GUI recovery:** visible web tasks use one verified GUI loop; failures expose their concrete reason and stop blind browser/AppleScript fallback chains. |
|
|
56
58
|
| **v5.64.4** | **Unified MiniMax media:** the existing MiniMax key now powers `MiniMax-VL-01` analysis, `image-01` generation, `MiniMax-Hailuo-2.3` video, and verified GUI vision—no second key required. |
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
# Third-party notices
|
|
2
|
+
|
|
3
|
+
NatureCo CLI's persistent browser-agent design was informed by the open-source
|
|
4
|
+
gstack browser project, particularly its persistent session, snapshot/reference,
|
|
5
|
+
and observe-act-observe patterns.
|
|
6
|
+
|
|
7
|
+
gstack is Copyright (c) 2026 Garry Tan and distributed under the MIT License:
|
|
8
|
+
https://github.com/garrytan/gstack
|
|
9
|
+
|
|
10
|
+
NatureCo's implementation is independently written for its CommonJS tool and
|
|
11
|
+
execution-gateway architecture. Playwright is used through `playwright-core` and
|
|
12
|
+
retains its own license and notices.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "natureco-cli",
|
|
3
|
-
"version": "5.
|
|
3
|
+
"version": "5.65.0",
|
|
4
4
|
"description": "Terminal-native AI agent CLI with bilingual TR/EN UI, multi-agent orchestration, persistent memory, secure tools and messaging integrations.",
|
|
5
5
|
"bin": {
|
|
6
6
|
"natureco": "bin/natureco.js"
|
|
@@ -15,6 +15,7 @@
|
|
|
15
15
|
"CHANGELOG.md",
|
|
16
16
|
"AUDIT.md",
|
|
17
17
|
"SECURITY_AUDIT_SUMMARY.md",
|
|
18
|
+
"THIRD_PARTY_NOTICES.md",
|
|
18
19
|
"DEPLOY_v2.0.0.md"
|
|
19
20
|
],
|
|
20
21
|
"scripts": {
|
|
@@ -28,6 +29,7 @@
|
|
|
28
29
|
"lint:errors-only": "eslint src/ bin/ test/ --quiet",
|
|
29
30
|
"smoke": "node --check bin/natureco.js && node bin/natureco.js help",
|
|
30
31
|
"smoke:context": "node scripts/e2e-context-smoke.js",
|
|
32
|
+
"smoke:browser": "node scripts/e2e-browser-smoke.js",
|
|
31
33
|
"postinstall": "node scripts/postinstall.js",
|
|
32
34
|
"prepublishOnly": "node --check bin/natureco.js && eslint src/ bin/ test/ --quiet && vitest run"
|
|
33
35
|
},
|
|
@@ -82,6 +84,7 @@
|
|
|
82
84
|
"node-telegram-bot-api": "^1.1.2",
|
|
83
85
|
"openai": "^6.45.0",
|
|
84
86
|
"pino": "^8.21.0",
|
|
87
|
+
"playwright-core": "^1.61.1",
|
|
85
88
|
"qrcode": "^1.5.4",
|
|
86
89
|
"qrcode-terminal": "^0.12.0",
|
|
87
90
|
"semver": "^7.8.1",
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
const browser = require('../src/tools/browser');
|
|
4
|
+
|
|
5
|
+
async function main() {
|
|
6
|
+
const opened = await browser.execute({ action: 'open', url: 'https://example.com', visible: false });
|
|
7
|
+
if (!opened.success) throw new Error('Open failed: ' + opened.error);
|
|
8
|
+
const snapshot = await browser.execute({ action: 'snapshot', visible: false });
|
|
9
|
+
if (!snapshot.success || snapshot.title !== 'Example Domain' || !snapshot.items.some(item => item.ref === '@e1')) {
|
|
10
|
+
throw new Error('Snapshot did not expose the expected page and element ref');
|
|
11
|
+
}
|
|
12
|
+
const closed = await browser.execute({ action: 'close' });
|
|
13
|
+
if (!closed.success) throw new Error('Close failed');
|
|
14
|
+
console.log(JSON.stringify({ ok: true, mode: opened.mode, title: snapshot.title, firstRef: snapshot.items[0] }));
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
main().catch(async error => {
|
|
18
|
+
try { await browser.execute({ action: 'close' }); } catch {}
|
|
19
|
+
console.error(error.message);
|
|
20
|
+
process.exitCode = 1;
|
|
21
|
+
});
|
package/src/tools/browser.js
CHANGED
|
@@ -1,112 +1,159 @@
|
|
|
1
|
-
const
|
|
1
|
+
const fs = require('fs');
|
|
2
|
+
const os = require('os');
|
|
3
|
+
const path = require('path');
|
|
2
4
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
const { chromium } = require('playwright');
|
|
6
|
-
const browser = await chromium.launch({ headless: true });
|
|
7
|
-
try {
|
|
8
|
-
const page = await browser.newPage();
|
|
9
|
-
|
|
10
|
-
if (action === 'open') {
|
|
11
|
-
await page.goto(params.url, { waitUntil: 'networkidle', timeout: 30000 });
|
|
12
|
-
const title = await page.title();
|
|
13
|
-
const content = await page.evaluate(() => document.body.innerText);
|
|
14
|
-
return { success: true, action, title, url: params.url, content: content.substring(0, 10000) };
|
|
15
|
-
}
|
|
5
|
+
let contextPromise = null;
|
|
6
|
+
let activePage = null;
|
|
16
7
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
8
|
+
function chromeCandidates(platform = process.platform) {
|
|
9
|
+
if (platform === 'darwin') return [
|
|
10
|
+
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
|
|
11
|
+
'/Applications/Chromium.app/Contents/MacOS/Chromium',
|
|
12
|
+
'/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge',
|
|
13
|
+
];
|
|
14
|
+
if (platform === 'win32') return [
|
|
15
|
+
path.join(process.env.PROGRAMFILES || '', 'Google/Chrome/Application/chrome.exe'),
|
|
16
|
+
path.join(process.env['PROGRAMFILES(X86)'] || '', 'Google/Chrome/Application/chrome.exe'),
|
|
17
|
+
path.join(process.env.LOCALAPPDATA || '', 'Google/Chrome/Application/chrome.exe'),
|
|
18
|
+
path.join(process.env.PROGRAMFILES || '', 'Microsoft/Edge/Application/msedge.exe'),
|
|
19
|
+
];
|
|
20
|
+
return ['/usr/bin/google-chrome', '/usr/bin/google-chrome-stable', '/usr/bin/chromium', '/usr/bin/chromium-browser', '/usr/bin/microsoft-edge'];
|
|
21
|
+
}
|
|
22
22
|
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
return { success: true, action, url: params.url, result: JSON.stringify(result) };
|
|
27
|
-
}
|
|
23
|
+
function findChrome(platform = process.platform, exists = fs.existsSync) {
|
|
24
|
+
return chromeCandidates(platform).find(candidate => candidate && exists(candidate)) || null;
|
|
25
|
+
}
|
|
28
26
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
27
|
+
function safeUrl(value) {
|
|
28
|
+
const parsed = new URL(String(value || ''));
|
|
29
|
+
if (!['http:', 'https:'].includes(parsed.protocol)) throw new Error('Only http/https URLs are allowed');
|
|
30
|
+
return parsed.href;
|
|
31
|
+
}
|
|
34
32
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
}
|
|
39
|
-
|
|
40
|
-
if (
|
|
41
|
-
|
|
42
|
-
}
|
|
43
|
-
|
|
44
|
-
|
|
33
|
+
async function getContext({ visible = true } = {}) {
|
|
34
|
+
if (contextPromise) return contextPromise;
|
|
35
|
+
contextPromise = (async () => {
|
|
36
|
+
const { chromium } = require('playwright-core');
|
|
37
|
+
const executablePath = findChrome();
|
|
38
|
+
if (!executablePath) throw new Error('Google Chrome, Chromium, or Microsoft Edge was not found');
|
|
39
|
+
const userDataDir = path.join(os.homedir(), '.natureco', 'browser-profile');
|
|
40
|
+
fs.mkdirSync(userDataDir, { recursive: true });
|
|
41
|
+
const context = await chromium.launchPersistentContext(userDataDir, {
|
|
42
|
+
executablePath,
|
|
43
|
+
headless: !visible,
|
|
44
|
+
viewport: null,
|
|
45
|
+
args: ['--no-first-run', '--no-default-browser-check'],
|
|
46
|
+
});
|
|
47
|
+
context.on('close', () => { contextPromise = null; activePage = null; });
|
|
48
|
+
const pages = context.pages();
|
|
49
|
+
activePage = pages[0] || await context.newPage();
|
|
50
|
+
return context;
|
|
51
|
+
})().catch(error => { contextPromise = null; throw error; });
|
|
52
|
+
return contextPromise;
|
|
45
53
|
}
|
|
46
54
|
|
|
47
|
-
async function
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
55
|
+
async function pageFor(params = {}) {
|
|
56
|
+
const context = await getContext({ visible: params.visible !== false });
|
|
57
|
+
if (!activePage || activePage.isClosed()) activePage = context.pages().find(page => !page.isClosed()) || await context.newPage();
|
|
58
|
+
return activePage;
|
|
59
|
+
}
|
|
52
60
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
61
|
+
async function snapshot(page) {
|
|
62
|
+
const result = await page.evaluate(() => {
|
|
63
|
+
document.querySelectorAll('[data-natureco-ref]').forEach(el => el.removeAttribute('data-natureco-ref'));
|
|
64
|
+
const selector = 'a[href],button,input,textarea,select,[role="button"],[role="link"],[contenteditable="true"],[tabindex]:not([tabindex="-1"])';
|
|
65
|
+
const elements = Array.from(document.querySelectorAll(selector)).filter(el => {
|
|
66
|
+
const style = getComputedStyle(el);
|
|
67
|
+
const rect = el.getBoundingClientRect();
|
|
68
|
+
return style.visibility !== 'hidden' && style.display !== 'none' && rect.width > 0 && rect.height > 0;
|
|
69
|
+
}).slice(0, 250);
|
|
70
|
+
const items = elements.map((el, index) => {
|
|
71
|
+
const ref = `e${index + 1}`;
|
|
72
|
+
el.setAttribute('data-natureco-ref', ref);
|
|
73
|
+
const role = el.getAttribute('role') || el.tagName.toLowerCase();
|
|
74
|
+
const name = (el.getAttribute('aria-label') || el.getAttribute('placeholder') || el.innerText || el.getAttribute('value') || el.getAttribute('title') || '').trim().replace(/\s+/g, ' ').slice(0, 160);
|
|
75
|
+
const type = el.getAttribute('type') || '';
|
|
76
|
+
return { ref: `@${ref}`, role, name, type, disabled: Boolean(el.disabled) };
|
|
65
77
|
});
|
|
66
|
-
|
|
67
|
-
|
|
78
|
+
return { title: document.title, url: location.href, items, text: (document.body?.innerText || '').trim().slice(0, 6000) };
|
|
79
|
+
});
|
|
80
|
+
return result;
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
function refSelector(ref) {
|
|
84
|
+
if (!/^@e\d+$/.test(String(ref || ''))) throw new Error('Use a fresh @e reference from browser snapshot');
|
|
85
|
+
return `[data-natureco-ref="${String(ref).slice(1)}"]`;
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
async function execute(params) {
|
|
89
|
+
const action = params.action === 'navigate' ? 'open' : params.action;
|
|
90
|
+
if (action === 'close') {
|
|
91
|
+
const context = contextPromise ? await contextPromise : null;
|
|
92
|
+
if (context) await context.close();
|
|
93
|
+
return { success: true, action: 'close' };
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
try {
|
|
97
|
+
const page = await pageFor(params);
|
|
98
|
+
if (action === 'open') {
|
|
99
|
+
const url = safeUrl(params.url);
|
|
100
|
+
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
|
|
101
|
+
await page.bringToFront();
|
|
102
|
+
return { success: true, action: 'open', url: page.url(), title: await page.title(), mode: params.visible === false ? 'headless-persistent' : 'headed-persistent' };
|
|
103
|
+
}
|
|
104
|
+
if (action === 'snapshot') return { success: true, action, ...(await snapshot(page)) };
|
|
105
|
+
if (action === 'click') {
|
|
106
|
+
await page.locator(refSelector(params.ref)).click({ timeout: 15000 });
|
|
107
|
+
await page.waitForTimeout(300);
|
|
108
|
+
return { success: true, action, ref: params.ref, url: page.url() };
|
|
109
|
+
}
|
|
110
|
+
if (action === 'fill') {
|
|
111
|
+
if (typeof params.text !== 'string') throw new Error('fill requires text');
|
|
112
|
+
await page.locator(refSelector(params.ref)).fill(params.text, { timeout: 15000 });
|
|
113
|
+
return { success: true, action, ref: params.ref };
|
|
114
|
+
}
|
|
115
|
+
if (action === 'type') {
|
|
116
|
+
if (typeof params.text !== 'string') throw new Error('type requires text');
|
|
117
|
+
await page.keyboard.type(params.text);
|
|
118
|
+
return { success: true, action };
|
|
119
|
+
}
|
|
120
|
+
if (action === 'press') {
|
|
121
|
+
if (!params.key) throw new Error('press requires key');
|
|
122
|
+
await page.keyboard.press(params.key);
|
|
123
|
+
await page.waitForTimeout(300);
|
|
124
|
+
return { success: true, action, key: params.key, url: page.url() };
|
|
125
|
+
}
|
|
126
|
+
if (action === 'text') return { success: true, action, url: page.url(), content: (await page.locator('body').innerText()).slice(0, 10000) };
|
|
127
|
+
if (action === 'current_url') return { success: true, action, url: page.url(), title: await page.title() };
|
|
128
|
+
if (action === 'screenshot') {
|
|
129
|
+
const buffer = await page.screenshot({ type: 'png', fullPage: params.fullPage !== false });
|
|
130
|
+
return { success: true, action, url: page.url(), screenshot: buffer.toString('base64'), format: 'png' };
|
|
131
|
+
}
|
|
132
|
+
if (action === 'html') return { success: true, action, url: page.url(), html: (await page.content()).slice(0, 50000) };
|
|
133
|
+
if (action === 'evaluate') {
|
|
134
|
+
if (!params.script) throw new Error('evaluate requires script');
|
|
135
|
+
const result = await page.evaluate(params.script);
|
|
136
|
+
return { success: true, action, url: page.url(), result };
|
|
137
|
+
}
|
|
138
|
+
return { success: false, error: `Unknown browser action: ${params.action}` };
|
|
139
|
+
} catch (error) {
|
|
140
|
+
return { success: false, error: error.message };
|
|
68
141
|
}
|
|
69
142
|
}
|
|
70
143
|
|
|
71
144
|
module.exports = {
|
|
72
145
|
name: 'browser',
|
|
73
|
-
description: '
|
|
146
|
+
description: 'Persistent Chrome/Chromium agent. Use open → snapshot → @ref click/fill → snapshot. Visible by default; preserves login and storage in the NatureCo browser profile.',
|
|
74
147
|
inputSchema: {
|
|
75
148
|
type: 'object',
|
|
76
149
|
properties: {
|
|
77
|
-
action: { type: 'string',
|
|
78
|
-
url: { type: 'string', description: '
|
|
79
|
-
|
|
80
|
-
|
|
150
|
+
action: { type: 'string', enum: ['open', 'navigate', 'snapshot', 'click', 'fill', 'type', 'press', 'text', 'current_url', 'screenshot', 'html', 'evaluate', 'close'] },
|
|
151
|
+
url: { type: 'string' }, ref: { type: 'string', description: '@e reference from the latest snapshot' },
|
|
152
|
+
text: { type: 'string' }, key: { type: 'string' }, script: { type: 'string' },
|
|
153
|
+
visible: { type: 'boolean', description: 'Show browser window (default true)' }, fullPage: { type: 'boolean' },
|
|
81
154
|
},
|
|
82
|
-
required: ['action',
|
|
155
|
+
required: ['action'],
|
|
83
156
|
},
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
if (params.action === 'open') {
|
|
87
|
-
const pw = await tryPlaywright('open', params);
|
|
88
|
-
if (pw.fallback) return httpFallback(params.url);
|
|
89
|
-
return pw;
|
|
90
|
-
}
|
|
91
|
-
|
|
92
|
-
if (params.action === 'screenshot') {
|
|
93
|
-
const pw = await tryPlaywright('screenshot', params);
|
|
94
|
-
if (pw.fallback) return { success: false, error: 'Screenshot için Playwright gerekli. Kur: npm install playwright' };
|
|
95
|
-
return pw;
|
|
96
|
-
}
|
|
97
|
-
|
|
98
|
-
if (params.action === 'evaluate') {
|
|
99
|
-
const pw = await tryPlaywright('evaluate', params);
|
|
100
|
-
if (pw.fallback) return { success: false, error: 'JS evaluate için Playwright gerekli. Kur: npm install playwright' };
|
|
101
|
-
return pw;
|
|
102
|
-
}
|
|
103
|
-
|
|
104
|
-
if (params.action === 'html') {
|
|
105
|
-
const pw = await tryPlaywright('html', params);
|
|
106
|
-
if (pw.fallback) return httpFallback(params.url);
|
|
107
|
-
return pw;
|
|
108
|
-
}
|
|
109
|
-
|
|
110
|
-
return { success: false, error: `Unknown action: ${params.action}` };
|
|
111
|
-
}
|
|
157
|
+
execute,
|
|
158
|
+
_test: { chromeCandidates, findChrome, safeUrl, refSelector },
|
|
112
159
|
};
|
|
@@ -133,17 +133,41 @@ function evaluateCompletionEvidence({ mutationCount, initialHash, currentHash, v
|
|
|
133
133
|
async function verifyGoal(providerUrl, apiKey, model, goal, screenshot) {
|
|
134
134
|
const prompt = `You are a strict independent verifier. Inspect only the screenshot. Goal: ${goal}\nReturn JSON only: {"verified":boolean,"confidence":0..1,"evidence":"exact visible evidence","reason":"why not"}. Never infer success from the goal text. If the requested sent message, booking, purchase, or confirmation is not visibly present, verified must be false.`;
|
|
135
135
|
const reply = await visionCall(providerUrl, apiKey, model, prompt, screenshot);
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
136
|
+
const parsed = parseVisionDecision(reply);
|
|
137
|
+
return parsed.success ? parsed.value : { verified: false, confidence: 0, reason: 'Verifier returned invalid JSON: ' + parsed.error };
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
function parseVisionDecision(reply) {
|
|
141
|
+
const raw = String(reply || '').trim().replace(/^```(?:json)?\s*/i, '').replace(/\s*```$/, '');
|
|
142
|
+
const candidates = [raw];
|
|
143
|
+
const start = raw.indexOf('{');
|
|
144
|
+
const end = raw.lastIndexOf('}');
|
|
145
|
+
if (start >= 0 && end > start) candidates.push(raw.slice(start, end + 1));
|
|
146
|
+
for (const candidate of candidates) {
|
|
147
|
+
try {
|
|
148
|
+
const value = JSON.parse(candidate);
|
|
149
|
+
if (value && typeof value === 'object' && !Array.isArray(value)) return { success: true, value };
|
|
150
|
+
} catch {}
|
|
140
151
|
}
|
|
152
|
+
return { success: false, error: raw ? 'malformed or truncated JSON' : 'empty vision response' };
|
|
153
|
+
}
|
|
154
|
+
|
|
155
|
+
function validateAction(action, params = {}) {
|
|
156
|
+
const finite = value => typeof value === 'number' && Number.isFinite(value);
|
|
157
|
+
if (!['click', 'type', 'keypress', 'mouse_move', 'scroll', 'wait', 'done'].includes(action)) return 'Unknown action: ' + action;
|
|
158
|
+
if ((action === 'click' || action === 'mouse_move') && (!finite(params.x) || !finite(params.y))) return action + ' requires finite numeric x and y';
|
|
159
|
+
if (action === 'scroll' && !finite(params.y)) return 'scroll requires a finite numeric y';
|
|
160
|
+
if (action === 'type' && typeof params.text !== 'string') return 'type requires text';
|
|
161
|
+
if (action === 'keypress' && (typeof params.key !== 'string' || !params.key.trim())) return 'keypress requires key';
|
|
162
|
+
return null;
|
|
141
163
|
}
|
|
142
164
|
|
|
143
165
|
function executeAction(action, params) {
|
|
144
166
|
const { spawnSync } = require('child_process');
|
|
145
167
|
const PLATFORM = os.platform();
|
|
146
168
|
|
|
169
|
+
const invalid = validateAction(action, params);
|
|
170
|
+
if (invalid) return { success: false, error: invalid };
|
|
147
171
|
const ESC = (s) => s.replace(/\\/g, '\\\\').replace(/"/g, '\\"');
|
|
148
172
|
|
|
149
173
|
function osaScript(script, timeoutMs = 10000) {
|
|
@@ -232,15 +256,17 @@ async function loop(goal, maxSteps) {
|
|
|
232
256
|
const history = steps.filter(s => s.action && s.action !== 'screenshot' && s.action !== 'done').slice(-5);
|
|
233
257
|
const historyText = history.length > 0 ? '\nOnceki adimlar:\n' + history.map(h => 'Adim ' + h.step + ': ' + JSON.stringify(h)).join('\n') : '';
|
|
234
258
|
const reply = await visionCall(providerUrl, apiKey, model, SYSTEM_PROMPT + '\n\nGorev: ' + goal + historyText + '\n\nEkran goruntusunu analiz et. Siradaki action ne?', screenshot);
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
259
|
+
const parsedDecision = parseVisionDecision(reply);
|
|
260
|
+
if (!parsedDecision.success) {
|
|
261
|
+
steps.push({ step: i + 1, action: 'vision_retry', error: parsedDecision.error });
|
|
262
|
+
continue;
|
|
263
|
+
}
|
|
264
|
+
const decision = parsedDecision.value;
|
|
265
|
+
const invalidDecision = validateAction(decision.action, decision);
|
|
266
|
+
if (invalidDecision) {
|
|
267
|
+
steps.push({ step: i + 1, action: 'vision_retry', error: invalidDecision });
|
|
268
|
+
continue;
|
|
241
269
|
}
|
|
242
|
-
|
|
243
|
-
if (!decision.action) decision.action = 'wait';
|
|
244
270
|
|
|
245
271
|
// 3. Execute
|
|
246
272
|
if (decision.action === 'done') {
|
|
@@ -307,4 +333,4 @@ async function execute(params) {
|
|
|
307
333
|
return await loop(params.goal, params.maxSteps || 30);
|
|
308
334
|
}
|
|
309
335
|
|
|
310
|
-
module.exports = { name, description, parameters, execute, executeAction, evaluateCompletionEvidence, resolveVisionConfig, visionCall };
|
|
336
|
+
module.exports = { name, description, parameters, execute, executeAction, evaluateCompletionEvidence, resolveVisionConfig, visionCall, parseVisionDecision, validateAction };
|
package/src/tools/workflow.js
CHANGED
|
@@ -243,13 +243,13 @@ async function workflow(params) {
|
|
|
243
243
|
'\n\nTAM MOD ACIK — su araclara da ERISIMIN VAR; o an ne gerekiyorsa dogrudan cagir:',
|
|
244
244
|
'- mac_app_open: macOS uygulamasi ac. parametre: appName (orn. "WhatsApp", "Google Chrome", "Spotify")',
|
|
245
245
|
'- mac_app_quit: macOS uygulamasi kapat. parametre: appName',
|
|
246
|
-
'- browser:
|
|
246
|
+
'- browser: kalici Chrome/Chromium ajani; varsayilan GORUNUR ve login/storage korur. Sira: open(url) → snapshot → snapshot\'taki @e ref ile click/fill → tekrar snapshot ile dogrula. action=open|snapshot|click|fill|type|press|text|current_url|screenshot|html|evaluate|close.',
|
|
247
247
|
'- browser_use: ayri bulut/CLI tarayici servisi; yalniz kurulu ve hazirsa kullan. Acik Chrome penceresini kontrol etmek icin kullanma.',
|
|
248
248
|
'- computer_use: GUI otomasyonu. parametreler: action ("screenshot"/"click"/"type"/"keypress"/"scroll"), x, y, text, key',
|
|
249
249
|
'- computer_use_loop: GORUNUR, cok-adimli GUI icin TEK tercih. p: goal, maxSteps?. Kendi screenshot→vision→action→dogrulama dongusunu yapar.',
|
|
250
250
|
'- social_open: muzik/video/sosyal ac. parametreler: query, platform (spotify/youtube...)',
|
|
251
251
|
'- macos_screenshot: ekran goruntusu al',
|
|
252
|
-
'\
|
|
252
|
+
'\nTARAYICI SECIMI: Kullanici "Chromium/tarayici ajani" derse kalici `browser` kullan. "Su anda acik olan kendi Chrome pencerem" derse yalniz `computer_use_loop` kullan; ayri browser profili acma. Muzik/video yalniz acilacaksa dogrudan URL\'yi `open` ile ac.',
|
|
253
253
|
'\nTum arac listesi (isimle cagir; parametre yanlissa <tool_results> duzeltir): ' + allNames.join(', '),
|
|
254
254
|
].join('\n');
|
|
255
255
|
}
|
|
@@ -302,7 +302,7 @@ async function workflow(params) {
|
|
|
302
302
|
'- Dosya degistirmeden ONCE read_file ile oku, edit_file ile hedefli degistir. Yerini bilmiyorsan file_search/grep_search ile kesfet.',
|
|
303
303
|
'- Kod yazinca/degistirince bash ile calistir/test et, hata varsa duzelt. Coklu dosya = her biri icin AYRI write_file. Hep TAM yol; "masaustu" = ' + desktop + '.',
|
|
304
304
|
'- Arac sonuclari <tool_results> icinde doner; is bitince arac cagirmadan tek cumlelik ozet yaz. Basit sohbette arac cagirma, dogrudan yanitla.',
|
|
305
|
-
execFull ? '-
|
|
305
|
+
execFull ? '- Tarayici gorevinde kullanicinin secimini izle: "Chromium/browser agent" → browser ile open→snapshot→ref action→snapshot; "acik kendi Chrome pencerem" → SADECE computer_use_loop. Bir yol hata verince ayni turda kor fallback zinciri kurma. Mesaj/gonderim/satin alma sonucunu son snapshot veya GUI kaniti olmadan tamamlandi deme.' : '',
|
|
306
306
|
fullToolsBlock,
|
|
307
307
|
treeDigest ? ('\n\nBILDIGIN KALICI HAFIZA (bu kullaniciya ait, onceki oturumlardan hatirladiklarin; kullaniciya ozel bir sey sorulursa ONCE BUNU KULLAN — dosya arama, uydurma):\n' + treeDigest) : '',
|
|
308
308
|
treeIndex ? ('\n\nHafiza agaci yapisi (yukarida olmayan detay icin memory_tree(action:read/search) ile ilgili kok/dali oku):\n' + treeIndex) : '',
|
package/src/utils/tools.js
CHANGED
|
@@ -132,10 +132,10 @@ const TOOLSET_MAP = {
|
|
|
132
132
|
};
|
|
133
133
|
|
|
134
134
|
// ── check_fn'ler (tool availability kontrolleri) ────────────────────────
|
|
135
|
-
function _checkBrowser() {
|
|
136
|
-
try {
|
|
137
|
-
require.resolve('playwright');
|
|
138
|
-
return true;
|
|
135
|
+
function _checkBrowser() {
|
|
136
|
+
try {
|
|
137
|
+
require.resolve('playwright-core');
|
|
138
|
+
return true;
|
|
139
139
|
} catch { return false; }
|
|
140
140
|
}
|
|
141
141
|
|