testronaut 2.1.1 → 2.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/TODO.md +10 -0
- package/package.json +1 -1
- package/tests/toolsTests/chromeBrowserClickText.test.js +40 -0
- package/tests/toolsTests/chromeBrowserGetDomFrames.test.js +33 -0
- package/tests/toolsTests/chromeBrowserPressKey.test.js +39 -0
- package/tests/toolsTests/domControl.test.js +1 -1
- package/tests/toolsTests/toolSchema.test.js +14 -0
- package/tools/chromeBrowser.js +103 -20
- package/tools/domControl.js +5 -3
- package/tools/toolSchema.js +28 -0
- package/todo.md +0 -25
package/TODO.md
CHANGED
|
@@ -12,6 +12,7 @@
|
|
|
12
12
|
## Browser Automation Follow-Ups
|
|
13
13
|
|
|
14
14
|
- [x] Make `click_text` match exact text, prefer the innermost visible element, and fail quickly instead of clicking an ancestor or partial match.
|
|
15
|
+
- [ ] Complete `press_key` turn-loop integration: classify it as fire-and-forget, automatically inject the updated DOM after successful key presses, and test that Enter/Escape state changes are available to the next reasoning turn without an explicit `get_dom` call.
|
|
15
16
|
- [ ] Replace framework/vendor-specific DOM class stripping with a framework-agnostic classifier that removes generated hashes, styling utilities, variants, and arbitrary selectors while preserving semantic component classes. Reconsider broad prefix rules such as `/^cl-/`, cap retained class count and length, and add representative fixtures for Tailwind, MUI, Ant Design, Chakra, Clerk, CSS Modules, and CSS-in-JS.
|
|
16
17
|
|
|
17
18
|
## Session and Reporting Follow-Ups
|
|
@@ -27,9 +28,18 @@
|
|
|
27
28
|
|
|
28
29
|
## Optimization and Rate-Limit Follow-Ups
|
|
29
30
|
|
|
31
|
+
- [ ] Reduce redundant screenshot capture in focused missions without weakening evidence: clarify or enforce when starting, transition, and final screenshots materially differ, and measure the effect on turns, tokens, and report usefulness.
|
|
30
32
|
- [ ] Run a history-informed model-routing benchmark before expanding cross-provider routing. Give Jev versioned, per-model/per-mission-class reliability, token, latency, and price observations from prior runs; require a configurable reliability floor; compare against the best fixed-model-plus-guardrails control with blocked repetitions; keep selection at the mission boundary; record the evidence and reason for every choice; and define cold-start, stale-data, low-confidence, and provider-failure fallbacks. Do not retry a failed mission on another provider in the benchmark because that would confound routing quality with retry recovery.
|
|
31
33
|
- [ ] Add an opt-in CLI/config policy that reroutes an imminent throttled LLM request to a compatible model as an alternative to the default backoff behavior. Keep backoff as the default; use observed TPM/RPM headroom and projected request tokens; constrain failover to explicitly configured provider/model families; preserve tool/history compatibility; prevent routing loops; fall back safely when the alternate model is unavailable; and record the original model, routed model, trigger, token usage, latency, and outcome in reports.
|
|
32
34
|
|
|
35
|
+
## OpenAI Provider Follow-Ups
|
|
36
|
+
|
|
37
|
+
- [ ] Migrate the OpenAI adapter from Chat Completions to the Responses API in a dedicated change. Preserve Testronaut's provider-neutral message and tool-call contract, add response-ID/reasoning-state handling, normalize usage and raw headers, and cover multi-turn tool continuation with unit and API smoke tests. Once migrated, remove the GPT-5.6 Chat Completions compatibility override that sets `reasoning_effort` to `none` for function-tool requests.
|
|
38
|
+
- [ ] Add active context budgeting after its behavior is designed and tested. Use model context/output metadata to reserve response and reasoning capacity, then apply explicit history compaction or summarization rules. Do not silently truncate mission instructions, current DOM state, or unmatched tool calls.
|
|
39
|
+
- [ ] Learn effective OpenAI TPM/RPM limits from successful response headers and rate-limit errors. Prefer observed project/model limits over inferring a named account tier; retain the environment override and conservative startup values.
|
|
40
|
+
- [ ] Persist provider-observed rate limits in a separate, expiring local cache keyed by provider, API project/account, endpoint/region, model, and traffic type. Never rewrite user config or infer a named usage tier from a TPM value; expose configured, cached, and live sources distinctly and discard stale observations.
|
|
41
|
+
- [ ] After dynamic rate limits are established, make DOM injection quota-aware. Budget DOM tokens against both remaining TPM and the model context window, preserve mission instructions and actionable controls, and fall back to semantic sections plus on-demand DOM chunks instead of silent truncation.
|
|
42
|
+
|
|
33
43
|
## Automated MFA Follow-Ups
|
|
34
44
|
|
|
35
45
|
- [ ] Add an integration smoke test against a local or staging API fixture for `get_mfa_code`.
|
package/package.json
CHANGED
|
@@ -69,4 +69,44 @@ describe('ChromeBrowser.check_text', () => {
|
|
|
69
69
|
|
|
70
70
|
await expect(browser.check_text({ text: 'Hidden message' })).resolves.toBe('NOT FOUND: Hidden message');
|
|
71
71
|
});
|
|
72
|
+
|
|
73
|
+
it('reports the iframe containing a visible match', async () => {
|
|
74
|
+
const noMatches = { count: vi.fn().mockResolvedValue(0) };
|
|
75
|
+
const frameMatches = {
|
|
76
|
+
count: vi.fn().mockResolvedValue(1),
|
|
77
|
+
nth: vi.fn().mockReturnValue({ isVisible: vi.fn().mockResolvedValue(true) }),
|
|
78
|
+
};
|
|
79
|
+
const mainFrame = { getByText: vi.fn().mockReturnValue(noMatches) };
|
|
80
|
+
const childFrame = {
|
|
81
|
+
getByText: vi.fn().mockReturnValue(frameMatches),
|
|
82
|
+
name: vi.fn().mockReturnValue(''),
|
|
83
|
+
frameElement: vi.fn().mockResolvedValue({ getAttribute: vi.fn().mockResolvedValue('webviewer-1') }),
|
|
84
|
+
url: vi.fn().mockReturnValue('https://example.test/viewer'),
|
|
85
|
+
};
|
|
86
|
+
const browser = Object.create(ChromeBrowser.prototype);
|
|
87
|
+
browser.page = {
|
|
88
|
+
mainFrame: vi.fn().mockReturnValue(mainFrame),
|
|
89
|
+
frames: vi.fn().mockReturnValue([mainFrame, childFrame]),
|
|
90
|
+
};
|
|
91
|
+
|
|
92
|
+
await expect(browser.check_text({ text: 'Quarterly report' }))
|
|
93
|
+
.resolves.toBe('FOUND in frame webviewer-1: Quarterly report');
|
|
94
|
+
});
|
|
95
|
+
});
|
|
96
|
+
|
|
97
|
+
describe('ChromeBrowser iframe interaction targeting', () => {
|
|
98
|
+
it('builds an explicit frame-qualified click locator', () => {
|
|
99
|
+
const locator = { first: vi.fn().mockReturnThis() };
|
|
100
|
+
const frameContext = { locator: vi.fn().mockReturnValue(locator) };
|
|
101
|
+
const browser = Object.create(ChromeBrowser.prototype);
|
|
102
|
+
browser.page = { frameLocator: vi.fn().mockReturnValue(frameContext) };
|
|
103
|
+
|
|
104
|
+
const [make] = browser._buildClickCandidates({
|
|
105
|
+
frame: 'iframe#webviewer-1', selector: 'button[aria-label="Settings"]', searchAllFrames: true,
|
|
106
|
+
});
|
|
107
|
+
|
|
108
|
+
expect(make()).toBe(locator);
|
|
109
|
+
expect(browser.page.frameLocator).toHaveBeenCalledWith('iframe#webviewer-1');
|
|
110
|
+
expect(frameContext.locator).toHaveBeenCalledWith('button[aria-label="Settings"]');
|
|
111
|
+
});
|
|
72
112
|
});
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
import { describe, expect, it, vi } from 'vitest';
|
|
2
|
+
import { ChromeBrowser } from '../../tools/chromeBrowser.js';
|
|
3
|
+
|
|
4
|
+
describe('ChromeBrowser.get_dom iframe serialization', () => {
|
|
5
|
+
it('keeps serialized same-origin content and inaccessible-frame markers', async () => {
|
|
6
|
+
const browser = Object.create(ChromeBrowser.prototype);
|
|
7
|
+
browser.domListLimit = 3;
|
|
8
|
+
browser.debugEnabled = false;
|
|
9
|
+
browser.context = null;
|
|
10
|
+
browser.page = {
|
|
11
|
+
isClosed: vi.fn().mockReturnValue(false),
|
|
12
|
+
waitForLoadState: vi.fn().mockResolvedValue(undefined),
|
|
13
|
+
content: vi.fn().mockResolvedValue('<html><body></body></html>'),
|
|
14
|
+
evaluate: vi.fn().mockResolvedValue(`
|
|
15
|
+
<html><body>
|
|
16
|
+
<testronaut-frame id="viewer" data-testronaut-frame="viewer" data-testronaut-frame-status="same-origin">
|
|
17
|
+
<html><body><button aria-label="Settings">Settings</button><p>Quarterly report</p></body></html>
|
|
18
|
+
</testronaut-frame>
|
|
19
|
+
<testronaut-frame id="payment" data-testronaut-frame="payment" data-testronaut-frame-status="cross-origin">
|
|
20
|
+
[cross-origin frame, content not accessible]
|
|
21
|
+
</testronaut-frame>
|
|
22
|
+
</body></html>`),
|
|
23
|
+
};
|
|
24
|
+
|
|
25
|
+
const result = await browser.get_dom();
|
|
26
|
+
|
|
27
|
+
expect(result).toContain('data-testronaut-frame="viewer"');
|
|
28
|
+
expect(result).toContain('<iframe');
|
|
29
|
+
expect(result).toContain('Quarterly report');
|
|
30
|
+
expect(result).toContain('aria-label="Settings"');
|
|
31
|
+
expect(result).toContain('[cross-origin frame, content not accessible]');
|
|
32
|
+
});
|
|
33
|
+
});
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
import { describe, expect, it, vi } from 'vitest';
|
|
2
|
+
import { CHROME_TOOL_MAP, ChromeBrowser } from '../../tools/chromeBrowser.js';
|
|
3
|
+
|
|
4
|
+
describe('ChromeBrowser.press_key', () => {
|
|
5
|
+
it('presses a key on the active page', async () => {
|
|
6
|
+
const browser = new ChromeBrowser();
|
|
7
|
+
browser.page = { keyboard: { press: vi.fn().mockResolvedValue(undefined) } };
|
|
8
|
+
|
|
9
|
+
await expect(browser.press_key({ key: 'Escape' })).resolves.toBe('pressed Escape');
|
|
10
|
+
expect(browser.page.keyboard.press).toHaveBeenCalledWith('Escape');
|
|
11
|
+
});
|
|
12
|
+
|
|
13
|
+
it('focuses a selected element before pressing Enter', async () => {
|
|
14
|
+
const browser = new ChromeBrowser();
|
|
15
|
+
const focus = vi.fn().mockResolvedValue(undefined);
|
|
16
|
+
const first = vi.fn().mockReturnValue({ focus });
|
|
17
|
+
browser.page = {
|
|
18
|
+
locator: vi.fn().mockReturnValue({ first }),
|
|
19
|
+
keyboard: { press: vi.fn().mockResolvedValue(undefined) },
|
|
20
|
+
};
|
|
21
|
+
|
|
22
|
+
await expect(browser.press_key({ key: 'Enter', selector: '#email' }))
|
|
23
|
+
.resolves.toBe('pressed Enter after focusing #email');
|
|
24
|
+
|
|
25
|
+
expect(browser.page.locator).toHaveBeenCalledWith('#email');
|
|
26
|
+
expect(first).toHaveBeenCalledOnce();
|
|
27
|
+
expect(focus).toHaveBeenCalledWith({ timeout: 3000 });
|
|
28
|
+
expect(browser.page.keyboard.press).toHaveBeenCalledWith('Enter');
|
|
29
|
+
expect(focus.mock.invocationCallOrder[0])
|
|
30
|
+
.toBeLessThan(browser.page.keyboard.press.mock.invocationCallOrder[0]);
|
|
31
|
+
});
|
|
32
|
+
|
|
33
|
+
it('is exposed through the browser tool dispatcher', async () => {
|
|
34
|
+
const press_key = vi.fn().mockResolvedValue('pressed Control+B');
|
|
35
|
+
await expect(CHROME_TOOL_MAP.press_key({ press_key }, { key: 'Control+B' }))
|
|
36
|
+
.resolves.toBe('pressed Control+B');
|
|
37
|
+
expect(press_key).toHaveBeenCalledWith({ key: 'Control+B' });
|
|
38
|
+
});
|
|
39
|
+
});
|
|
@@ -56,7 +56,7 @@ describe('domControl helpers', () => {
|
|
|
56
56
|
expect(result).not.toContain('<style');
|
|
57
57
|
expect(result).not.toContain('<meta');
|
|
58
58
|
expect(result).not.toContain('<link');
|
|
59
|
-
expect(result).
|
|
59
|
+
expect(result).toContain('<iframe src="frame.html"></iframe>');
|
|
60
60
|
expect(result).not.toContain('<canvas');
|
|
61
61
|
expect(result).not.toContain('<svg');
|
|
62
62
|
expect(result).not.toContain('<!-- comment -->');
|
|
@@ -21,6 +21,20 @@ describe('tools/toolSchema', () => {
|
|
|
21
21
|
const entry = toolsSchema.find(tool => tool.function.name === 'get_dom');
|
|
22
22
|
expect(entry.function.parameters.properties.selector.type).toBe('string');
|
|
23
23
|
});
|
|
24
|
+
it('includes press_key with a required key and optional selector', () => {
|
|
25
|
+
const entry = toolsSchema.find(tool => tool.function.name === 'press_key');
|
|
26
|
+
expect(entry?.function.parameters.required).toEqual(['key']);
|
|
27
|
+
expect(entry.function.parameters.properties.key.type).toBe('string');
|
|
28
|
+
expect(entry.function.parameters.properties.selector.type).toBe('string');
|
|
29
|
+
});
|
|
30
|
+
it('exposes iframe targeting and iframe text search', () => {
|
|
31
|
+
const fill = toolsSchema.find(tool => tool.function.name === 'fill');
|
|
32
|
+
const click = toolsSchema.find(tool => tool.function.name === 'click');
|
|
33
|
+
const checkText = toolsSchema.find(tool => tool.function.name === 'check_text');
|
|
34
|
+
expect(fill.function.parameters.properties.frame.type).toBe('string');
|
|
35
|
+
expect(click.function.parameters.properties.frame.type).toBe('string');
|
|
36
|
+
expect(checkText.function.parameters.properties.searchAllFrames.default).toBe(true);
|
|
37
|
+
});
|
|
24
38
|
it('includes resource_progress tool with empty params', () => {
|
|
25
39
|
const entry = toolsSchema.find(t => t.function?.name === 'resource_progress');
|
|
26
40
|
expect(entry?.type).toBe('function');
|
package/tools/chromeBrowser.js
CHANGED
|
@@ -180,8 +180,13 @@ export class ChromeBrowser {
|
|
|
180
180
|
}
|
|
181
181
|
|
|
182
182
|
// Build candidate locator MAKERS (Page|Frame-aware)
|
|
183
|
-
|
|
184
|
-
|
|
183
|
+
_interactionContexts(frame, searchAllFrames) {
|
|
184
|
+
if (frame) return [this.page.frameLocator(frame)];
|
|
185
|
+
return searchAllFrames ? this._allFrames({ includeMain: true }) : [this.page.mainFrame()];
|
|
186
|
+
}
|
|
187
|
+
|
|
188
|
+
_buildClickCandidates({ selector, text, role, attrs, scope, searchAllFrames, frame }) {
|
|
189
|
+
const frames = this._interactionContexts(frame, searchAllFrames);
|
|
185
190
|
const makers = [];
|
|
186
191
|
const add = (fn) => fn && makers.push(fn);
|
|
187
192
|
|
|
@@ -211,8 +216,9 @@ export class ChromeBrowser {
|
|
|
211
216
|
}
|
|
212
217
|
}
|
|
213
218
|
|
|
214
|
-
|
|
215
|
-
|
|
219
|
+
// Do not de-duplicate by function source: closures for different frames have
|
|
220
|
+
// identical source text but resolve against different documents.
|
|
221
|
+
return makers;
|
|
216
222
|
}
|
|
217
223
|
|
|
218
224
|
|
|
@@ -459,8 +465,8 @@ export class ChromeBrowser {
|
|
|
459
465
|
}
|
|
460
466
|
|
|
461
467
|
// Build candidates for text entry (Page|Frame-aware)
|
|
462
|
-
_buildFillCandidates({ selector, placeholder, role = 'textbox', attrs, scope, searchAllFrames }) {
|
|
463
|
-
const frames =
|
|
468
|
+
_buildFillCandidates({ selector, placeholder, role = 'textbox', attrs, scope, searchAllFrames, frame }) {
|
|
469
|
+
const frames = this._interactionContexts(frame, searchAllFrames);
|
|
464
470
|
const makers = [];
|
|
465
471
|
const add = (fn) => fn && makers.push(fn);
|
|
466
472
|
|
|
@@ -499,9 +505,9 @@ export class ChromeBrowser {
|
|
|
499
505
|
}
|
|
500
506
|
}
|
|
501
507
|
|
|
502
|
-
// de-
|
|
503
|
-
|
|
504
|
-
return makers
|
|
508
|
+
// Do not de-duplicate by function source: closures for different frames have
|
|
509
|
+
// identical source text but resolve against different documents.
|
|
510
|
+
return makers;
|
|
505
511
|
}
|
|
506
512
|
|
|
507
513
|
|
|
@@ -714,6 +720,7 @@ export class ChromeBrowser {
|
|
|
714
720
|
async fill({
|
|
715
721
|
selector,
|
|
716
722
|
text,
|
|
723
|
+
frame,
|
|
717
724
|
placeholder, // string | RegExp
|
|
718
725
|
role = 'textbox', // generic default
|
|
719
726
|
attrs,
|
|
@@ -729,9 +736,9 @@ export class ChromeBrowser {
|
|
|
729
736
|
|
|
730
737
|
const makeCandidates = () => {
|
|
731
738
|
if (strategy === 'strict') {
|
|
732
|
-
return this._buildFillCandidates({ selector, scope, searchAllFrames });
|
|
739
|
+
return this._buildFillCandidates({ selector, scope, searchAllFrames, frame });
|
|
733
740
|
}
|
|
734
|
-
return this._buildFillCandidates({ selector, placeholder, role, attrs, scope, searchAllFrames });
|
|
741
|
+
return this._buildFillCandidates({ selector, placeholder, role, attrs, scope, searchAllFrames, frame });
|
|
735
742
|
};
|
|
736
743
|
|
|
737
744
|
return this._withRetries(async () => {
|
|
@@ -764,6 +771,18 @@ export class ChromeBrowser {
|
|
|
764
771
|
}, { attempts: 3, baseMs: 250 });
|
|
765
772
|
}
|
|
766
773
|
|
|
774
|
+
async press_key({ key, selector }) {
|
|
775
|
+
if (selector) {
|
|
776
|
+
const target = this.page.locator(selector).first();
|
|
777
|
+
await target.focus({ timeout: 3000 });
|
|
778
|
+
}
|
|
779
|
+
|
|
780
|
+
await this.page.keyboard.press(key);
|
|
781
|
+
return selector
|
|
782
|
+
? `pressed ${key} after focusing ${selector}`
|
|
783
|
+
: `pressed ${key}`;
|
|
784
|
+
}
|
|
785
|
+
|
|
767
786
|
|
|
768
787
|
|
|
769
788
|
/**
|
|
@@ -778,7 +797,7 @@ export class ChromeBrowser {
|
|
|
778
797
|
* @param {number} [opts.delayMs=200] // micro settle
|
|
779
798
|
*/
|
|
780
799
|
async click({
|
|
781
|
-
selector, text, role, attrs, scope,
|
|
800
|
+
selector, text, role, attrs, scope, frame,
|
|
782
801
|
searchAllFrames = true,
|
|
783
802
|
strategy = 'auto',
|
|
784
803
|
waitFor,
|
|
@@ -794,8 +813,8 @@ export class ChromeBrowser {
|
|
|
794
813
|
|
|
795
814
|
// Build your candidates the way you already do (generic/hint-driven)
|
|
796
815
|
const makers = (strategy === 'strict')
|
|
797
|
-
? this._buildClickCandidates({ selector, scope, searchAllFrames })
|
|
798
|
-
: this._buildClickCandidates({ selector, text, role, attrs, scope, searchAllFrames });
|
|
816
|
+
? this._buildClickCandidates({ selector, scope, searchAllFrames, frame })
|
|
817
|
+
: this._buildClickCandidates({ selector, text, role, attrs, scope, searchAllFrames, frame });
|
|
799
818
|
|
|
800
819
|
// choose & click (same as your generic version)
|
|
801
820
|
let chosen = null;
|
|
@@ -1014,9 +1033,57 @@ export class ChromeBrowser {
|
|
|
1014
1033
|
// Fallback to empty string if still nothing (keeps caller logic simple)
|
|
1015
1034
|
if (!html) html = '';
|
|
1016
1035
|
|
|
1036
|
+
// page.content() only serializes iframe elements, not their documents. Build a
|
|
1037
|
+
// read-only clone that embeds accessible frame documents so selectors and the
|
|
1038
|
+
// global output limit apply to the complete snapshot.
|
|
1039
|
+
try {
|
|
1040
|
+
html = await this.page.evaluate(() => {
|
|
1041
|
+
const serializeDocument = (doc) => {
|
|
1042
|
+
const clone = doc.documentElement.cloneNode(true);
|
|
1043
|
+
const sourceFrames = Array.from(doc.querySelectorAll('iframe'));
|
|
1044
|
+
const clonedFrames = Array.from(clone.querySelectorAll('iframe'));
|
|
1045
|
+
|
|
1046
|
+
sourceFrames.forEach((sourceFrame, index) => {
|
|
1047
|
+
const clonedFrame = clonedFrames[index];
|
|
1048
|
+
if (!clonedFrame) return;
|
|
1049
|
+
// iframe is a raw-text element in HTML parsers. Use a temporary
|
|
1050
|
+
// custom element so Cheerio parses the embedded controls as markup.
|
|
1051
|
+
const serializedFrame = doc.createElement('testronaut-frame');
|
|
1052
|
+
for (const attribute of Array.from(clonedFrame.attributes)) {
|
|
1053
|
+
serializedFrame.setAttribute(attribute.name, attribute.value);
|
|
1054
|
+
}
|
|
1055
|
+
clonedFrame.replaceWith(serializedFrame);
|
|
1056
|
+
const label = sourceFrame.id || sourceFrame.name || sourceFrame.getAttribute('src') || `frame-${index + 1}`;
|
|
1057
|
+
serializedFrame.setAttribute('data-testronaut-frame', label);
|
|
1058
|
+
try {
|
|
1059
|
+
const childDocument = sourceFrame.contentDocument;
|
|
1060
|
+
if (!childDocument?.documentElement) throw new Error('inaccessible');
|
|
1061
|
+
serializedFrame.setAttribute('data-testronaut-frame-status', 'same-origin');
|
|
1062
|
+
serializedFrame.innerHTML = serializeDocument(childDocument);
|
|
1063
|
+
} catch {
|
|
1064
|
+
serializedFrame.setAttribute('data-testronaut-frame-status', 'cross-origin');
|
|
1065
|
+
serializedFrame.textContent = '[cross-origin frame, content not accessible]';
|
|
1066
|
+
}
|
|
1067
|
+
});
|
|
1068
|
+
|
|
1069
|
+
return clone.outerHTML;
|
|
1070
|
+
};
|
|
1071
|
+
return serializeDocument(document);
|
|
1072
|
+
});
|
|
1073
|
+
} catch {
|
|
1074
|
+
// Preserve the main-document snapshot if serialization races a navigation.
|
|
1075
|
+
}
|
|
1076
|
+
|
|
1017
1077
|
// --- 4) Optional DOM reduction/cleanup (your existing steps) ---
|
|
1018
1078
|
let $ = cheerio.load(html);
|
|
1019
1079
|
|
|
1080
|
+
// Restore the public marker shape after the HTML parser has materialized
|
|
1081
|
+
// the temporary element's descendants.
|
|
1082
|
+
$('testronaut-frame').each((_, element) => {
|
|
1083
|
+
element.name = 'iframe';
|
|
1084
|
+
element.tagName = 'iframe';
|
|
1085
|
+
});
|
|
1086
|
+
|
|
1020
1087
|
if (selector) {
|
|
1021
1088
|
let selection;
|
|
1022
1089
|
try {
|
|
@@ -1141,12 +1208,27 @@ export class ChromeBrowser {
|
|
|
1141
1208
|
}
|
|
1142
1209
|
|
|
1143
1210
|
|
|
1144
|
-
async check_text({ text }) {
|
|
1145
|
-
const
|
|
1146
|
-
const
|
|
1147
|
-
|
|
1148
|
-
|
|
1149
|
-
|
|
1211
|
+
async check_text({ text, searchAllFrames = true }) {
|
|
1212
|
+
const mainFrame = this.page.mainFrame?.();
|
|
1213
|
+
const frames = searchAllFrames && this.page.frames
|
|
1214
|
+
? [mainFrame, ...this.page.frames().filter(frame => frame !== mainFrame)].filter(Boolean)
|
|
1215
|
+
: [mainFrame || this.page];
|
|
1216
|
+
|
|
1217
|
+
for (const frame of frames) {
|
|
1218
|
+
const matches = frame.getByText(text);
|
|
1219
|
+
const count = await matches.count();
|
|
1220
|
+
for (let index = 0; index < count; index++) {
|
|
1221
|
+
if (!await matches.nth(index).isVisible().catch(() => false)) continue;
|
|
1222
|
+
if (frame === mainFrame || frame === this.page) return `FOUND: ${text}`;
|
|
1223
|
+
|
|
1224
|
+
let label = frame.name?.() || '';
|
|
1225
|
+
if (!label) {
|
|
1226
|
+
label = await frame.frameElement?.()
|
|
1227
|
+
.then(element => element.getAttribute('id'))
|
|
1228
|
+
.catch(() => '') || '';
|
|
1229
|
+
}
|
|
1230
|
+
label ||= frame.url?.() || 'unnamed';
|
|
1231
|
+
return `FOUND in frame ${label}: ${text}`;
|
|
1150
1232
|
}
|
|
1151
1233
|
}
|
|
1152
1234
|
return `NOT FOUND: ${text}`;
|
|
@@ -1455,6 +1537,7 @@ export const CHROME_TOOL_MAP = {
|
|
|
1455
1537
|
reload: (b) => b.reload(),
|
|
1456
1538
|
fill: (b, args) => b.fill(args),
|
|
1457
1539
|
click: (b, args) => b.click(args),
|
|
1540
|
+
press_key: (b, args) => b.press_key(args),
|
|
1458
1541
|
get_dom: (b, args, agentMemory) => b.get_dom(args, agentMemory),
|
|
1459
1542
|
check_text: (b, args) => b.check_text(args),
|
|
1460
1543
|
click_text: async (browser, args, agentMemory) => {
|
package/tools/domControl.js
CHANGED
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
* normalize, and compress HTML so it’s cheaper and clearer for LLMs.
|
|
8
8
|
*
|
|
9
9
|
* Responsibilities:
|
|
10
|
-
* - Remove non-content tags (scripts, styles,
|
|
10
|
+
* - Remove non-content tags (scripts, styles, etc.) and noisy attributes.
|
|
11
11
|
* - Collapse overly long lists/sections into truncated markers.
|
|
12
12
|
* - Reduce repeated div blocks to a representative sample.
|
|
13
13
|
* - Strip obfuscated/utility CSS classes that don’t help reasoning.
|
|
@@ -23,7 +23,8 @@
|
|
|
23
23
|
|
|
24
24
|
/**
|
|
25
25
|
* Remove tags and attributes that are unlikely to help LLM reasoning.
|
|
26
|
-
* - Strips: script, style, meta, link, noscript,
|
|
26
|
+
* - Strips: script, style, meta, link, noscript, canvas, svg
|
|
27
|
+
* - Keeps iframe shells and any same-origin content injected by get_dom.
|
|
27
28
|
* - Keeps only a safe whitelist of attributes on remaining elements.
|
|
28
29
|
* - Removes HTML comments.
|
|
29
30
|
*
|
|
@@ -31,7 +32,7 @@
|
|
|
31
32
|
* @returns {$} same Cheerio instance for chaining
|
|
32
33
|
*/
|
|
33
34
|
export const removeNonContentTags = async ($) => {
|
|
34
|
-
$('script, style, meta, link, noscript,
|
|
35
|
+
$('script, style, meta, link, noscript, canvas, svg').remove();
|
|
35
36
|
|
|
36
37
|
$('*').each((_, el) => {
|
|
37
38
|
const $el = $(el);
|
|
@@ -39,6 +40,7 @@ export const removeNonContentTags = async ($) => {
|
|
|
39
40
|
'href', 'src', 'type', 'alt', 'title',
|
|
40
41
|
'placeholder', 'name', 'id', 'class', 'value',
|
|
41
42
|
'data-testid', 'disabled', 'role', 'data-state',
|
|
43
|
+
'data-testronaut-frame', 'data-testronaut-frame-status',
|
|
42
44
|
// keep key data-* attributes that help identify rows/items without bloating too much
|
|
43
45
|
'data-type', 'data-id', 'data-title', 'data-parentid', 'data-order'
|
|
44
46
|
];
|
package/tools/toolSchema.js
CHANGED
|
@@ -44,6 +44,7 @@ const toolsSchema = [
|
|
|
44
44
|
properties: {
|
|
45
45
|
selector: { type: 'string' },
|
|
46
46
|
text: { type: 'string' },
|
|
47
|
+
frame: { type: 'string', description: 'Optional CSS selector for the containing iframe.' },
|
|
47
48
|
},
|
|
48
49
|
required: ['selector', 'text'],
|
|
49
50
|
},
|
|
@@ -58,6 +59,7 @@ const toolsSchema = [
|
|
|
58
59
|
type: 'object',
|
|
59
60
|
properties: {
|
|
60
61
|
selector: { type: 'string' },
|
|
62
|
+
frame: { type: 'string', description: 'Optional CSS selector for the containing iframe.' },
|
|
61
63
|
waitFor: {
|
|
62
64
|
type: 'string',
|
|
63
65
|
enum: ['load', 'domcontentloaded', 'networkidle'],
|
|
@@ -73,6 +75,27 @@ const toolsSchema = [
|
|
|
73
75
|
},
|
|
74
76
|
},
|
|
75
77
|
},
|
|
78
|
+
{
|
|
79
|
+
type: 'function',
|
|
80
|
+
function: {
|
|
81
|
+
name: 'press_key',
|
|
82
|
+
description: 'Press a keyboard key or shortcut, optionally focusing an element by CSS selector first',
|
|
83
|
+
parameters: {
|
|
84
|
+
type: 'object',
|
|
85
|
+
properties: {
|
|
86
|
+
key: {
|
|
87
|
+
type: 'string',
|
|
88
|
+
description: 'Playwright key or shortcut, such as Enter, Escape, Control+B, or Meta+B',
|
|
89
|
+
},
|
|
90
|
+
selector: {
|
|
91
|
+
type: 'string',
|
|
92
|
+
description: 'Optional CSS selector of the element to focus before pressing the key',
|
|
93
|
+
},
|
|
94
|
+
},
|
|
95
|
+
required: ['key'],
|
|
96
|
+
},
|
|
97
|
+
},
|
|
98
|
+
},
|
|
76
99
|
{
|
|
77
100
|
type: 'function',
|
|
78
101
|
function: {
|
|
@@ -108,6 +131,11 @@ const toolsSchema = [
|
|
|
108
131
|
type: 'object',
|
|
109
132
|
properties: {
|
|
110
133
|
text: { type: 'string' },
|
|
134
|
+
searchAllFrames: {
|
|
135
|
+
type: 'boolean',
|
|
136
|
+
default: true,
|
|
137
|
+
description: 'Search iframe documents as well as the main document.',
|
|
138
|
+
},
|
|
111
139
|
},
|
|
112
140
|
required: ['text'],
|
|
113
141
|
},
|
package/todo.md
DELETED
|
@@ -1,25 +0,0 @@
|
|
|
1
|
-
# Testronaut CLI TODO
|
|
2
|
-
|
|
3
|
-
## OpenAI provider follow-ups
|
|
4
|
-
|
|
5
|
-
- Migrate the OpenAI adapter from Chat Completions to the Responses API in a
|
|
6
|
-
dedicated change. Preserve Testronaut's provider-neutral message and tool-call
|
|
7
|
-
contract, add response-ID/reasoning-state handling, normalize usage and raw
|
|
8
|
-
headers, and cover multi-turn tool continuation with unit and API smoke tests.
|
|
9
|
-
Once migrated, remove the GPT-5.6 Chat Completions compatibility override that
|
|
10
|
-
sets `reasoning_effort` to `none` for function-tool requests.
|
|
11
|
-
- Add active context budgeting after its behavior is designed and tested. Use
|
|
12
|
-
model context/output metadata to reserve response and reasoning capacity, then
|
|
13
|
-
apply explicit history compaction or summarization rules. Do not silently
|
|
14
|
-
truncate mission instructions, current DOM state, or unmatched tool calls.
|
|
15
|
-
- Learn effective OpenAI TPM/RPM limits from successful response headers and
|
|
16
|
-
rate-limit errors. Prefer observed project/model limits over inferring a named
|
|
17
|
-
account tier; retain the environment override and conservative startup values.
|
|
18
|
-
- Persist provider-observed rate limits in a separate, expiring local cache keyed
|
|
19
|
-
by provider, API project/account, endpoint/region, model, and traffic type.
|
|
20
|
-
Never rewrite user config or infer a named usage tier from a TPM value; expose
|
|
21
|
-
configured, cached, and live sources distinctly and discard stale observations.
|
|
22
|
-
- After dynamic rate limits are established, make DOM injection quota-aware.
|
|
23
|
-
Budget DOM tokens against both remaining TPM and the model context window,
|
|
24
|
-
preserve mission instructions and actionable controls, and fall back to
|
|
25
|
-
semantic sections plus on-demand DOM chunks instead of silent truncation.
|