jevway 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +37 -2
- package/{INSTALL-FOR-AGENTS.md → SETUP.md} +94 -14
- package/dist/browser/downloads.js +103 -0
- package/dist/browser/frames.js +21 -1
- package/dist/browser/snapshot.js +53 -0
- package/dist/browser/tab.js +59 -10
- package/dist/cli/dispatch.js +2 -0
- package/dist/cli/inbox-command.js +199 -0
- package/dist/cli/look-command.js +10 -3
- package/dist/cli/run-command.js +51 -3
- package/dist/daemon/server.js +22 -3
- package/dist/inbox/inbox-file.js +43 -0
- package/dist/inbox/inbox-registry.js +33 -0
- package/dist/inbox/inbox.js +127 -0
- package/dist/inbox/mail-api.js +52 -0
- package/dist/inbox/message.js +34 -0
- package/dist/jev/secrets.js +4 -3
- package/dist/mcp/inbox-tools.js +82 -0
- package/dist/mcp/instructions.js +11 -2
- package/dist/mcp/server.js +15 -8
- package/dist/mcp/task-tools.js +24 -8
- package/dist/notify/desktop.js +40 -0
- package/dist/notify/person-messages.js +50 -0
- package/dist/output/layout.js +37 -1
- package/dist/output/manifest.js +6 -0
- package/dist/output/run-folder.js +44 -2
- package/dist/recording/describe-step.js +3 -0
- package/dist/session/session-service.js +31 -11
- package/dist/setup/skill-command.js +44 -21
- package/dist/task/ask-user.js +182 -0
- package/dist/task/downloads.js +63 -0
- package/dist/task/person-wait.js +104 -0
- package/dist/task/present.js +32 -11
- package/dist/task/run-task.js +265 -28
- package/dist/task/start-task.js +49 -6
- package/dist/task/totp.js +124 -0
- package/dist/task/upload-files.js +237 -5
- package/package.json +2 -2
- package/skill/jevway/SKILL.md +5 -2
- package/skill/jevway-inbox/SKILL.md +61 -0
- package/skill/jevway-inbox/scripts/inbox.mjs +209 -0
package/README.md
CHANGED
|
@@ -40,6 +40,7 @@ optional tools unlock more.
|
|
|
40
40
|
| **Google Chrome** (or Chromium) | every run: your running Chrome, or a throwaway one jevway starts | runs fail; set `CHROME_PATH` if Chrome isn't in the usual place |
|
|
41
41
|
| **An API key**: `TYPESAFE_API_KEY` (recommended) or `OPENROUTER_API_KEY` | every decision Jev makes | the server still starts and lists its tools, and `jevway_look` works, but tasks and journeys answer with the setup fix |
|
|
42
42
|
| `ffmpeg` *(optional)* | the `run.mp4` video in a recording | recordings keep their step screenshots and HTML page, without the video |
|
|
43
|
+
| `notify-send` *(optional, Linux)* | the desktop notification when a person is needed (macOS has it built in) | the agent still asks you; there's just no pop-up |
|
|
43
44
|
| chrome-devtools MCP *(optional)* | inspecting what jevway left on screen: console, network, performance, DOM details | nothing is lost in jevway itself; your agent just has no deeper inspection tool |
|
|
44
45
|
|
|
45
46
|
**Which key?** Both run the same Jev model. The TypeSafe key adds **Box**, which locates controls on the screenshot:
|
|
@@ -78,6 +79,20 @@ control (opt-in), and marks each journey screen's primary action and the control
|
|
|
78
79
|
|
|
79
80
|
`npx -y jevway doctor` checks the setup at any time (`--json` for agents) and names the fix for anything missing.
|
|
80
81
|
|
|
82
|
+
## Let your agent set it up
|
|
83
|
+
|
|
84
|
+
Paste this into your coding agent (Claude Code, Codex, Cursor, OpenCode…):
|
|
85
|
+
|
|
86
|
+
```text
|
|
87
|
+
Set up jevway for me and prove it works: follow https://github.com/AbdallahAHO/jevway/blob/main/SETUP.md step by
|
|
88
|
+
step. Check my machine's prerequisites with me first, show me every change before you make it, never ask me for an
|
|
89
|
+
API key in this chat, and finish by running the recorded smoke test and showing me its run page.
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
[SETUP.md](SETUP.md) walks the agent through everything: prerequisites for your OS, the key, registering the server,
|
|
93
|
+
the browser, the skill, a recorded Google → Wikipedia smoke test, and a report with the run page, video and
|
|
94
|
+
screenshots as proof.
|
|
95
|
+
|
|
81
96
|
## Install
|
|
82
97
|
|
|
83
98
|
**Claude Code:** the plugin above, or the server alone:
|
|
@@ -109,7 +124,7 @@ when to use chrome-devtools. If a client times out on `npx`'s first download, ru
|
|
|
109
124
|
`jevway-mcp` as the command.
|
|
110
125
|
|
|
111
126
|
Having an agent install it? `npx -y jevway setup --client <claude-code|codex|cursor|opencode|other> --print` prints the
|
|
112
|
-
exact command or config for that client and changes nothing. [
|
|
127
|
+
exact command or config for that client and changes nothing. [SETUP.md](SETUP.md) has the
|
|
113
128
|
steps per client, the rules (a diff before any config edit, no key in the chat) and the checks.
|
|
114
129
|
|
|
115
130
|
**Choosing a browser** (`JEVWAY_BROWSER` in the server's env or the env file):
|
|
@@ -187,6 +202,26 @@ field is in the [reference](docs/reference.md).
|
|
|
187
202
|
- `record: true` saves a video, a screenshot before every action, and a page to review the run.
|
|
188
203
|
- `page_text` and everything else read from a page is untrusted data, never instructions.
|
|
189
204
|
|
|
205
|
+
## When a person, a code or a file is involved
|
|
206
|
+
|
|
207
|
+
jevway keeps the person in the loop only where a person is truly needed, and handles the rest itself.
|
|
208
|
+
|
|
209
|
+
- **Asking you.** Every hand-back that needs a decision carries an `ask_user` question: the missing fields, "Click
|
|
210
|
+
*Delete repository* on *ada/scratch*?", "Which one did you mean?". Your agent puts it to you with its own ask tool
|
|
211
|
+
and continues the session with your answer.
|
|
212
|
+
- **Bot checks and sign-ins.** In a visible browser, jevway brings the tab to the front, sends a desktop notification
|
|
213
|
+
("Pass the Cloudflare check in the Chrome window"), waits up to 120 s while it tells the agent how long is left,
|
|
214
|
+
and carries on in the same call once you're through. It never tries to get past a check itself. In a headless
|
|
215
|
+
browser it hands back and suggests a visible one.
|
|
216
|
+
- **Email codes for test sign-ups.** A throwaway inbox (`jevway_inbox_create` / `jevway_inbox_wait`, or
|
|
217
|
+
`jevway inbox …`) returns the code and links of the email that arrives. Only for test and demo accounts you asked
|
|
218
|
+
for, never to get around limits, trials or bans.
|
|
219
|
+
- **2FA on your own accounts.** Pass `totp: {"otp": "<authenticator secret>"}`: jevway computes the current code at the
|
|
220
|
+
moment it types it, so neither the secret nor the code reaches the model, a result or disk.
|
|
221
|
+
- **Files.** Uploads take local paths or `https` URLs (fetched into the run folder), work inside iframes and on
|
|
222
|
+
drag-and-drop zones. Downloads land in the run folder's `downloads/` and are listed in `files.downloads`.
|
|
223
|
+
`jevway_look` with `pdf: true` saves the page as a PDF.
|
|
224
|
+
|
|
190
225
|
## Recordings
|
|
191
226
|
|
|
192
227
|
Every session gets a folder under `./.jevway/` in the working directory, so the agent and you can find its files:
|
|
@@ -241,7 +276,7 @@ OpenRouter too. Jev is priced at $0.042 per 1M input tokens, so a typical 10-ste
|
|
|
241
276
|
| [docs/reference.md](docs/reference.md) | every tool, parameter, status, CLI flag, environment variable and model |
|
|
242
277
|
| [docs/how-it-works.md](docs/how-it-works.md) | the technical specification: one step end to end, guards, safety |
|
|
243
278
|
| [docs/journeys.md](docs/journeys.md) | the journey-map tools |
|
|
244
|
-
| [
|
|
279
|
+
| [SETUP.md](SETUP.md) | agents setting jevway up for a person, with a recorded smoke test |
|
|
245
280
|
| [skill/jevway/SKILL.md](skill/jevway/SKILL.md) | the skill agents load: when and how to call jevway |
|
|
246
281
|
| [docs/verification.md](docs/verification.md) | the test suites and their results |
|
|
247
282
|
| [docs/prior-art.md](docs/prior-art.md) | other open-source Jev browser projects, and what was taken from them |
|
|
@@ -1,20 +1,34 @@
|
|
|
1
|
-
#
|
|
2
|
-
|
|
3
|
-
You are an agent asked to
|
|
4
|
-
It hands browser goals to TypeSafe Jev
|
|
5
|
-
|
|
6
|
-
|
|
1
|
+
# Setting up jevway (for agents)
|
|
2
|
+
|
|
3
|
+
You are an agent asked to set up jevway for a person and prove it works. jevway is an MCP server, `jevway-mcp`, plus a
|
|
4
|
+
CLI, `jevway`, both from the npm package `jevway`. It hands browser goals to TypeSafe Jev and returns one checked
|
|
5
|
+
summary per goal.
|
|
6
|
+
|
|
7
|
+
Read this whole file before you run anything. You will:
|
|
8
|
+
1. check the machine's prerequisites with the person and help install what's missing;
|
|
9
|
+
2. get the API key in place (the person does that part);
|
|
10
|
+
3. register the MCP server with the client you're running in;
|
|
11
|
+
4. pick the browser with the person;
|
|
12
|
+
5. install the skill;
|
|
13
|
+
6. check the setup;
|
|
14
|
+
7. run a recorded smoke test, from Google to Wikipedia;
|
|
15
|
+
8. report back with the run page, its video and screenshots, as proof.
|
|
16
|
+
|
|
17
|
+
Run only these yourself:
|
|
18
|
+
- `node --version`, `ffmpeg -version`, and the person-approved install commands from step 1;
|
|
7
19
|
- `npx -y jevway doctor --json` (or `--plan`, or `jevway doctor`);
|
|
8
20
|
- `npx -y jevway setup --client <name> --print [--json]`, which prints how to register jevway with a client and
|
|
9
21
|
changes nothing;
|
|
10
22
|
- `npx -y jevway skill --install <skills folder>`;
|
|
11
23
|
- `npm install -g jevway`;
|
|
12
|
-
- the client's own `mcp add` command, or an edit to its MCP config file that the person approved
|
|
24
|
+
- the client's own `mcp add` command, or an edit to its MCP config file that the person approved;
|
|
25
|
+
- the jevway tools themselves (or `npx -y jevway run …`) for the smoke test.
|
|
13
26
|
|
|
14
27
|
Never run plain `jevway setup` yourself: it is the person's interactive setup, and without a terminal it exits 2 with
|
|
15
28
|
`{ "ok": false, "error": { "code": "needs_tty", … } }`.
|
|
16
29
|
|
|
17
|
-
Done means: `jevway doctor --json` prints `"ready": true
|
|
30
|
+
Done means: `jevway doctor --json` prints `"ready": true`, the client lists the `jevway_*` tools, and the smoke test
|
|
31
|
+
ended `done` with a run page you showed the person.
|
|
18
32
|
|
|
19
33
|
## Rules
|
|
20
34
|
|
|
@@ -45,6 +59,19 @@ It needs no key to run and never prints one. It checks:
|
|
|
45
59
|
- the browser source;
|
|
46
60
|
- ffmpeg (optional: only for recordings).
|
|
47
61
|
|
|
62
|
+
`npx` itself needs Node, so check `node --version` first. For anything missing, propose the command that fits the
|
|
63
|
+
person's OS and package manager, and run it only after they say yes (or wait while they install it themselves, then
|
|
64
|
+
check again):
|
|
65
|
+
|
|
66
|
+
| Tool | Needed for | macOS (Homebrew) | Debian / Ubuntu | Fedora | Windows (winget) |
|
|
67
|
+
| --- | --- | --- | --- | --- | --- |
|
|
68
|
+
| Node.js 24+ | running jevway (required) | `brew install node` | Node 24 from NodeSource or nvm | `sudo dnf install nodejs` (check it's 24+) | `winget install OpenJS.NodeJS` |
|
|
69
|
+
| Chrome or Chromium | every run (required) | `brew install --cask google-chrome` | Google's `.deb`, or `sudo apt install chromium` | `sudo dnf install chromium` | `winget install Google.Chrome` |
|
|
70
|
+
| ffmpeg | the video in a recording (optional) | `brew install ffmpeg` | `sudo apt install ffmpeg` | `sudo dnf install ffmpeg-free` | `winget install Gyan.FFmpeg` |
|
|
71
|
+
| notify-send | the desktop notification when a person is needed (optional, Linux) | built in | `sudo apt install libnotify-bin` | `sudo dnf install libnotify` | not used |
|
|
72
|
+
|
|
73
|
+
A Chrome in an unusual place: set `CHROME_PATH` in `~/.config/jevway/env`.
|
|
74
|
+
|
|
48
75
|
If the first `npx` start is slow, or a client times out starting the server, install once with `npm install -g jevway`.
|
|
49
76
|
Then use `jevway-mcp` as the command instead of `npx -y -p jevway jevway-mcp`.
|
|
50
77
|
|
|
@@ -97,7 +124,8 @@ npx -y jevway setup --client cursor --print --json # { client, kind, command?, f
|
|
|
97
124
|
|
|
98
125
|
## 4. The browser
|
|
99
126
|
|
|
100
|
-
`JEVWAY_BROWSER`
|
|
127
|
+
Ask the person which browser jevway should use, and set `JEVWAY_BROWSER` in the server's env (or `--browser` for
|
|
128
|
+
the CLI) when it isn't `auto`. For the smoke test a visible one is best, so they can pass a check if one appears.
|
|
101
129
|
|
|
102
130
|
| Value | What it uses | What the person does |
|
|
103
131
|
| --- | --- | --- |
|
|
@@ -117,11 +145,13 @@ npx -y jevway skill --install ~/.codex/skills # or ~/.claude/skills, ~/.cursor
|
|
|
117
145
|
It prints `{ ok, status, path, files }`. A different `jevway` skill already there (or a symlink to one) is left alone
|
|
118
146
|
with `"code": "skill_exists"` and exit 1; replace it with `--force` only if the person agrees.
|
|
119
147
|
|
|
120
|
-
## 6. Check the
|
|
148
|
+
## 6. Check the setup
|
|
121
149
|
|
|
122
|
-
1. `npx -y jevway doctor --json` prints `"ready": true` and exits 0 (every finding `ok` or `warn`).
|
|
123
|
-
|
|
124
|
-
|
|
150
|
+
1. `npx -y jevway doctor --json` prints `"ready": true` and exits 0 (every finding `ok` or `warn`). For every finding
|
|
151
|
+
that isn't `ok`, tell the person the fix in one line and wait; re-run it after each fix (`--plan` lists the steps
|
|
152
|
+
left).
|
|
153
|
+
2. The client lists the tools: `jevway_task`, `jevway_look`, `jevway_end`, the five `jevway_journey_*` tools and
|
|
154
|
+
`jevway_inbox_create` / `jevway_inbox_wait`.
|
|
125
155
|
- In Claude Code: `/mcp`.
|
|
126
156
|
- Elsewhere, the client's MCP list.
|
|
127
157
|
- Or the MCP Inspector:
|
|
@@ -129,4 +159,54 @@ with `"code": "skill_exists"` and exit 1; replace it with `--force` only if the
|
|
|
129
159
|
3. Optional, costs a fraction of a cent: `npx -y jevway doctor --live` makes one real Jev request and reports its
|
|
130
160
|
latency.
|
|
131
161
|
|
|
132
|
-
|
|
162
|
+
## 7. The smoke test
|
|
163
|
+
|
|
164
|
+
Call `jevway_task` with:
|
|
165
|
+
|
|
166
|
+
```json
|
|
167
|
+
{
|
|
168
|
+
"url": "https://www.google.com",
|
|
169
|
+
"goal": "Search Google for the Rosetta Stone and open its English Wikipedia article",
|
|
170
|
+
"values": { "query": "Rosetta Stone Wikipedia" },
|
|
171
|
+
"done_when": "The English Wikipedia article about the Rosetta Stone is open",
|
|
172
|
+
"allowed_hosts": ["google.com", "*.google.com", "en.wikipedia.org"],
|
|
173
|
+
"record": true,
|
|
174
|
+
"detail": "full"
|
|
175
|
+
}
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
What to expect, and what to do:
|
|
179
|
+
- **A cookie banner** is handled by jevway (it declines when it can).
|
|
180
|
+
- **A reCAPTCHA ("unusual traffic")** is common for automated browsers on Google. In a visible browser jevway brings
|
|
181
|
+
the tab to the front, sends the person a desktop notification and waits up to 2 minutes for them to pass it; it
|
|
182
|
+
never solves it. Tell the person it may happen before you start.
|
|
183
|
+
- **A result with `ask_user`** (a bot check, a sign-in, a missing value): put that question to the person with your
|
|
184
|
+
own ask tool, then do what its `then` says.
|
|
185
|
+
- **Google keeps blocking**: run the same goal from `https://duckduckgo.com` with
|
|
186
|
+
`allowed_hosts: ["duckduckgo.com", "en.wikipedia.org"]`. It usually finishes in 5 steps.
|
|
187
|
+
- **No MCP in your client**: run the same test with the CLI, which prints one JSON result on stdout:
|
|
188
|
+
|
|
189
|
+
```bash
|
|
190
|
+
npx -y jevway run --browser launch --record --detail full --url https://www.google.com \
|
|
191
|
+
--goal "Search Google for the Rosetta Stone and open its English Wikipedia article" \
|
|
192
|
+
--values '{"query":"Rosetta Stone Wikipedia"}' \
|
|
193
|
+
--done-when "The English Wikipedia article about the Rosetta Stone is open" \
|
|
194
|
+
--allowed-hosts "google.com,*.google.com,en.wikipedia.org"
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
End the session with `jevway_end` when you have what you need.
|
|
198
|
+
|
|
199
|
+
## 8. Report back
|
|
200
|
+
|
|
201
|
+
Tell the person, in this shape:
|
|
202
|
+
- **Outcome:** `status` and `reason`, and the `story` as a numbered list.
|
|
203
|
+
- **Proof:** the run page `files.page` (open it for them: `open` on macOS, `xdg-open` on Linux, `start` on Windows),
|
|
204
|
+
the video `files.video` (present when ffmpeg is installed), the end screenshot `files.screenshot` (show it if you
|
|
205
|
+
can display images), and the run folder `files.run_dir` with its `manifest.json`.
|
|
206
|
+
- **Cost:** `jev.requests` and `jev.est_cost_usd` (an estimate; a typical smoke test is well under a cent).
|
|
207
|
+
- **Setup:** what you installed or changed, with the person's yes for each.
|
|
208
|
+
- **Anything that needed them**, and anything that didn't work, with the fix.
|
|
209
|
+
|
|
210
|
+
The run page opens with the goal and how it ended, then the video with every step on a timeline and a strip of
|
|
211
|
+
screenshots kept in sync. It is the proof that the whole chain works: the client, the MCP server, the key, the
|
|
212
|
+
browser and Jev.
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
import { mkdtempSync, rmSync } from 'node:fs';
|
|
2
|
+
import { tmpdir } from 'node:os';
|
|
3
|
+
import { join, relative, isAbsolute } from 'node:path';
|
|
4
|
+
const STAGING = new WeakMap();
|
|
5
|
+
/**
|
|
6
|
+
* Sets how the browser behind `cdp` downloads, once per connection: jevway's launched browser saves every download under
|
|
7
|
+
* its guid in a staging folder of its own; the person's Chrome (`auto`, a debugging URL) keeps its own behaviour and only
|
|
8
|
+
* reports downloads. A browser that refuses never fails a run: its downloads go unreported.
|
|
9
|
+
*
|
|
10
|
+
* @example
|
|
11
|
+
* await useDownloads(cdp, 'launch-headless'); // Browser.setDownloadBehavior { behavior: 'allowAndName', downloadPath: '/tmp/jevway-downloads-…' }
|
|
12
|
+
*/
|
|
13
|
+
export const useDownloads = async (cdp, source) => {
|
|
14
|
+
if (source !== 'launch' && source !== 'launch-headless') {
|
|
15
|
+
await cdp.send('Browser.setDownloadBehavior', { behavior: 'default', eventsEnabled: true }).catch(() => undefined);
|
|
16
|
+
return;
|
|
17
|
+
}
|
|
18
|
+
const downloadPath = mkdtempSync(join(tmpdir(), 'jevway-downloads-'));
|
|
19
|
+
STAGING.set(cdp, downloadPath);
|
|
20
|
+
await cdp.send('Browser.setDownloadBehavior', { behavior: 'allowAndName', downloadPath, eventsEnabled: true }).catch(() => undefined);
|
|
21
|
+
};
|
|
22
|
+
/** Removes the staging folder of a launched browser's connection, with whatever was never kept. */
|
|
23
|
+
export const dropStagedDownloads = (cdp) => {
|
|
24
|
+
const dir = STAGING.get(cdp);
|
|
25
|
+
if (dir)
|
|
26
|
+
rmSync(dir, { recursive: true, force: true });
|
|
27
|
+
STAGING.delete(cdp);
|
|
28
|
+
};
|
|
29
|
+
const inside = (dir, path) => {
|
|
30
|
+
if (!dir || !path)
|
|
31
|
+
return false;
|
|
32
|
+
const rel = relative(dir, path);
|
|
33
|
+
return Boolean(rel) && !rel.startsWith('..') && !isAbsolute(rel);
|
|
34
|
+
};
|
|
35
|
+
/**
|
|
36
|
+
* The downloads one tab started: `owns` says whether a frame id is the tab's (its page, one of its frames, a tab it
|
|
37
|
+
* opened). Every other download, in the person's other tabs too, is never recorded.
|
|
38
|
+
*
|
|
39
|
+
* @example
|
|
40
|
+
* const downloads = new TabDownloads(cdp, (frameId) => frameId === targetId);
|
|
41
|
+
* const finished = await downloads.settle({ waitMs: 30_000 });
|
|
42
|
+
*/
|
|
43
|
+
export class TabDownloads {
|
|
44
|
+
#cdp;
|
|
45
|
+
#owns;
|
|
46
|
+
#all = new Map();
|
|
47
|
+
#untaken = [];
|
|
48
|
+
#changed = new Set();
|
|
49
|
+
constructor(cdp, owns) {
|
|
50
|
+
this.#cdp = cdp;
|
|
51
|
+
this.#owns = owns;
|
|
52
|
+
}
|
|
53
|
+
/** Starts listening; returns the unsubscribe. */
|
|
54
|
+
watch() {
|
|
55
|
+
return this.#cdp.onEvent((method, params, sessionId) => {
|
|
56
|
+
if (sessionId || !method.startsWith('Browser.download'))
|
|
57
|
+
return;
|
|
58
|
+
if (method === 'Browser.downloadWillBegin' && this.#owns(params.frameId)) {
|
|
59
|
+
this.#all.set(params.guid, { guid: params.guid, name: params.suggestedFilename, url: params.url, state: 'inProgress', bytes: 0, staged: false });
|
|
60
|
+
this.#untaken.push(params.guid);
|
|
61
|
+
}
|
|
62
|
+
const known = method === 'Browser.downloadProgress' && this.#all.get(params.guid);
|
|
63
|
+
if (!known)
|
|
64
|
+
return;
|
|
65
|
+
const filePath = params.filePath ?? known.filePath;
|
|
66
|
+
this.#all.set(params.guid, {
|
|
67
|
+
...known,
|
|
68
|
+
state: params.state,
|
|
69
|
+
bytes: params.receivedBytes ?? known.bytes,
|
|
70
|
+
...(filePath && { filePath }),
|
|
71
|
+
staged: inside(STAGING.get(this.#cdp), filePath),
|
|
72
|
+
});
|
|
73
|
+
for (const changed of this.#changed)
|
|
74
|
+
changed();
|
|
75
|
+
});
|
|
76
|
+
}
|
|
77
|
+
/**
|
|
78
|
+
* Takes the downloads begun since the last take, after waiting up to `waitMs` (or until `signal`) for those still
|
|
79
|
+
* running. Each comes back as it stands then: completed, canceled, or still in progress.
|
|
80
|
+
*/
|
|
81
|
+
async settle({ waitMs, signal }) {
|
|
82
|
+
const taken = this.#untaken.splice(0);
|
|
83
|
+
const running = () => taken.some((guid) => this.#all.get(guid)?.state === 'inProgress');
|
|
84
|
+
if (taken.length && running() && !signal?.aborted) {
|
|
85
|
+
await new Promise((resolve) => {
|
|
86
|
+
const done = () => {
|
|
87
|
+
clearTimeout(timer);
|
|
88
|
+
signal?.removeEventListener('abort', done);
|
|
89
|
+
this.#changed.delete(check);
|
|
90
|
+
resolve();
|
|
91
|
+
};
|
|
92
|
+
const check = () => {
|
|
93
|
+
if (!running())
|
|
94
|
+
done();
|
|
95
|
+
};
|
|
96
|
+
const timer = setTimeout(done, waitMs);
|
|
97
|
+
signal?.addEventListener('abort', done, { once: true });
|
|
98
|
+
this.#changed.add(check);
|
|
99
|
+
});
|
|
100
|
+
}
|
|
101
|
+
return taken.map((guid) => ({ ...this.#all.get(guid) }));
|
|
102
|
+
}
|
|
103
|
+
}
|
package/dist/browser/frames.js
CHANGED
|
@@ -134,6 +134,7 @@ export class Frames {
|
|
|
134
134
|
#sessionId;
|
|
135
135
|
#counter;
|
|
136
136
|
#touch = false;
|
|
137
|
+
#files = false;
|
|
137
138
|
#mainFrameId = '';
|
|
138
139
|
#viewport = { w: 0, h: 0 };
|
|
139
140
|
/** Out-of-process frames attached so far, by target id (= frame id). */
|
|
@@ -164,6 +165,20 @@ export class Frames {
|
|
|
164
165
|
for (const { sessionId } of this.#children.values())
|
|
165
166
|
void this.#emulateTouch(sessionId);
|
|
166
167
|
}
|
|
168
|
+
/**
|
|
169
|
+
* The caller declared files: frame reads list file inputs and drop zones, and out-of-process frames keep native file
|
|
170
|
+
* pickers closed too (the page's own session does that for same-process frames).
|
|
171
|
+
*/
|
|
172
|
+
set files(enabled) {
|
|
173
|
+
if (enabled && !this.#files)
|
|
174
|
+
for (const { sessionId } of this.#children.values())
|
|
175
|
+
void this.#interceptFileChooser(sessionId);
|
|
176
|
+
this.#files = enabled;
|
|
177
|
+
}
|
|
178
|
+
/** Whether a protocol frame id is this tab's: its page, a frame read on it, or an out-of-process frame attached to it. */
|
|
179
|
+
holds(frameId) {
|
|
180
|
+
return Boolean(frameId) && (frameId === this.#mainFrameId || this.#known.has(frameId) || this.#children.has(frameId));
|
|
181
|
+
}
|
|
167
182
|
/** The frame holding an element id: undefined for the main frame and for a frame no longer on the page. */
|
|
168
183
|
ofNode(node) {
|
|
169
184
|
const base = idBaseOf(node);
|
|
@@ -228,7 +243,7 @@ export class Frames {
|
|
|
228
243
|
// The text walk keeps to the part of the frame the page shows, in the frame's own coordinates.
|
|
229
244
|
const { offset, region } = placement;
|
|
230
245
|
const view = { x: region.x - offset.x, y: region.y - offset.y, w: region.w, h: region.h };
|
|
231
|
-
const response = await this.evaluate(frame, `window.__jevIdBase=${frame.base};window.__jevView=${JSON.stringify(view)};window.__jevWanted=${JSON.stringify(wanted)};${script}`, { timeout: FRAME_READ_MS });
|
|
246
|
+
const response = await this.evaluate(frame, `window.__jevIdBase=${frame.base};window.__jevView=${JSON.stringify(view)};window.__jevWanted=${JSON.stringify(wanted)};window.__jevFiles=${this.#files};${script}`, { timeout: FRAME_READ_MS });
|
|
232
247
|
const state = response.exceptionDetails ? null : response.result?.value;
|
|
233
248
|
if (!state)
|
|
234
249
|
return null;
|
|
@@ -404,6 +419,8 @@ export class Frames {
|
|
|
404
419
|
await this.#send('Page.addScriptToEvaluateOnNewDocument', { source: this.#counter, worldName: WORLD }, sessionId);
|
|
405
420
|
await this.#send('Emulation.setFocusEmulationEnabled', { enabled: true }, sessionId).catch(() => undefined);
|
|
406
421
|
await this.#emulateTouch(sessionId);
|
|
422
|
+
if (this.#files)
|
|
423
|
+
await this.#interceptFileChooser(sessionId);
|
|
407
424
|
await this.#autoAttach(sessionId);
|
|
408
425
|
}
|
|
409
426
|
catch {
|
|
@@ -418,6 +435,9 @@ export class Frames {
|
|
|
418
435
|
const enabled = this.#touch;
|
|
419
436
|
return this.#send('Emulation.setTouchEmulationEnabled', { enabled, maxTouchPoints: enabled ? 5 : 1 }, sessionId).catch(() => undefined);
|
|
420
437
|
}
|
|
438
|
+
#interceptFileChooser(sessionId) {
|
|
439
|
+
return this.#send('Page.setInterceptFileChooserDialog', { enabled: true }, sessionId).catch(() => undefined);
|
|
440
|
+
}
|
|
421
441
|
#autoAttach(sessionId) {
|
|
422
442
|
return this.#send('Target.setAutoAttach', { autoAttach: true, waitForDebuggerOnStart: true, flatten: true, filter: [{ type: 'iframe' }] }, sessionId);
|
|
423
443
|
}
|
package/dist/browser/snapshot.js
CHANGED
|
@@ -424,10 +424,12 @@
|
|
|
424
424
|
}
|
|
425
425
|
return null;
|
|
426
426
|
};
|
|
427
|
+
const anchors = [];
|
|
427
428
|
for (const input of fileInputs) {
|
|
428
429
|
if (input.matches(':disabled') || input.closest('[aria-disabled="true"],[inert]')) continue;
|
|
429
430
|
const anchor = standInFor(input);
|
|
430
431
|
if (!anchor || outOfView(anchor.getBoundingClientRect())) continue;
|
|
432
|
+
anchors.push(anchor);
|
|
431
433
|
const r = cache.clickRect(anchor),
|
|
432
434
|
own = (anchor === input ? name(input) : name(input) || name(anchor)).replace(/\s+/g, ' ').trim();
|
|
433
435
|
actions.push({
|
|
@@ -439,6 +441,57 @@
|
|
|
439
441
|
rect: { x: r.x, y: r.y, w: r.width, h: r.height },
|
|
440
442
|
});
|
|
441
443
|
}
|
|
444
|
+
// jevway: a drop zone with no file input in it is listed only on clear markers: an ondrop attribute, a drop-zone
|
|
445
|
+
// name in its class, id or a data- attribute, or a short text asking for a file to be dropped ("Drag and drop",
|
|
446
|
+
// "Drop your CV here"; its nearest box of 40×40 or more). The innermost zone wins; one holding or inside a file field's stand-in
|
|
447
|
+
// is that field already. The file is dropped onto it.
|
|
448
|
+
const DROP_NAME = /drop-?zone|drop-?area|file-?drop|drop-?target/i;
|
|
449
|
+
const DROP_TEXT =
|
|
450
|
+
/\bdrag\s*(?:and|&|'?n'?)\s*drop\b|\bdrop\s+(?:your\s+|a\s+|an\s+|the\s+)?(?:files?|cv|resume|résumé|images?|photos?|documents?|attachments?)\b/i;
|
|
451
|
+
const named = (e) =>
|
|
452
|
+
e.hasAttribute('ondrop') ||
|
|
453
|
+
DROP_NAME.test(typeof e.className === 'string' ? e.className : '') ||
|
|
454
|
+
DROP_NAME.test(e.id) ||
|
|
455
|
+
[...e.attributes].some((a) => a.name.startsWith('data-') && DROP_NAME.test(a.name));
|
|
456
|
+
const big = (e) => {
|
|
457
|
+
const r = e.getBoundingClientRect();
|
|
458
|
+
return r.width >= 40 && r.height >= 40;
|
|
459
|
+
};
|
|
460
|
+
const zones = new Set(
|
|
461
|
+
[...document.querySelectorAll('[ondrop],[class*="drop" i],[id*="drop" i]')].filter(named),
|
|
462
|
+
);
|
|
463
|
+
const texts = document.createTreeWalker(document.body ?? document.documentElement, NodeFilter.SHOW_TEXT, {
|
|
464
|
+
acceptNode: (t) => (t.data.length <= 200 && DROP_TEXT.test(t.data) ? NodeFilter.FILTER_ACCEPT : NodeFilter.FILTER_SKIP),
|
|
465
|
+
});
|
|
466
|
+
for (let t = texts.nextNode(), seen = 0; t && seen < 20; t = texts.nextNode(), seen++) {
|
|
467
|
+
let box = t.parentElement;
|
|
468
|
+
for (let up = 0; box && up < 4 && !big(box); up++) box = box.parentElement;
|
|
469
|
+
if (box && big(box)) zones.add(box);
|
|
470
|
+
}
|
|
471
|
+
const listed = [...zones].filter(
|
|
472
|
+
(zone) =>
|
|
473
|
+
zone !== document.body &&
|
|
474
|
+
zone !== document.documentElement &&
|
|
475
|
+
visible(zone) &&
|
|
476
|
+
big(zone) &&
|
|
477
|
+
!outOfView(zone.getBoundingClientRect()) &&
|
|
478
|
+
!zone.querySelector('input[type=file]') &&
|
|
479
|
+
!anchors.some((anchor) => anchor.contains(zone) || zone.contains(anchor)),
|
|
480
|
+
);
|
|
481
|
+
for (const zone of listed) {
|
|
482
|
+
if (listed.some((other) => other !== zone && zone.contains(other))) continue;
|
|
483
|
+
const r = cache.clickRect(zone),
|
|
484
|
+
text = (zone.innerText || '').replace(/\s+/g, ' ').trim();
|
|
485
|
+
actions.push({
|
|
486
|
+
node: identity(zone),
|
|
487
|
+
role: 'file',
|
|
488
|
+
kind: 'upload',
|
|
489
|
+
drop: true,
|
|
490
|
+
label: cap(name(zone).replace(/\s+/g, ' ').trim() || text || 'drop zone', 120) + ' (drop zone)',
|
|
491
|
+
value: '',
|
|
492
|
+
rect: { x: r.x, y: r.y, w: r.width, h: r.height },
|
|
493
|
+
});
|
|
494
|
+
}
|
|
442
495
|
}
|
|
443
496
|
// jevway: controls sharing a role and label (a product grid's "Add to cart" buttons) are told apart by the card or
|
|
444
497
|
// row each sits in, as a person reads the page: "Add to cart — Sauce Labs Bike Light". Only colliding labels change;
|
package/dist/browser/tab.js
CHANGED
|
@@ -4,6 +4,7 @@ import { readFileSync } from 'node:fs';
|
|
|
4
4
|
import { isDeepStrictEqual } from 'node:util';
|
|
5
5
|
import { CdpError } from './cdp.js';
|
|
6
6
|
import { DEVICES } from './devices.js';
|
|
7
|
+
import { TabDownloads } from './downloads.js';
|
|
7
8
|
import { Frames, idBaseOf, mergeFrames, whenAborted } from './frames.js';
|
|
8
9
|
import { messageNames } from './words.js';
|
|
9
10
|
// Atomic DOM read from browser-use/jev-ultrafast (MIT): visible text + indexed controls + freshness guards.
|
|
@@ -72,6 +73,9 @@ export class Tab {
|
|
|
72
73
|
#device;
|
|
73
74
|
/** Page tabs this tab opened (target=_blank, window.open) that the run has not taken yet, oldest first. */
|
|
74
75
|
#opened = [];
|
|
76
|
+
/** Every page tab this tab opened, taken or not: a download one of them starts is this tab's. */
|
|
77
|
+
#openedEver = new Set();
|
|
78
|
+
#downloads;
|
|
75
79
|
#closed = false;
|
|
76
80
|
#unsubscribe = [];
|
|
77
81
|
/** The running call's stop signal: the waits after an input end early on it (see `stopSignal`). */
|
|
@@ -85,6 +89,7 @@ export class Tab {
|
|
|
85
89
|
this.targetId = targetId;
|
|
86
90
|
this.sessionId = sessionId;
|
|
87
91
|
this.#frames = new Frames(cdp, sessionId, INPUT_COUNTER);
|
|
92
|
+
this.#downloads = new TabDownloads(cdp, (frameId) => frameId === targetId || this.#openedEver.has(frameId) || this.#frames.holds(frameId));
|
|
88
93
|
}
|
|
89
94
|
/**
|
|
90
95
|
* Opens a background tab so the user's visible tab stays put. `newWindow` keeps the page rendering
|
|
@@ -233,13 +238,16 @@ export class Tab {
|
|
|
233
238
|
// opened (the user's own, in their Chrome) is never attached, read or changed.
|
|
234
239
|
tab.#unsubscribe.push(cdp.onEvent((method, params, sessionId) => {
|
|
235
240
|
const info = params?.targetInfo;
|
|
236
|
-
if (method === 'Target.targetCreated' && info?.type === 'page' && info.openerId === targetId)
|
|
241
|
+
if (method === 'Target.targetCreated' && info?.type === 'page' && info.openerId === targetId) {
|
|
237
242
|
tab.#opened.push(info.targetId);
|
|
243
|
+
tab.#openedEver.add(info.targetId);
|
|
244
|
+
}
|
|
238
245
|
if (method === 'Target.targetDestroyed' && params?.targetId === targetId)
|
|
239
246
|
tab.#closed = true;
|
|
240
247
|
if (method === 'Target.detachedFromTarget' && params?.sessionId === tab.sessionId && !sessionId)
|
|
241
248
|
tab.#closed = true;
|
|
242
249
|
}));
|
|
250
|
+
tab.#unsubscribe.push(tab.#downloads.watch());
|
|
243
251
|
await cdp.send('Target.setDiscoverTargets', { discover: true }).catch(() => undefined);
|
|
244
252
|
return tab;
|
|
245
253
|
}
|
|
@@ -376,29 +384,64 @@ export class Tab {
|
|
|
376
384
|
if (files && !this.#files)
|
|
377
385
|
void this.#call('Page.setInterceptFileChooserDialog', { enabled: true }).catch(() => undefined);
|
|
378
386
|
this.#files = files;
|
|
387
|
+
this.#frames.files = files;
|
|
379
388
|
}
|
|
380
389
|
/**
|
|
381
|
-
* Sets a declared file on an observed file input (hidden ones too
|
|
382
|
-
* as a click. The path comes from the caller's
|
|
390
|
+
* Sets a declared file on an observed file input (hidden ones too, in the page or in a frame), or drops it onto an
|
|
391
|
+
* observed drop zone, after the same element-level freshness check as a click. The path comes from the caller's
|
|
392
|
+
* `files`, never from the model or `values`.
|
|
383
393
|
*
|
|
384
394
|
* @example
|
|
385
395
|
* await tab.upload(page.actions.find((a) => a.kind === 'upload')!, page, '/Users/ada/me.png');
|
|
386
396
|
*/
|
|
387
397
|
async upload(action, page, path) {
|
|
388
|
-
// The file is set through the page's own session; a frame's node lives in another context, so it is refused by name.
|
|
389
|
-
if (action.node !== undefined && idBaseOf(action.node)) {
|
|
390
|
-
throw new TargetUnreachable(`"${action.label}" is inside a frame, and jevway uploads only to file fields in the page itself`);
|
|
391
|
-
}
|
|
392
398
|
if (!(await this.fresh(page, action)))
|
|
393
399
|
throw new StalePage('Page changed since this decision');
|
|
394
|
-
|
|
395
|
-
|
|
400
|
+
if (action.drop)
|
|
401
|
+
return this.#drop(action, path);
|
|
402
|
+
// A frame's node lives in the frame's own world, and its file is set through the session that owns the frame.
|
|
403
|
+
const frame = this.#frameOf(action.node);
|
|
404
|
+
const expression = `(() => { const e = window.__jevFast?.nodes.get(${Number(action.node)}); return e?.isConnected && e.matches('input[type=file]:not(:disabled)') ? e : null; })()`;
|
|
405
|
+
const { result } = await (frame ? this.#frames.evaluate(frame, expression, { returnByValue: false }) : this.#call('Runtime.evaluate', { expression })).catch((error) => {
|
|
406
|
+
throw error instanceof CdpError && !TIMED_OUT.test(error.message) ? new StalePage(error.message) : error;
|
|
396
407
|
});
|
|
397
408
|
if (!result?.objectId)
|
|
398
409
|
throw new TargetUnreachable('The file field is gone or disabled');
|
|
399
|
-
await this
|
|
410
|
+
await this.cdp.send('DOM.setFileInputFiles', { files: [path], objectId: result.objectId }, frame?.sessionId ?? this.sessionId);
|
|
400
411
|
this.#afterInput = action;
|
|
401
412
|
}
|
|
413
|
+
// A drop zone with no file input: a trusted drag of the file onto its visible centre, checked to have reached the page.
|
|
414
|
+
async #drop(action, path) {
|
|
415
|
+
const frame = this.#frameOf(action.node);
|
|
416
|
+
const resolved = await this.#evaluateFor(frame, `(${RESOLVE_TARGET})(${JSON.stringify(action)})`);
|
|
417
|
+
if (!resolved || resolved.covered)
|
|
418
|
+
throw new TargetUnreachable('Target changed or is covered', Boolean(resolved?.byPopup), resolved?.coverNodes ?? []);
|
|
419
|
+
const target = frame ? await this.#frames.toViewport(frame, resolved) : resolved;
|
|
420
|
+
if ('covered' in target)
|
|
421
|
+
throw new TargetUnreachable('Target changed or is covered', false, target.coverNodes);
|
|
422
|
+
await this.#evaluateFor(frame, WATCH_DROP);
|
|
423
|
+
const data = { items: [], files: [path], dragOperationsMask: DRAG_COPY };
|
|
424
|
+
for (const type of ['dragEnter', 'dragOver', 'drop'])
|
|
425
|
+
await this.#call('Input.dispatchDragEvent', { type, x: target.x, y: target.y, data });
|
|
426
|
+
if (!(await this.#evaluateFor(frame, 'window.__jevDropped===true')))
|
|
427
|
+
throw new TargetUnreachable('The drop zone did not take the file');
|
|
428
|
+
this.#afterInput = action;
|
|
429
|
+
}
|
|
430
|
+
/**
|
|
431
|
+
* The downloads this tab (its page, its frames, a tab it opened) started since the last call, after waiting up to
|
|
432
|
+
* `waitMs` for those still running; the run's stop signal ends the wait early.
|
|
433
|
+
*
|
|
434
|
+
* @example
|
|
435
|
+
* await tab.downloadsAfterInput(); // [{ name: 'invoice-1042.pdf', state: 'completed', bytes: 86016, filePath, staged: true, … }]
|
|
436
|
+
*/
|
|
437
|
+
downloadsAfterInput({ waitMs = DOWNLOAD_WAIT_MS } = {}) {
|
|
438
|
+
return this.#downloads.settle({ waitMs, signal: this.#stop });
|
|
439
|
+
}
|
|
440
|
+
/** The page as a PDF (its print styles, backgrounds included); headless and headed Chrome both print. */
|
|
441
|
+
async pdf() {
|
|
442
|
+
const { data } = await this.#call('Page.printToPDF', { printBackground: true });
|
|
443
|
+
return Buffer.from(data, 'base64');
|
|
444
|
+
}
|
|
402
445
|
async observe() {
|
|
403
446
|
if (this.#closed)
|
|
404
447
|
throw new StalePage('The tab was closed');
|
|
@@ -845,6 +888,12 @@ const INTERACTIVE = 'a,button,input,select,textarea,[contenteditable="true"],' +
|
|
|
845
888
|
const FIELD_LIKE = 'input:not([type="checkbox"],[type="radio"],[type="button"],[type="submit"],[type="reset"],[type="image"]),textarea,' +
|
|
846
889
|
'[contenteditable="true"],[role="combobox"],[role="textbox"],[role="searchbox"],[role="spinbutton"]';
|
|
847
890
|
// Resolves the observed node, rejects hidden/disabled/covered targets, and applies native selects in-page.
|
|
891
|
+
// How long a step waits for the downloads it started to finish.
|
|
892
|
+
const DOWNLOAD_WAIT_MS = 30_000;
|
|
893
|
+
// Input.dispatchDragEvent's operation mask for a copy, as a file dragged from the desktop offers.
|
|
894
|
+
const DRAG_COPY = 1;
|
|
895
|
+
// Notes a drop reaching the document (in the target's world): a page that refuses the drag never gets one.
|
|
896
|
+
const WATCH_DROP = `(() => { window.__jevDropped = false; window.addEventListener('drop', () => { window.__jevDropped = true; }, { capture: true, once: true }); return true; })()`;
|
|
848
897
|
const RESOLVE_TARGET = `action => {
|
|
849
898
|
const e=window.__jevFast?.nodes.get(action.node);
|
|
850
899
|
if (!e?.isConnected || e.matches(':disabled') || e.closest('[aria-disabled="true"],[inert]') ||
|
package/dist/cli/dispatch.js
CHANGED
|
@@ -3,6 +3,7 @@ import { SETUP_USAGE, runSetup } from '../setup/setup-command.js';
|
|
|
3
3
|
import { SKILL_USAGE, runSkill } from '../setup/skill-command.js';
|
|
4
4
|
import { closestName, usageError } from './flags.js';
|
|
5
5
|
import { topHelp } from './help.js';
|
|
6
|
+
import { inboxCommand } from './inbox-command.js';
|
|
6
7
|
import { journeyCommand } from './journey-command.js';
|
|
7
8
|
import { lookCommand } from './look-command.js';
|
|
8
9
|
import { printFailure } from './output.js';
|
|
@@ -12,6 +13,7 @@ const COMMANDS = {
|
|
|
12
13
|
run: { summary: 'hand a browser goal to Jev (one-shot, or a session with --session)', run: runCommand },
|
|
13
14
|
look: { summary: 'print what a page or a session\'s tab shows, with no model call', run: lookCommand },
|
|
14
15
|
journey: { summary: 'map mobile user journeys: start, step, autopilot, annotate, finish', run: journeyCommand },
|
|
16
|
+
inbox: { summary: 'a throwaway email inbox for test and demo sign-ups: create, wait, list, read', run: inboxCommand },
|
|
15
17
|
sessions: { summary: 'list the open sessions and journeys', run: sessionsCommand },
|
|
16
18
|
end: { summary: 'end a session and close the tabs it opened', run: endCommand },
|
|
17
19
|
serve: { summary: 'run the session daemon in the foreground', run: serveCommand },
|