@revyl/verify 0.4.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +31 -3
- package/README.md +19 -14
- package/dist/agent/loop.d.ts +43 -1
- package/dist/agent/loop.js +118 -20
- package/dist/computer-session.d.ts +5 -0
- package/dist/computer-session.js +15 -0
- package/dist/index.d.ts +1 -1
- package/dist/session.d.ts +37 -5
- package/dist/session.js +66 -9
- package/docs/agents.md +8 -4
- package/docs/api.md +15 -7
- package/docs/errors.md +1 -1
- package/docs/llms.txt +3 -1
- package/docs/mobile.md +31 -6
- package/docs/own-model.md +4 -4
- package/docs/web.md +35 -5
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,34 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
## 0.
|
|
3
|
+
## 0.5.0 (unreleased)
|
|
4
|
+
|
|
5
|
+
- Early access: more exact actions on Web, iOS and Android. `doubleTap(at)` double-taps (double-clicks on Web), `longPress(at, { duration })` presses and holds for `duration` milliseconds (1 to 5000, 1500 by default), and `drag(from, to)` presses at one location, moves to the other and releases. They take a description or screenshot pixels like `tap`, and `doubleTap` and `longPress` resolve with the point.
|
|
6
|
+
- Web: `typeFocused(text, { clear })` types into the field that already has focus, without clicking, and replaces its text unless `clear` is `false`, the same default as `type()`. `pressKey` also takes `'tab'`, `'escape'`, `'delete'`, `'space'`, `'arrowup'`, `'arrowdown'`, `'arrowleft'`, `'arrowright'`, `'home'`, `'end'`, `'pageup'` and `'pagedown'`, and `{ modifiers }`: up to four of `'ctrl'`, `'shift'`, `'alt'` and `'meta'`, with which a lowercase letter or digit works too, as in `pressKey('a', { modifiers: ['ctrl'] })`. The `Key` and `KeyModifier` types are exported.
|
|
7
|
+
- iOS and Android: `goHome()` goes to the home screen.
|
|
8
|
+
- A call the platform lacks throws `NOT_AVAILABLE_ON_PLATFORM` before anything is sent, and the message names what to use: `typeFocused` on iOS and Android (use `type(location, text)`), keys other than `'enter'` and `'backspace'` or any modifiers on iOS and Android, and `goHome` on Web. Computers refuse all of them, like the other exact actions.
|
|
9
|
+
- Your own model can choose `double_tap`, `long_press` and `drag` on every platform, `type_focused` and the wider `press_key` on Web, and `go_home` on iOS and Android, with both groundings. The model sees these extra tools in its tool list; the tools it had before send the same requests as 0.4.1.
|
|
10
|
+
- Every existing call sends the same requests as 0.4.1. Your own model choosing `open_url` with a deep link on Web gets the `openUrl()` wording.
|
|
11
|
+
- Docs: the platform table, `api.md`, the web and mobile guides with a script for the new calls, `agents.md` and `llms.txt`.
|
|
12
|
+
|
|
13
|
+
## Go 0.4.0 (unreleased)
|
|
14
|
+
|
|
15
|
+
Published as the `go/v0.4.0` tag.
|
|
16
|
+
|
|
17
|
+
- Early access: more exact actions on Web, iOS and Android, as in TypeScript 0.5.0. `DoubleTap(ctx, at)` and `LongPress(ctx, at, LongPressOptions{Duration})` return the point; `Duration` is whole milliseconds from 1 millisecond to 5 seconds, and zero leaves Revyl's 1.5 seconds. `Drag(ctx, from, to)` presses, moves and releases.
|
|
18
|
+
- Web: `TypeFocused(ctx, text, TypeOptions{Clear})` types into the field that already has focus and replaces its text unless `Clear` points to `false`. `PressKey(ctx, key, KeyOptions{Modifiers})` adds `KeyTab`, `KeyEscape`, `KeyDelete`, `KeySpace`, `KeyArrowUp`, `KeyArrowDown`, `KeyArrowLeft`, `KeyArrowRight`, `KeyHome`, `KeyEnd`, `KeyPageUp` and `KeyPageDown`, and modifiers (`ModifierCtrl`, `ModifierShift`, `ModifierAlt`, `ModifierMeta`), with which `Key("a")` and other lowercase letters and digits work too. Existing `PressKey(ctx, key)` calls compile unchanged.
|
|
19
|
+
- iOS and Android: `GoHome(ctx)`.
|
|
20
|
+
- A call the platform lacks returns `ErrNotAvailableOnPlatform` before anything is sent, naming the alternative. Your own model can choose the new actions with both groundings, so it sees more tools; the tools it had send the same requests as v0.3.1, as does every existing call.
|
|
21
|
+
- `ErrInstructionFailed` names the step and the reason, as TypeScript does: `Instruction failed: "<text>". <reason> See <link>`. The uncertain-action message says "The step deadline passed". Your own model choosing `open_url` with a deep link on Web gets the `OpenURL` wording.
|
|
22
|
+
|
|
23
|
+
## Python 0.3.0 (unreleased)
|
|
24
|
+
|
|
25
|
+
- Early access: more exact actions on Web, iOS and Android, as in TypeScript 0.5.0. `double_tap(at)` and `long_press(at, duration=None)` return the point; `duration` is seconds, 0.001 to 5 in whole milliseconds, 1.5 by default. `drag(start, end)` presses, moves and releases.
|
|
26
|
+
- Web: `type_focused(text, clear=True)` types into the field that already has focus and replaces its text unless `clear` is `False`. `press_key(key, modifiers=())` adds `"tab"`, `"escape"`, `"delete"`, `"space"`, the arrows, `"home"`, `"end"`, `"pageup"` and `"pagedown"`, and modifiers (`"ctrl"`, `"shift"`, `"alt"`, `"meta"`), with which a lowercase letter or digit works too. `Key` and `KeyModifier` are exported.
|
|
27
|
+
- iOS and Android: `go_home()`. `AsyncSession` has every new method.
|
|
28
|
+
- A call the platform lacks raises `NOT_AVAILABLE_ON_PLATFORM` before anything is sent, naming the alternative. Your own model can choose the new actions with both groundings, so it sees more tools; the tools it had send the same requests as 0.2.1, as does every existing call.
|
|
29
|
+
- `INSTRUCTION_FAILED` names the step and the reason, as TypeScript does: `Instruction failed: "<text>". <reason> See <link>`, and the progress line for a failed step on your own model ends with the reason. Your own model choosing `open_url` with a deep link on Web gets the `open_url()` wording.
|
|
30
|
+
|
|
31
|
+
## 0.4.1 (2026-11-12)
|
|
4
32
|
|
|
5
33
|
- **Breaking for iOS exact actions (early access):** coordinates are now screenshot pixels on every platform. On iOS, `tap`, `type`, `swipe`'s `from`, `locate`, `screenSize` and the points `tap` and `type` return used the screen's points (440 × 956 on an iPhone whose screenshots are 1320 × 2868); they now use the screenshot's pixels, and Verify converts them to points for you. A script that passed iOS points must pass screenshot pixels instead: multiply by the screenshot's width over 0.4.0's `screenSize()` width (3 on that iPhone). Android and Web coordinates are unchanged, because their screenshots already match the screen.
|
|
6
34
|
- A point outside the screenshot is refused with `INVALID_ARGUMENT` before any action is sent, and the message names the valid range. The first call in a session that takes or returns a point reads the screen size once (on iOS, plus one screenshot), and later calls reuse it. Before refusing a point, Verify reads the size again, so a rotated device or a resized Web viewport doesn't get a valid point refused. On Android and Web, if the size can't be read, the action is sent as in 0.4.0; on iOS the call fails, because the scale is needed. Calls by description send the same requests as 0.4.0 on Android and Web.
|
|
@@ -8,7 +36,7 @@
|
|
|
8
36
|
- `instruction()` and `validation()` on your own model are unchanged: the model still sees screenshots, and Verify still maps its answers to the device.
|
|
9
37
|
- Docs: one coordinate rule in the same words for TypeScript, Python and Go, in the README, `api.md`, the web and mobile guides, `agents.md` and `llms.txt`. Web coordinates are screenshot pixels, which are CSS pixels at the default device scale factor of 1.
|
|
10
38
|
|
|
11
|
-
## Go 0.3.1 (
|
|
39
|
+
## Go 0.3.1 (2026-11-12)
|
|
12
40
|
|
|
13
41
|
Published as the `go/v0.3.1` tag.
|
|
14
42
|
|
|
@@ -16,7 +44,7 @@ Published as the `go/v0.3.1` tag.
|
|
|
16
44
|
- A point outside the screenshot returns `ErrInvalidArgument` before any action is sent, naming the valid range, after Verify reads the size again in case the screen rotated. The screen size is read once per session. On Android and Web a failed size read skips the check and sends the action as before; on iOS it returns the error.
|
|
17
45
|
- README: the same coordinate rule as TypeScript and Python.
|
|
18
46
|
|
|
19
|
-
## Python 0.2.1 (
|
|
47
|
+
## Python 0.2.1 (2026-11-12)
|
|
20
48
|
|
|
21
49
|
- **Breaking for iOS exact actions (early access):** coordinates are now screenshot pixels on every platform, as in TypeScript 0.4.1. On iOS, `tap`, `type`, `swipe`'s `from_`, `locate`, `screen_size` and the points `tap` and `type` return used the screen's points; they now use the screenshot's pixels, and Verify converts them to points for you. Android and Web coordinates are unchanged. `AsyncSession` behaves the same.
|
|
22
50
|
- A point outside the screenshot raises `INVALID_ARGUMENT` before any action is sent, naming the valid range, after Verify reads the size again in case the screen rotated. The screen size is read once per session. On Android and Web a failed size read skips the check and sends the action as before; on iOS it raises.
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Verify
|
|
2
2
|
|
|
3
|
-

|
|
4
4
|
|
|
5
5
|
Verify runs end-to-end checks in a cloud browser, iOS simulator, or Android emulator on Revyl. You write what the agent should do and what must be true afterward, in plain sentences, and call it from a script, a coding agent, CI, or the test runner you already use. When you know exactly what to press, you can tap, type and swipe yourself instead. Verify also opens Mac computers for commands, files, and screenshots.
|
|
6
6
|
|
|
@@ -41,24 +41,26 @@ Or set it up yourself:
|
|
|
41
41
|
npx @revyl/verify init
|
|
42
42
|
```
|
|
43
43
|
|
|
44
|
-
`init` adds Verify to your project, asks what to open, writes `verify.config.ts` and an example script, signs you in through the browser on the first interactive run, and prints the command that runs the example. With no app id, the iOS and Android examples open the built-in Settings app, so they run on a new account before you upload anything. To skip the config, run `npm install @revyl/verify` and save the example above as `verify/check.mts`, then run `node verify/check.mts`. [Getting started](https://github.com/RevylAI/verify/blob/v0.
|
|
44
|
+
`init` adds Verify to your project, asks what to open, writes `verify.config.ts` and an example script, signs you in through the browser on the first interactive run, and prints the command that runs the example. With no app id, the iOS and Android examples open the built-in Settings app, so they run on a new account before you upload anything. To skip the config, run `npm install @revyl/verify` and save the example above as `verify/check.mts`, then run `node verify/check.mts`. [Getting started](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/getting-started.md) covers sign-in, your own app, and [CI](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/ci.md).
|
|
45
45
|
|
|
46
|
-
For finished projects, see [`examples/`](https://github.com/RevylAI/verify/tree/v0.
|
|
46
|
+
For finished projects, see [`examples/`](https://github.com/RevylAI/verify/tree/v0.5.0/examples/): a [web store](https://github.com/RevylAI/verify/tree/v0.5.0/examples/web-store/), [mobile Settings](https://github.com/RevylAI/verify/tree/v0.5.0/examples/mobile-settings/), a [Mac workspace](https://github.com/RevylAI/verify/tree/v0.5.0/examples/mac-workspace/), a small [agent platform](https://github.com/RevylAI/verify/tree/v0.5.0/examples/agent-platform/) that runs web and iOS side by side, and [your own model](https://github.com/RevylAI/verify/tree/v0.5.0/examples/byo-model/) (with [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/examples/byo_model.py) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/examples/byomodel/main.go) versions).
|
|
47
47
|
|
|
48
48
|
## Exact actions
|
|
49
49
|
|
|
50
|
-
Tap, type, swipe and press keys yourself, by a description Revyl locates on screen or by coordinates, in the same script as your steps:
|
|
50
|
+
Tap, double-tap, long-press, drag, type, swipe and press keys yourself, by a description Revyl locates on screen or by coordinates, in the same script as your steps:
|
|
51
51
|
|
|
52
52
|
```ts
|
|
53
53
|
await session.type({ target: 'Email field' }, 'demo@example.com'); // Revyl finds the element you describe
|
|
54
54
|
await session.pressKey('enter');
|
|
55
55
|
await session.swipe('up'); // scrolls the content down
|
|
56
56
|
await session.tap({ x: 196, y: 640 }); // or a point in screenshot pixels
|
|
57
|
+
await session.longPress({ target: 'First message row' }); // press and hold, 1.5 s by default
|
|
58
|
+
await session.drag({ target: 'Third item handle' }, { target: 'First item' }); // press, move, release
|
|
57
59
|
await session.instruction('Accept the terms if they are shown'); // the agent where it's fuzzy
|
|
58
60
|
await session.validation('The welcome screen is shown');
|
|
59
61
|
```
|
|
60
62
|
|
|
61
|
-
`back`, `openUrl`, `launchApp`, `locate` and `screenSize` round them out; [which call works where](#which-call-works-where) shows the platform limits. Early access.
|
|
63
|
+
`doubleTap`, `typeFocused`, `back`, `goHome`, `openUrl`, `launchApp`, `locate` and `screenSize` round them out; [which call works where](#which-call-works-where) shows the platform limits. Early access. Since TypeScript (`@revyl/verify`) 0.4.0, Python (`revyl-verify`) 0.2.0 and Go v0.3.0. `doubleTap`, `longPress`, `drag`, `typeFocused`, `goHome`, and Web keys and modifiers in `pressKey`, are new in TypeScript 0.5.0, Python 0.3.0 and Go v0.4.0. The [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#exact-actions) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#exact-actions) READMEs show them with each language's names. The [web](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/web.md#exact-actions) and [mobile](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/mobile.md#exact-actions) guides have runnable scripts, and [api.md](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/api.md#exact-actions) lists every call.
|
|
62
64
|
|
|
63
65
|
## Your own model
|
|
64
66
|
|
|
@@ -72,7 +74,7 @@ await session.instruction('Open Privacy', { model: 'openai/gpt-5.5' });
|
|
|
72
74
|
await open('ios', { model: { id: 'qwen-3.8-27b', baseURL: 'https://api.cerebras.ai/v1', apiKey: process.env.CEREBRAS_API_KEY } });
|
|
73
75
|
```
|
|
74
76
|
|
|
75
|
-
|
|
77
|
+
Since TypeScript (`@revyl/verify`) 0.4.0, Python (`revyl-verify`) 0.2.0 and Go v0.3.0; the [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#your-own-model) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#your-own-model) READMEs show them with each language's names. Built in: `anthropic`, `openai`, `google`, `openrouter`, `groq`, `cerebras`, `xai` and `ollama`; anything OpenAI-compatible works through `baseURL`. Early access; [Use your own model](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/own-model.md) covers the providers, picking a model, what Revyl still sees, limits and troubleshooting.
|
|
76
78
|
|
|
77
79
|
## Packages
|
|
78
80
|
|
|
@@ -80,8 +82,8 @@ In TypeScript (`@revyl/verify` 0.4.0), Python (`revyl-verify` 0.2.0) and Go (v0.
|
|
|
80
82
|
| --- | --- | --- |
|
|
81
83
|
| [`@revyl/verify`](https://www.npmjs.com/package/@revyl/verify) | `npm install @revyl/verify` | TypeScript SDK and the `verify` CLI (`init`, `login`, `logout`, `whoami`). Published on npm, Node.js 22.18 or later. |
|
|
82
84
|
| [`revyl-verify`](https://www.npmjs.com/package/revyl-verify) (npm) | `npm install revyl-verify` | Alias of `@revyl/verify` at the same version. |
|
|
83
|
-
| [Go module](https://github.com/RevylAI/verify/tree/go/v0.
|
|
84
|
-
| [`revyl-verify`](https://pypi.org/project/revyl-verify/) (PyPI) | `pip install revyl-verify` or `uv add revyl-verify` | Python SDK, published on PyPI. Python 3.10 or later, sync and asyncio APIs, no CLI; see the [Python guide](https://github.com/RevylAI/verify/blob/v0.
|
|
85
|
+
| [Go module](https://github.com/RevylAI/verify/tree/go/v0.4.0/go) | `go get github.com/RevylAI/verify/go@v0.4.0` | Published as the `go/v0.4.0` tag, Go 1.26.9 or later. Calls return errors instead of throwing. |
|
|
86
|
+
| [`revyl-verify`](https://pypi.org/project/revyl-verify/) (PyPI) | `pip install revyl-verify` or `uv add revyl-verify` | Python SDK, published on PyPI. Python 3.10 or later, sync and asyncio APIs, no CLI; see the [Python guide](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md). |
|
|
85
87
|
|
|
86
88
|
## Platforms
|
|
87
89
|
|
|
@@ -96,8 +98,11 @@ In TypeScript (`@revyl/verify` 0.4.0), Python (`revyl-verify` 0.2.0) and Go (v0.
|
|
|
96
98
|
|
|
97
99
|
| Call | Web | iOS | Android | Mac computer |
|
|
98
100
|
| --- | --- | --- | --- | --- |
|
|
99
|
-
| `tap`, `type`, `swipe`, `
|
|
101
|
+
| `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `locate`, `screenSize` | Yes | Yes | Yes | No, use `exec()` and files |
|
|
102
|
+
| `pressKey(key, { modifiers })` | Yes: `'enter'`, `'backspace'`, `'tab'`, `'escape'`, arrows and more, with modifiers | `'enter'` and `'backspace'` only | `'enter'` and `'backspace'` only | No |
|
|
103
|
+
| `typeFocused(text)` | Yes, types into the focused field | No, use `type(location, text)` | No, use `type(location, text)` | No |
|
|
100
104
|
| `back()` | Yes, browser history | No, tap the on-screen back control | Yes, the system back button | No |
|
|
105
|
+
| `goHome()` | No, use `openUrl` or `navigate` | Yes, the home screen | Yes, the home screen | No |
|
|
101
106
|
| `openUrl(url)` | Yes, loads an http(s) URL in the tab | Yes, a URL or deep link | Yes, a URL or deep link | No |
|
|
102
107
|
| `launchApp(bundleId)` | No, use `openUrl` or `navigate` | Yes | Yes | No |
|
|
103
108
|
| `navigate(url)` | Yes | No, use `openUrl` | No, use `openUrl` | No |
|
|
@@ -105,20 +110,20 @@ In TypeScript (`@revyl/verify` 0.4.0), Python (`revyl-verify` 0.2.0) and Go (v0.
|
|
|
105
110
|
| `screenshot()` | Yes | Yes | Yes | Yes |
|
|
106
111
|
| `exec`, files, processes | No, open a Mac computer | No, open a Mac computer | No, open a Mac computer | Yes |
|
|
107
112
|
|
|
108
|
-
**Coordinates.** Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()
|
|
113
|
+
**Coordinates.** Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()`, `swipe()` and every other exact action unchanged. `screenSize()` returns the screenshot's width and height, and `locate()` and the points `tap()`, `doubleTap()`, `longPress()` and `type()` return use the same pixels. A point outside the screenshot is refused with `INVALID_ARGUMENT` before any action is sent, and the message names the valid range. On iOS, Verify converts the pixels to the screen's points for you. On Web they equal CSS pixels at the default device scale factor of 1, and you still pass screenshot pixels at any other. This holds from TypeScript 0.4.1, Python 0.2.1 and Go v0.3.1; earlier versions took iOS points.
|
|
109
114
|
|
|
110
115
|
A call a platform doesn't support throws `NOT_AVAILABLE_ON_PLATFORM`, and its message names the alternative.
|
|
111
116
|
|
|
112
|
-
Web is available to every organization. Mac computers are early access and have a server-bounded lifetime; see [computers](https://github.com/RevylAI/verify/blob/v0.
|
|
117
|
+
Web is available to every organization. Mac computers are early access and have a server-bounded lifetime; see [computers](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/computers.md).
|
|
113
118
|
|
|
114
119
|
## Documentation
|
|
115
120
|
|
|
116
|
-
The [documentation](https://github.com/RevylAI/verify/tree/v0.
|
|
121
|
+
The [documentation](https://github.com/RevylAI/verify/tree/v0.5.0/packages/verify/docs/) covers the [API](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/api.md), [configuration](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/configuration.md), [web](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/web.md), [mobile](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/mobile.md), [computers](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/computers.md), [CI](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/ci.md), [your own model](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/own-model.md), [errors and troubleshooting](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/errors.md), [coding agents](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/agents.md), and [live end-to-end results](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/end-to-end.md). The npm package ships these pages in `node_modules/@revyl/verify/docs/`, with an [llms.txt](https://github.com/RevylAI/verify/blob/v0.5.0/packages/verify/docs/llms.txt) index, so a coding agent can read the docs for the version you installed. The Revyl docs site has an overview at [docs.revyl.com/verify](https://docs.revyl.com/verify).
|
|
117
122
|
|
|
118
123
|
## Status
|
|
119
124
|
|
|
120
|
-
Verify is at 0.x, so APIs can still change between minor releases; the [changelog](https://github.com/RevylAI/verify/blob/v0.
|
|
125
|
+
Verify is at 0.x, so APIs can still change between minor releases; the [changelog](https://github.com/RevylAI/verify/blob/v0.5.0/CHANGELOG.md) records each release. Mac computers are early access. See [Contributing](https://github.com/RevylAI/verify/blob/v0.5.0/CONTRIBUTING.md) and [Releasing](https://github.com/RevylAI/verify/blob/v0.5.0/RELEASING.md) to work on Verify.
|
|
121
126
|
|
|
122
127
|
## License
|
|
123
128
|
|
|
124
|
-
[Apache 2.0](https://github.com/RevylAI/verify/blob/v0.
|
|
129
|
+
[Apache 2.0](https://github.com/RevylAI/verify/blob/v0.5.0/LICENSE).
|
package/dist/agent/loop.d.ts
CHANGED
|
@@ -10,27 +10,69 @@ export interface Target {
|
|
|
10
10
|
}
|
|
11
11
|
export type Location = Point | Target;
|
|
12
12
|
export type SwipeDirection = 'up' | 'down' | 'left' | 'right';
|
|
13
|
+
/** Keys pressKey() takes on Web. iOS and Android take only 'enter' and 'backspace'. */
|
|
14
|
+
export declare const WEB_KEYS: {
|
|
15
|
+
readonly enter: "Enter";
|
|
16
|
+
readonly backspace: "Backspace";
|
|
17
|
+
readonly tab: "Tab";
|
|
18
|
+
readonly escape: "Escape";
|
|
19
|
+
readonly delete: "Delete";
|
|
20
|
+
readonly space: "Space";
|
|
21
|
+
readonly arrowup: "ArrowUp";
|
|
22
|
+
readonly arrowdown: "ArrowDown";
|
|
23
|
+
readonly arrowleft: "ArrowLeft";
|
|
24
|
+
readonly arrowright: "ArrowRight";
|
|
25
|
+
readonly home: "Home";
|
|
26
|
+
readonly end: "End";
|
|
27
|
+
readonly pageup: "PageUp";
|
|
28
|
+
readonly pagedown: "PageDown";
|
|
29
|
+
};
|
|
30
|
+
export type Key = keyof typeof WEB_KEYS;
|
|
31
|
+
export type KeyModifier = 'ctrl' | 'shift' | 'alt' | 'meta';
|
|
13
32
|
export type DeviceAction = {
|
|
14
33
|
kind: 'tap';
|
|
15
34
|
at: Location;
|
|
35
|
+
} | {
|
|
36
|
+
kind: 'double_tap';
|
|
37
|
+
at: Location;
|
|
38
|
+
} | {
|
|
39
|
+
kind: 'long_press';
|
|
40
|
+
at: Location;
|
|
41
|
+
durationMs?: number;
|
|
16
42
|
} | {
|
|
17
43
|
kind: 'type';
|
|
18
44
|
at: Location;
|
|
19
45
|
text: string;
|
|
20
46
|
clear: boolean;
|
|
47
|
+
} | {
|
|
48
|
+
kind: 'type_focused';
|
|
49
|
+
text: string;
|
|
50
|
+
clear: boolean;
|
|
21
51
|
} | {
|
|
22
52
|
kind: 'swipe';
|
|
23
53
|
direction: SwipeDirection;
|
|
24
54
|
from?: Location;
|
|
55
|
+
} | {
|
|
56
|
+
kind: 'drag';
|
|
57
|
+
from: Location;
|
|
58
|
+
to: Location;
|
|
25
59
|
} | {
|
|
26
60
|
kind: 'key';
|
|
27
|
-
key:
|
|
61
|
+
key: string;
|
|
62
|
+
modifiers?: KeyModifier[];
|
|
28
63
|
} | {
|
|
29
64
|
kind: 'back';
|
|
65
|
+
} | {
|
|
66
|
+
kind: 'go_home';
|
|
30
67
|
} | {
|
|
31
68
|
kind: 'open_url';
|
|
32
69
|
url: string;
|
|
33
70
|
};
|
|
71
|
+
/** Checks a key and its modifiers for the platform, before any request. */
|
|
72
|
+
export declare function checkKey(platform: string, key: unknown, modifiers?: unknown): {
|
|
73
|
+
key: string;
|
|
74
|
+
modifiers: KeyModifier[];
|
|
75
|
+
};
|
|
34
76
|
/** What the agent loop needs from a session. DeviceSession provides it. */
|
|
35
77
|
export interface AgentControls {
|
|
36
78
|
readonly platform: 'ios' | 'android' | 'web';
|
package/dist/agent/loop.js
CHANGED
|
@@ -1,6 +1,40 @@
|
|
|
1
1
|
import { setTimeout as delay } from 'node:timers/promises';
|
|
2
2
|
import { VerifyError, errorMessage, isVerifyError } from '../errors.js';
|
|
3
3
|
import { fitPng } from './image.js';
|
|
4
|
+
/** Keys pressKey() takes on Web. iOS and Android take only 'enter' and 'backspace'. */
|
|
5
|
+
export const WEB_KEYS = {
|
|
6
|
+
enter: 'Enter',
|
|
7
|
+
backspace: 'Backspace',
|
|
8
|
+
tab: 'Tab',
|
|
9
|
+
escape: 'Escape',
|
|
10
|
+
delete: 'Delete',
|
|
11
|
+
space: 'Space',
|
|
12
|
+
arrowup: 'ArrowUp',
|
|
13
|
+
arrowdown: 'ArrowDown',
|
|
14
|
+
arrowleft: 'ArrowLeft',
|
|
15
|
+
arrowright: 'ArrowRight',
|
|
16
|
+
home: 'Home',
|
|
17
|
+
end: 'End',
|
|
18
|
+
pageup: 'PageUp',
|
|
19
|
+
pagedown: 'PageDown',
|
|
20
|
+
};
|
|
21
|
+
/** Checks a key and its modifiers for the platform, before any request. */
|
|
22
|
+
export function checkKey(platform, key, modifiers = []) {
|
|
23
|
+
if (!Array.isArray(modifiers) || modifiers.length > 4 || !modifiers.every((modifier) => ['ctrl', 'shift', 'alt', 'meta'].includes(modifier))) {
|
|
24
|
+
throw new VerifyError('INVALID_ARGUMENT', "pressKey() modifiers are up to four of 'ctrl', 'shift', 'alt' and 'meta'.");
|
|
25
|
+
}
|
|
26
|
+
if (platform !== 'web') {
|
|
27
|
+
if ((key === 'enter' || key === 'backspace') && !modifiers.length)
|
|
28
|
+
return { key, modifiers: [] };
|
|
29
|
+
if (modifiers.length || (typeof key === 'string' && key in WEB_KEYS)) {
|
|
30
|
+
throw new VerifyError('NOT_AVAILABLE_ON_PLATFORM', "On iOS and Android, pressKey() takes only 'enter' or 'backspace', with no modifiers; tap the on-screen control instead.");
|
|
31
|
+
}
|
|
32
|
+
throw new VerifyError('INVALID_ARGUMENT', "pressKey() takes 'enter' or 'backspace'.");
|
|
33
|
+
}
|
|
34
|
+
if (typeof key === 'string' && (key in WEB_KEYS || (/^[a-z0-9]$/.test(key) && modifiers.length)))
|
|
35
|
+
return { key, modifiers: [...modifiers] };
|
|
36
|
+
throw new VerifyError('INVALID_ARGUMENT', `pressKey() takes one of ${Object.keys(WEB_KEYS).map((name) => `'${name}'`).join(', ')}, or a lowercase letter or digit with modifiers, such as pressKey('a', { modifiers: ['ctrl'] }).`);
|
|
37
|
+
}
|
|
4
38
|
/** Runs instructions and validations on a model you choose, on your machine. Revyl only receives device actions. */
|
|
5
39
|
export function localAgent(options) {
|
|
6
40
|
if (!options || typeof options.model?.complete !== 'function' || typeof options.model.name !== 'string') {
|
|
@@ -93,45 +127,67 @@ export async function runLocalStep(type, task, agent, controls, signal, log) {
|
|
|
93
127
|
}
|
|
94
128
|
function toAction(call, platform, grounding, image, size) {
|
|
95
129
|
const args = call.arguments;
|
|
96
|
-
const location = (required) => {
|
|
130
|
+
const location = (required, prefix = '') => {
|
|
131
|
+
const [target, x, y] = [args[`${prefix}target`], args[`${prefix}x`], args[`${prefix}y`]];
|
|
97
132
|
if (grounding === 'revyl') {
|
|
98
|
-
if (typeof
|
|
99
|
-
return { target:
|
|
133
|
+
if (typeof target === 'string' && target.trim())
|
|
134
|
+
return { target: target.trim().slice(0, 500) };
|
|
100
135
|
if (required)
|
|
101
|
-
throw new VerifyError('INVALID_ARGUMENT',
|
|
136
|
+
throw new VerifyError('INVALID_ARGUMENT', `${prefix}target is required: describe the element by its visible text or appearance.`);
|
|
102
137
|
return undefined;
|
|
103
138
|
}
|
|
104
|
-
if (typeof
|
|
139
|
+
if (typeof x !== 'number' || typeof y !== 'number') {
|
|
105
140
|
if (required)
|
|
106
|
-
throw new VerifyError('INVALID_ARGUMENT',
|
|
141
|
+
throw new VerifyError('INVALID_ARGUMENT', `${prefix}x and ${prefix}y are required.`);
|
|
107
142
|
return undefined;
|
|
108
143
|
}
|
|
109
|
-
if (
|
|
110
|
-
throw new VerifyError('INVALID_ARGUMENT', `(${
|
|
144
|
+
if (x < 0 || y < 0 || x >= image.width || y >= image.height) {
|
|
145
|
+
throw new VerifyError('INVALID_ARGUMENT', `(${x}, ${y}) is outside the ${image.width}x${image.height} screenshot.`);
|
|
111
146
|
}
|
|
112
|
-
return { x: Math.round((
|
|
147
|
+
return { x: Math.round((x * size.width) / image.width), y: Math.round((y * size.height) / image.height) };
|
|
113
148
|
};
|
|
114
149
|
switch (call.name) {
|
|
115
150
|
case 'tap':
|
|
116
151
|
return { kind: 'tap', at: location(true) };
|
|
152
|
+
case 'double_tap':
|
|
153
|
+
return { kind: 'double_tap', at: location(true) };
|
|
154
|
+
case 'long_press':
|
|
155
|
+
return { kind: 'long_press', at: location(true) };
|
|
117
156
|
case 'type':
|
|
118
157
|
if (typeof args.text !== 'string')
|
|
119
158
|
throw new VerifyError('INVALID_ARGUMENT', 'text is required.');
|
|
120
159
|
return { kind: 'type', at: location(true), text: args.text, clear: args.clear !== false };
|
|
160
|
+
case 'type_focused':
|
|
161
|
+
if (platform !== 'web')
|
|
162
|
+
throw new VerifyError('NOT_AVAILABLE_ON_PLATFORM', 'type_focused is available only in a web browser; use type, which taps the field first.');
|
|
163
|
+
if (typeof args.text !== 'string')
|
|
164
|
+
throw new VerifyError('INVALID_ARGUMENT', 'text is required.');
|
|
165
|
+
return { kind: 'type_focused', text: args.text, clear: args.clear !== false };
|
|
166
|
+
case 'drag':
|
|
167
|
+
return { kind: 'drag', from: location(true, 'from_'), to: location(true, 'to_') };
|
|
121
168
|
case 'swipe': {
|
|
122
169
|
if (!['up', 'down', 'left', 'right'].includes(String(args.direction)))
|
|
123
170
|
throw new VerifyError('INVALID_ARGUMENT', 'direction must be up, down, left or right.');
|
|
124
171
|
const from = location(false);
|
|
125
172
|
return { kind: 'swipe', direction: args.direction, ...(from ? { from } : {}) };
|
|
126
173
|
}
|
|
127
|
-
case 'press_key':
|
|
128
|
-
if (
|
|
129
|
-
|
|
130
|
-
|
|
174
|
+
case 'press_key': {
|
|
175
|
+
if (platform !== 'web') {
|
|
176
|
+
if (args.key !== 'enter' && args.key !== 'backspace')
|
|
177
|
+
throw new VerifyError('INVALID_ARGUMENT', 'key must be enter or backspace.');
|
|
178
|
+
return { kind: 'key', key: args.key };
|
|
179
|
+
}
|
|
180
|
+
const { key, modifiers } = checkKey(platform, args.key, args.modifiers ?? []);
|
|
181
|
+
return { kind: 'key', key, ...(modifiers.length ? { modifiers } : {}) };
|
|
182
|
+
}
|
|
131
183
|
case 'back':
|
|
132
184
|
if (platform === 'ios')
|
|
133
185
|
throw new VerifyError('NOT_AVAILABLE_ON_PLATFORM', 'iOS has no back button; tap the on-screen back control.');
|
|
134
186
|
return { kind: 'back' };
|
|
187
|
+
case 'go_home':
|
|
188
|
+
if (platform === 'web')
|
|
189
|
+
throw new VerifyError('NOT_AVAILABLE_ON_PLATFORM', 'A web browser has no home screen; use open_url to load a page.');
|
|
190
|
+
return { kind: 'go_home' };
|
|
135
191
|
case 'open_url':
|
|
136
192
|
if (typeof args.url !== 'string' || !args.url)
|
|
137
193
|
throw new VerifyError('INVALID_ARGUMENT', 'url is required.');
|
|
@@ -145,14 +201,24 @@ function describe(action) {
|
|
|
145
201
|
switch (action.kind) {
|
|
146
202
|
case 'tap':
|
|
147
203
|
return `tap${where(action.at)}`;
|
|
204
|
+
case 'double_tap':
|
|
205
|
+
return `double tap${where(action.at)}`;
|
|
206
|
+
case 'long_press':
|
|
207
|
+
return `long press${where(action.at)}`;
|
|
148
208
|
case 'type':
|
|
149
209
|
return `type ${JSON.stringify(action.text.length > 40 ? `${action.text.slice(0, 40)}…` : action.text)} into${where(action.at)}`;
|
|
210
|
+
case 'type_focused':
|
|
211
|
+
return `type ${JSON.stringify(action.text.length > 40 ? `${action.text.slice(0, 40)}…` : action.text)} into the focused field`;
|
|
150
212
|
case 'swipe':
|
|
151
213
|
return `swipe ${action.direction}${action.from ? ` from${where(action.from)}` : ''}`;
|
|
214
|
+
case 'drag':
|
|
215
|
+
return `drag from${where(action.from)} to${where(action.to)}`;
|
|
152
216
|
case 'key':
|
|
153
|
-
return `press ${action.key}`;
|
|
217
|
+
return `press ${[...(action.modifiers ?? []), action.key].join('+')}`;
|
|
154
218
|
case 'back':
|
|
155
219
|
return 'back';
|
|
220
|
+
case 'go_home':
|
|
221
|
+
return 'home';
|
|
156
222
|
case 'open_url':
|
|
157
223
|
return `open ${action.url}`;
|
|
158
224
|
}
|
|
@@ -188,25 +254,57 @@ function instructionTools(platform, grounding) {
|
|
|
188
254
|
const at = grounding === 'revyl'
|
|
189
255
|
? { properties: { target: { type: 'string', description: 'The element, by its visible text or look.' } }, required: ['target'] }
|
|
190
256
|
: { properties: { x: { type: 'integer', description: 'Screenshot pixel x.' }, y: { type: 'integer', description: 'Screenshot pixel y.' } }, required: ['x', 'y'] };
|
|
191
|
-
const
|
|
257
|
+
const prefixed = (prefix) => ({
|
|
258
|
+
properties: Object.fromEntries(Object.entries(at.properties).map(([name, schema]) => [`${prefix}${name}`, schema])),
|
|
259
|
+
required: at.required.map((name) => `${prefix}${name}`),
|
|
260
|
+
});
|
|
261
|
+
const [from, to] = [prefixed('from_'), prefixed('to_')];
|
|
262
|
+
const web = platform === 'web';
|
|
263
|
+
return [
|
|
192
264
|
{ name: 'tap', description: 'Tap an element.', parameters: { type: 'object', ...at } },
|
|
265
|
+
{ name: 'double_tap', description: 'Double-tap an element.', parameters: { type: 'object', ...at } },
|
|
266
|
+
{ name: 'long_press', description: 'Press and hold an element, for example to open a context menu.', parameters: { type: 'object', ...at } },
|
|
193
267
|
{
|
|
194
268
|
name: 'type',
|
|
195
269
|
description: 'Tap a text field and type into it. Clears the field first unless clear is false.',
|
|
196
270
|
parameters: { type: 'object', properties: { ...at.properties, text: { type: 'string' }, clear: { type: 'boolean' } }, required: [...at.required, 'text'] },
|
|
197
271
|
},
|
|
272
|
+
...(web
|
|
273
|
+
? [{
|
|
274
|
+
name: 'type_focused',
|
|
275
|
+
description: 'Type into the field that already has focus, without tapping. Clears the field first unless clear is false.',
|
|
276
|
+
parameters: { type: 'object', properties: { text: { type: 'string' }, clear: { type: 'boolean' } }, required: ['text'] },
|
|
277
|
+
}]
|
|
278
|
+
: []),
|
|
198
279
|
{
|
|
199
280
|
name: 'swipe',
|
|
200
281
|
description: "Swipe from a point, or from the screen's center. up moves the finger up, which scrolls content down.",
|
|
201
282
|
parameters: { type: 'object', properties: { direction: { type: 'string', enum: ['up', 'down', 'left', 'right'] }, ...at.properties }, required: ['direction'] },
|
|
202
283
|
},
|
|
203
|
-
{
|
|
204
|
-
|
|
284
|
+
{
|
|
285
|
+
name: 'drag',
|
|
286
|
+
description: 'Press an element, move it to another place and release, for example to reorder a list or move a slider.',
|
|
287
|
+
parameters: { type: 'object', properties: { ...from.properties, ...to.properties }, required: [...from.required, ...to.required] },
|
|
288
|
+
},
|
|
289
|
+
web
|
|
290
|
+
? {
|
|
291
|
+
name: 'press_key',
|
|
292
|
+
description: 'Press a key on the focused element, optionally with modifiers held, such as escape, tab, arrowdown, or a with ctrl.',
|
|
293
|
+
parameters: {
|
|
294
|
+
type: 'object',
|
|
295
|
+
properties: {
|
|
296
|
+
key: { type: 'string', description: `One of ${Object.keys(WEB_KEYS).join(', ')}, or a lowercase letter or digit when modifiers are given.` },
|
|
297
|
+
modifiers: { type: 'array', items: { type: 'string', enum: ['ctrl', 'shift', 'alt', 'meta'] } },
|
|
298
|
+
},
|
|
299
|
+
required: ['key'],
|
|
300
|
+
},
|
|
301
|
+
}
|
|
302
|
+
: { name: 'press_key', description: 'Press a key on the focused field.', parameters: { type: 'object', properties: { key: { type: 'string', enum: ['enter', 'backspace'] } }, required: ['key'] } },
|
|
303
|
+
...(platform !== 'ios' ? [{ name: 'back', description: web ? 'Go back in browser history.' : 'Press the Android back button.', parameters: { type: 'object', properties: {} } }] : []),
|
|
304
|
+
...(!web ? [{ name: 'go_home', description: 'Go to the home screen.', parameters: { type: 'object', properties: {} } }] : []),
|
|
305
|
+
{ name: 'open_url', description: web ? 'Load a URL in the current tab.' : 'Open a URL or deep link on the device.', parameters: { type: 'object', properties: { url: { type: 'string' } }, required: ['url'] } },
|
|
205
306
|
{ name: 'done', description: 'Finish the task.', parameters: { type: 'object', properties: { success: { type: 'boolean' }, reason: { type: 'string' } }, required: ['success', 'reason'] } },
|
|
206
307
|
];
|
|
207
|
-
if (platform !== 'ios')
|
|
208
|
-
tools.splice(4, 0, { name: 'back', description: platform === 'web' ? 'Go back in browser history.' : 'Press the Android back button.', parameters: { type: 'object', properties: {} } });
|
|
209
|
-
return tools;
|
|
210
308
|
}
|
|
211
309
|
function bounded(value, min, max, name) {
|
|
212
310
|
if (!Number.isInteger(value) || value < min || value > max)
|
|
@@ -29,9 +29,14 @@ export declare class ComputerSession implements Session {
|
|
|
29
29
|
navigate(): Promise<never>;
|
|
30
30
|
tap(): Promise<never>;
|
|
31
31
|
type(): Promise<never>;
|
|
32
|
+
typeFocused(): Promise<never>;
|
|
33
|
+
doubleTap(): Promise<never>;
|
|
34
|
+
longPress(): Promise<never>;
|
|
32
35
|
swipe(): Promise<never>;
|
|
36
|
+
drag(): Promise<never>;
|
|
33
37
|
pressKey(): Promise<never>;
|
|
34
38
|
back(): Promise<never>;
|
|
39
|
+
goHome(): Promise<never>;
|
|
35
40
|
openUrl(): Promise<never>;
|
|
36
41
|
launchApp(): Promise<never>;
|
|
37
42
|
locate(): Promise<never>;
|
package/dist/computer-session.js
CHANGED
|
@@ -129,15 +129,30 @@ export class ComputerSession {
|
|
|
129
129
|
type() {
|
|
130
130
|
return this.unavailable('type');
|
|
131
131
|
}
|
|
132
|
+
typeFocused() {
|
|
133
|
+
return this.unavailable('typeFocused');
|
|
134
|
+
}
|
|
135
|
+
doubleTap() {
|
|
136
|
+
return this.unavailable('doubleTap');
|
|
137
|
+
}
|
|
138
|
+
longPress() {
|
|
139
|
+
return this.unavailable('longPress');
|
|
140
|
+
}
|
|
132
141
|
swipe() {
|
|
133
142
|
return this.unavailable('swipe');
|
|
134
143
|
}
|
|
144
|
+
drag() {
|
|
145
|
+
return this.unavailable('drag');
|
|
146
|
+
}
|
|
135
147
|
pressKey() {
|
|
136
148
|
return this.unavailable('pressKey');
|
|
137
149
|
}
|
|
138
150
|
back() {
|
|
139
151
|
return this.unavailable('back');
|
|
140
152
|
}
|
|
153
|
+
goHome() {
|
|
154
|
+
return this.unavailable('goHome');
|
|
155
|
+
}
|
|
141
156
|
openUrl() {
|
|
142
157
|
return this.unavailable('openUrl');
|
|
143
158
|
}
|
package/dist/index.d.ts
CHANGED
|
@@ -7,7 +7,7 @@ export type { Platform, ComputerPlatform, MobilePlatform } from './api/types.js'
|
|
|
7
7
|
export { VerifyError, isVerifyError } from './errors.js';
|
|
8
8
|
export type { RefusalReason, VerifyErrorCode } from './errors.js';
|
|
9
9
|
export { localAgent } from './agent/loop.js';
|
|
10
|
-
export type { Grounding, LocalAgent, LocalAgentOptions, Location, ModelRun, ModelTurn, Point, SwipeDirection, Target } from './agent/loop.js';
|
|
10
|
+
export type { Grounding, Key, KeyModifier, LocalAgent, LocalAgentOptions, Location, ModelRun, ModelTurn, Point, SwipeDirection, Target } from './agent/loop.js';
|
|
11
11
|
export type { CustomModel, ModelOption } from './agent/providers.js';
|
|
12
12
|
export { anthropic, openaiCompatible } from './agent/model.js';
|
|
13
13
|
export type { AnthropicOptions, JsonSchema, ModelAdapter, ModelReply, ModelRequest, ModelTool, ModelToolCall, ModelUsage, OpenAICompatibleOptions } from './agent/model.js';
|
package/dist/session.d.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { RevylClient } from './api/client.js';
|
|
2
2
|
import type { LiveStepType, Platform } from './api/types.js';
|
|
3
3
|
import type { HostedTargetConfig, ResolvedDeviceTarget } from './config.js';
|
|
4
|
-
import { type LocalAgent, type Location, type ModelRun, type Point, type SwipeDirection } from './agent/loop.js';
|
|
4
|
+
import { type Key, type KeyModifier, type LocalAgent, type Location, type ModelRun, type Point, type SwipeDirection } from './agent/loop.js';
|
|
5
5
|
import { type ModelOption } from './agent/providers.js';
|
|
6
6
|
export type StepType = LiveStepType;
|
|
7
7
|
/** The outcome of one instruction or validation, as Revyl's agent reported it. */
|
|
@@ -109,14 +109,30 @@ export interface OpenedSession extends Session {
|
|
|
109
109
|
type(at: Location, text: string, options?: {
|
|
110
110
|
clear?: boolean;
|
|
111
111
|
}): Promise<Point>;
|
|
112
|
+
/** Types into the field that already has focus, without tapping, replacing its content unless clear is false. Web only; on iOS and Android, use type(location, text). */
|
|
113
|
+
typeFocused(text: string, options?: {
|
|
114
|
+
clear?: boolean;
|
|
115
|
+
}): Promise<void>;
|
|
116
|
+
/** Double-taps a point in screenshot pixels, or an element Revyl locates from a description. Returns the point, in screenshot pixels. Browsers and devices only. */
|
|
117
|
+
doubleTap(at: Location): Promise<Point>;
|
|
118
|
+
/** Presses and holds a point or element for duration milliseconds (1 to 5000, default 1500). Returns the point, in screenshot pixels. Browsers and devices only. */
|
|
119
|
+
longPress(at: Location, options?: {
|
|
120
|
+
duration?: number;
|
|
121
|
+
}): Promise<Point>;
|
|
112
122
|
/** Swipes from a point or the screen's center. 'up' moves the finger up, which scrolls content down. Browsers and devices only. */
|
|
113
123
|
swipe(direction: SwipeDirection, options?: {
|
|
114
124
|
from?: Location;
|
|
115
125
|
}): Promise<void>;
|
|
116
|
-
/** Presses
|
|
117
|
-
|
|
126
|
+
/** Presses at one point or element, moves to another and releases. Browsers and devices only. */
|
|
127
|
+
drag(from: Location, to: Location): Promise<void>;
|
|
128
|
+
/** Presses a key on the focused element. iOS and Android take 'enter' and 'backspace'; Web also takes the other Key names, and modifiers, with which a lowercase letter or digit also works. */
|
|
129
|
+
pressKey(key: Key | string, options?: {
|
|
130
|
+
modifiers?: KeyModifier[];
|
|
131
|
+
}): Promise<void>;
|
|
118
132
|
/** Goes back: browser history on Web, the back button on Android. Not on iOS. */
|
|
119
133
|
back(): Promise<void>;
|
|
134
|
+
/** Goes to the home screen on iOS and Android. Not on Web. */
|
|
135
|
+
goHome(): Promise<void>;
|
|
120
136
|
/** Opens a URL or deep link on a device, or loads a URL in a browser. */
|
|
121
137
|
openUrl(url: string): Promise<void>;
|
|
122
138
|
/** Launches an installed app by bundle ID or package name. Devices only. */
|
|
@@ -203,14 +219,30 @@ export declare class DeviceSession implements Session {
|
|
|
203
219
|
type(at: Location, text: string, options?: {
|
|
204
220
|
clear?: boolean;
|
|
205
221
|
}): Promise<Point>;
|
|
222
|
+
/** Types into the field that already has focus, without tapping, replacing its content unless clear is false. Web only. */
|
|
223
|
+
typeFocused(text: string, options?: {
|
|
224
|
+
clear?: boolean;
|
|
225
|
+
}): Promise<void>;
|
|
226
|
+
/** Double-taps a point in screenshot pixels, or an element Revyl locates from a description. Returns the point, in screenshot pixels. */
|
|
227
|
+
doubleTap(at: Location): Promise<Point>;
|
|
228
|
+
/** Presses and holds a point or element for duration milliseconds (1 to 5000, default 1500). Returns the point, in screenshot pixels. */
|
|
229
|
+
longPress(at: Location, options?: {
|
|
230
|
+
duration?: number;
|
|
231
|
+
}): Promise<Point>;
|
|
206
232
|
/** Swipes from a point or the screen's center. 'up' moves the finger up, which scrolls content down. */
|
|
207
233
|
swipe(direction: SwipeDirection, options?: {
|
|
208
234
|
from?: Location;
|
|
209
235
|
}): Promise<void>;
|
|
210
|
-
/** Presses
|
|
211
|
-
|
|
236
|
+
/** Presses at one point or element, moves to another and releases. */
|
|
237
|
+
drag(from: Location, to: Location): Promise<void>;
|
|
238
|
+
/** Presses a key on the focused element, with modifiers held on Web. */
|
|
239
|
+
pressKey(key: Key | string, options?: {
|
|
240
|
+
modifiers?: KeyModifier[];
|
|
241
|
+
}): Promise<void>;
|
|
212
242
|
/** Goes back: browser history on Web, the back button on Android. Not on iOS. */
|
|
213
243
|
back(): Promise<void>;
|
|
244
|
+
/** Goes to the home screen on iOS and Android. */
|
|
245
|
+
goHome(): Promise<void>;
|
|
214
246
|
/** Opens a URL or deep link on a device, or loads a URL in a browser. */
|
|
215
247
|
openUrl(url: string): Promise<void>;
|
|
216
248
|
/** Launches an installed app by bundle ID or package name. Devices only. */
|
package/dist/session.js
CHANGED
|
@@ -6,7 +6,7 @@ import { apiError, edgeBlockedError, parseJson, responseDetail, responseErrorCod
|
|
|
6
6
|
import { trackSession, untrackSession } from './cleanup.js';
|
|
7
7
|
import { VerifyError, errorMessage, isVerifyError } from './errors.js';
|
|
8
8
|
import { screenHasContent } from './screen-readiness.js';
|
|
9
|
-
import { runLocalStep } from './agent/loop.js';
|
|
9
|
+
import { WEB_KEYS, checkKey, runLocalStep } from './agent/loop.js';
|
|
10
10
|
import { chooseAgent } from './agent/providers.js';
|
|
11
11
|
const POLL_INTERVAL_MS = 2_000;
|
|
12
12
|
const DEVICE_CONNECT_TIMEOUT_MS = 90_000;
|
|
@@ -441,28 +441,61 @@ export class DeviceSession {
|
|
|
441
441
|
await this.readScreenshotSize();
|
|
442
442
|
return this.toScreenshot((await this.perform({ kind: 'type', at: location, text, clear: options.clear !== false })));
|
|
443
443
|
}
|
|
444
|
+
/** Types into the field that already has focus, without tapping, replacing its content unless clear is false. Web only. */
|
|
445
|
+
async typeFocused(text, options = {}) {
|
|
446
|
+
if (this.platform !== 'web')
|
|
447
|
+
throw new VerifyError('NOT_AVAILABLE_ON_PLATFORM', 'typeFocused() is available only on web sessions; on iOS and Android, use type(location, text), which taps the field first.');
|
|
448
|
+
if (typeof text !== 'string' || text.length > 16_384)
|
|
449
|
+
throw new VerifyError('INVALID_ARGUMENT', 'typeFocused() takes text of at most 16384 characters.');
|
|
450
|
+
await this.perform({ kind: 'type_focused', text, clear: options.clear !== false });
|
|
451
|
+
}
|
|
452
|
+
/** Double-taps a point in screenshot pixels, or an element Revyl locates from a description. Returns the point, in screenshot pixels. */
|
|
453
|
+
async doubleTap(at) {
|
|
454
|
+
const location = await this.toDevice(checkLocation(at));
|
|
455
|
+
if (this.platform === 'ios')
|
|
456
|
+
await this.readScreenshotSize();
|
|
457
|
+
return this.toScreenshot((await this.perform({ kind: 'double_tap', at: location })));
|
|
458
|
+
}
|
|
459
|
+
/** Presses and holds a point or element for duration milliseconds (1 to 5000, default 1500). Returns the point, in screenshot pixels. */
|
|
460
|
+
async longPress(at, options = {}) {
|
|
461
|
+
const { duration } = options;
|
|
462
|
+
if (duration !== undefined && (!Number.isInteger(duration) || duration < 1 || duration > 5_000))
|
|
463
|
+
throw new VerifyError('INVALID_ARGUMENT', 'longPress() duration is whole milliseconds from 1 to 5000.');
|
|
464
|
+
const location = await this.toDevice(checkLocation(at));
|
|
465
|
+
if (this.platform === 'ios')
|
|
466
|
+
await this.readScreenshotSize();
|
|
467
|
+
return this.toScreenshot((await this.perform({ kind: 'long_press', at: location, ...(duration !== undefined ? { durationMs: duration } : {}) })));
|
|
468
|
+
}
|
|
444
469
|
/** Swipes from a point or the screen's center. 'up' moves the finger up, which scrolls content down. */
|
|
445
470
|
async swipe(direction, options = {}) {
|
|
446
471
|
if (!['up', 'down', 'left', 'right'].includes(direction))
|
|
447
472
|
throw new VerifyError('INVALID_ARGUMENT', "swipe() takes 'up', 'down', 'left' or 'right'.");
|
|
448
473
|
await this.perform({ kind: 'swipe', direction, ...(options.from ? { from: await this.toDevice(checkLocation(options.from)) } : {}) });
|
|
449
474
|
}
|
|
450
|
-
/** Presses
|
|
451
|
-
async
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
475
|
+
/** Presses at one point or element, moves to another and releases. */
|
|
476
|
+
async drag(from, to) {
|
|
477
|
+
const [start, end] = [checkLocation(from), checkLocation(to)];
|
|
478
|
+
await this.perform({ kind: 'drag', from: await this.toDevice(start), to: await this.toDevice(end) });
|
|
479
|
+
}
|
|
480
|
+
/** Presses a key on the focused element, with modifiers held on Web. */
|
|
481
|
+
async pressKey(key, options = {}) {
|
|
482
|
+
const checked = checkKey(this.platform, key, options.modifiers ?? []);
|
|
483
|
+
await this.perform({ kind: 'key', key: checked.key, ...(checked.modifiers.length ? { modifiers: checked.modifiers } : {}) });
|
|
455
484
|
}
|
|
456
485
|
/** Goes back: browser history on Web, the back button on Android. Not on iOS. */
|
|
457
486
|
async back() {
|
|
458
487
|
await this.perform({ kind: 'back' });
|
|
459
488
|
}
|
|
489
|
+
/** Goes to the home screen on iOS and Android. */
|
|
490
|
+
async goHome() {
|
|
491
|
+
if (this.platform === 'web')
|
|
492
|
+
throw new VerifyError('NOT_AVAILABLE_ON_PLATFORM', 'goHome() is available only on iOS and Android sessions; on web, use openUrl() or navigate() to load a page.');
|
|
493
|
+
await this.perform({ kind: 'go_home' });
|
|
494
|
+
}
|
|
460
495
|
/** Opens a URL or deep link on a device, or loads a URL in a browser. */
|
|
461
496
|
async openUrl(url) {
|
|
462
497
|
if (typeof url !== 'string' || url.length === 0 || url.length > 8_192)
|
|
463
498
|
throw new VerifyError('INVALID_ARGUMENT', 'openUrl() takes a URL of 1 to 8192 characters.');
|
|
464
|
-
if (this.platform === 'web' && !/^https?:/i.test(url))
|
|
465
|
-
throw new VerifyError('INVALID_ARGUMENT', 'On web, openUrl() takes an http(s) URL; deep links such as myapp://orders/42 open only on iOS and Android.');
|
|
466
499
|
await this.perform({ kind: 'open_url', url });
|
|
467
500
|
}
|
|
468
501
|
/** Launches an installed app by bundle ID or package name. Devices only. */
|
|
@@ -577,11 +610,30 @@ export class DeviceSession {
|
|
|
577
610
|
await this.act('tap', pointBody(point, action.at), signal);
|
|
578
611
|
return { x: point.x, y: point.y };
|
|
579
612
|
}
|
|
613
|
+
case 'double_tap': {
|
|
614
|
+
const point = await this.resolve(action.at, screenToken, signal);
|
|
615
|
+
await this.act('double_tap', pointBody(point, action.at), signal);
|
|
616
|
+
return { x: point.x, y: point.y };
|
|
617
|
+
}
|
|
618
|
+
case 'long_press': {
|
|
619
|
+
const point = await this.resolve(action.at, screenToken, signal);
|
|
620
|
+
await this.act('longpress', { ...pointBody(point, action.at), ...(action.durationMs !== undefined ? { duration_ms: action.durationMs } : {}) }, signal);
|
|
621
|
+
return { x: point.x, y: point.y };
|
|
622
|
+
}
|
|
580
623
|
case 'type': {
|
|
581
624
|
const point = await this.resolve(action.at, screenToken, signal);
|
|
582
625
|
await this.act('input', { ...pointBody(point, action.at), text: action.text, clear_first: action.clear }, signal);
|
|
583
626
|
return { x: point.x, y: point.y };
|
|
584
627
|
}
|
|
628
|
+
case 'type_focused':
|
|
629
|
+
await this.act('input', { text: action.text, clear_first: action.clear }, signal);
|
|
630
|
+
return undefined;
|
|
631
|
+
case 'drag': {
|
|
632
|
+
const from = await this.resolve(action.from, screenToken, signal);
|
|
633
|
+
const to = await this.resolve(action.to, screenToken, signal);
|
|
634
|
+
await this.act('drag', { start_x: from.x, start_y: from.y, end_x: to.x, end_y: to.y }, signal);
|
|
635
|
+
return undefined;
|
|
636
|
+
}
|
|
585
637
|
case 'swipe': {
|
|
586
638
|
const point = action.from ? await this.resolve(action.from, screenToken, signal) : await this.readScreenSize(signal).then((size) => ({ x: Math.floor(size.width / 2), y: Math.floor(size.height / 2) }));
|
|
587
639
|
const direction = this.platform === 'web' ? { up: 'down', down: 'up', left: 'right', right: 'left' }[action.direction] : action.direction;
|
|
@@ -589,7 +641,10 @@ export class DeviceSession {
|
|
|
589
641
|
return undefined;
|
|
590
642
|
}
|
|
591
643
|
case 'key':
|
|
592
|
-
await this.act('key', { key: this.platform === 'web' ? (action.key
|
|
644
|
+
await this.act('key', { key: this.platform === 'web' ? (WEB_KEYS[action.key] ?? action.key) : action.key.toUpperCase(), ...(action.modifiers?.length ? { modifiers: action.modifiers } : {}) }, signal);
|
|
645
|
+
return undefined;
|
|
646
|
+
case 'go_home':
|
|
647
|
+
await this.act('go_home', undefined, signal);
|
|
593
648
|
return undefined;
|
|
594
649
|
case 'back':
|
|
595
650
|
if (this.platform === 'ios')
|
|
@@ -598,6 +653,8 @@ export class DeviceSession {
|
|
|
598
653
|
return undefined;
|
|
599
654
|
case 'open_url':
|
|
600
655
|
signal?.throwIfAborted();
|
|
656
|
+
if (this.platform === 'web' && !/^https?:/i.test(action.url))
|
|
657
|
+
throw new VerifyError('INVALID_ARGUMENT', 'On web, openUrl() takes an http(s) URL; deep links such as myapp://orders/42 open only on iOS and Android.');
|
|
601
658
|
if (this.platform === 'web')
|
|
602
659
|
await this.navigateTo(action.url, signal);
|
|
603
660
|
else
|
package/docs/agents.md
CHANGED
|
@@ -24,11 +24,11 @@ This page is for AI coding agents (Claude Code, Cursor, Codex, and others) that
|
|
|
24
24
|
|
|
25
25
|
## Exact actions and coordinates
|
|
26
26
|
|
|
27
|
-
Use `instruction()` when the step needs judgement, and exact actions (`tap`, `type`, `swipe`, `pressKey`, `locate`, `screenSize`) when you know exactly what to press. They work on Web, iOS and Android, not on computers.
|
|
27
|
+
Use `instruction()` when the step needs judgement, and exact actions (`tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `pressKey`, `locate`, `screenSize`) when you know exactly what to press. They work on Web, iOS and Android, not on computers. On Web, `typeFocused(text)` types into the field that has focus, and `pressKey` also takes keys such as `'escape'`, `'tab'` and arrows, with `modifiers`. On iOS and Android, `goHome()` goes to the home screen, `pressKey` takes only `'enter'` and `'backspace'`, and you type with `type(location, text)`. A call the platform lacks throws `NOT_AVAILABLE_ON_PLATFORM` before anything is sent, and the message names what to use instead.
|
|
28
28
|
|
|
29
|
-
Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()
|
|
29
|
+
Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()`, `swipe()` and every other exact action unchanged. `screenSize()` returns the screenshot's width and height, and `locate()` and the points `tap()`, `doubleTap()`, `longPress()` and `type()` return use the same pixels. A point outside the screenshot is refused with `INVALID_ARGUMENT` before any action is sent, and the message names the valid range. On iOS, Verify converts the pixels to the screen's points for you. On Web they equal CSS pixels at the default device scale factor of 1, and you still pass screenshot pixels at any other. This holds from TypeScript 0.4.1, Python 0.2.1 and Go v0.3.1; earlier versions took iOS points.
|
|
30
30
|
|
|
31
|
-
`swipe('up')` moves the finger up, so content scrolls down, on every platform. Actions return once Revyl accepts them, so after a swipe, `openUrl` or `launchApp`, wait for the screen to settle before you `locate` or tap by coordinates.
|
|
31
|
+
`swipe('up')` moves the finger up, so content scrolls down, on every platform. Actions return once Revyl accepts them, so after a swipe, drag, `goHome`, `openUrl` or `launchApp`, wait for the screen to settle before you `locate` or tap by coordinates.
|
|
32
32
|
|
|
33
33
|
```ts
|
|
34
34
|
import { open } from '@revyl/verify';
|
|
@@ -44,12 +44,16 @@ try {
|
|
|
44
44
|
await session.swipe('up', { from: { x: 660, y: 2000 } }); // swipe from a screenshot position too
|
|
45
45
|
const { width, height } = await session.screenSize(); // the screenshot's size
|
|
46
46
|
console.log(`x is 0 to ${width - 1}, y is 0 to ${height - 1}`);
|
|
47
|
+
|
|
48
|
+
await session.longPress({ target: 'Wi-Fi row' }); // press and hold, 1.5 s by default
|
|
49
|
+
await session.drag({ x: 660, y: 2000 }, { x: 660, y: 800 }); // press, move, release: the list follows the finger
|
|
50
|
+
await session.goHome(); // the home screen, on iOS and Android
|
|
47
51
|
} finally {
|
|
48
52
|
await session.close();
|
|
49
53
|
}
|
|
50
54
|
```
|
|
51
55
|
|
|
52
|
-
The [Python](https://github.com/RevylAI/verify/blob/v0.
|
|
56
|
+
The [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#exact-actions) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#exact-actions) READMEs state the same rule with each language's names. [api.md](api.md#exact-actions) lists every call and its errors.
|
|
53
57
|
|
|
54
58
|
## Rules
|
|
55
59
|
|
package/docs/api.md
CHANGED
|
@@ -84,12 +84,15 @@ Web only, hosted or normal. Sends one deterministic `open_url` action and checks
|
|
|
84
84
|
|
|
85
85
|
## Exact actions
|
|
86
86
|
|
|
87
|
-
**
|
|
87
|
+
**Early access.** Since TypeScript (`@revyl/verify`) 0.4.0, Python (`revyl-verify`) 0.2.0 and Go v0.3.0. `doubleTap`, `longPress`, `drag`, `typeFocused`, `goHome`, and Web keys and modifiers in `pressKey`, are new in TypeScript 0.5.0, Python 0.3.0 and Go v0.4.0. The [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#exact-actions) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#exact-actions) READMEs show them with each language's names. Web, iOS and Android sessions take device actions that need no agent.
|
|
88
88
|
|
|
89
89
|
| Call | Web | iOS | Android | Mac computer |
|
|
90
90
|
| --- | --- | --- | --- | --- |
|
|
91
|
-
| `tap`, `type`, `swipe`, `
|
|
91
|
+
| `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `locate`, `screenSize` | Yes | Yes | Yes | No, use `exec()` and files |
|
|
92
|
+
| `pressKey(key, { modifiers })` | Yes: `'enter'`, `'backspace'`, `'tab'`, `'escape'`, arrows and more, with modifiers | `'enter'` and `'backspace'` only | `'enter'` and `'backspace'` only | No |
|
|
93
|
+
| `typeFocused(text)` | Yes, types into the focused field | No, use `type(location, text)` | No, use `type(location, text)` | No |
|
|
92
94
|
| `back()` | Yes, browser history | No, tap the on-screen back control | Yes, the system back button | No |
|
|
95
|
+
| `goHome()` | No, use `openUrl` or `navigate` | Yes, the home screen | Yes, the home screen | No |
|
|
93
96
|
| `openUrl(url)` | Yes, loads an http(s) URL in the tab | Yes, a URL or deep link | Yes, a URL or deep link | No |
|
|
94
97
|
| `launchApp(bundleId)` | No, use `openUrl` or `navigate` | Yes | Yes | No |
|
|
95
98
|
| `navigate(url)` | Yes | No, use `openUrl` | No, use `openUrl` | No |
|
|
@@ -104,10 +107,15 @@ A location is `{ x, y }` in screenshot pixels (see [Coordinates](#coordinates))
|
|
|
104
107
|
| Call | What it does |
|
|
105
108
|
| --- | --- |
|
|
106
109
|
| `tap(at)` | Taps and resolves with the point tapped |
|
|
110
|
+
| `doubleTap(at)` | Double-taps (double-clicks on Web) and resolves with the point |
|
|
111
|
+
| `longPress(at, { duration })` | Presses and holds for `duration` milliseconds, 1 to 5000 (default 1500), and resolves with the point |
|
|
107
112
|
| `type(at, text, { clear })` | Taps a text field and types into it, clearing it first unless `clear` is `false` |
|
|
113
|
+
| `typeFocused(text, { clear })` | Web only: types into the field that already has focus, without clicking, replacing its text unless `clear` is `false`. Focus the field first, for example with `tap` or `pressKey('tab')`. With no field focused nothing is typed and no error is thrown, so check the result. On iOS and Android it throws `NOT_AVAILABLE_ON_PLATFORM`; use `type(location, text)` |
|
|
108
114
|
| `swipe(direction, { from })` | Swipes from a location or the screen's center; `'up'` moves the finger up, which scrolls content down |
|
|
109
|
-
| `
|
|
115
|
+
| `drag(from, to)` | Presses at one location, moves to the other and releases, for example to reorder a list, move a slider or select text |
|
|
116
|
+
| `pressKey(key, { modifiers })` | Presses a key on the focused element. Every platform takes `'enter'` and `'backspace'`. Web also takes `'tab'`, `'escape'`, `'delete'`, `'space'`, `'arrowup'`, `'arrowdown'`, `'arrowleft'`, `'arrowright'`, `'home'`, `'end'`, `'pageup'` and `'pagedown'`, and `modifiers`: up to four of `'ctrl'`, `'shift'`, `'alt'` and `'meta'`, held while the key is pressed. With modifiers a lowercase letter or digit works too, such as `pressKey('a', { modifiers: ['ctrl'] })`. Other keys and any modifiers throw `NOT_AVAILABLE_ON_PLATFORM` on iOS and Android |
|
|
110
117
|
| `back()` | Browser history on Web, the back button on Android; iOS throws `NOT_AVAILABLE_ON_PLATFORM` |
|
|
118
|
+
| `goHome()` | Goes to the home screen on iOS and Android; Web throws `NOT_AVAILABLE_ON_PLATFORM` |
|
|
111
119
|
| `openUrl(url)` | Opens a URL or deep link on a device, or loads a URL in a browser |
|
|
112
120
|
| `launchApp(bundleId)` | Launches an installed app on a device |
|
|
113
121
|
| `locate(target)` | Resolves with Revyl's best match on the current screen, without acting |
|
|
@@ -115,11 +123,11 @@ A location is `{ x, y }` in screenshot pixels (see [Coordinates](#coordinates))
|
|
|
115
123
|
|
|
116
124
|
A failed action throws `ACTION_FAILED`, including when Revyl cannot find a target or nothing at a point can take the action. One whose outcome Revyl can't confirm throws `ACTION_UNCERTAIN` or `TIMEOUT`, so check the session before acting again. Actions are never repeated. If Verify can't reach Revyl, the action fails with `API_ERROR` and isn't resent either, so check the session before trying it again.
|
|
117
125
|
|
|
118
|
-
Exact actions return when Revyl has accepted them, not when the screen has finished changing. After `swipe`, `openUrl` or `launchApp`, let the screen settle or check it before you `locate` or tap coordinates. `locate` returns Revyl's best match on the current screen and can return a point for an element that is off screen, so bring the element on screen first. A point outside `screenSize()` is off screen, and an action at it is refused.
|
|
126
|
+
Exact actions return when Revyl has accepted them, not when the screen has finished changing. After `swipe`, `drag`, `goHome`, `openUrl` or `launchApp`, let the screen settle or check it before you `locate` or tap coordinates. `locate` returns Revyl's best match on the current screen and can return a point for an element that is off screen, so bring the element on screen first. A point outside `screenSize()` is off screen, and an action at it is refused.
|
|
119
127
|
|
|
120
128
|
### Coordinates
|
|
121
129
|
|
|
122
|
-
Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()
|
|
130
|
+
Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()`, `swipe()` and every other exact action unchanged. `screenSize()` returns the screenshot's width and height, and `locate()` and the points `tap()`, `doubleTap()`, `longPress()` and `type()` return use the same pixels. A point outside the screenshot is refused with `INVALID_ARGUMENT` before any action is sent, and the message names the valid range. On iOS, Verify converts the pixels to the screen's points for you. On Web they equal CSS pixels at the default device scale factor of 1, and you still pass screenshot pixels at any other. This holds from TypeScript 0.4.1, Python 0.2.1 and Go v0.3.1; earlier versions took iOS points.
|
|
123
131
|
|
|
124
132
|
```ts
|
|
125
133
|
await session.screenshot({ path: '.verify/screen.png' }); // open it and find the element
|
|
@@ -172,7 +180,7 @@ When inspecting `revyl device report --session-id <id>`, read validation verdict
|
|
|
172
180
|
|
|
173
181
|
## Your own model
|
|
174
182
|
|
|
175
|
-
**
|
|
183
|
+
**Early access.** Since TypeScript (`@revyl/verify`) 0.4.0, Python (`revyl-verify`) 0.2.0 and Go v0.3.0; the [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#your-own-model) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#your-own-model) READMEs show them with each language's names. Runs instructions and validations on a model you choose, in your process, instead of Revyl's agent. Revyl still provides the browser or device. Leave `model` out and Revyl's agent runs the step, as before.
|
|
176
184
|
|
|
177
185
|
```ts
|
|
178
186
|
const session = await open('ios', { model: 'anthropic/claude-sonnet-5-5' }); // every step on your model
|
|
@@ -208,7 +216,7 @@ const session = await open('ios', { agent });
|
|
|
208
216
|
```
|
|
209
217
|
|
|
210
218
|
- `model`: a `ModelAdapter`. `anthropic({ model, apiKey, baseURL, maxTokens, maxRetries })` uses `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL` for what you leave out. `openaiCompatible({ model, baseURL, apiKey, headers, provider, maxTokens, maxRetries })` uses `OPENAI_BASE_URL` and `OPENAI_API_KEY` only when `baseURL` is left out, so a key never reaches another provider; it then sends OpenAI's `max_completion_tokens`. To wrap another SDK or runtime, implement `{ name, complete(request, signal) }`: it gets the system prompt, text, screenshot and tools, and returns the chosen tool call and token usage.
|
|
211
|
-
- `grounding`: `'revyl'` (default) as above. `'model'` has the model give screenshot coordinates instead, so Revyl receives only taps, typing and swipes, never element descriptions; it needs a model that points accurately.
|
|
219
|
+
- `grounding`: `'revyl'` (default) as above. `'model'` has the model give screenshot coordinates instead, so Revyl receives only device actions such as taps, typing and swipes, never element descriptions; it needs a model that points accurately.
|
|
212
220
|
- `maxTurns` (default 20, up to 100), `modelTimeout` per model call (default 60000), `maxImageSide` for screenshots sent to the model (default 1280).
|
|
213
221
|
|
|
214
222
|
## session.screenshot(options?)
|
package/docs/errors.md
CHANGED
|
@@ -48,7 +48,7 @@ try {
|
|
|
48
48
|
| `SESSION_ENDED` | The session was closed, stopped, or expired, or Revyl answered 404 for the session itself | Close explicitly; computer expiry does not prove release |
|
|
49
49
|
| `SESSION_BUSY` | A step conflict, computer operation limit, or recording requested before terminal state and confirmed release | Inspect the state before another call; a terminal computer alone does not prove release |
|
|
50
50
|
| `COMPUTER_LIMIT` | Computer admission was refused due to queued work (including iOS) or the concurrency limit | Wait for capacity; Verify does not retry or stop a definitely refused start |
|
|
51
|
-
| `NOT_AVAILABLE_ON_PLATFORM` | The call is unavailable: computer agent steps, device shell/file/recording methods, computer access or runtime unavailable, or a backend without web steps | See [web.md](web.md) and [computers.md](computers.md#availability-and-errors) |
|
|
51
|
+
| `NOT_AVAILABLE_ON_PLATFORM` | The call is unavailable: computer agent steps, device shell/file/recording methods, an exact action the platform lacks (such as `back()` on iOS, `goHome()` or `launchApp()` on Web, `typeFocused()` or keys other than Enter and Backspace on iOS and Android), computer access or runtime unavailable, or a backend without web steps | See [web.md](web.md) and [computers.md](computers.md#availability-and-errors) |
|
|
52
52
|
| `COMMAND_FAILED` | A computer command exited nonzero | Read its reason; use `exec(argv, { check: false })` to inspect the exit code and output instead |
|
|
53
53
|
| `NOT_FOUND` | A computer file or resource was not found | Check the workspace-relative path |
|
|
54
54
|
| `ACCESS_DENIED` | The computer refused access | Check permissions; do not assume host/root access |
|
package/docs/llms.txt
CHANGED
|
@@ -20,4 +20,6 @@ Mac computers are early access; ask support@revyl.ai about access. See computers
|
|
|
20
20
|
|
|
21
21
|
## Exact actions and coordinates
|
|
22
22
|
|
|
23
|
-
Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()
|
|
23
|
+
Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()`, `swipe()` and every other exact action unchanged. `screenSize()` returns the screenshot's width and height, and `locate()` and the points `tap()`, `doubleTap()`, `longPress()` and `type()` return use the same pixels. A point outside the screenshot is refused with `INVALID_ARGUMENT` before any action is sent, and the message names the valid range. On iOS, Verify converts the pixels to the screen's points for you. On Web they equal CSS pixels at the default device scale factor of 1, and you still pass screenshot pixels at any other. This holds from TypeScript 0.4.1, Python 0.2.1 and Go v0.3.1; earlier versions took iOS points. See [For coding agents](agents.md#exact-actions-and-coordinates) for a runnable example.
|
|
24
|
+
|
|
25
|
+
Exact actions are `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `pressKey`, `locate` and `screenSize` on Web, iOS and Android. Web adds `typeFocused(text)`, which types into the field that has focus, and `pressKey` keys such as `'escape'`, `'tab'` and arrows with `modifiers`. iOS and Android add `goHome()`; their `pressKey` takes only `'enter'` and `'backspace'`, and you type with `type(location, text)`. A call the platform lacks throws `NOT_AVAILABLE_ON_PLATFORM` before anything is sent, naming what to use instead. See [api.md](api.md#exact-actions) for every call.
|
package/docs/mobile.md
CHANGED
|
@@ -50,7 +50,7 @@ For Android Settings, instruct the agent to scroll or navigate until the OS vers
|
|
|
50
50
|
|
|
51
51
|
## Exact actions
|
|
52
52
|
|
|
53
|
-
**
|
|
53
|
+
**Early access.** Since TypeScript (`@revyl/verify`) 0.4.0, Python (`revyl-verify`) 0.2.0 and Go v0.3.0. `doubleTap`, `longPress`, `drag`, `typeFocused`, `goHome`, and Web keys and modifiers in `pressKey`, are new in TypeScript 0.5.0, Python 0.3.0 and Go v0.4.0. The [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#exact-actions) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#exact-actions) READMEs show them with each language's names. When you know exactly what to press, skip the agent. Exact actions run on the device as given, with no model, and mix freely with `instruction()` and `validation()` in one script:
|
|
54
54
|
|
|
55
55
|
```ts
|
|
56
56
|
import { open } from '@revyl/verify';
|
|
@@ -75,14 +75,39 @@ try {
|
|
|
75
75
|
}
|
|
76
76
|
```
|
|
77
77
|
|
|
78
|
-
A location is `{ target: 'a description' }`, which Revyl's grounding model finds on the current screen, or `{ x, y }` in screenshot pixels from the top-left corner; `screenSize()` gives the bounds. `type()` taps the field and clears it first unless you pass `{ clear: false }`. `pressKey()` takes `'enter'` or `'backspace'
|
|
78
|
+
A location is `{ target: 'a description' }`, which Revyl's grounding model finds on the current screen, or `{ x, y }` in screenshot pixels from the top-left corner; `screenSize()` gives the bounds. `type()` taps the field and clears it first unless you pass `{ clear: false }`. `pressKey()` takes `'enter'` or `'backspace'`; for other keys, tap the on-screen control.
|
|
79
79
|
|
|
80
|
-
|
|
80
|
+
More actions, new in 0.5.0, on the same Settings app:
|
|
81
|
+
|
|
82
|
+
```ts
|
|
83
|
+
import { open } from '@revyl/verify';
|
|
84
|
+
|
|
85
|
+
const session = await open({ platform: 'ios', app: { bundleId: 'com.apple.Preferences' } });
|
|
86
|
+
try {
|
|
87
|
+
await session.goHome(); // the home screen
|
|
88
|
+
await session.longPress({ target: 'Settings app icon' }); // press and hold, 1.5 s by default
|
|
89
|
+
await session.validation('A menu for the Settings app is shown');
|
|
90
|
+
await session.goHome(); // closes the menu
|
|
91
|
+
await session.launchApp('com.apple.Preferences');
|
|
92
|
+
await session.drag({ target: 'General row' }, { target: 'Search field' }); // the list follows the finger
|
|
93
|
+
await session.doubleTap({ target: 'Settings title' });
|
|
94
|
+
await session.longPress({ x: 660, y: 1500 }, { duration: 800 }); // a point in screenshot pixels, held for 800 ms
|
|
95
|
+
} finally {
|
|
96
|
+
await session.close();
|
|
97
|
+
}
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
`goHome()` goes to the home screen on iOS and Android. `longPress()` holds for 1500 ms unless you pass `duration`, from 1 to 5000. `drag(from, to)` presses at one location, moves to the other and releases. On iOS and Android, type into a field with `type(location, text)`, which taps it first; `typeFocused()` is Web only and throws `NOT_AVAILABLE_ON_PLATFORM` here.
|
|
101
|
+
|
|
102
|
+
On Android, `back()` presses the system back button. On both platforms, `goHome()` goes to the home screen, `launchApp(bundleId)` brings an installed app to the front and `openUrl('myapp://orders/42')` opens a deep link. A failed action throws `ACTION_FAILED`. One whose outcome Revyl can't confirm throws `ACTION_UNCERTAIN` or `TIMEOUT`, so check the session before acting again. Actions are never repeated. If Verify can't reach Revyl, the action fails with `API_ERROR` and isn't resent either, so check the session before trying it again. [api.md](api.md#exact-actions) lists every call.
|
|
81
103
|
|
|
82
104
|
| Call | Web | iOS | Android | Mac computer |
|
|
83
105
|
| --- | --- | --- | --- | --- |
|
|
84
|
-
| `tap`, `type`, `swipe`, `
|
|
106
|
+
| `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `locate`, `screenSize` | Yes | Yes | Yes | No, use `exec()` and files |
|
|
107
|
+
| `pressKey(key, { modifiers })` | Yes: `'enter'`, `'backspace'`, `'tab'`, `'escape'`, arrows and more, with modifiers | `'enter'` and `'backspace'` only | `'enter'` and `'backspace'` only | No |
|
|
108
|
+
| `typeFocused(text)` | Yes, types into the focused field | No, use `type(location, text)` | No, use `type(location, text)` | No |
|
|
85
109
|
| `back()` | Yes, browser history | No, tap the on-screen back control | Yes, the system back button | No |
|
|
110
|
+
| `goHome()` | No, use `openUrl` or `navigate` | Yes, the home screen | Yes, the home screen | No |
|
|
86
111
|
| `openUrl(url)` | Yes, loads an http(s) URL in the tab | Yes, a URL or deep link | Yes, a URL or deep link | No |
|
|
87
112
|
| `launchApp(bundleId)` | No, use `openUrl` or `navigate` | Yes | Yes | No |
|
|
88
113
|
| `navigate(url)` | Yes | No, use `openUrl` | No, use `openUrl` | No |
|
|
@@ -90,8 +115,8 @@ On Android, `back()` presses the system back button. On both platforms, `launchA
|
|
|
90
115
|
| `screenshot()` | Yes | Yes | Yes | Yes |
|
|
91
116
|
| `exec`, files, processes | No, open a Mac computer | No, open a Mac computer | No, open a Mac computer | Yes |
|
|
92
117
|
|
|
93
|
-
**Coordinates.** Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()
|
|
118
|
+
**Coordinates.** Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()`, `swipe()` and every other exact action unchanged. `screenSize()` returns the screenshot's width and height, and `locate()` and the points `tap()`, `doubleTap()`, `longPress()` and `type()` return use the same pixels. A point outside the screenshot is refused with `INVALID_ARGUMENT` before any action is sent, and the message names the valid range. On iOS, Verify converts the pixels to the screen's points for you. On Web they equal CSS pixels at the default device scale factor of 1, and you still pass screenshot pixels at any other. This holds from TypeScript 0.4.1, Python 0.2.1 and Go v0.3.1; earlier versions took iOS points.
|
|
94
119
|
|
|
95
120
|
A call a platform doesn't support throws `NOT_AVAILABLE_ON_PLATFORM`, and its message names the alternative.
|
|
96
121
|
|
|
97
|
-
When you pass a [`model`](own-model.md), your model uses `tap`, `type`, `swipe`, `pressKey`, `openUrl` and, where the platform has
|
|
122
|
+
When you pass a [`model`](own-model.md), your model uses `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `pressKey`, `openUrl` and, where the platform has them, `typeFocused`, `back` and `goHome` inside `instruction()`.
|
package/docs/own-model.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# Use your own model
|
|
2
2
|
|
|
3
|
-
**
|
|
3
|
+
**Early access.** Since TypeScript (`@revyl/verify`) 0.4.0, Python (`revyl-verify`) 0.2.0 and Go v0.3.0; the [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#your-own-model) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#your-own-model) READMEs show them with each language's names. Verify normally runs `instruction()` and `validation()` on Revyl's agent. Pass a `model` and the agent loop runs in your process instead, on a model you choose, with your own key. Revyl still provides the cloud browser or device and, by default, finds the elements your model names on screen.
|
|
4
4
|
|
|
5
5
|
Use it when you want a specific model, your own prompts and key, or faster steps on a cheap model. Leave `model` out and nothing changes.
|
|
6
6
|
|
|
7
|
-
Your model uses the same [exact actions](api.md#exact-actions) you can call yourself: `tap`, `type`, `swipe`, `pressKey`, `openUrl` and, where the platform has
|
|
7
|
+
Your model uses the same [exact actions](api.md#exact-actions) you can call yourself: `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `pressKey`, `openUrl` and, where the platform has them, `typeFocused` and Web keys with modifiers (Web), `back` (Web and Android) and `goHome` (iOS and Android). So a script can mix your own actions, steps on your model and steps on Revyl's agent.
|
|
8
8
|
|
|
9
9
|
## The one-line form
|
|
10
10
|
|
|
@@ -52,7 +52,7 @@ log.Println(step.Model.ID, len(step.Model.Turns), step.Model.InputTokens, step.M
|
|
|
52
52
|
_, err = s.Validation(ctx, "The screen shows the iOS version")
|
|
53
53
|
```
|
|
54
54
|
|
|
55
|
-
The [Python](https://github.com/RevylAI/verify/blob/v0.
|
|
55
|
+
The [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#your-own-model) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#your-own-model) READMEs have the full reference for each.
|
|
56
56
|
|
|
57
57
|
Set `model` on `open()` to run every step on it, or on one step to override it:
|
|
58
58
|
|
|
@@ -169,7 +169,7 @@ const session = await open('ios', { agent }); // or session.instruction(text, {
|
|
|
169
169
|
|
|
170
170
|
**Your model sees** the system prompt, the step text, a short log of the actions it took so far and whether they worked, and the current screenshot, downscaled so its longest side is at most 1280 pixels (`maxImageSide`).
|
|
171
171
|
|
|
172
|
-
**Revyl sees** what it always sees for a session it hosts: the device and its screen. On top of that it receives each device action (taps, typed text, swipes, key presses, URLs), the element descriptions your model writes, such as "General row" (not with `grounding: 'model'`), and the model's name, such as `anthropic/claude-sonnet-5-5`, for the session report.
|
|
172
|
+
**Revyl sees** what it always sees for a session it hosts: the device and its screen. On top of that it receives each device action (taps, long presses, drags, typed text, swipes, key presses, URLs), the element descriptions your model writes, such as "General row" (not with `grounding: 'model'`), and the model's name, such as `anthropic/claude-sonnet-5-5`, for the session report.
|
|
173
173
|
|
|
174
174
|
**Revyl does not receive** the step text, the prompts, your model's output or reasoning, or your key. Keys go only to the provider's endpoint, are never logged, and are replaced with `[redacted]` in error messages, as are custom header values. A custom endpoint's address is never sent to Revyl.
|
|
175
175
|
|
package/docs/web.md
CHANGED
|
@@ -53,7 +53,7 @@ Revyl runs one step at a time on a session. A step sent while another is still r
|
|
|
53
53
|
|
|
54
54
|
## Exact actions
|
|
55
55
|
|
|
56
|
-
**
|
|
56
|
+
**Early access.** Since TypeScript (`@revyl/verify`) 0.4.0, Python (`revyl-verify`) 0.2.0 and Go v0.3.0. `doubleTap`, `longPress`, `drag`, `typeFocused`, `goHome`, and Web keys and modifiers in `pressKey`, are new in TypeScript 0.5.0, Python 0.3.0 and Go v0.4.0. The [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#exact-actions) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#exact-actions) READMEs show them with each language's names. When you know exactly what to click, skip the agent. Exact actions run in the browser as given, with no model, and mix freely with `instruction()` and `validation()`:
|
|
57
57
|
|
|
58
58
|
```ts
|
|
59
59
|
import { open } from '@revyl/verify';
|
|
@@ -76,12 +76,42 @@ try {
|
|
|
76
76
|
}
|
|
77
77
|
```
|
|
78
78
|
|
|
79
|
-
A location is `{ target: 'a description' }`, which Revyl's grounding model finds on the current page, or `{ x, y }` in screenshot pixels from the top-left corner; `screenSize()` gives the bounds and `locate(target)` gives an element's point without acting. `type()` clicks the field and clears it first unless you pass `{ clear: false }`. `
|
|
79
|
+
A location is `{ target: 'a description' }`, which Revyl's grounding model finds on the current page, or `{ x, y }` in screenshot pixels from the top-left corner; `screenSize()` gives the bounds and `locate(target)` gives an element's point without acting. `type()` clicks the field and clears it first unless you pass `{ clear: false }`. `openUrl(url)` loads a page like `navigate(url)`.
|
|
80
|
+
|
|
81
|
+
More actions, new in 0.5.0, on the same store:
|
|
82
|
+
|
|
83
|
+
```ts
|
|
84
|
+
import { open } from '@revyl/verify';
|
|
85
|
+
|
|
86
|
+
const session = await open('web'); // https://www.saucedemo.com in examples/web-store
|
|
87
|
+
try {
|
|
88
|
+
await session.tap({ target: 'Username field' });
|
|
89
|
+
await session.typeFocused('standard_user'); // types into the field that has focus, replacing its text
|
|
90
|
+
await session.pressKey('tab'); // focus moves to the password field
|
|
91
|
+
await session.typeFocused('secret_sauce');
|
|
92
|
+
await session.pressKey('enter');
|
|
93
|
+
await session.validation('A list of products is shown');
|
|
94
|
+
await session.tap({ target: 'Open Menu button' });
|
|
95
|
+
await session.pressKey('escape'); // closes the menu
|
|
96
|
+
await session.validation('The side menu is closed');
|
|
97
|
+
await session.doubleTap({ target: 'Products heading' }); // a double-click
|
|
98
|
+
await session.pressKey('a', { modifiers: ['ctrl'] }); // a chord: a lowercase letter or digit with modifiers
|
|
99
|
+
await session.longPress({ target: 'Products heading' }, { duration: 800 }); // press and hold for 800 ms
|
|
100
|
+
await session.drag({ target: 'Sauce Labs Backpack description' }, { target: 'Sauce Labs Bike Light description' }); // selects the text between
|
|
101
|
+
} finally {
|
|
102
|
+
await session.close();
|
|
103
|
+
}
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
`typeFocused(text)` types where the focus already is, without clicking, and replaces the field's text unless you pass `{ clear: false }`. Focus the field first, with `tap()` or `pressKey('tab')`: with no field focused, nothing is typed and no error is thrown. `pressKey()` takes `'enter'`, `'backspace'`, `'tab'`, `'escape'`, `'delete'`, `'space'`, `'arrowup'`, `'arrowdown'`, `'arrowleft'`, `'arrowright'`, `'home'`, `'end'`, `'pageup'` and `'pagedown'`, with up to four `modifiers` held (`'ctrl'`, `'shift'`, `'alt'`, `'meta'`); with modifiers a lowercase letter or digit works too. `longPress()` holds for 1500 ms unless you pass `duration`, from 1 to 5000. `drag(from, to)` presses at one location, moves to the other and releases. A failed action throws `ACTION_FAILED`. One whose outcome Revyl can't confirm throws `ACTION_UNCERTAIN` or `TIMEOUT`, so check the session before acting again. Actions are never repeated. If Verify can't reach Revyl, the action fails with `API_ERROR` and isn't resent either, so check the session before trying it again. [api.md](api.md#exact-actions) lists every call.
|
|
80
107
|
|
|
81
108
|
| Call | Web | iOS | Android | Mac computer |
|
|
82
109
|
| --- | --- | --- | --- | --- |
|
|
83
|
-
| `tap`, `type`, `swipe`, `
|
|
110
|
+
| `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `locate`, `screenSize` | Yes | Yes | Yes | No, use `exec()` and files |
|
|
111
|
+
| `pressKey(key, { modifiers })` | Yes: `'enter'`, `'backspace'`, `'tab'`, `'escape'`, arrows and more, with modifiers | `'enter'` and `'backspace'` only | `'enter'` and `'backspace'` only | No |
|
|
112
|
+
| `typeFocused(text)` | Yes, types into the focused field | No, use `type(location, text)` | No, use `type(location, text)` | No |
|
|
84
113
|
| `back()` | Yes, browser history | No, tap the on-screen back control | Yes, the system back button | No |
|
|
114
|
+
| `goHome()` | No, use `openUrl` or `navigate` | Yes, the home screen | Yes, the home screen | No |
|
|
85
115
|
| `openUrl(url)` | Yes, loads an http(s) URL in the tab | Yes, a URL or deep link | Yes, a URL or deep link | No |
|
|
86
116
|
| `launchApp(bundleId)` | No, use `openUrl` or `navigate` | Yes | Yes | No |
|
|
87
117
|
| `navigate(url)` | Yes | No, use `openUrl` | No, use `openUrl` | No |
|
|
@@ -89,8 +119,8 @@ A location is `{ target: 'a description' }`, which Revyl's grounding model finds
|
|
|
89
119
|
| `screenshot()` | Yes | Yes | Yes | Yes |
|
|
90
120
|
| `exec`, files, processes | No, open a Mac computer | No, open a Mac computer | No, open a Mac computer | Yes |
|
|
91
121
|
|
|
92
|
-
**Coordinates.** Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()
|
|
122
|
+
**Coordinates.** Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()`, `swipe()` and every other exact action unchanged. `screenSize()` returns the screenshot's width and height, and `locate()` and the points `tap()`, `doubleTap()`, `longPress()` and `type()` return use the same pixels. A point outside the screenshot is refused with `INVALID_ARGUMENT` before any action is sent, and the message names the valid range. On iOS, Verify converts the pixels to the screen's points for you. On Web they equal CSS pixels at the default device scale factor of 1, and you still pass screenshot pixels at any other. This holds from TypeScript 0.4.1, Python 0.2.1 and Go v0.3.1; earlier versions took iOS points.
|
|
93
123
|
|
|
94
124
|
A call a platform doesn't support throws `NOT_AVAILABLE_ON_PLATFORM`, and its message names the alternative.
|
|
95
125
|
|
|
96
|
-
When you pass a [`model`](own-model.md), your model uses `tap`, `type`, `swipe`, `pressKey`, `openUrl` and, where the platform has
|
|
126
|
+
When you pass a [`model`](own-model.md), your model uses `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `pressKey`, `openUrl` and, where the platform has them, `typeFocused`, `back` and `goHome` inside `instruction()`.
|