@revyl/verify 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +48 -3
- package/README.md +128 -61
- package/dist/api/client.d.ts +6 -4
- package/dist/api/client.js +64 -14
- package/dist/api/types.d.ts +7 -0
- package/dist/cli/init.js +2 -3
- package/dist/cli/install.d.ts +2 -0
- package/dist/cli/install.js +64 -0
- package/dist/cli/login.d.ts +6 -0
- package/dist/cli/login.js +200 -0
- package/dist/cli/main.js +48 -14
- package/dist/computer-session.d.ts +21 -5
- package/dist/computer-session.js +214 -35
- package/dist/config.d.ts +8 -1
- package/dist/config.js +16 -2
- package/dist/errors.d.ts +6 -1
- package/dist/errors.js +4 -0
- package/dist/index.d.ts +5 -5
- package/dist/index.js +1 -1
- package/dist/open.d.ts +11 -3
- package/dist/open.js +37 -11
- package/dist/screen-readiness.d.ts +1 -0
- package/dist/screen-readiness.js +27 -0
- package/dist/session.d.ts +30 -2
- package/dist/session.js +165 -12
- package/docs/README.md +2 -2
- package/docs/agents.md +14 -9
- package/docs/api.md +86 -12
- package/docs/ci.md +1 -1
- package/docs/computers.md +28 -17
- package/docs/configuration.md +14 -12
- package/docs/end-to-end.md +105 -0
- package/docs/errors.md +31 -5
- package/docs/getting-started.md +49 -26
- package/docs/llms.txt +6 -6
- package/docs/mobile.md +6 -0
- package/docs/web.md +1 -7
- package/package.json +7 -4
- package/docs/linux-computers-contract.md +0 -107
package/CHANGELOG.md
CHANGED
|
@@ -1,15 +1,60 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.3.0 (2026-10-31)
|
|
4
|
+
|
|
5
|
+
- Sign in automatically on the first interactive `open()`, `attach()`, or `currentComputer()` call using the Revyl CLI browser-approval flow; CI remains key-only and browser-free.
|
|
6
|
+
- Make `npx @revyl/verify init` install Verify, scaffold a project, sign in through the Revyl CLI browser approval flow when needed, and print the example command. Non-interactive runs use `REVYL_API_KEY` without prompting or opening a browser.
|
|
7
|
+
- Add `currentComputer()`, Python `current_computer()` / `async_current_computer()`, and Go `CurrentComputer` to attach non-owning to `REVYL_COMPUTER_ID`, with explicit ownership opt-in and a typed error when no current computer is configured.
|
|
8
|
+
- Add explicit non-owning computer attachment with `stopOnClose: false`, Python `stop_on_close=False`, and Go `OpenOptions.StopOnClose`. Closing these attachments still cleans up hosted children but leaves the computer running for its owner to release; the default remains owning.
|
|
9
|
+
|
|
10
|
+
- Removed environment overrides.
|
|
11
|
+
|
|
12
|
+
- **Breaking wire change:** encode computer-operation arguments as `argv_b64` and file paths as `path_b64` using padded URL-safe base64 of UTF-8, without changing public APIs, argument boundaries, or existing stdin/data fields. Verify 0.3.0 requires the encoded-only backend, which is already live on revyl.ai; no plaintext fallback or automatic replay is provided.
|
|
13
|
+
|
|
14
|
+
- Classify HTTP 403 responses with a `text/html` content type and nonempty `cf-ray` as `EDGE_BLOCKED` across TypeScript, Go and Python, unless the body is syntactically valid JSON. Preserve origin authentication errors, omit response content from edge-block messages, and do not retry blocked requests.
|
|
15
|
+
|
|
16
|
+
- Map hosted Web navigation `request_timeout` and `protocol_error` responses to `ACTION_UNCERTAIN` with safe reasons and HTTP status. Timeout messages explain that navigation may still commit and the session remains open; Verify never replays it.
|
|
17
|
+
|
|
18
|
+
- Treat ordinary Web `409 web_runtime_unavailable` admission refusals as definite pre-dispatch failures: preserve `API_ERROR` plus safe reason/status details and do not send a stop request. Allocated or otherwise uncertain Web failures retain cleanup.
|
|
19
|
+
|
|
20
|
+
- Bound parent computer cleanup's child phase to five seconds so slow child stops cannot prevent requesting parent release. Recognize `web_runtime_not_configured` as a definite hosted refusal, reject reports without a known session ID locally, preserve safe TypeScript startup diagnostics, and keep the original TypeScript `Session` interface compatible through the extended `OpenedSession` interface.
|
|
21
|
+
|
|
22
|
+
- Add public computer attachment, computer-hosted child sessions and deterministic Web navigation in TypeScript, Go and Python. Child sessions reuse the parent's authenticated client and expire with it; parent close attempts bounded child cleanup before requesting computer release. Initial hosted Web builds remain HTTPS; localhost is reached by navigation after start, subject to runtime availability.
|
|
23
|
+
- Preserve typed server refusal reasons and HTTP status alongside existing error codes, including Web `web_runtime_unavailable`, computer `computer_limit` and `runtime_unavailable`, without replaying starts or navigation.
|
|
24
|
+
|
|
25
|
+
- Preserve Python startup session identity, viewer links and separate cleanup outcomes on typed errors; add optional lifecycle logging to sync/async opening. Clean up identifiable malformed or uncertain starts, interrupt cancelled async startup, validate device stop acknowledgments, and preserve original failures when cleanup also fails.
|
|
26
|
+
|
|
27
|
+
- Prepare Python SDK 0.1.0 (`revyl-verify`, import `revyl_verify`) for Python ≥3.10, with typed sync and thread-adapted async sessions, seconds-based timeouts, web/mobile steps and early-access computer operations. Include fake-server coverage, wheel/sdist checks and an isolated installed-wheel smoke; Python has not been published or verified live. Its independent `py/v*` release workflow validates without publishing on dispatch, skips exact published versions, and requires administrator-enabled, protected-environment PyPI trusted publishing for new tag releases.
|
|
28
|
+
|
|
29
|
+
- Add the source-free `revyl-verify` npm alias with matching API, TypeScript types and CLI, pinned to the exact `@revyl/verify` version. Validate and pack both packages; optional CI publication checks each version independently and publishes the scoped package before the alias.
|
|
30
|
+
- Add a standard-library-only Go SDK for web, iOS, Android, and early-access Mac and Linux computers. Computer commands, binary files, process handles, screenshots, recording metadata, and confirmed-release cleanup require computer access through support@revyl.ai. Capacity refusals have a distinct `COMPUTER_LIMIT` code; no start or operation is automatically replayed.
|
|
31
|
+
|
|
32
|
+
## 0.2.0
|
|
33
|
+
|
|
34
|
+
Prepared for release; see npm for publication status.
|
|
35
|
+
|
|
36
|
+
- Enable Linux named and inline computer targets with the shared Mac command, file, process, screenshot, and confirmed-release cleanup surface. Both operating systems remain early access; contact support@revyl.ai for access.
|
|
37
|
+
- Add `COMPUTER_LIMIT` for queued work (including iOS) or concurrency admission refusals, without retrying or stopping a definitely refused start. Preserve `SESSION_BUSY` for operation limits.
|
|
38
|
+
- Explain the 120-second unclaimed-start failure without shortening the default 10-minute start timeout.
|
|
39
|
+
- Expose optional `expiresAt` and bound operation/screenshot observation by the hard 60-minute lifetime from create, while keeping explicit close possible.
|
|
40
|
+
- Add computer `recording()` metadata retrieval after terminal state and confirmed release, including after close. Missing recordings return null; signed URLs remain private, with no automatic waiting or download.
|
|
41
|
+
- Keep Linux CLI init intentionally unavailable; use inline targets or config entries. Computer instruction/validation remain locally unavailable.
|
|
42
|
+
- Observe mobile first-screen content best-effort for up to 10 seconds after connection/launch, without pretending the runtime provides rendered-readiness guarantees. Generated mobile examples begin with an instruction to wait for the first screen before validation.
|
|
43
|
+
- Retry structured computer screen-not-ready responses with bounded backoff during the fixed startup window; do not replay mutations or unrelated failures.
|
|
44
|
+
- Skip CI publication when the exact version is already on npm, require explicit administrator opt-in for new CI publications, fail closed on registry lookup errors, and document manual Mac publication, optional trusted publishing, and independent Go module tags.
|
|
45
|
+
|
|
46
|
+
Regression validation uses fake API responses. The [end-to-end record](https://github.com/RevylAI/verify/blob/v0.2.0/packages/verify/docs/end-to-end.md) distinguishes baseline live runs from startup fixes still awaiting live re-verification.
|
|
47
|
+
|
|
3
48
|
## 0.1.0
|
|
4
49
|
|
|
5
|
-
|
|
50
|
+
Published manually from a maintainer's Mac without provenance.
|
|
6
51
|
|
|
7
52
|
- Open Revyl cloud web browsers, iOS simulators, and Android emulators from TypeScript.
|
|
8
53
|
- Run natural-language instruction and validation steps, with typed failures and optional soft validation results.
|
|
9
54
|
- Save screenshots, share session links, and close sessions explicitly or during best-effort process cleanup.
|
|
10
55
|
- Configure named targets and initialize a script with the `verify` or `revyl-verify` command.
|
|
11
56
|
- Use environment credentials or an existing Revyl CLI login, with documentation included in the package.
|
|
12
|
-
- Include **early-access Mac computers
|
|
57
|
+
- Include **early-access Mac computers** for commands, binary files, process handles, screenshots, and confirmed-release cleanup. Ask support@revyl.ai about access.
|
|
13
58
|
- Reserve Linux target types and document the future contract; opening Linux is not available in this release.
|
|
14
59
|
|
|
15
|
-
Requires Node.js 22.18 or later and a Revyl account. There is no bundled test runner or step replay. Mac operations remain covered by source-aligned fake-API tests;
|
|
60
|
+
Requires Node.js 22.18 or later and a Revyl account. There is no bundled test runner or step replay. Mac operations remain covered by source-aligned fake-API tests; the attempted live start never reached active, so it did not prove any operation or confirmed cleanup.
|
package/README.md
CHANGED
|
@@ -1,101 +1,168 @@
|
|
|
1
1
|
# Verify
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Verify runs end-to-end checks in a real browser, iOS simulator, or Android emulator on Revyl. You describe what the agent should do and what must be true afterward, instead of maintaining selectors and coordinates that break when a screen changes. Use it from a script, a coding agent, or the test runner you already have. Mac and Linux computers are also available for commands, files, and screenshots, not natural-language steps.
|
|
4
|
+
|
|
5
|
+
Every run gives you a link to watch the target. The session keeps its report and available recording, and the SDK keeps instruction and validation results in order. A failed check leaves evidence you can inspect.
|
|
6
|
+
|
|
7
|
+
## Install and run
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npm install @revyl/verify
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Save this as `verify/check.ts`:
|
|
4
14
|
|
|
5
15
|
```ts
|
|
6
16
|
import { open } from '@revyl/verify';
|
|
7
17
|
|
|
8
|
-
const session = await open('ios');
|
|
9
|
-
|
|
10
|
-
await session.
|
|
11
|
-
await session.
|
|
18
|
+
const session = await open({ platform: 'ios', app: { appId: 'YOUR_REVYL_APP_ID' } });
|
|
19
|
+
try {
|
|
20
|
+
await session.instruction('Open General, then About');
|
|
21
|
+
await session.validation('The screen shows the operating system version');
|
|
22
|
+
await session.screenshot({ path: '.verify/about.png' });
|
|
23
|
+
} finally {
|
|
24
|
+
await session.close();
|
|
25
|
+
}
|
|
12
26
|
```
|
|
13
27
|
|
|
28
|
+
Run the script:
|
|
29
|
+
|
|
14
30
|
```bash
|
|
15
|
-
|
|
16
|
-
|
|
31
|
+
node verify/check.ts
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
On the first interactive run, Verify opens the Revyl browser approval flow once and saves the login in `~/.revyl/credentials.json` for the CLI and SDK to share.
|
|
35
|
+
|
|
36
|
+
Verify prints a live session URL and progress like this:
|
|
37
|
+
|
|
38
|
+
```text
|
|
39
|
+
[verify] ios: starting ios session https://app.revyl.ai/sessions/…
|
|
40
|
+
[verify] ios: ready
|
|
41
|
+
[verify] ios: ✓ instruction "Open General, then About" (12.0s)
|
|
42
|
+
[verify] ios: ✓ validation "The screen shows the operating system version" (9.8s)
|
|
43
|
+
[verify] ios: closed
|
|
17
44
|
```
|
|
18
45
|
|
|
19
|
-
|
|
46
|
+
The URL is live during the run and keeps the report and available recording after it closes. A failed instruction or validation throws a typed error and includes the same URL. In CI, Verify never opens a browser: provide `REVYL_API_KEY` as a secret.
|
|
20
47
|
|
|
21
|
-
|
|
22
|
-
- **A coding agent.** An agent can check its own change on a real device in five lines, then read a one-line outcome per step and a link to the recording.
|
|
23
|
-
- **CI.** Run the script in a job. The browsers and devices run in Revyl's cloud, so the runner needs nothing but Node and an API key.
|
|
24
|
-
- **Your test runner.** Call it inside node:test, Vitest, Jest, or Playwright Test, next to the tests you already have.
|
|
48
|
+
For Web, use `open({ platform: 'web', app: { appId, url } })`; for a computer, use `open({ platform: 'mac' })`.
|
|
25
49
|
|
|
26
|
-
##
|
|
50
|
+
## Scaffold a config (optional)
|
|
27
51
|
|
|
28
|
-
|
|
52
|
+
Use `init` when you want named targets in `verify.config.ts` and a generated `verify/example.ts` script:
|
|
29
53
|
|
|
30
54
|
```bash
|
|
31
55
|
npx @revyl/verify init
|
|
32
|
-
npm install --save-dev @revyl/verify
|
|
33
|
-
export REVYL_API_KEY=... # or sign in with the Revyl CLI: revyl auth login
|
|
34
|
-
node verify/example.ts
|
|
35
56
|
```
|
|
36
57
|
|
|
37
|
-
|
|
58
|
+
`init` detects your package manager, asks before adding `@revyl/verify` as a dev dependency, creates `verify.config.ts` and `verify/example.ts`, and prints the command that runs the example. Use `--yes` to accept installation without a prompt, or `--no-install` when Verify is already installed. It is optional; installing the package is enough to use inline targets.
|
|
38
59
|
|
|
39
|
-
|
|
60
|
+
The generated project starts with a named target. Use `--platform web`, `--platform ios`, `--platform android`, or `--platform mac` when you want to skip the platform question. Web and mobile targets need a Revyl app id; a Mac target does not. The generated config is ordinary TypeScript, so edit it when the example needs a pinned build, a public Web URL, or a different target.
|
|
40
61
|
|
|
41
|
-
|
|
42
|
-
| --- | --- |
|
|
43
|
-
| `open(target)` | Starts a Revyl cloud browser, device, or Mac computer, from a target name in `verify.config.ts` or a target object, and waits until it is ready |
|
|
44
|
-
| `session.exec(argv, options?)` | Runs structured argv on a Mac computer; returns `{ exitCode, stdout, stderr }` |
|
|
45
|
-
| `session.readFile(path)`, `writeFile(path, data)`, `stat(path)` | Reads a `Buffer`, writes a string or bytes, or returns `{ kind, size }` on a computer |
|
|
46
|
-
| `session.processes()`, `terminate(id)` | Lists run-owned process handles or terminates one on a computer |
|
|
47
|
-
| `session.instruction(text)` | Revyl's agent does what the text says. Throws `INSTRUCTION_FAILED` if it cannot. |
|
|
48
|
-
| `session.validation(text)` | Revyl's agent checks that the text holds on screen. Throws `VALIDATION_FAILED` if it does not, or returns the result with `{ soft: true }`. |
|
|
49
|
-
| `session.screenshot({ path })` | A PNG of the screen |
|
|
50
|
-
| `session.viewerUrl` | The session in Revyl: live while it runs, its recording after |
|
|
51
|
-
| `session.close()` | Stops the session. Verify also closes open sessions on Ctrl+C, on an uncaught error, and when a script ends without closing. |
|
|
62
|
+
See the [TypeScript getting-started guide](https://github.com/RevylAI/verify/blob/v0.3.0/packages/verify/docs/getting-started.md), the [Python SDK guide](https://github.com/RevylAI/verify/blob/v0.3.0/python/README.md), and the [Go SDK guide](https://github.com/RevylAI/verify/tree/go/v0.2.0/go).
|
|
52
63
|
|
|
53
|
-
|
|
64
|
+
## CI
|
|
54
65
|
|
|
55
|
-
|
|
66
|
+
For a normal GitHub Actions job, install from the lockfile with `npm ci` and pass `REVYL_API_KEY` from repository secrets to the script that runs Verify:
|
|
56
67
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
68
|
+
```yaml
|
|
69
|
+
- uses: actions/checkout@v5
|
|
70
|
+
- uses: actions/setup-node@v5
|
|
71
|
+
with:
|
|
72
|
+
node-version: 24
|
|
73
|
+
cache: npm
|
|
74
|
+
- run: npm ci
|
|
75
|
+
- run: node verify/example.ts
|
|
76
|
+
env:
|
|
77
|
+
REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}
|
|
78
|
+
```
|
|
64
79
|
|
|
65
|
-
|
|
80
|
+
On a `revyl-macos` job, `currentComputer()` attaches to the runner's existing computer without taking ownership. The runner releases the computer; closing the Verify attachment still cleans up hosted children. See the [CI guide](https://github.com/RevylAI/verify/blob/v0.3.0/packages/verify/docs/ci.md) and [computer API](https://github.com/RevylAI/verify/blob/v0.3.0/packages/verify/docs/api.md#inside-a-revyl-macos-github-actions-job).
|
|
66
81
|
|
|
67
|
-
|
|
82
|
+
## Reference authentication
|
|
68
83
|
|
|
69
|
-
|
|
84
|
+
Use manual setup when you need to control the package manager or keep the generated files out of the project:
|
|
70
85
|
|
|
71
|
-
|
|
86
|
+
```bash
|
|
87
|
+
revyl auth login
|
|
88
|
+
npx @revyl/verify whoami
|
|
89
|
+
```
|
|
72
90
|
|
|
73
|
-
|
|
74
|
-
- [Configuration and CLI](https://github.com/RevylAI/verify/blob/main/packages/verify/docs/configuration.md)
|
|
75
|
-
- [Web](https://github.com/RevylAI/verify/blob/main/packages/verify/docs/web.md) and [iOS and Android](https://github.com/RevylAI/verify/blob/main/packages/verify/docs/mobile.md)
|
|
76
|
-
- [Mac computers](https://github.com/RevylAI/verify/blob/main/packages/verify/docs/computers.md) and the [deferred Linux contract](https://github.com/RevylAI/verify/blob/main/packages/verify/docs/linux-computers-contract.md)
|
|
77
|
-
- [CI and test runners](https://github.com/RevylAI/verify/blob/main/packages/verify/docs/ci.md) and [error codes](https://github.com/RevylAI/verify/blob/main/packages/verify/docs/errors.md)
|
|
78
|
-
- [For coding agents](https://github.com/RevylAI/verify/blob/main/packages/verify/docs/agents.md)
|
|
91
|
+
The SDK resolves credentials in this order: a usable stored CLI login marked `local_auth_override`, `REVYL_API_KEY`, then another usable stored CLI login. In CI, use a secret manager rather than putting a key in source, command arguments, or logs. `whoami` checks the credentials without starting a session.
|
|
79
92
|
|
|
80
|
-
|
|
93
|
+
## TypeScript
|
|
81
94
|
|
|
82
|
-
|
|
95
|
+
```ts
|
|
96
|
+
import { open } from '@revyl/verify';
|
|
83
97
|
|
|
84
|
-
|
|
98
|
+
const session = await open('ios');
|
|
99
|
+
try {
|
|
100
|
+
await session.instruction('Open General, then About');
|
|
101
|
+
await session.validation('The screen shows the operating system version');
|
|
102
|
+
} finally {
|
|
103
|
+
await session.close();
|
|
104
|
+
}
|
|
105
|
+
```
|
|
85
106
|
|
|
86
|
-
For
|
|
107
|
+
For Web, use a public HTTPS URL in the target config. For iOS and Android, point the target at an app or build in Revyl. The [API guide](https://github.com/RevylAI/verify/blob/v0.3.0/packages/verify/docs/api.md) covers target options, step results, screenshots, reports, and cleanup.
|
|
87
108
|
|
|
88
|
-
##
|
|
109
|
+
## Go
|
|
89
110
|
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
111
|
+
Inside a function returning `error`, with a bounded `ctx`:
|
|
112
|
+
|
|
113
|
+
```go
|
|
114
|
+
session, err := verify.Open(ctx, verify.Target{Platform: verify.IOS, BundleID: "com.apple.Preferences"})
|
|
115
|
+
if err != nil {
|
|
116
|
+
return err
|
|
117
|
+
}
|
|
118
|
+
defer func() { _ = session.Close(ctx) }()
|
|
119
|
+
if _, err := session.Instruction(ctx, "Open General, then About"); err != nil {
|
|
120
|
+
return err
|
|
121
|
+
}
|
|
122
|
+
_, err = session.Validation(ctx, "The screen shows the operating system version")
|
|
123
|
+
return err
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Go calls return errors instead of throwing. The [Go reference](https://github.com/RevylAI/verify/tree/go/v0.2.0/go) covers context deadlines, screenshots, computer operations, and hosted sessions.
|
|
127
|
+
|
|
128
|
+
## Python
|
|
129
|
+
|
|
130
|
+
```python
|
|
131
|
+
from revyl_verify import Target, open
|
|
132
|
+
|
|
133
|
+
with open(Target(platform="ios", bundle_id="com.apple.Preferences")) as session:
|
|
134
|
+
session.instruction("Open General, then About")
|
|
135
|
+
session.validation("The screen shows the operating system version")
|
|
95
136
|
```
|
|
96
137
|
|
|
97
|
-
|
|
138
|
+
Python supports synchronous and asyncio APIs, but it has no bundled CLI. The [Python reference](https://github.com/RevylAI/verify/blob/v0.3.0/python/README.md) covers both APIs and computer operations.
|
|
139
|
+
|
|
140
|
+
## Platforms
|
|
141
|
+
|
|
142
|
+
| Platform | Instructions / validations | Commands / files / processes | PNG screenshot |
|
|
143
|
+
| --- | :---: | :---: | :---: |
|
|
144
|
+
| Web | Yes | — | Yes |
|
|
145
|
+
| iOS simulator | Yes | — | Yes |
|
|
146
|
+
| Android emulator | Yes | — | Yes |
|
|
147
|
+
| Mac computer | — | Yes | Yes |
|
|
148
|
+
| Linux computer | — | Yes | Yes |
|
|
149
|
+
|
|
150
|
+
Computer targets are early access and use commands, files, processes, and screenshots rather than natural-language agent steps. They have a server-bounded lifetime. See [computer setup](https://github.com/RevylAI/verify/blob/v0.3.0/packages/verify/docs/computers.md).
|
|
151
|
+
|
|
152
|
+
## Documentation
|
|
153
|
+
|
|
154
|
+
- [Documentation index](https://github.com/RevylAI/verify/tree/v0.3.0/packages/verify/docs/) · [API](https://github.com/RevylAI/verify/blob/v0.3.0/packages/verify/docs/api.md) · [CI](https://github.com/RevylAI/verify/blob/v0.3.0/packages/verify/docs/ci.md) · [Coding agents](https://github.com/RevylAI/verify/blob/v0.3.0/packages/verify/docs/agents.md)
|
|
155
|
+
- [End-to-end results and limitations](https://github.com/RevylAI/verify/blob/v0.3.0/packages/verify/docs/end-to-end.md)
|
|
156
|
+
- [Examples](https://github.com/RevylAI/verify/tree/v0.3.0/examples/): [Web store](https://github.com/RevylAI/verify/tree/v0.3.0/examples/web-store/) · [Mobile Settings](https://github.com/RevylAI/verify/tree/v0.3.0/examples/mobile-settings/) · [Mac workspace](https://github.com/RevylAI/verify/tree/v0.3.0/examples/mac-workspace/)
|
|
157
|
+
- [Contributing](https://github.com/RevylAI/verify/blob/v0.3.0/CONTRIBUTING.md) · [Security](https://github.com/RevylAI/verify/blob/v0.3.0/SECURITY.md) · [Releasing](https://github.com/RevylAI/verify/blob/v0.3.0/RELEASING.md)
|
|
158
|
+
|
|
159
|
+
## Troubleshooting
|
|
160
|
+
|
|
161
|
+
- **`NOT_AVAILABLE_ON_PLATFORM`:** the requested method is not available for that target. Computer targets do not run instruction or validation steps; Web and mobile targets do not expose computer commands or files.
|
|
162
|
+
- **`COMPUTER_LIMIT`:** computer admission was refused because of queued work or a concurrency limit. Verify does not retry creation or stop a definitely refused start.
|
|
163
|
+
- **Blank first screen:** mobile startup uses a bounded, best-effort blank-frame observation, not a rendered-readiness guarantee. Begin with an instruction that reaches the screen your workflow needs.
|
|
164
|
+
- **A validation report looks successful but the assertion failed:** inspect each step's `effective_status` and `validation_result`, and compare `validations_passed` with `total_validations` rather than transport counters.
|
|
98
165
|
|
|
99
166
|
## License
|
|
100
167
|
|
|
101
|
-
[Apache 2.0](https://github.com/RevylAI/verify/blob/
|
|
168
|
+
[Apache 2.0](https://github.com/RevylAI/verify/blob/v0.3.0/LICENSE).
|
package/dist/api/client.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { VerifyError } from '../errors.js';
|
|
1
|
+
import { VerifyError, type RefusalReason } from '../errors.js';
|
|
2
2
|
import type { AppResponse, BuildResponse, DeviceSessionDetail, DeviceSessionStopResponse, StartDeviceRequest, StartDeviceResponse, WhoAmI, WorkerConnection } from './types.js';
|
|
3
3
|
export declare const DEFAULT_BACKEND_URL = "https://backend.revyl.ai";
|
|
4
4
|
export declare const DEFAULT_APP_URL = "https://app.revyl.ai";
|
|
@@ -34,7 +34,7 @@ export declare class RevylClient {
|
|
|
34
34
|
startDevice(request: StartDeviceRequest, signal?: AbortSignal): Promise<StartDeviceResponse>;
|
|
35
35
|
getDeviceSession(sessionId: string, signal?: AbortSignal): Promise<DeviceSessionDetail>;
|
|
36
36
|
getWorkerConnection(workflowRunId: string, signal?: AbortSignal): Promise<WorkerConnection>;
|
|
37
|
-
stopDeviceSession(sessionId: string): Promise<DeviceSessionStopResponse>;
|
|
37
|
+
stopDeviceSession(sessionId: string, retry?: boolean): Promise<DeviceSessionStopResponse>;
|
|
38
38
|
cancelDevice(workflowRunId: string): Promise<unknown>;
|
|
39
39
|
getApp(appId: string, signal?: AbortSignal): Promise<AppResponse>;
|
|
40
40
|
registerWebUrlBuild(appId: string, request: {
|
|
@@ -55,9 +55,11 @@ export declare class RevylClient {
|
|
|
55
55
|
}): Promise<RawResponse>;
|
|
56
56
|
private sendOnce;
|
|
57
57
|
}
|
|
58
|
-
export declare function appUrl(): string;
|
|
59
|
-
export declare function sessionViewerUrl(sessionId: string, kind?: 'computer'): string;
|
|
58
|
+
export declare function appUrl(origin?: string): string;
|
|
59
|
+
export declare function sessionViewerUrl(sessionId: string, kind?: 'computer', origin?: string): string;
|
|
60
60
|
export declare function parseJson<T>(response: RawResponse, what: string): T;
|
|
61
61
|
export declare function responseDetail(response: RawResponse): string | undefined;
|
|
62
62
|
export declare function responseErrorCode(response: RawResponse): string | undefined;
|
|
63
63
|
export declare function apiError(method: string, path: string, response: RawResponse): VerifyError;
|
|
64
|
+
export declare function edgeBlockedError(response: RawResponse): VerifyError | undefined;
|
|
65
|
+
export declare function responseRefusalReason(response: RawResponse): RefusalReason | undefined;
|
package/dist/api/client.js
CHANGED
|
@@ -15,7 +15,7 @@ export class RevylClient {
|
|
|
15
15
|
requestTimeoutMs;
|
|
16
16
|
constructor(options) {
|
|
17
17
|
this.token = options.token;
|
|
18
|
-
this.backendUrl = validateBackendUrl(options.backendUrl ??
|
|
18
|
+
this.backendUrl = validateBackendUrl(options.backendUrl ?? DEFAULT_BACKEND_URL);
|
|
19
19
|
this.fetchImpl = options.fetch ?? fetch;
|
|
20
20
|
this.requestTimeoutMs = options.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS;
|
|
21
21
|
}
|
|
@@ -55,8 +55,8 @@ export class RevylClient {
|
|
|
55
55
|
getWorkerConnection(workflowRunId, signal) {
|
|
56
56
|
return this.json('GET', `/api/v1/execution/streaming/worker-connection/${encodeURIComponent(workflowRunId)}`, { signal });
|
|
57
57
|
}
|
|
58
|
-
stopDeviceSession(sessionId) {
|
|
59
|
-
return this.json('POST', `/api/v1/execution/device/sessions/${encodeURIComponent(sessionId)}/stop`, { retry
|
|
58
|
+
stopDeviceSession(sessionId, retry = true) {
|
|
59
|
+
return this.json('POST', `/api/v1/execution/device/sessions/${encodeURIComponent(sessionId)}/stop`, { retry });
|
|
60
60
|
}
|
|
61
61
|
cancelDevice(workflowRunId) {
|
|
62
62
|
return this.json('POST', `/api/v1/execution/device/status/cancel/${encodeURIComponent(workflowRunId)}`, { retry: true });
|
|
@@ -127,11 +127,12 @@ export class RevylClient {
|
|
|
127
127
|
}
|
|
128
128
|
}
|
|
129
129
|
}
|
|
130
|
-
export function appUrl() {
|
|
131
|
-
|
|
130
|
+
export function appUrl(origin = DEFAULT_APP_URL) {
|
|
131
|
+
validateBackendUrl(origin, 'viewer');
|
|
132
|
+
return origin.replace(/\/+$/, '');
|
|
132
133
|
}
|
|
133
|
-
export function sessionViewerUrl(sessionId, kind) {
|
|
134
|
-
return `${appUrl()}/sessions/${encodeURIComponent(sessionId)}${kind ? `?kind=${kind}` : ''}`;
|
|
134
|
+
export function sessionViewerUrl(sessionId, kind, origin) {
|
|
135
|
+
return `${appUrl(origin)}/sessions/${encodeURIComponent(sessionId)}${kind ? `?kind=${kind}` : ''}`;
|
|
135
136
|
}
|
|
136
137
|
export function parseJson(response, what) {
|
|
137
138
|
try {
|
|
@@ -175,27 +176,76 @@ export function responseErrorCode(response) {
|
|
|
175
176
|
}
|
|
176
177
|
}
|
|
177
178
|
export function apiError(method, path, response) {
|
|
178
|
-
const
|
|
179
|
+
const blocked = edgeBlockedError(response);
|
|
180
|
+
if (blocked)
|
|
181
|
+
return blocked;
|
|
182
|
+
const detail = path === '/api/v1/execution/start_device' ? responseKnownDetail(response, START_ERROR_DETAILS) : responseDetail(response);
|
|
179
183
|
const suffix = detail ? `: ${detail}` : '';
|
|
180
184
|
if (response.status === 401 || response.status === 403) {
|
|
181
|
-
return new VerifyError('AUTH_REJECTED', `Revyl rejected the request ${method} ${path} (HTTP ${response.status})${suffix}. Check REVYL_API_KEY or run 'revyl auth login'
|
|
185
|
+
return new VerifyError('AUTH_REJECTED', `Revyl rejected the request ${method} ${path} (HTTP ${response.status})${suffix}. Check REVYL_API_KEY or run 'revyl auth login'.`, { reason: responseRefusalReason(response), httpStatus: response.status });
|
|
182
186
|
}
|
|
183
|
-
return new VerifyError('API_ERROR', `Revyl request ${method} ${path} failed with HTTP ${response.status}${suffix}
|
|
187
|
+
return new VerifyError('API_ERROR', `Revyl request ${method} ${path} failed with HTTP ${response.status}${suffix}`, { reason: responseRefusalReason(response), httpStatus: response.status });
|
|
184
188
|
}
|
|
185
|
-
function
|
|
189
|
+
export function edgeBlockedError(response) {
|
|
190
|
+
if (response.status !== 403 || response.headers.get('content-type')?.split(';')[0]?.trim().toLowerCase() !== 'text/html' || !response.headers.get('cf-ray')?.trim())
|
|
191
|
+
return undefined;
|
|
192
|
+
try {
|
|
193
|
+
JSON.parse(new TextDecoder().decode(response.body));
|
|
194
|
+
return undefined;
|
|
195
|
+
}
|
|
196
|
+
catch { }
|
|
197
|
+
return new VerifyError('EDGE_BLOCKED', 'Revyl request was blocked at the network edge (HTTP 403). Check network access or contact support. Verify will not retry this request.', { httpStatus: response.status });
|
|
198
|
+
}
|
|
199
|
+
const REFUSAL_REASONS = new Set([
|
|
200
|
+
'web_runtime_unavailable', 'web_runtime_not_configured', 'computer_limit', 'runtime_unavailable', 'computer_runtime_unavailable',
|
|
201
|
+
'computer_at_capacity', 'computer_not_running', 'computer_cannot_host_platform', 'computer_not_found',
|
|
202
|
+
'web_build_required', 'web_build_not_supported', 'ios_build_required', 'ios_build_not_supported',
|
|
203
|
+
'android_build_required', 'android_build_not_supported', 'web_session_conflict', 'ios_session_conflict',
|
|
204
|
+
'android_session_conflict', 'not_available_on_platform', 'request_timeout', 'protocol_error',
|
|
205
|
+
]);
|
|
206
|
+
const START_ERROR_DETAILS = new Set([
|
|
207
|
+
...REFUSAL_REASONS,
|
|
208
|
+
'web_start_actor_invalid', 'web_app_required', 'web_app_not_found', 'web_idle_timeout_invalid',
|
|
209
|
+
'web_preview_overrides_unsupported', 'web_env_override_invalid', 'web_env_override_secret',
|
|
210
|
+
'web_sign_in_config_invalid', 'web_sign_in_override_not_secret', 'web_env_overrides_not_supported',
|
|
211
|
+
'web_build_selection_invalid', 'web_build_not_found', 'web_build_invalid', 'web_billing_not_configured',
|
|
212
|
+
'web_concurrency_unavailable', 'web_worker_unavailable', 'web_session_fence_conflict',
|
|
213
|
+
'web_callback_conflict', 'web_start_in_progress', 'web_start_unconfirmed', 'web_session_inputs_unavailable',
|
|
214
|
+
'sign_in_secret_not_found', 'sign_in_secret_unavailable', 'proof_sign_in_not_configured',
|
|
215
|
+
]);
|
|
216
|
+
export function responseRefusalReason(response) {
|
|
217
|
+
return responseKnownDetail(response, REFUSAL_REASONS);
|
|
218
|
+
}
|
|
219
|
+
function responseKnownDetail(response, codes) {
|
|
220
|
+
try {
|
|
221
|
+
const parsed = JSON.parse(new TextDecoder().decode(response.body));
|
|
222
|
+
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed))
|
|
223
|
+
return undefined;
|
|
224
|
+
const record = parsed;
|
|
225
|
+
const values = [record.detail];
|
|
226
|
+
if (record.detail && typeof record.detail === 'object')
|
|
227
|
+
values.push(record.detail.error);
|
|
228
|
+
values.push(record.error);
|
|
229
|
+
return values.find((value) => typeof value === 'string' && codes.has(value));
|
|
230
|
+
}
|
|
231
|
+
catch {
|
|
232
|
+
return undefined;
|
|
233
|
+
}
|
|
234
|
+
}
|
|
235
|
+
function validateBackendUrl(raw, label = 'backend') {
|
|
186
236
|
let url;
|
|
187
237
|
try {
|
|
188
238
|
url = new URL(raw);
|
|
189
239
|
}
|
|
190
240
|
catch {
|
|
191
|
-
throw new VerifyError('CONFIG_INVALID',
|
|
241
|
+
throw new VerifyError('CONFIG_INVALID', `The ${label} URL is not a valid URL.`);
|
|
192
242
|
}
|
|
193
243
|
const loopback = ['localhost', '127.0.0.1', '[::1]'].includes(url.hostname);
|
|
194
244
|
if (url.protocol !== 'https:' && !(loopback && url.protocol === 'http:')) {
|
|
195
|
-
throw new VerifyError('CONFIG_INVALID',
|
|
245
|
+
throw new VerifyError('CONFIG_INVALID', `The ${label} URL must use HTTPS (plain HTTP is allowed only on loopback).`);
|
|
196
246
|
}
|
|
197
247
|
if (url.username || url.password || url.search || url.hash) {
|
|
198
|
-
throw new VerifyError('CONFIG_INVALID',
|
|
248
|
+
throw new VerifyError('CONFIG_INVALID', `The ${label} URL must be an origin without credentials, query, or fragment.`);
|
|
199
249
|
}
|
|
200
250
|
return url.origin;
|
|
201
251
|
}
|
package/dist/api/types.d.ts
CHANGED
|
@@ -15,6 +15,7 @@ export interface StartDeviceRequest {
|
|
|
15
15
|
platform: MobilePlatform | 'web';
|
|
16
16
|
is_simulation?: boolean;
|
|
17
17
|
session_id?: string;
|
|
18
|
+
computer_session_id?: string;
|
|
18
19
|
app_id?: string;
|
|
19
20
|
build_id?: string;
|
|
20
21
|
app_url?: string;
|
|
@@ -130,6 +131,12 @@ export interface ComputerReport {
|
|
|
130
131
|
state: ComputerState;
|
|
131
132
|
release: 'pending' | 'confirmed' | 'unknown';
|
|
132
133
|
failure_reason?: string | null;
|
|
134
|
+
expires_at?: string | null;
|
|
135
|
+
}
|
|
136
|
+
export interface ComputerRecordingResponse {
|
|
137
|
+
url: string;
|
|
138
|
+
duration_seconds?: number | null;
|
|
139
|
+
started_at?: string | null;
|
|
133
140
|
}
|
|
134
141
|
export interface ComputerProcessHandle {
|
|
135
142
|
computer_session_id: string;
|
package/dist/cli/init.js
CHANGED
|
@@ -8,7 +8,7 @@ const CONFIG_FILE = 'verify.config.ts';
|
|
|
8
8
|
const EXAMPLE_SCRIPT = join('verify', 'example.ts');
|
|
9
9
|
export function validateInitAnswers(answers) {
|
|
10
10
|
if (!['web', 'ios', 'android', 'mac'].includes(answers.platform)) {
|
|
11
|
-
throw new VerifyError('INVALID_ARGUMENT', "--platform must be 'web', 'ios', 'android', or 'mac'. Linux
|
|
11
|
+
throw new VerifyError('INVALID_ARGUMENT', "--platform must be 'web', 'ios', 'android', or 'mac'. Linux init is unavailable; use open({ platform: 'linux' }) or a Linux config target.");
|
|
12
12
|
}
|
|
13
13
|
if (answers.platform === 'mac' && answers.appId !== undefined) {
|
|
14
14
|
throw new VerifyError('INVALID_ARGUMENT', '--app-id does not apply to Mac computers.');
|
|
@@ -101,8 +101,7 @@ try {
|
|
|
101
101
|
const steps = platform === 'web'
|
|
102
102
|
? `await session.instruction('Follow the link on the page');
|
|
103
103
|
await session.validation('A page about domains reserved for documentation is shown');`
|
|
104
|
-
:
|
|
105
|
-
// await session.instruction('Sign in as the demo user and open Settings');
|
|
104
|
+
: `await session.instruction('Wait for the app to finish opening and show its first screen');
|
|
106
105
|
await session.validation('The app has opened and shows its first screen without an error');`;
|
|
107
106
|
return `import { open } from '${PACKAGE_NAME}';
|
|
108
107
|
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
import { existsSync } from 'node:fs';
|
|
2
|
+
import { readFile, writeFile } from 'node:fs/promises';
|
|
3
|
+
import { spawn } from 'node:child_process';
|
|
4
|
+
import { dirname, join } from 'node:path';
|
|
5
|
+
import { VerifyError } from '../errors.js';
|
|
6
|
+
import { PACKAGE_NAME } from '../version.js';
|
|
7
|
+
export async function packageHasVerify(cwd) {
|
|
8
|
+
let raw;
|
|
9
|
+
try {
|
|
10
|
+
raw = await readFile(join(cwd, 'package.json'), 'utf8');
|
|
11
|
+
}
|
|
12
|
+
catch (error) {
|
|
13
|
+
if (error.code === 'ENOENT')
|
|
14
|
+
return false;
|
|
15
|
+
throw new VerifyError('CONFIG_INVALID', 'Could not read package.json.', { cause: error });
|
|
16
|
+
}
|
|
17
|
+
let manifest;
|
|
18
|
+
try {
|
|
19
|
+
manifest = JSON.parse(raw);
|
|
20
|
+
}
|
|
21
|
+
catch {
|
|
22
|
+
throw new VerifyError('CONFIG_INVALID', 'package.json is not valid JSON.');
|
|
23
|
+
}
|
|
24
|
+
if (!manifest || typeof manifest !== 'object' || Array.isArray(manifest))
|
|
25
|
+
throw new VerifyError('CONFIG_INVALID', 'package.json must contain an object.');
|
|
26
|
+
return ['dependencies', 'devDependencies', 'optionalDependencies', 'peerDependencies'].some((field) => {
|
|
27
|
+
const dependencies = manifest[field];
|
|
28
|
+
return dependencies && typeof dependencies === 'object' && !Array.isArray(dependencies) && Object.hasOwn(dependencies, PACKAGE_NAME);
|
|
29
|
+
});
|
|
30
|
+
}
|
|
31
|
+
export async function installVerify(cwd) {
|
|
32
|
+
const manager = detectPackageManager(cwd);
|
|
33
|
+
if (!existsSync(join(cwd, 'package.json')))
|
|
34
|
+
await writeFile(join(cwd, 'package.json'), '{\n "private": true,\n "type": "module"\n}\n', { flag: 'wx' });
|
|
35
|
+
const args = manager === 'npm'
|
|
36
|
+
? ['install', '--save-dev', '--ignore-scripts']
|
|
37
|
+
: manager === 'yarn'
|
|
38
|
+
? ['add', '--dev', '--mode=skip-builds']
|
|
39
|
+
: ['add', '--dev', '--ignore-scripts'];
|
|
40
|
+
args.push(PACKAGE_NAME);
|
|
41
|
+
process.stdout.write(`Installing ${PACKAGE_NAME} with ${manager}...\n`);
|
|
42
|
+
await new Promise((resolve, reject) => {
|
|
43
|
+
const env = Object.fromEntries(['PATH', 'HOME', 'USERPROFILE', 'TMPDIR', 'TEMP', 'TMP'].flatMap((key) => {
|
|
44
|
+
const value = process.env[key];
|
|
45
|
+
return value === undefined ? [] : [[key, value]];
|
|
46
|
+
}));
|
|
47
|
+
const child = spawn(manager, args, { cwd, env, stdio: 'inherit', shell: process.platform === 'win32', timeout: 180_000, killSignal: 'SIGKILL' });
|
|
48
|
+
const fail = () => reject(new VerifyError('CONFIG_INVALID', `Could not install ${PACKAGE_NAME} with ${manager}. Install it manually, then retry init --no-install.`));
|
|
49
|
+
child.once('error', fail);
|
|
50
|
+
child.once('exit', (code) => code === 0 ? resolve() : fail());
|
|
51
|
+
});
|
|
52
|
+
}
|
|
53
|
+
function detectPackageManager(cwd) {
|
|
54
|
+
const lockfiles = [['pnpm-lock.yaml', 'pnpm'], ['yarn.lock', 'yarn'], ['bun.lock', 'bun'], ['bun.lockb', 'bun'], ['package-lock.json', 'npm'], ['npm-shrinkwrap.json', 'npm']];
|
|
55
|
+
for (let directory = cwd;; directory = dirname(directory)) {
|
|
56
|
+
const managers = [...new Set(lockfiles.filter(([file]) => existsSync(join(directory, file))).map(([, manager]) => manager))];
|
|
57
|
+
if (managers.length > 1)
|
|
58
|
+
throw new VerifyError('CONFIG_INVALID', 'Found lockfiles for multiple package managers. Keep the project lockfile and retry init.');
|
|
59
|
+
if (managers[0])
|
|
60
|
+
return managers[0];
|
|
61
|
+
if (dirname(directory) === directory)
|
|
62
|
+
return 'npm';
|
|
63
|
+
}
|
|
64
|
+
}
|