@revyl/verify 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +15 -0
- package/LICENSE +201 -0
- package/README.md +100 -2
- package/SECURITY.md +13 -0
- package/dist/api/client.d.ts +63 -0
- package/dist/api/client.js +201 -0
- package/dist/api/types.d.ts +216 -0
- package/dist/api/types.js +1 -0
- package/dist/auth.d.ts +17 -0
- package/dist/auth.js +65 -0
- package/dist/cleanup.d.ts +13 -0
- package/dist/cleanup.js +72 -0
- package/dist/cli/init.d.ts +14 -0
- package/dist/cli/init.js +116 -0
- package/dist/cli/main.d.ts +3 -0
- package/dist/cli/main.js +127 -0
- package/dist/computer-session.d.ts +37 -0
- package/dist/computer-session.js +322 -0
- package/dist/config.d.ts +88 -0
- package/dist/config.js +212 -0
- package/dist/errors.d.ts +9 -0
- package/dist/errors.js +14 -0
- package/dist/index.d.ts +8 -0
- package/dist/index.js +3 -0
- package/dist/open.d.ts +13 -0
- package/dist/open.js +36 -0
- package/dist/session.d.ts +144 -0
- package/dist/session.js +411 -0
- package/dist/version.d.ts +2 -0
- package/dist/version.js +6 -0
- package/docs/README.md +18 -0
- package/docs/agents.md +38 -0
- package/docs/api.md +108 -0
- package/docs/ci.md +70 -0
- package/docs/computers.md +73 -0
- package/docs/configuration.md +91 -0
- package/docs/errors.md +38 -0
- package/docs/getting-started.md +79 -0
- package/docs/linux-computers-contract.md +107 -0
- package/docs/llms.txt +18 -0
- package/docs/mobile.md +43 -0
- package/docs/web.md +43 -0
- package/package.json +68 -4
package/docs/README.md
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# Verify documentation
|
|
2
|
+
|
|
3
|
+
These files ship inside the npm package at `node_modules/@revyl/verify/docs/`, so a coding agent working in your project can read the docs for the exact version you installed.
|
|
4
|
+
|
|
5
|
+
| File | Read it when you want to |
|
|
6
|
+
| --- | --- |
|
|
7
|
+
| [getting-started.md](getting-started.md) | Install Verify, authenticate, and run a first script |
|
|
8
|
+
| [api.md](api.md) | Look up session lifecycle, agent steps, screenshots, commands, files, and processes |
|
|
9
|
+
| [configuration.md](configuration.md) | Look up every option in `verify.config.ts`, the environment variables, and the CLI |
|
|
10
|
+
| [web.md](web.md) | Open a web app in a Revyl cloud browser |
|
|
11
|
+
| [mobile.md](mobile.md) | Open an iOS or Android app on a Revyl cloud device |
|
|
12
|
+
| [computers.md](computers.md) | Open a Mac computer for commands, files, processes, and screenshots |
|
|
13
|
+
| [linux-computers-contract.md](linux-computers-contract.md) | Read the deferred Linux request/response contract and unconfirmed assumptions |
|
|
14
|
+
| [ci.md](ci.md) | Call Verify from CI or from a test runner you already use |
|
|
15
|
+
| [errors.md](errors.md) | Understand an error code and fix it |
|
|
16
|
+
| [agents.md](agents.md) | Use Verify as a coding agent |
|
|
17
|
+
|
|
18
|
+
[llms.txt](llms.txt) lists the same files in the llms.txt format.
|
package/docs/agents.md
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# For coding agents
|
|
2
|
+
|
|
3
|
+
This page is for AI coding agents (Claude Code, Cursor, Codex, and others) that use Verify to check an app on a cloud browser/device or run commands and files on a Mac computer. The rest of this folder describes the API in full; this page is the workflow.
|
|
4
|
+
|
|
5
|
+
## Workflow
|
|
6
|
+
|
|
7
|
+
1. Read `verify.config.ts` to learn the target names and their platforms. If there is none, run `npx @revyl/verify init --platform <web|ios|android> --app-id <id>`, or `npx @revyl/verify init --platform mac` without an app id. You can also pass a target object to `open()`.
|
|
8
|
+
2. Check the credentials once with `npx verify whoami`. It starts no session.
|
|
9
|
+
3. Write a short script, or extend an existing one, that opens one target, runs the steps you need, and closes:
|
|
10
|
+
```ts
|
|
11
|
+
import { open } from '@revyl/verify';
|
|
12
|
+
|
|
13
|
+
const session = await open('ios');
|
|
14
|
+
await session.instruction('Sign in as the demo user');
|
|
15
|
+
await session.validation('The dashboard shows at least one project card');
|
|
16
|
+
await session.screenshot({ path: '.verify/dashboard.png' });
|
|
17
|
+
await session.close();
|
|
18
|
+
```
|
|
19
|
+
4. Run it with `node path/to/script.ts`. Share the session link from the first `[verify]` line with the user, so they can watch.
|
|
20
|
+
5. Read the outcome from the exit code and stderr. Each step prints `✓` or `✗` with its text; a failure prints `CODE: message`, ending with the session link. Look at any screenshot you saved before changing the script or the app.
|
|
21
|
+
|
|
22
|
+
## Rules
|
|
23
|
+
|
|
24
|
+
Mac computers are early access and not yet available in Revyl production. They work only on Revyl staging for enabled organizations through the optional `REVYL_BACKEND_URL` override with matching staging credentials. Read [availability and errors](computers.md#availability-and-errors) before starting a computer; do not try production or repeat an uncertain start/stop to probe availability.
|
|
25
|
+
|
|
26
|
+
For Mac work, use structured `exec(['program', 'arg'])`, `readFile`, `writeFile`, `stat`, and run-owned handles from `processes()`/`terminate(id)`. Check command output with ordinary assertions and close in `finally`; screenshots are supported, but instruction/validation are not. Do not fall back to agent steps on computers or shell/file methods on mobile/web. See [computers.md](computers.md) for limits and staging setup. Linux is a contract only: opening it fails before auth/network, so do not probe it with live calls. Starting a computer is a paid external action and needs the same approval as starting a device.
|
|
27
|
+
|
|
28
|
+
- Write steps the way you would brief a person: what to do, or what should be true, in one sentence. Name visible things ("the blue Save button in the footer"), not implementation details.
|
|
29
|
+
- Use `instruction` for doing and `validation` for checking. Do not ask an instruction to check something, or a validation to change anything.
|
|
30
|
+
- Prefer several small steps to one long one. Small steps fail with clearer reasons and time out less.
|
|
31
|
+
- Each step stands alone: Revyl does not carry values from one step to the next. Put everything a step needs into its text.
|
|
32
|
+
- Every step runs Revyl's agent again; nothing is cached or replayed. Do not rerun a whole script to probe one step.
|
|
33
|
+
- Await each step. Two steps at once on one session throw `INVALID_ARGUMENT`.
|
|
34
|
+
- Do not catch `VerifyError` to make a check pass. A `NOT_AVAILABLE_ON_PLATFORM` error means the call is not offered on that platform; see [web.md](web.md) and [computers.md](computers.md). A `STEP_FAILED` error means the step did not run to an outcome, not that the app is wrong.
|
|
35
|
+
- Never automatically repeat a computer mutation after `ACTION_UNCERTAIN` or a timeout: the command may have run, and timeout does not prove termination or rollback. Do not log argv, stdin, file contents, or credentials.
|
|
36
|
+
- Always close what you open. Each open session is a paid cloud browser, device, or computer.
|
|
37
|
+
- Never put secrets in scripts or config. Credentials come from `REVYL_API_KEY` or a Revyl CLI login; web sign-in uses the **name** of a Revyl organization secret.
|
|
38
|
+
- Step descriptions are sent to Revyl and logged to stderr by default. Do not include private credentials or customer data in instruction or validation text; use test-only sign-in and synthetic data instead. `log: false` suppresses local progress, not remote processing or reports.
|
package/docs/api.md
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
# API
|
|
2
|
+
|
|
3
|
+
```ts
|
|
4
|
+
import { open, defineConfig, VerifyError, isVerifyError } from '@revyl/verify';
|
|
5
|
+
```
|
|
6
|
+
|
|
7
|
+
## open(target, options?)
|
|
8
|
+
|
|
9
|
+
Starts a Revyl cloud browser, device, or Mac computer and resolves with a `Session` once it is ready. Starting usually takes from a few seconds to a couple of minutes. Linux targets are accepted in config/types but opening them throws `NOT_AVAILABLE_ON_PLATFORM` before credentials or network access.
|
|
10
|
+
|
|
11
|
+
Mac computers are early access on staging only for enabled organizations, not yet available in Revyl production. Use the optional `REVYL_BACKEND_URL` override and staging credentials described in [computer availability](computers.md#availability-and-errors).
|
|
12
|
+
|
|
13
|
+
```ts
|
|
14
|
+
const web = await open('web'); // a target named in verify.config.ts
|
|
15
|
+
const ios = await open({ platform: 'ios', app: { appId: '…' } }); // or a target object, no config needed
|
|
16
|
+
const mac = await open({ platform: 'mac' }); // computer, no app needed
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
- `target`: a target name from `verify.config.ts`, or a target object with the same shape as a config entry (see [configuration.md](configuration.md)). Verify reads the nearest `verify.config.ts` at or above the current folder.
|
|
20
|
+
- `options.config`: the path of a config file to read instead.
|
|
21
|
+
- `options.log`: where progress lines go. By default Verify writes the session link and each step's outcome to stderr, prefixed with `[verify]`. Pass a function to receive the lines yourself, or `false` to turn them off.
|
|
22
|
+
|
|
23
|
+
`open` needs credentials: `REVYL_API_KEY` or a Revyl CLI login. It throws `AUTH_MISSING` without them, `CONFIG_NOT_FOUND` or `CONFIG_INVALID` for config problems, and `SESSION_START_FAILED` or `TIMEOUT` when the browser or device does not start.
|
|
24
|
+
|
|
25
|
+
## session.instruction(text, options?)
|
|
26
|
+
|
|
27
|
+
Available on web/iOS/Android only. Computers throw `NOT_AVAILABLE_ON_PLATFORM` without sending an agent request.
|
|
28
|
+
|
|
29
|
+
Asks Revyl's agent to do something in the app, described the way you would tell a person: `'Sign in as the demo user and open Settings'`. The agent looks at the screen and acts until it is done or decides it cannot.
|
|
30
|
+
|
|
31
|
+
- Resolves with a `StepResult` when the agent finishes.
|
|
32
|
+
- Throws `INSTRUCTION_FAILED`, with Revyl's reason and the session link, when it cannot, and `STEP_FAILED` when the step is cancelled on Revyl.
|
|
33
|
+
- `options.timeout`: milliseconds before the step is cancelled on Revyl (`TIMEOUT`). Defaults to `stepTimeout` in the config, or 180000.
|
|
34
|
+
|
|
35
|
+
## session.validation(text, options?)
|
|
36
|
+
|
|
37
|
+
Available on web/iOS/Android only, not computers.
|
|
38
|
+
|
|
39
|
+
Asks Revyl's agent whether something holds on screen right now: `'The cart shows 2 items and a total of $40'`.
|
|
40
|
+
|
|
41
|
+
- Resolves with a `StepResult` whose `passed` is `true` when it holds.
|
|
42
|
+
- Throws `VALIDATION_FAILED`, with Revyl's reason and the session link, when it does not. This is what makes a plain script exit non-zero.
|
|
43
|
+
- Throws `STEP_FAILED` when Revyl cannot reach a verdict, for example because the step errored or was cancelled on Revyl.
|
|
44
|
+
- `options.soft: true` returns a failed result instead of throwing `VALIDATION_FAILED`, for scripts that want to collect several outcomes. It does not turn `STEP_FAILED` into a result.
|
|
45
|
+
- `options.timeout`: as for `instruction`.
|
|
46
|
+
|
|
47
|
+
## StepResult
|
|
48
|
+
|
|
49
|
+
```ts
|
|
50
|
+
interface StepResult {
|
|
51
|
+
type: 'instruction' | 'validation';
|
|
52
|
+
text: string; // the step as sent, trimmed
|
|
53
|
+
passed: boolean;
|
|
54
|
+
reason?: string; // Revyl's explanation, when it gave one
|
|
55
|
+
durationMs: number;
|
|
56
|
+
}
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
`session.steps` lists every step run on the session, in order.
|
|
60
|
+
|
|
61
|
+
## session.screenshot(options?)
|
|
62
|
+
|
|
63
|
+
Resolves with a PNG of the screen as a `Uint8Array`. With `options.path`, also writes it there, creating folders as needed.
|
|
64
|
+
|
|
65
|
+
## session.close()
|
|
66
|
+
|
|
67
|
+
Stops the session on Revyl. It is safe to call more than once, and steps after it throw `SESSION_ENDED`. A `Session` also works with `await using`, which closes it at the end of the block.
|
|
68
|
+
|
|
69
|
+
For computers, cleanup must be confirmed; a terminal state alone does not prove physical release. Uncertain release throws `ACTION_UNCERTAIN`.
|
|
70
|
+
|
|
71
|
+
## Computer commands, files, and processes
|
|
72
|
+
|
|
73
|
+
Mac sessions provide `exec(argv: string[], options?)`, `readFile(path, options?)` returning `Buffer`, `writeFile(path, string | Uint8Array, options?)`, `stat(path, options?)` returning `{ kind, size }`, `processes(options?)` returning `{ id, state, exitCode? }[]`, and `terminate(id, options?)`. They throw `NOT_AVAILABLE_ON_PLATFORM` on web/iOS/Android. See [computers.md](computers.md) for complete signatures, limits, timeouts, and examples. Mac also supports `screenshot()`; instruction/validation do not run on computers.
|
|
74
|
+
|
|
75
|
+
## Other session properties
|
|
76
|
+
|
|
77
|
+
| Property | Meaning |
|
|
78
|
+
| --- | --- |
|
|
79
|
+
| `viewerUrl` | The session in Revyl: a live view while it runs, its recording/report after. Use `revyl device report --session-id <id>` for web/mobile, or `revyl computer report <id>` for computers. |
|
|
80
|
+
| `id` | Revyl's session id |
|
|
81
|
+
| `platform` | `'web'`, `'ios'`, `'android'`, `'mac'`, or the reserved `'linux'` type (opening Linux is unavailable) |
|
|
82
|
+
| `target` | The target name, or the platform for a target object |
|
|
83
|
+
| `closed` | Whether `close()` was called |
|
|
84
|
+
|
|
85
|
+
## Rules worth knowing
|
|
86
|
+
|
|
87
|
+
- **One step at a time.** Await each step before starting the next on the same session; a second concurrent step throws `INVALID_ARGUMENT`. Separate sessions can run in parallel.
|
|
88
|
+
- **No hidden retries.** Verify never repeats a step or a device action. A step whose outcome is unknown because the connection failed mid-request throws `ACTION_UNCERTAIN`.
|
|
89
|
+
- **No memory between steps.** Each step is independent: Revyl does not carry values from one step to the next. Put everything a step needs into its text.
|
|
90
|
+
- **No replay or cache.** Every step runs Revyl's agent again, every time.
|
|
91
|
+
|
|
92
|
+
## Cleanup
|
|
93
|
+
|
|
94
|
+
While a session is open, Verify closes it before the process ends:
|
|
95
|
+
|
|
96
|
+
- on Ctrl+C or SIGTERM: it closes open sessions, then exits with 130 or 143 (a second Ctrl+C exits at once);
|
|
97
|
+
- on an uncaught error, such as a failed validation that nothing catches: it prints the error, closes open sessions, then exits with 1;
|
|
98
|
+
- when the script finishes without calling `close()`: it closes the sessions and prints a reminder.
|
|
99
|
+
|
|
100
|
+
Verify only takes over a signal or error event that nothing else in the process listens to, so test runners and apps that manage their own shutdown keep control of it; there, close sessions in `finally` or an `after` hook. A process that is killed outright, or calls `process.exit()` with sessions open, leaves browser/mobile sessions running until their idle timeout (10 minutes by default). Computers have a server-bounded one-hour lifetime instead; no idle timeout is configurable.
|
|
101
|
+
|
|
102
|
+
## defineConfig(config)
|
|
103
|
+
|
|
104
|
+
Returns the config unchanged; it exists for type checking in `verify.config.ts`. See [configuration.md](configuration.md).
|
|
105
|
+
|
|
106
|
+
## Errors
|
|
107
|
+
|
|
108
|
+
Everything Verify throws on purpose is a `VerifyError` with a stable `code`; `isVerifyError(error)` narrows to it. See [errors.md](errors.md).
|
package/docs/ci.md
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# CI and test runners
|
|
2
|
+
|
|
3
|
+
Verify is a library, so it runs wherever your code runs. Browsers, devices, and computers run in Revyl's cloud, so the runner needs nothing but Node.js and an API key. A Mac target works from a Linux runner; no local macOS host is needed. Use the same staging URL overrides and organization enablement as [computers.md](computers.md).
|
|
4
|
+
|
|
5
|
+
## A script in GitHub Actions
|
|
6
|
+
|
|
7
|
+
Add `REVYL_API_KEY` as a repository secret, then run your script with `node`. A failed instruction or validation makes the script exit non-zero, which fails the job.
|
|
8
|
+
|
|
9
|
+
```yaml
|
|
10
|
+
name: Verify
|
|
11
|
+
on:
|
|
12
|
+
pull_request:
|
|
13
|
+
|
|
14
|
+
jobs:
|
|
15
|
+
verify:
|
|
16
|
+
runs-on: ubuntu-latest
|
|
17
|
+
timeout-minutes: 30
|
|
18
|
+
steps:
|
|
19
|
+
- uses: actions/checkout@v5
|
|
20
|
+
- uses: actions/setup-node@v5
|
|
21
|
+
with:
|
|
22
|
+
node-version: 24
|
|
23
|
+
cache: npm
|
|
24
|
+
- run: npm ci
|
|
25
|
+
- run: node verify/checkout.ts
|
|
26
|
+
env:
|
|
27
|
+
REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}
|
|
28
|
+
- if: ${{ !cancelled() }}
|
|
29
|
+
uses: actions/upload-artifact@v4
|
|
30
|
+
with:
|
|
31
|
+
name: verify-screenshots
|
|
32
|
+
path: .verify/
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
To open each pull request's preview deployment on web, read the URL from the environment in `verify.config.ts`:
|
|
36
|
+
|
|
37
|
+
```ts
|
|
38
|
+
web: { platform: 'web', app: { appId: process.env.REVYL_APP_ID!, url: process.env.PREVIEW_URL!, branch: process.env.GITHUB_HEAD_REF } }
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
## Inside a test runner
|
|
42
|
+
|
|
43
|
+
Verify has no test runner of its own. Call it from the one you already use, and close the session in `finally` or an `after` hook: test runners handle errors and shutdown themselves, so Verify's automatic cleanup steps aside for them.
|
|
44
|
+
|
|
45
|
+
### node:test
|
|
46
|
+
|
|
47
|
+
```ts
|
|
48
|
+
import { test } from 'node:test';
|
|
49
|
+
import { open } from '@revyl/verify';
|
|
50
|
+
|
|
51
|
+
test('a shopper can check out', async () => {
|
|
52
|
+
const session = await open('ios');
|
|
53
|
+
try {
|
|
54
|
+
await session.instruction('Add the first product to the cart and check out with the saved card');
|
|
55
|
+
await session.validation('An order confirmation with an order number is shown');
|
|
56
|
+
} finally {
|
|
57
|
+
await session.close();
|
|
58
|
+
}
|
|
59
|
+
});
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
### Vitest, Jest, or Playwright Test
|
|
63
|
+
|
|
64
|
+
The same code works with `import { test } from 'vitest'`, Jest's global `test`, or `import { test } from '@playwright/test'`. Give the test a timeout long enough for a session to start and the steps to run, for example `test('…', async () => { … }, 600_000)` in Vitest and Jest, or `test.setTimeout(600_000)` in Playwright Test.
|
|
65
|
+
|
|
66
|
+
A thrown `VerifyError` fails the test with its code, Revyl's reason, and the session link.
|
|
67
|
+
|
|
68
|
+
## Cost and parallelism
|
|
69
|
+
|
|
70
|
+
Each `open()` holds one Revyl session until `close()`. Sessions run in parallel when you open several at once, so close each one as soon as you are done with it. A browser/mobile session that loses its process stops itself after `idleTimeoutSeconds` (10 minutes by default). Computers instead have a server-bounded one-hour lifetime and no configurable idle timeout; close in `finally` and do not treat a timed-out CI job as proof of physical release.
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
# Mac computers (early access)
|
|
2
|
+
|
|
3
|
+
Verify opens a Revyl cloud macOS computer for commands, files, processes, and screenshots. It does not open an assigned physical Mac. Mac computers are in **early access and not yet available in Revyl production**. They work only on Revyl staging for enabled organizations, which also need compute credits and a free computer slot. Ask [support@revyl.ai](mailto:support@revyl.ai) about staging access. No production availability date is promised.
|
|
4
|
+
|
|
5
|
+
```bash
|
|
6
|
+
export REVYL_BACKEND_URL=https://backend-staging.revyl.ai
|
|
7
|
+
export REVYL_APP_URL=https://staging.revyl.ai
|
|
8
|
+
export REVYL_API_KEY=... # staging credentials for your enabled organization
|
|
9
|
+
npx @revyl/verify init --platform mac
|
|
10
|
+
npm install --save-dev @revyl/verify
|
|
11
|
+
node verify/example.ts
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
`init` only writes local files; running the script starts a paid computer. No app id, build, device selector, image selector, or local Mac is needed. The generated script checks commands and a file round trip without agent steps.
|
|
15
|
+
|
|
16
|
+
`REVYL_BACKEND_URL` is an optional environment override for Verify generally, but currently required for Mac early access on staging. Production origins remain the defaults; never mix credentials between environments.
|
|
17
|
+
|
|
18
|
+
## Commands and files
|
|
19
|
+
|
|
20
|
+
```ts
|
|
21
|
+
import { open } from '@revyl/verify';
|
|
22
|
+
|
|
23
|
+
const computer = await open({ platform: 'mac', startTimeout: 600_000 });
|
|
24
|
+
try {
|
|
25
|
+
const result = await computer.exec(['uname', '-s']);
|
|
26
|
+
console.log(result.stdout);
|
|
27
|
+
await computer.writeFile('hello.txt', 'Hello!\n');
|
|
28
|
+
const bytes = await computer.readFile('hello.txt');
|
|
29
|
+
console.log(bytes.toString('utf8'));
|
|
30
|
+
console.log(await computer.stat('hello.txt'));
|
|
31
|
+
await computer.screenshot({ path: '.verify/mac.png' });
|
|
32
|
+
} finally {
|
|
33
|
+
await computer.close();
|
|
34
|
+
}
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
| Call | Result |
|
|
38
|
+
| --- | --- |
|
|
39
|
+
| `exec(argv: string[], { timeout?, stdin?, check? }?)` | `{ exitCode: number, stdout: string, stderr: string }`. Output strings are UTF-8 decoded. Nonzero exit throws `COMMAND_FAILED` unless `check: false`. |
|
|
40
|
+
| `readFile(path, { timeout? }?)` | A `Buffer`, up to the first 1 MiB. Use this for exact binary bytes. |
|
|
41
|
+
| `writeFile(path, data: string \| Uint8Array, { timeout? }?)` | `void`. Strings are UTF-8 encoded. Writes up to 64 KiB. |
|
|
42
|
+
| `stat(path, { timeout? }?)` | `{ kind: 'file' \| 'directory', size: number }`, with size in bytes. |
|
|
43
|
+
| `processes({ timeout? }?)` | Up to 64 `{ id: string, state: 'running' \| 'exited', exitCode?: number }` entries from commands on this computer. |
|
|
44
|
+
| `terminate(id, { timeout? }?)` | `void` after confirmed termination. Use an id from `processes()`, not an operating-system PID. |
|
|
45
|
+
| `screenshot({ path? }?)` | PNG `Uint8Array`; optionally saves it locally. |
|
|
46
|
+
|
|
47
|
+
`exec` takes structured argv, not a shell line. For shell syntax, explicitly invoke a shell, such as `['/bin/sh', '-c', 'printf hello']`. Never interpolate untrusted input into shell text. `stdin` accepts a string or `Uint8Array`. At most 64 arguments and 16,384 UTF-8 argv plus stdin bytes are allowed. Revyl keeps only the first 1 MiB of combined stdout/stderr; the command still runs to completion. Commands are synchronous at the library boundary; there is no interactive terminal or background-process creation API.
|
|
48
|
+
|
|
49
|
+
Paths are relative to the computer's workspace: no leading slash, empty components, `.` or `..`, backslashes, colons, or control characters. A path is at most 1,024 UTF-8 bytes. To make a directory, run a command such as `['mkdir', '-p', 'output']` before writing into it. There is no recursive sync or snapshot API.
|
|
50
|
+
|
|
51
|
+
Operation `timeout` is milliseconds, defaults to 30,000, and must be an integer from 1 to 300,000. It is also bounded by the run's remaining lifetime. A computer runs for at most one hour; `idleTimeoutSeconds` and the agent `stepTimeout` do not configure computer lifetime or commands. Revyl allows at most 256 operations per run.
|
|
52
|
+
|
|
53
|
+
## Availability and errors
|
|
54
|
+
|
|
55
|
+
Mac computers are early access on Revyl staging only, not yet available in Revyl production. A computer-create `503 runtime_unavailable` or typed `computer_not_enabled` becomes `NOT_AVAILABLE_ON_PLATFORM` with this availability guidance; Verify does not retry the create or send a stop after these definite refusals. Other admission and runtime failures retain their own error codes. Availability is determined by the server, not a client-side production-URL block.
|
|
56
|
+
|
|
57
|
+
The default `startTimeout` remains 600000 ms to allow a cold staging host to become ready; there is no measured cold-host baseline that justifies shortening it. Override it explicitly if needed. Progress announces availability immediately. A start timeout explains that no Mac computer may have been assigned; an unconfirmed close also links here. Neither proves that resources were released. Inspect the session and contact support rather than blindly repeating a start or stop.
|
|
58
|
+
|
|
59
|
+
`instruction()` and `validation()` throw `NOT_AVAILABLE_ON_PLATFORM` on computers without sending agent requests. Screenshot is supported on Mac; agent control of the screen is not part of this release. On web, iOS, and Android, shell/file/process methods throw `NOT_AVAILABLE_ON_PLATFORM`.
|
|
60
|
+
|
|
61
|
+
Linux is accepted in target types and config, but `open({ platform: 'linux' })` rejects with `NOT_AVAILABLE_ON_PLATFORM` before credentials or network access. A named target with `platform: 'linux'` behaves the same. `init --platform linux` is intentionally unavailable. See [linux-computers-contract.md](linux-computers-contract.md) for the future implementation contract and unconfirmed Linux assumptions.
|
|
62
|
+
|
|
63
|
+
Missing files and access denials throw `NOT_FOUND` and `ACCESS_DENIED`. An operation timeout with confirmed termination throws `TIMEOUT`; unconfirmed termination or execution throws `ACTION_UNCERTAIN`. Neither implies rollback. Execution outcome, evidence recording, and physical release are independent: an observed successful command can return its result while logging pending/failed report evidence. Do not repeat a mutation merely because its response was lost. Check the computer in Revyl before deciding what to do next. See [errors.md](errors.md).
|
|
64
|
+
|
|
65
|
+
## Lifecycle and transport
|
|
66
|
+
|
|
67
|
+
Verify mints a session UUID, creates `/api/v1/computer-runs` with `os: 'macos'`, then polls `GET /computer-runs/{id}/report` until `state: 'active'`, which means the owner route is ready. `starting` alone is not readiness. The public target spelling is `mac`, not `macos`.
|
|
68
|
+
|
|
69
|
+
Each command is submitted once to `/computer-runs/{id}/operations` with its own UUID `request_id` and `Prefer: respond-async`. The operation id is UUIDv5 with the computer UUID as namespace and the request UUID string as name. A `202` response has JSON `null`; Verify reads the operation by id until it settles instead of submitting it again. Binary operation fields use padded URL-safe base64 on the wire, with standard base64 accepted as input. Screenshot uses standard base64 PNG instead.
|
|
70
|
+
|
|
71
|
+
Verify bounds operation observation and explicit stop requests to 310 seconds. This is separate from the command's `timeout`: the backend may hold the async submission for 60 seconds before returning `202`. Automatic process-exit cleanup has a shorter 15-second budget and is best effort; use explicit `await close()` in `finally` to observe confirmed release.
|
|
72
|
+
|
|
73
|
+
`close()` posts `/computer-runs/{id}/stop`. Always close in `finally` or use `await using`; the same automatic process cleanup as other sessions applies. A terminal state alone is not confirmed physical release. Keep computer output and screenshots private: they can contain application data and credentials. Verify does not log file contents, argv, stdin, or command output automatically.
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# Configuration
|
|
2
|
+
|
|
3
|
+
`open('name')` reads targets from `verify.config.ts` (or `.mts`, `.js`, `.mjs`), looking in the current folder and then each folder above it. You can skip the config and pass a target object to `open()` instead.
|
|
4
|
+
|
|
5
|
+
```ts
|
|
6
|
+
import { defineConfig } from '@revyl/verify';
|
|
7
|
+
|
|
8
|
+
export default defineConfig({
|
|
9
|
+
targets: {
|
|
10
|
+
web: { platform: 'web', app: { appId: '…', url: 'https://staging.example.com' }, viewport: { width: 1280, height: 800 } },
|
|
11
|
+
ios: { platform: 'ios', app: { appId: '…' }, device: { model: 'iPhone 16', osVersion: '18.5' } },
|
|
12
|
+
pixel: { platform: 'android', app: { buildId: '…' }, device: { model: 'Pixel 8' } },
|
|
13
|
+
mac: { platform: 'mac' },
|
|
14
|
+
},
|
|
15
|
+
stepTimeout: 240_000,
|
|
16
|
+
});
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Unknown keys are rejected, so a typo fails loudly instead of being ignored.
|
|
20
|
+
|
|
21
|
+
## Top-level options
|
|
22
|
+
|
|
23
|
+
Mac targets currently require staging early access for an enabled organization through the optional `REVYL_BACKEND_URL` override and matching credentials; they are not yet available in Revyl production. See [computer availability](computers.md#availability-and-errors).
|
|
24
|
+
|
|
25
|
+
| Option | Default | Meaning |
|
|
26
|
+
| --- | --- | --- |
|
|
27
|
+
| `targets` | required | Named targets. The key is the name you pass to `open()`: letters, digits, dots, dashes, and underscores. |
|
|
28
|
+
| `stepTimeout` | `180000` | Milliseconds one instruction or validation may run before it is cancelled |
|
|
29
|
+
|
|
30
|
+
## Web targets
|
|
31
|
+
|
|
32
|
+
| Option | Meaning |
|
|
33
|
+
| --- | --- |
|
|
34
|
+
| `platform: 'web'` | |
|
|
35
|
+
| `app.appId` | Required. The Revyl web app. |
|
|
36
|
+
| `app.url` | A public HTTPS URL to open, registered as a build each time a session opens. |
|
|
37
|
+
| `app.buildId` | An exact Revyl web build to open instead of a URL. |
|
|
38
|
+
| `app.branch` | Branch to register the URL on, or to take the newest build from. |
|
|
39
|
+
| `viewport` | `{ width, height, deviceScaleFactor? }` in CSS pixels |
|
|
40
|
+
| `signIn` | `{ path, secret, persona? }`: a test-only sign-in route. See [web.md](web.md). |
|
|
41
|
+
| `startTimeout` | Milliseconds to wait for the browser. Default 600000. |
|
|
42
|
+
| `idleTimeoutSeconds` | Revyl stops the session after this much inactivity. Default 600. |
|
|
43
|
+
|
|
44
|
+
## iOS and Android targets
|
|
45
|
+
|
|
46
|
+
| Option | Meaning |
|
|
47
|
+
| --- | --- |
|
|
48
|
+
| `platform: 'ios'` or `'android'` | |
|
|
49
|
+
| `app.appId` | A Revyl app; each session installs its newest build. |
|
|
50
|
+
| `app.buildId` | An exact Revyl build to install. |
|
|
51
|
+
| `app.artifactUrl` | An HTTPS URL of an `.apk`, `.app.zip`, or `.ipa` to install. |
|
|
52
|
+
| `app.bundleId` | An app already on the device to launch instead, such as a system app. |
|
|
53
|
+
| `device` | `{ model?, osVersion? }`, for example `{ model: 'Pixel 8', osVersion: '15' }` |
|
|
54
|
+
| `startTimeout` | Milliseconds to wait for the device. Default 600000. |
|
|
55
|
+
| `idleTimeoutSeconds` | Default 600. |
|
|
56
|
+
|
|
57
|
+
Give exactly one of `appId`, `buildId`, `artifactUrl`, or `bundleId`.
|
|
58
|
+
|
|
59
|
+
## Computer targets
|
|
60
|
+
|
|
61
|
+
| Option | Meaning |
|
|
62
|
+
| --- | --- |
|
|
63
|
+
| `platform: 'mac'` | A cloud macOS computer, without `app` or `device` configuration |
|
|
64
|
+
| `platform: 'linux'` | Accepted contract value only; `open()` throws `NOT_AVAILABLE_ON_PLATFORM` before credentials/network |
|
|
65
|
+
| `startTimeout` | Milliseconds to wait for an active owner route; default 600000, range 10000..1800000 |
|
|
66
|
+
|
|
67
|
+
Computers do not accept `idleTimeoutSeconds`, image, provider, cwd, or lifetime selectors. The server bounds a run to one hour; per-operation timeout defaults to 30000 ms and accepts 1..300000 ms. See [computers.md](computers.md).
|
|
68
|
+
|
|
69
|
+
## Environment variables
|
|
70
|
+
|
|
71
|
+
| Variable | Meaning |
|
|
72
|
+
| --- | --- |
|
|
73
|
+
| `REVYL_API_KEY` | API key. Takes precedence over a Revyl CLI login. |
|
|
74
|
+
| `REVYL_BACKEND_URL` | Optional API origin override. Defaults to production `https://backend.revyl.ai`. Optional staging access uses `https://backend-staging.revyl.ai` with staging credentials. |
|
|
75
|
+
| `REVYL_APP_URL` | Optional session-link origin override. Defaults to production `https://app.revyl.ai`. Optional staging access uses `https://staging.revyl.ai`. |
|
|
76
|
+
|
|
77
|
+
## Command line
|
|
78
|
+
|
|
79
|
+
The `verify` command (also installed as `revyl-verify`) has two commands. Scripts run with `node`, not through the CLI.
|
|
80
|
+
|
|
81
|
+
```text
|
|
82
|
+
verify init [--platform web|ios|android|mac] [--app-id <id>] [--url <https-url>] [--force]
|
|
83
|
+
verify whoami
|
|
84
|
+
verify --help | --version
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
- `init` writes `verify.config.ts` and `verify/example.ts`, and adds `.verify/` to `.gitignore`. Without `--platform` it asks its questions interactively. It never overwrites files unless you pass `--force`.
|
|
88
|
+
- `init --platform mac` needs no app id and generates a command/file script. `--app-id` is rejected for Mac, `--url` applies only to web, and Linux init is unavailable.
|
|
89
|
+
- `whoami` prints the Revyl account and organization your credentials belong to.
|
|
90
|
+
|
|
91
|
+
Exit codes: `0` success, `1` a Revyl or file error, `2` a usage or authentication problem.
|
package/docs/errors.md
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Errors
|
|
2
|
+
|
|
3
|
+
Everything Verify throws on purpose is a `VerifyError` with a stable `code`. Messages name the problem and, for step failures, end with the session link.
|
|
4
|
+
|
|
5
|
+
```ts
|
|
6
|
+
import { isVerifyError } from '@revyl/verify';
|
|
7
|
+
|
|
8
|
+
try {
|
|
9
|
+
await session.validation('The cart is empty');
|
|
10
|
+
} catch (error) {
|
|
11
|
+
if (isVerifyError(error) && error.code === 'VALIDATION_FAILED') { /* … */ }
|
|
12
|
+
throw error;
|
|
13
|
+
}
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
| Code | What happened | What to do |
|
|
17
|
+
| --- | --- | --- |
|
|
18
|
+
| `INVALID_ARGUMENT` | A call got a bad value: an unknown target name, an empty step, two steps at once on one session | The message names the argument |
|
|
19
|
+
| `CONFIG_NOT_FOUND` | `open('name')` found no `verify.config.ts` at or above the current folder | Run `npx @revyl/verify init`, pass `options.config`, or pass a target object |
|
|
20
|
+
| `CONFIG_INVALID` | The config failed validation or did not load | The message names the exact option and the rule |
|
|
21
|
+
| `AUTH_MISSING` | No `REVYL_API_KEY` and no Revyl CLI login | Set `REVYL_API_KEY` or run `revyl auth login` |
|
|
22
|
+
| `AUTH_REJECTED` | Revyl rejected the credentials | Create a new API key, or log in again |
|
|
23
|
+
| `API_ERROR` | Revyl returned an unexpected response | Try again; if it persists, report it with the session link |
|
|
24
|
+
| `SESSION_START_FAILED` | The browser or device did not start | The message has Revyl's reason, for example no build for the app |
|
|
25
|
+
| `SESSION_ENDED` | The session was closed, or Revyl stopped it, for example after the idle timeout | Open a new session |
|
|
26
|
+
| `SESSION_BUSY` | The session refused a step because another one was still running on it, for example a step that just timed out | Wait for it to finish, then send the step again |
|
|
27
|
+
| `NOT_AVAILABLE_ON_PLATFORM` | The call is unavailable: Linux open, computer agent steps, device shell/file methods, a Mac computer before staging assignment, or a backend without web steps | See [web.md](web.md) and [computers.md](computers.md#availability-and-errors) |
|
|
28
|
+
| `COMMAND_FAILED` | A computer command exited nonzero | Read its reason; use `exec(argv, { check: false })` to inspect the exit code and output instead |
|
|
29
|
+
| `NOT_FOUND` | A computer file or resource was not found | Check the workspace-relative path |
|
|
30
|
+
| `ACCESS_DENIED` | The computer refused access | Check permissions; do not assume host/root access |
|
|
31
|
+
| `ACTION_FAILED` | Revyl refused or failed a request on the session | Read the reason in the message |
|
|
32
|
+
| `ACTION_UNCERTAIN` | The connection failed mid-request, so the step may or may not have run | Verify does not repeat it; check the session link before deciding what to do |
|
|
33
|
+
| `INSTRUCTION_FAILED` | Revyl's agent could not do what the instruction asked | Make the instruction more specific, or split it into smaller steps |
|
|
34
|
+
| `VALIDATION_FAILED` | The validation did not hold | The message has Revyl's reason; watch the recording at the session link |
|
|
35
|
+
| `STEP_FAILED` | Revyl could not run the step to an outcome: it errored, it was cancelled on Revyl, or a validation came back without a verdict. This says nothing about your app, and `soft` does not cover it. | Watch the session at the link, then run the step again |
|
|
36
|
+
| `TIMEOUT` | A step, a session start, or a request ran out of time | Raise `stepTimeout` or `startTimeout`, or split the step. A Mac start timeout can mean no production computer was assigned; see [computer availability](computers.md#availability-and-errors) before retrying |
|
|
37
|
+
|
|
38
|
+
Computer operations use `TIMEOUT` for bounded runtime with confirmed termination, and `ACTION_UNCERTAIN` when termination, execution, or cleanup is unconfirmed. An observed successful operation can return its result while logging pending/failed evidence. Timeout never proves rollback, and failed evidence does not prove the command never ran. Never automatically retry a mutation with a fresh request id; inspect the computer first. See [computers.md](computers.md).
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Getting started
|
|
2
|
+
|
|
3
|
+
Verify needs Node.js 22.18 or later, which runs TypeScript files directly, and a Revyl account.
|
|
4
|
+
|
|
5
|
+
## 1. Create a config and an example script
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
npx @revyl/verify init
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
`init` asks what you want to open (web, iOS, Android, or Mac), your Revyl app id for web/mobile, and for web, the URL to open. It writes two files and adds `.verify/` to `.gitignore`:
|
|
12
|
+
|
|
13
|
+
- `verify.config.ts`: the targets `open()` can start, by name
|
|
14
|
+
- `verify/example.ts`: a script that opens the target, runs steps and takes a screenshot (web/mobile) or checks commands and files (Mac), and closes
|
|
15
|
+
|
|
16
|
+
In scripts and CI, pass the answers as flags:
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
npx @revyl/verify init --platform web --app-id <revyl-app-id> --url https://staging.example.com
|
|
20
|
+
npx @revyl/verify init --platform ios --app-id <revyl-app-id>
|
|
21
|
+
npx @revyl/verify init --platform mac
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## 2. Install Verify
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
npm install --save-dev @revyl/verify
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
## 3. Authenticate
|
|
31
|
+
|
|
32
|
+
Verify uses the same credentials as the Revyl CLI, in the same order:
|
|
33
|
+
|
|
34
|
+
1. `REVYL_API_KEY`, an API key you create in Revyl's settings.
|
|
35
|
+
2. A login stored by the Revyl CLI in `~/.revyl/credentials.json` (`revyl auth login`).
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
export REVYL_API_KEY=... # or: revyl auth login
|
|
39
|
+
npx verify whoami # checks the credentials without starting a session
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## 4. Point the config at your app
|
|
43
|
+
|
|
44
|
+
Web and mobile targets need an app. Open the app in Revyl and copy its id into `app.appId`, or set `REVYL_APP_ID`, which the generated config reads. Mac targets need no app id.
|
|
45
|
+
|
|
46
|
+
- **Web:** set `app.url` to a public HTTPS URL, such as a staging or preview deployment. Revyl's cloud browser cannot reach `localhost`. See [web.md](web.md).
|
|
47
|
+
- **iOS and Android:** upload a build to the app in Revyl, for example with `revyl build upload`; each session installs the newest one. See [mobile.md](mobile.md).
|
|
48
|
+
- **Mac:** early access is not yet available in Revyl production. It works only on Revyl staging for enabled organizations through the optional `REVYL_BACKEND_URL` override and matching staging credentials. Agent instruction/validation are unavailable. See [computers.md](computers.md#availability-and-errors). Linux is a contract only and cannot be opened yet.
|
|
49
|
+
|
|
50
|
+
## 5. Run the script
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
node verify/example.ts
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
If Node warns that the module type is not specified, add `"type": "module"` to your `package.json` or rename the script to `example.mts`.
|
|
57
|
+
|
|
58
|
+
```text
|
|
59
|
+
[verify] ios: starting ios session https://app.revyl.ai/sessions/…
|
|
60
|
+
[verify] ios: ready
|
|
61
|
+
[verify] ios: ✓ validation "The app has opened and shows its first screen without an error" (9.8s)
|
|
62
|
+
[verify] ios: closed
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Open the link to watch the session live. After the session ends, the link keeps its recording/report. The Revyl CLI prints the stored report with `revyl device report --session-id <id>` for web/mobile, or `revyl computer report <id>` for Mac.
|
|
66
|
+
|
|
67
|
+
## The whole API
|
|
68
|
+
|
|
69
|
+
```ts
|
|
70
|
+
import { open } from '@revyl/verify';
|
|
71
|
+
|
|
72
|
+
const session = await open('ios'); // a target from verify.config.ts
|
|
73
|
+
await session.instruction('Add the first product to the cart'); // Revyl's agent does it
|
|
74
|
+
await session.validation('The cart badge shows 1'); // throws if it does not hold
|
|
75
|
+
await session.screenshot({ path: '.verify/cart.png' });
|
|
76
|
+
await session.close();
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
A failed instruction or validation throws a `VerifyError`. In a plain script, that ends the script with exit code 1, and Verify closes the session before the process exits. See [api.md](api.md) for every option, and [ci.md](ci.md) for calling Verify from a test runner.
|