@revyl/verify 0.4.0 → 0.5.0

package/docs/mobile.md

@@ -50,7 +50,7 @@ For Android Settings, instruct the agent to scroll or navigate until the OS vers
 
 ## Exact actions
 
-**Since 0.4.0. Early access.** In TypeScript (`@revyl/verify` 0.4.0), Python (`revyl-verify` 0.2.0) and Go (v0.3.0); the [Python](https://github.com/RevylAI/verify/blob/v0.4.0/python/README.md#exact-actions) and [Go](https://github.com/RevylAI/verify/blob/v0.4.0/go/README.md#exact-actions) READMEs show them with each language's names. When you know exactly what to press, skip the agent. Exact actions run on the device as given, with no model, and mix freely with `instruction()` and `validation()` in one script:
+**Early access.** Since TypeScript (`@revyl/verify`) 0.4.0, Python (`revyl-verify`) 0.2.0 and Go v0.3.0. `doubleTap`, `longPress`, `drag`, `typeFocused`, `goHome`, and Web keys and modifiers in `pressKey`, are new in TypeScript 0.5.0, Python 0.3.0 and Go v0.4.0. The [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#exact-actions) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#exact-actions) READMEs show them with each language's names. When you know exactly what to press, skip the agent. Exact actions run on the device as given, with no model, and mix freely with `instruction()` and `validation()` in one script:
 
 ```ts
 import { open } from '@revyl/verify';
@@ -64,7 +64,7 @@ try {
   await session.tap({ target: 'Back button' });
 
   await session.swipe('down');                                   // the finger moves down, so the list scrolls to the top
-  const search = await session.locate('Search field');           // where Revyl finds it, in iOS points
+  const search = await session.locate('Search field');           // a point in screenshot pixels
   await session.tap(search);                                     // a tap by coordinates
   await session.type(search, 'Wallpaper');
   await session.pressKey('enter');
@@ -75,14 +75,39 @@ try {
 }
 ```
 
-A location is `{ target: 'a description' }`, which Revyl's grounding model finds on the current screen, or `{ x, y }` from the top-left corner in iOS points or Android pixels; `screenSize()` gives the bounds. `type()` taps the field and clears it first unless you pass `{ clear: false }`. `pressKey()` takes `'enter'` or `'backspace'`.
+A location is `{ target: 'a description' }`, which Revyl's grounding model finds on the current screen, or `{ x, y }` in screenshot pixels from the top-left corner; `screenSize()` gives the bounds. `type()` taps the field and clears it first unless you pass `{ clear: false }`. `pressKey()` takes `'enter'` or `'backspace'`; for other keys, tap the on-screen control.
 
-On Android, `back()` presses the system back button. On both platforms, `launchApp(bundleId)` brings an installed app to the front and `openUrl('myapp://orders/42')` opens a deep link. A failed action throws `ACTION_FAILED`. One whose outcome Revyl can't confirm throws `ACTION_UNCERTAIN` or `TIMEOUT`, so check the session before acting again. Actions are never repeated. If Verify can't reach Revyl, the action fails with `API_ERROR` and isn't resent either, so check the session before trying it again. [api.md](api.md#exact-actions) lists every call.
+More actions, new in 0.5.0, on the same Settings app:
+
+```ts
+import { open } from '@revyl/verify';
+
+const session = await open({ platform: 'ios', app: { bundleId: 'com.apple.Preferences' } });
+try {
+  await session.goHome();                                        // the home screen
+  await session.longPress({ target: 'Settings app icon' });      // press and hold, 1.5 s by default
+  await session.validation('A menu for the Settings app is shown');
+  await session.goHome();                                        // closes the menu
+  await session.launchApp('com.apple.Preferences');
+  await session.drag({ target: 'General row' }, { target: 'Search field' });  // the list follows the finger
+  await session.doubleTap({ target: 'Settings title' });
+  await session.longPress({ x: 660, y: 1500 }, { duration: 800 });  // a point in screenshot pixels, held for 800 ms
+} finally {
+  await session.close();
+}
+```
+
+`goHome()` goes to the home screen on iOS and Android. `longPress()` holds for 1500 ms unless you pass `duration`, from 1 to 5000. `drag(from, to)` presses at one location, moves to the other and releases. On iOS and Android, type into a field with `type(location, text)`, which taps it first; `typeFocused()` is Web only and throws `NOT_AVAILABLE_ON_PLATFORM` here.
+
+On Android, `back()` presses the system back button. On both platforms, `goHome()` goes to the home screen, `launchApp(bundleId)` brings an installed app to the front and `openUrl('myapp://orders/42')` opens a deep link. A failed action throws `ACTION_FAILED`. One whose outcome Revyl can't confirm throws `ACTION_UNCERTAIN` or `TIMEOUT`, so check the session before acting again. Actions are never repeated. If Verify can't reach Revyl, the action fails with `API_ERROR` and isn't resent either, so check the session before trying it again. [api.md](api.md#exact-actions) lists every call.
 
 | Call | Web | iOS | Android | Mac computer |
 | --- | --- | --- | --- | --- |
-| `tap`, `type`, `swipe`, `pressKey`, `locate`, `screenSize` | Yes | Yes | Yes | No, use `exec()` and files |
+| `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `locate`, `screenSize` | Yes | Yes | Yes | No, use `exec()` and files |
+| `pressKey(key, { modifiers })` | Yes: `'enter'`, `'backspace'`, `'tab'`, `'escape'`, arrows and more, with modifiers | `'enter'` and `'backspace'` only | `'enter'` and `'backspace'` only | No |
+| `typeFocused(text)` | Yes, types into the focused field | No, use `type(location, text)` | No, use `type(location, text)` | No |
 | `back()` | Yes, browser history | No, tap the on-screen back control | Yes, the system back button | No |
+| `goHome()` | No, use `openUrl` or `navigate` | Yes, the home screen | Yes, the home screen | No |
 | `openUrl(url)` | Yes, loads an http(s) URL in the tab | Yes, a URL or deep link | Yes, a URL or deep link | No |
 | `launchApp(bundleId)` | No, use `openUrl` or `navigate` | Yes | Yes | No |
 | `navigate(url)` | Yes | No, use `openUrl` | No, use `openUrl` | No |
@@ -90,6 +115,8 @@ On Android, `back()` presses the system back button. On both platforms, `launchA
 | `screenshot()` | Yes | Yes | Yes | Yes |
 | `exec`, files, processes | No, open a Mac computer | No, open a Mac computer | No, open a Mac computer | Yes |
 
-Coordinates are in iOS points, Android pixels and Web CSS pixels, from the top-left corner. A call a platform doesn't support throws `NOT_AVAILABLE_ON_PLATFORM`, and its message names the alternative.
+**Coordinates.** Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()`, `swipe()` and every other exact action unchanged. `screenSize()` returns the screenshot's width and height, and `locate()` and the points `tap()`, `doubleTap()`, `longPress()` and `type()` return use the same pixels. A point outside the screenshot is refused with `INVALID_ARGUMENT` before any action is sent, and the message names the valid range. On iOS, Verify converts the pixels to the screen's points for you. On Web they equal CSS pixels at the default device scale factor of 1, and you still pass screenshot pixels at any other. This holds from TypeScript 0.4.1, Python 0.2.1 and Go v0.3.1; earlier versions took iOS points.
+
+A call a platform doesn't support throws `NOT_AVAILABLE_ON_PLATFORM`, and its message names the alternative.
 
-When you pass a [`model`](own-model.md), your model uses `tap`, `type`, `swipe`, `pressKey`, `openUrl` and, where the platform has it, `back` inside `instruction()`.
+When you pass a [`model`](own-model.md), your model uses `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `pressKey`, `openUrl` and, where the platform has them, `typeFocused`, `back` and `goHome` inside `instruction()`.

package/docs/own-model.md

@@ -1,10 +1,10 @@
 # Use your own model
 
-**Since 0.4.0. Early access.** In TypeScript (`@revyl/verify` 0.4.0), Python (`revyl-verify` 0.2.0) and Go (v0.3.0); the [Python](https://github.com/RevylAI/verify/blob/v0.4.0/python/README.md#your-own-model) and [Go](https://github.com/RevylAI/verify/blob/v0.4.0/go/README.md#your-own-model) READMEs show them with each language's names. Verify normally runs `instruction()` and `validation()` on Revyl's agent. Pass a `model` and the agent loop runs in your process instead, on a model you choose, with your own key. Revyl still provides the cloud browser or device and, by default, finds the elements your model names on screen.
+**Early access.** Since TypeScript (`@revyl/verify`) 0.4.0, Python (`revyl-verify`) 0.2.0 and Go v0.3.0; the [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#your-own-model) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#your-own-model) READMEs show them with each language's names. Verify normally runs `instruction()` and `validation()` on Revyl's agent. Pass a `model` and the agent loop runs in your process instead, on a model you choose, with your own key. Revyl still provides the cloud browser or device and, by default, finds the elements your model names on screen.
 
 Use it when you want a specific model, your own prompts and key, or faster steps on a cheap model. Leave `model` out and nothing changes.
 
-Your model uses the same [exact actions](api.md#exact-actions) you can call yourself: `tap`, `type`, `swipe`, `pressKey`, `openUrl` and, where the platform has it, `back`. So a script can mix your own actions, steps on your model and steps on Revyl's agent.
+Your model uses the same [exact actions](api.md#exact-actions) you can call yourself: `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `pressKey`, `openUrl` and, where the platform has them, `typeFocused` and Web keys with modifiers (Web), `back` (Web and Android) and `goHome` (iOS and Android). So a script can mix your own actions, steps on your model and steps on Revyl's agent.
 
 ## The one-line form
 
@@ -52,7 +52,7 @@ log.Println(step.Model.ID, len(step.Model.Turns), step.Model.InputTokens, step.M
 _, err = s.Validation(ctx, "The screen shows the iOS version")
 ```
 
-The [Python](https://github.com/RevylAI/verify/blob/v0.4.0/python/README.md#your-own-model) and [Go](https://github.com/RevylAI/verify/blob/v0.4.0/go/README.md#your-own-model) READMEs have the full reference for each.
+The [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#your-own-model) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#your-own-model) READMEs have the full reference for each.
 
 Set `model` on `open()` to run every step on it, or on one step to override it:
 
@@ -169,7 +169,7 @@ const session = await open('ios', { agent }); // or session.instruction(text, {
 
 **Your model sees** the system prompt, the step text, a short log of the actions it took so far and whether they worked, and the current screenshot, downscaled so its longest side is at most 1280 pixels (`maxImageSide`).
 
-**Revyl sees** what it always sees for a session it hosts: the device and its screen. On top of that it receives each device action (taps, typed text, swipes, key presses, URLs), the element descriptions your model writes, such as "General row" (not with `grounding: 'model'`), and the model's name, such as `anthropic/claude-sonnet-5-5`, for the session report.
+**Revyl sees** what it always sees for a session it hosts: the device and its screen. On top of that it receives each device action (taps, long presses, drags, typed text, swipes, key presses, URLs), the element descriptions your model writes, such as "General row" (not with `grounding: 'model'`), and the model's name, such as `anthropic/claude-sonnet-5-5`, for the session report.
 
 **Revyl does not receive** the step text, the prompts, your model's output or reasoning, or your key. Keys go only to the provider's endpoint, are never logged, and are replaced with `[redacted]` in error messages, as are custom header values. A custom endpoint's address is never sent to Revyl.
 

package/docs/web.md

@@ -53,7 +53,7 @@ Revyl runs one step at a time on a session. A step sent while another is still r
 
 ## Exact actions
 
-**Since 0.4.0. Early access.** In TypeScript (`@revyl/verify` 0.4.0), Python (`revyl-verify` 0.2.0) and Go (v0.3.0); the [Python](https://github.com/RevylAI/verify/blob/v0.4.0/python/README.md#exact-actions) and [Go](https://github.com/RevylAI/verify/blob/v0.4.0/go/README.md#exact-actions) READMEs show them with each language's names. When you know exactly what to click, skip the agent. Exact actions run in the browser as given, with no model, and mix freely with `instruction()` and `validation()`:
+**Early access.** Since TypeScript (`@revyl/verify`) 0.4.0, Python (`revyl-verify`) 0.2.0 and Go v0.3.0. `doubleTap`, `longPress`, `drag`, `typeFocused`, `goHome`, and Web keys and modifiers in `pressKey`, are new in TypeScript 0.5.0, Python 0.3.0 and Go v0.4.0. The [Python](https://github.com/RevylAI/verify/blob/v0.5.0/python/README.md#exact-actions) and [Go](https://github.com/RevylAI/verify/blob/v0.5.0/go/README.md#exact-actions) READMEs show them with each language's names. When you know exactly what to click, skip the agent. Exact actions run in the browser as given, with no model, and mix freely with `instruction()` and `validation()`:
 
 ```ts
 import { open } from '@revyl/verify';
@@ -66,7 +66,7 @@ try {
   await session.validation('A list of products is shown');
   await session.swipe('up');                                     // the finger moves up, so the page scrolls down
   await session.instruction('Add the Sauce Labs Onesie to the cart');  // hand the fuzzy part to the agent
-  const cart = await session.locate('Shopping cart icon');      // where Revyl finds it, in CSS pixels
+  const cart = await session.locate('Shopping cart icon');      // a point in screenshot pixels
   await session.tap(cart);                                       // a tap by coordinates
   await session.validation('The cart lists the Sauce Labs Onesie');
   await session.back();                                          // browser history
@@ -76,12 +76,42 @@ try {
 }
 ```
 
-A location is `{ target: 'a description' }`, which Revyl's grounding model finds on the current page, or `{ x, y }` from the top-left corner of the viewport; `screenSize()` gives the bounds and `locate(target)` gives an element's point without acting. `type()` clicks the field and clears it first unless you pass `{ clear: false }`. `pressKey()` takes `'enter'` or `'backspace'`, and `openUrl(url)` loads a page like `navigate(url)`. A failed action throws `ACTION_FAILED`. One whose outcome Revyl can't confirm throws `ACTION_UNCERTAIN` or `TIMEOUT`, so check the session before acting again. Actions are never repeated. If Verify can't reach Revyl, the action fails with `API_ERROR` and isn't resent either, so check the session before trying it again. [api.md](api.md#exact-actions) lists every call.
+A location is `{ target: 'a description' }`, which Revyl's grounding model finds on the current page, or `{ x, y }` in screenshot pixels from the top-left corner; `screenSize()` gives the bounds and `locate(target)` gives an element's point without acting. `type()` clicks the field and clears it first unless you pass `{ clear: false }`. `openUrl(url)` loads a page like `navigate(url)`.
+
+More actions, new in 0.5.0, on the same store:
+
+```ts
+import { open } from '@revyl/verify';
+
+const session = await open('web');                               // https://www.saucedemo.com in examples/web-store
+try {
+  await session.tap({ target: 'Username field' });
+  await session.typeFocused('standard_user');                    // types into the field that has focus, replacing its text
+  await session.pressKey('tab');                                 // focus moves to the password field
+  await session.typeFocused('secret_sauce');
+  await session.pressKey('enter');
+  await session.validation('A list of products is shown');
+  await session.tap({ target: 'Open Menu button' });
+  await session.pressKey('escape');                              // closes the menu
+  await session.validation('The side menu is closed');
+  await session.doubleTap({ target: 'Products heading' });       // a double-click
+  await session.pressKey('a', { modifiers: ['ctrl'] });          // a chord: a lowercase letter or digit with modifiers
+  await session.longPress({ target: 'Products heading' }, { duration: 800 });  // press and hold for 800 ms
+  await session.drag({ target: 'Sauce Labs Backpack description' }, { target: 'Sauce Labs Bike Light description' });  // selects the text between
+} finally {
+  await session.close();
+}
+```
+
+`typeFocused(text)` types where the focus already is, without clicking, and replaces the field's text unless you pass `{ clear: false }`. Focus the field first, with `tap()` or `pressKey('tab')`: with no field focused, nothing is typed and no error is thrown. `pressKey()` takes `'enter'`, `'backspace'`, `'tab'`, `'escape'`, `'delete'`, `'space'`, `'arrowup'`, `'arrowdown'`, `'arrowleft'`, `'arrowright'`, `'home'`, `'end'`, `'pageup'` and `'pagedown'`, with up to four `modifiers` held (`'ctrl'`, `'shift'`, `'alt'`, `'meta'`); with modifiers a lowercase letter or digit works too. `longPress()` holds for 1500 ms unless you pass `duration`, from 1 to 5000. `drag(from, to)` presses at one location, moves to the other and releases. A failed action throws `ACTION_FAILED`. One whose outcome Revyl can't confirm throws `ACTION_UNCERTAIN` or `TIMEOUT`, so check the session before acting again. Actions are never repeated. If Verify can't reach Revyl, the action fails with `API_ERROR` and isn't resent either, so check the session before trying it again. [api.md](api.md#exact-actions) lists every call.
 
 | Call | Web | iOS | Android | Mac computer |
 | --- | --- | --- | --- | --- |
-| `tap`, `type`, `swipe`, `pressKey`, `locate`, `screenSize` | Yes | Yes | Yes | No, use `exec()` and files |
+| `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `locate`, `screenSize` | Yes | Yes | Yes | No, use `exec()` and files |
+| `pressKey(key, { modifiers })` | Yes: `'enter'`, `'backspace'`, `'tab'`, `'escape'`, arrows and more, with modifiers | `'enter'` and `'backspace'` only | `'enter'` and `'backspace'` only | No |
+| `typeFocused(text)` | Yes, types into the focused field | No, use `type(location, text)` | No, use `type(location, text)` | No |
 | `back()` | Yes, browser history | No, tap the on-screen back control | Yes, the system back button | No |
+| `goHome()` | No, use `openUrl` or `navigate` | Yes, the home screen | Yes, the home screen | No |
 | `openUrl(url)` | Yes, loads an http(s) URL in the tab | Yes, a URL or deep link | Yes, a URL or deep link | No |
 | `launchApp(bundleId)` | No, use `openUrl` or `navigate` | Yes | Yes | No |
 | `navigate(url)` | Yes | No, use `openUrl` | No, use `openUrl` | No |
@@ -89,6 +119,8 @@ A location is `{ target: 'a description' }`, which Revyl's grounding model finds
 | `screenshot()` | Yes | Yes | Yes | Yes |
 | `exec`, files, processes | No, open a Mac computer | No, open a Mac computer | No, open a Mac computer | Yes |
 
-Coordinates are in iOS points, Android pixels and Web CSS pixels, from the top-left corner. A call a platform doesn't support throws `NOT_AVAILABLE_ON_PLATFORM`, and its message names the alternative.
+**Coordinates.** Prefer a description, such as `tap({ target: 'Sign in button' })`: Revyl finds the element on the current screen, and you never handle coordinates. Otherwise a point is `{ x, y }` in screenshot pixels on every platform: whole numbers from the top-left corner of the image `screenshot()` returns. So a position you read off a screenshot goes to `tap()`, `type()`, `swipe()` and every other exact action unchanged. `screenSize()` returns the screenshot's width and height, and `locate()` and the points `tap()`, `doubleTap()`, `longPress()` and `type()` return use the same pixels. A point outside the screenshot is refused with `INVALID_ARGUMENT` before any action is sent, and the message names the valid range. On iOS, Verify converts the pixels to the screen's points for you. On Web they equal CSS pixels at the default device scale factor of 1, and you still pass screenshot pixels at any other. This holds from TypeScript 0.4.1, Python 0.2.1 and Go v0.3.1; earlier versions took iOS points.
+
+A call a platform doesn't support throws `NOT_AVAILABLE_ON_PLATFORM`, and its message names the alternative.
 
-When you pass a [`model`](own-model.md), your model uses `tap`, `type`, `swipe`, `pressKey`, `openUrl` and, where the platform has it, `back` inside `instruction()`.
+When you pass a [`model`](own-model.md), your model uses `tap`, `doubleTap`, `longPress`, `type`, `swipe`, `drag`, `pressKey`, `openUrl` and, where the platform has them, `typeFocused`, `back` and `goHome` inside `instruction()`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@revyl/verify",
-  "version": "0.4.0",
+  "version": "0.5.0",
   "description": "Open Revyl cloud browsers and mobile devices for agent steps, or Mac computers for commands and files, from TypeScript.",
   "keywords": [
     "e2e",