ravensight-playtest 0.2.9 → 0.3.1

package/README.md

@@ -35,7 +35,7 @@ which is why a machine with no keychain still installs cleanly.
 ravensight-playtest login                  # device code, approved in the browser
 cd path/to/your/game
 ravensight-playtest init --game <gameId>   # links this repo
-ravensight-playtest check                  # confirms the machine and the balance
+ravensight-playtest check                  # one row per setup step, reserves nothing
 ravensight-playtest brief                  # four questions, two minutes
 ravensight-playtest run                    # prints the price, asks, then plays
 ```
@@ -81,11 +81,57 @@ with other jobs.
 
 ### check
 
-The doctor. Node version, server reachable, CLI new enough, credential valid and
-which store it came from, game visible, balance against the next job's price,
-content pack verified by sha256, disk, recommended concurrency, and the two
-optional tools. Exit code 3 when any row fails; a missing optional tool is a
-warning and never a failure, so CI does not break on it.
+The doctor. Each setup question is its own row, so a problem in one is never
+reported as another: `node`, `api reachable`, `cli version` (the installed
+version against the server's `latest` and `min_supported`), `credential store`,
+`login` (a token is present, the server accepts it, and which games it is scoped
+to), `project binding` (the game and environment this directory is linked to,
+and whether that game is in the token's scope), `brief` (present, complete, and
+which of goals, success criteria, controls and how to start are empty), `build`
+(the URL, Godot project or native build a run would use), `automation server`
+(native drivers only, see below), then the balance against the next job's price,
+the content pack verified by sha256, disk, recommended concurrency, and the two
+optional tools.
+
+Every row is `pass`, `warn` or `fail`, and every row that is not `pass` prints
+its fix. Exit code 3 when any row fails; a warning never fails, so CI does not
+break on a missing optional tool or a newer release. `check` reserves no units
+and starts no job, and says so at the end of its output: everything it asks the
+server is a read, including the price estimate.
+
+For a native driver (`ios_driver`, `android_driver`, `desktop_driver`,
+`unity_driver`, `unreal_driver`) the `build` row reads the driver config and
+confirms the app it names exists on disk (an app named by bundle id or package
+can only be confirmed by the server, so that is a warning), and the
+`automation server` row is the free capability probe: `GET /status` on the
+config's loopback endpoint, `POST /session` with the run's own capabilities,
+`GET /session/:id/source` to confirm the app opened and answers, then
+`DELETE /session/:id`. The detail names the step that failed. A session the
+probe could not close is a warning. This talks only to your own local server.
+[docs/NATIVE_RUNNER.md](docs/NATIVE_RUNNER.md) walks through an iOS simulator
+setup with the expected rows at each step.
+
+`check --json` prints one object:
+
+```json
+{
+  "ok": true,
+  "reserves_units": false,
+  "note": "check reserves no units and starts no job: ...",
+  "cli": { "installed": "0.3.1", "latest": "0.3.1", "min_supported": "0.1.0" },
+  "login": { "present": true, "valid": true, "games": [{ "gameId": "g1", "name": "Pocket Knight" }] },
+  "project": { "game": "g1", "environment": null, "in_scope": true },
+  "brief": { "present": true, "complete": true, "missing": [] },
+  "build": { "driver": "ios_driver", "ref": "/absolute/path/PocketKnight.app" },
+  "automation": { "endpoint": "http://127.0.0.1:4723", "reachable": true, "session": true, "app": true },
+  "checks": [{ "name": "login", "status": "pass", "detail": "...", "fix": null }]
+}
+```
+
+`automation` is null for a driver that is not native. A value of null inside it
+means that step was not reached because an earlier one failed. `environment`
+comes from the server when it reports one, otherwise from an `environment` key
+in `.ravensight/config.json`, otherwise null.
 
 For a web game there is also a `browser` row: `check` imports Playwright,
 launches a headless Chromium and closes it, which is exactly how a web run
@@ -114,6 +160,8 @@ automatically. `check` shows that same warning as a row when it applies.
 | `--personas <list>` | Price this persona list. Without it the balance row prices no personas, so the quote is lower than a real job's. |
 | `--godot-path <path>` | Where Godot is, if it is not on the PATH. |
 | `--web` | Check the browser even when this repo is not linked to a web build. |
+| `--driver <name>` | Check the setup for this driver instead of the configured one. |
+| `--driver-config <file>` | The native driver config to probe, instead of the configured build. |
 
 ### init
 
@@ -184,6 +232,14 @@ run options below.
 Registers a job, plays it, writes the reports, uploads them, finishes the job.
 Prints the price and asks before spending.
 
+With a native driver, `run` first runs the `build` and `automation server` rows
+of `check`. When either fails it exits 1 before the estimate, the confirmation
+and the register, so no job exists and nothing is reserved, and the message
+names the failing row and its fix. `--skip-preflight` registers without the
+probe, and then the job is registered and charged even if the automation server
+turns out to be down. The probe prints `probing <endpoint>, this can take up to
+3 minutes on a fresh simulator` before it starts, and each row as it completes.
+
 There is no product limit on how many personas one job can name; an unplayed
 run is refunded automatically when the job settles, so a large job never costs
 more than what actually got played. `--personas` past 100, or the same slug
@@ -197,9 +253,11 @@ splitting the job in two is what gets everything played.
 |---|---|
 | `--game <gameId>` | The game, when this repo is not linked. |
 | `--personas <list>` | Comma separated persona slugs. Defaults to the pack's set. |
-| `--driver <name>` | `playwright_web` or `godot_driver`. |
+| `--driver <name>` | `playwright_web`, `godot_driver`, `cli_stdio`, or a native driver: `desktop_driver`, `unity_driver`, `unreal_driver`, `android_driver`, `ios_driver`. |
 | `--build-url <url>` | The web build to play. |
 | `--godot-project <dir>` | The Godot project directory. |
+| `--driver-config <file>` | The native driver config, or the JSON-RPC bridge config for `cli_stdio`. |
+| `--skip-preflight` | Native drivers only: register without running the free automation probe first. The job is registered and charged even if the automation server turns out to be down. |
 | `--cli <command>` | The command that starts a text build. |
 | `--max-actions <n>` | Lower the actions one persona may take. It can only lower the persona's own patience (and the server's per run ceiling), never raise it. |
 | `--concurrency <n>` | Personas at once. Default 2; `check` recommends a number for this machine. |
@@ -248,9 +306,21 @@ How the report step reads what happened:
 
 - Console errors are recorded as automatic notes. A 4xx on a request (`Failed
   to load resource`, `HTTP 404`, `status: 401`, `401 Unauthorized` and similar)
-  is a network note: context, not a bug. A finding whose only evidence is
-  automatic network notes is capped at `minor` by the CLI, and its description
-  says so.
+  is a network note: context, not a bug. On the web a `Failed to load
+  resource` note names the request's method, URL (with its query string and
+  fragment removed) and status, and the report lists those requests under
+  network notes. A finding whose only evidence is automatic network notes is
+  capped at `minor` by the CLI, and its description says so.
+- The report step is given every action that went through (the newest 60):
+  what was clicked or typed into, by name or ref, and whether the screen
+  changed. A claim that something was never done or did nothing has to agree
+  with that list. A finding that says a named control "did nothing", "does
+  nothing", "never kicked off", "never started", "no response" or "nothing
+  happened", when the most recent click
+  on that control changed the screen without bringing up an error or a dialog,
+  at or after the screen the finding cites, is downgraded one severity in
+  report.json and report.md with a note saying at which step. It is never
+  dropped.
 - Every note carries `after`, the URL and title of the screen that followed
   it. A blocker or major finding has to cite the observation where the failure
   is visible and must not be contradicted by that next screen or by a later
@@ -293,6 +363,24 @@ macOS, Windows and Linux; ordinary typing is keystrokes. An editing shortcut
 such as select all uses the platform's own modifier (Command on a Mac, Control
 elsewhere, or `--key-modifier`).
 
+On the web each button, link and field the persona sees carries a short ref
+such as `[e12]`, and its centre `@(x,y)` when it is on screen. The persona
+clicks and types by ref, which names exactly one element even when two share
+a name; two such elements are both shown, each with where it is ("in header
+strip", "in main"). A name that matches twice is answered with the refs to
+choose from, and a target written as locator code (`getByRole(...)`,
+`page.locator(...)`, `css=`, `xpath=`) is refused at once. Failures like these
+are the driver's, not the game's: they are recorded as `driver_error`, and a
+blocker or major resting only on them is dropped before upload, as is any
+finding resting only on the persona's "waiting for X to finish" notes.
+
+Every field, checkbox, radio, slider and select the persona sees carries what
+it holds now, read from the page rather than from the accessibility text:
+`value="..."` (cut at 80 characters), `checked` or `not checked`, or
+`selected option="..."`. A password field shows `value=(hidden)`, and a field
+the page could not answer for shows no value at all, so a typed password never
+reaches the model.
+
 On a long web page the persona's text view of the screen is capped at 4,000
 characters. Over the cap, buttons, links and fields are kept first, a long run
 of look-alike items (a list of 249 cards) is shown as its first two and "and
@@ -301,7 +389,10 @@ after a long list still reaches the persona.
 
 Each persona starts at the game's URL as a fresh player with no account, so a
 sign-up form with validation can use up a whole unit before the game itself is
-reached. Point `--build-url` at a guest or demo path, or say where to start in
+reached. Every run gets its own identity to sign up with, an email
+`persona-<persona>-<short run id>@example.test` and a display name ending in the
+short run id, so a second run never hits "that email is taken" from the first.
+`example.test` is a reserved name and never delivers mail. Point `--build-url` at a guest or demo path, or say where to start in
 the brief's `how_to_start` note (for example "choose Play as guest"), when the
 sign-up is not what you want tested.
 
@@ -599,6 +690,8 @@ import { createClient, packs, uploadRunDir } from 'ravensight-playtest';
 |---|---|
 | `docs/api-contract.md` | Every server route this CLI calls, with the headers it sends and the status codes it handles. The list to check a server change against. |
 | `docs/godot-driver.md` | The Godot driver protocol, its method set, its error codes and its safety rules. |
+| `docs/ADAPTERS.md` | Every driver, its target and how far it has been validated. |
+| `docs/NATIVE_RUNNER.md` | A minimal iOS simulator runner through a local W3C automation server, step by step, with the `check` rows expected at each step. |
 | `src/api/README.md` | The module other code may import, and what the runner half expects from it. |
 
 ## License
@@ -629,4 +722,4 @@ feedback and timing-sensitive gameplay.
 
 ## Build adapters
 
-Browser and Godot runners share the paid playtest lifecycle with desktop, Unity/Unreal export, Android/iOS and custom JSON-RPC adapters. Native adapters require a locally configured automation server and remain experimental until validated against your build. See [adapter setup and validation boundaries](docs/ADAPTERS.md).
+Browser and Godot runners share the paid playtest lifecycle with desktop, Unity/Unreal export, Android/iOS and custom JSON-RPC adapters. Native adapters require a locally configured automation server and remain experimental until validated against your build. See [adapter setup and validation boundaries](docs/ADAPTERS.md) and the [native runner starter](docs/NATIVE_RUNNER.md); `check` probes the automation server for free before a paid native run.

package/docs/ADAPTERS.md

@@ -24,6 +24,8 @@ ravensight-playtest run --driver godot_driver --godot-project ./your-game
 
 ## Desktop and mobile
 
+Before anything else, `ravensight-playtest check --driver <name> --driver-config <file>` probes the automation server for free (status, a session with your capabilities, the app answering, the session closed), and `run` repeats that probe before it registers a paid native job. [NATIVE_RUNNER.md](NATIVE_RUNNER.md) is a step by step iOS simulator starter with the expected `check` rows.
+
 Install and configure an Appium-compatible driver for your platform. Start its server locally. Supply `ravensight-driver.json` with a loopback endpoint and capabilities appropriate to the installed driver:
 
 ```json

package/docs/NATIVE_RUNNER.md

@@ -0,0 +1,243 @@
+# Native runner starter: an iOS simulator app
+
+The smallest setup that lets a persona play a native or engine-exported iOS
+game on a simulator through `ios_driver`. Each step ends with the
+`ravensight-playtest check` rows you should see before moving on. `check` never
+reserves units or starts a job, so you can run it as often as you like.
+
+## Read this first: what is and is not validated
+
+Native persona playtesting needs a local W3C automation server, and it is
+**not yet validated on real hardware or a real simulator session**. A game that
+already sends events through a Ravensight SDK is not, by that alone, something a
+persona can play: the SDK token covers telemetry, and the persona needs the
+automation server to see the screen and tap it.
+
+| Step | Status on the machine this guide was written on |
+|---|---|
+| `check` rows for CLI version, login, project binding and brief | Tested by the CLI's own test suite against an in process server. |
+| `build` row reading a driver config and finding the `.app` on disk | Tested, including by running the CLI. |
+| `automation server` row with nothing listening | Tested by running the CLI. The output below is copied from that run. |
+| `automation server` row against a server that refuses the session, and against one that opens it | Tested by running the CLI against a small stub that answers the W3C endpoints the probe calls. **Not** tested against a real automation server. |
+| `run` refusing to register when the probe fails | Tested by the CLI's test suite. |
+| Building the app, booting the simulator, installing and starting a real automation server with its iOS driver | **Untested.** The commands are the standard ones, and they were not run for this guide. |
+| A persona actually playing an iOS app to a finished report | **Untested.** No paid native job has been run for this guide. |
+
+Treat the untested steps as a starting point, and tell us what you had to change.
+
+## What a persona can and cannot do on iOS
+
+The persona sees two things after every action: a screenshot (PNG) and the
+page source the automation server returns, cut to the model's text limit. On iOS
+that source is built from the accessibility tree, so a control with an
+accessibility label shows up by name and one without shows up as an unnamed
+element the persona can only find in the screenshot.
+
+It can:
+
+- tap a point on the screen (a W3C touch pointer: move, down, up);
+- swipe from one point to another with a single finger, 100 ms to 2 seconds;
+- type text into whatever already has focus, and press Enter, Escape, Tab,
+  Backspace, Space and the arrow keys;
+- wait up to 5 seconds.
+
+It cannot:
+
+- read engine logs, console output or crash logs (the native driver reports
+  none, so a crash is only seen as a screen that changed or stopped answering);
+- record video (screenshots only);
+- clear a text field before typing, or type into a field it has not tapped;
+- long press, pinch, rotate, use two fingers, shake the device, or press the
+  hardware buttons;
+- run more than one persona on the device at a time (native runs are
+  concurrency 1);
+- navigate by URL, install anything, or change the server, the capabilities or
+  the app: those come only from your driver config.
+
+Every device command times out after 30 seconds and responses are capped at
+8 MB. See [ADAPTERS.md](ADAPTERS.md) for the other drivers.
+
+## Step 1: the account side
+
+```bash
+npm install -g ravensight-playtest@0.3.1
+ravensight-playtest login
+cd path/to/your/game
+ravensight-playtest init --game <gameId>
+ravensight-playtest check --driver ios_driver
+```
+
+Expected rows (your email, game and version differ):
+
+```
+pass  cli version        installed 0.3.1; server latest 0.3.1, min_supported 0.1.0
+pass  login              you@example.com, valid, scoped to 1 game: Pocket Knight (g1)
+pass  project binding    linked to Pocket Knight (g1), in the token's scope
+fail  brief              this game has no expectations brief, and a persona run needs one
+fail  build              no driver config: a native run has nothing to open
+fail  automation server  no driver config, so there is no automation server to probe for ios_driver
+```
+
+`cli version` warns when the server lists a newer release and fails only below
+`min_supported`. `login` passing means the stored token is valid; it does not
+mean the game can be played. If `login` passes and `project binding` fails,
+the token is valid but was not approved for this game: run `login` again and
+approve it, rather than looking for a login problem.
+
+## Step 2: the brief
+
+```bash
+ravensight-playtest brief
+```
+
+A run needs four fields; for a native game these are the ones that decide
+whether a persona gets anywhere:
+
+| Field | What to write for a native game |
+|---|---|
+| `goals` | One or two. `onboarding` and `find_bugs` suit a first native run. |
+| `success_criteria` | Something visible on screen, one per line: "Reaches the map screen", "Finishes the tutorial battle". The persona has no logs to prove anything else. |
+| `build_notes.controls` | Taps and swipes only, named by what is on screen: "Tap a tile to move. Swipe left or right on the deck to pick a card." Say if anything needs a long press or two fingers, since the persona cannot do it. |
+| `build_notes.how_to_start` | The first screen after launch and what to tap: "Launches to the title screen; tap Start, then skip the intro with the X in the top right." Mention any system prompt (notifications, tracking) the persona will meet. |
+| `known_issues` | Anything already known, so it is not reported again. |
+| `design_intent_notes` | Choices that look like bugs: a deliberately hidden menu, an unskippable intro. |
+
+Expected after saving it:
+
+```
+pass  brief              version 1, complete (goals, success criteria, controls, how to start)
+```
+
+## Step 3: a simulator build (untested here)
+
+Build for the simulator and note the path of the `.app` it produces:
+
+```bash
+xcodebuild -scheme <YourScheme> -sdk iphonesimulator -configuration Debug \
+  -derivedDataPath build
+ls build/Build/Products/Debug-iphonesimulator/
+xcrun simctl list devices available
+xcrun simctl boot "<simulator name>"
+```
+
+## Step 4: the driver config
+
+Save `ravensight-driver.json` in the game's directory. The endpoint must be a
+loopback URL with no credentials; `check` and `run` refuse anything else. Use
+an absolute path for the app so `check` can confirm it exists:
+
+```json
+{
+  "endpoint": "http://127.0.0.1:4723",
+  "capabilities": {
+    "platformName": "iOS",
+    "appium:automationName": "XCUITest",
+    "appium:deviceName": "<simulator name>",
+    "appium:platformVersion": "<simulator iOS version>",
+    "appium:app": "/absolute/path/to/build/Build/Products/Debug-iphonesimulator/YourGame.app"
+  }
+}
+```
+
+For an app already installed on the simulator, use `"appium:bundleId":
+"com.example.yourgame"` in place of `appium:app`. Then `check` cannot confirm
+it on disk, so the `build` row is a warning and the `automation server` row is
+what confirms it.
+
+```bash
+ravensight-playtest check --driver ios_driver --driver-config ravensight-driver.json
+```
+
+With no automation server running yet, this is the output copied from running
+the CLI (paths shortened):
+
+```
+pass  build              appium:app /.../PocketKnight.app (from /.../ravensight-driver.json)
+fail  automation server  nothing answered at http://127.0.0.1:4723 (ECONNREFUSED)
+
+automation server: start your local W3C automation server (see docs/NATIVE_RUNNER.md) and confirm the endpoint in the driver config
+```
+
+A missing app reads `fail  build  appium:app /.../YourGame.app does not exist`.
+
+## Step 5: the automation server (untested here)
+
+Install a W3C automation server and its iOS simulator driver (the one that
+accepts `"appium:automationName": "XCUITest"`), following that server's own
+documentation, and start it listening on `127.0.0.1:4723`. ravensight-playtest
+does not install, bundle or start one. Do not expose it beyond loopback. The
+first session on a fresh simulator builds and installs the server's on device
+agent, which can take several minutes; the probe waits up to 3 minutes for a
+session.
+
+Run the same `check` again. If the server answers but cannot drive the
+simulator, the output is (copied from a run against the stub):
+
+```
+fail  automation server  the server did not create a session: session not created: Unknown device or simulator UDID
+
+automation server: boot the simulator and check the capabilities: platformName, deviceName, platformVersion and automationName must match a device the server can drive
+```
+
+When the server opens a session, launches the app and closes the session again
+(copied from a run against the stub, not a real server):
+
+```
+pass  build              appium:app /.../PocketKnight.app (from /.../ravensight-driver.json)
+pass  automation server  http://127.0.0.1:4723 reachable, session created and closed, app /.../PocketKnight.app opened
+```
+
+and `check --json` includes:
+
+```json
+"automation": { "endpoint": "http://127.0.0.1:4723", "reachable": true, "session": true, "app": true }
+```
+
+The other failures you may see, each naming its step: `... answered but says
+it is not ready`, `the capabilities name no app`, `... did not answer in the
+session` (the session opened but the app did not), and a `warn` reading
+`session created and NOT closed` when the probe could not close its session. If creating the session takes longer than 3 minutes the row
+says `a session may still be open on the server; restart it or wait before
+retrying`: when the server lists its sessions the probe asks it to close any that
+appeared meanwhile, and otherwise it cannot tell which one is its own.
+
+## Step 6: link the native build to the repo
+
+So `run` and `check` need no flags, set these in `.ravensight/config.json`:
+
+```json
+{
+  "driver": "ios_driver",
+  "build": { "kind": "driver_config", "ref": "ravensight-driver.json" }
+}
+```
+
+`run` resolves a relative `ref` against the directory you run it from, so run it
+from the game's directory. Now plain `ravensight-playtest check` should end with
+every row `pass` or `warn` and exit 0.
+
+## Step 7: the run (untested here)
+
+```bash
+ravensight-playtest run --personas curious-kid --max-actions 15
+```
+
+Before it asks the server for a price, `run` repeats the `build` and
+`automation server` rows as a free preflight, printing each row as it completes
+and a line before the slow step:
+
+```
+preflight pass build: appium:app /.../YourGame.app (from /.../ravensight-driver.json)
+probing http://127.0.0.1:4723, this can take up to 3 minutes on a fresh simulator
+preflight pass automation server: http://127.0.0.1:4723 reachable, session created and closed, app /.../YourGame.app opened
+```
+
+If either row fails, `run` exits 1 with `Preflight failed at "<row>": ...` and
+`No job was registered and nothing was reserved.` Only after the preflight
+passes does it print the price and ask. `--skip-preflight` registers without the
+probe: the job is then registered and charged even if the automation server
+turns out to be down, so use it only when you know the server is up and the
+probe's own session would get in the way.
+
+One persona and a low `--max-actions` keep the first native run cheap while you
+find out what the persona can reach.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ravensight-playtest",
-  "version": "0.2.9",
+  "version": "0.3.1",
   "description": "AI personas play your game and file reports to Ravensight.",
   "license": "MIT",
   "homepage": "https://ravensight.io/docs/#playtest",
@@ -26,6 +26,7 @@
     "src",
     "addons",
     "docs/ADAPTERS.md",
+    "docs/NATIVE_RUNNER.md",
     "README.md",
     "LICENSE",
     "!src/**/*.test.js",

package/src/cli.js

@@ -67,6 +67,8 @@ Run options (parsed by the runner, see src/run/args.js):
   --headed                  show the browser
   --headless                hide it
   --verbose                 print each turn
+  --skip-preflight          native drivers: register without probing the automation server first;
+                            the job is registered and charged even if the server turns out to be down
 
 Accepted and reported as not applied by this release:
   --quality, --offline-upload, --build-dir
@@ -162,10 +164,14 @@ export function buildProgram() {
     .addOption(new Option('--personas <list>', 'price this persona list').argParser(list))
     .option('--godot-path <path>', 'where Godot is, if it is not on the PATH')
     .option('--web', 'check the browser even when this repo is not linked to a web build')
+    .option('--driver <name>', 'check the setup for this driver instead of the configured one')
+    .option('--driver-config <file>', 'the native driver config to probe instead of the configured one')
     .action(capture(slot, async options => {
       const { check } = await import('./commands/check.js');
       return check(flagsFrom(options, {
         game: options.game,
+        driver: options.driver,
+        driverConfig: options.driverConfig,
         modules: options.modules,
         personas: options.personas,
         godotPath: options.godotPath,