ravensight-playtest 0.4.0 → 0.5.0

package/README.md

@@ -7,7 +7,7 @@ the reports and renders them.
 Simulation Runner adds seeded headless batches for mechanics, economy and
 progression checks through your own game adapter. It runs locally without
 model calls and saves bounded results to the same Playtest workspace. See
-[Simulation Runner](docs/SIMULATION.md) for the held candidate and setup.
+[Simulation Runner](docs/SIMULATION.md) for setup.
 
 You need a Ravensight account and Node 22 or newer. You do not need an
 Anthropic API key: model calls go through Ravensight under the same token you
@@ -374,7 +374,10 @@ splitting the job in two is what gets everything played.
 | `--max-actions <n>` | Lower the actions one persona may take. It can only lower the persona's own patience (and the server's per run ceiling), never raise it. |
 | `--long` | A long persona run: twice the actions, model budget and wall time of a standard run, at twice the price. The price is read from the estimate and confirmed before any charge. |
 | `--concurrency <n>` | Personas at once. Default 2; `check` recommends a number for this machine. |
-| `--upload-video` | Upload `session.webm` as well as the reports. |
+| `--controller <name>` | One playthrough using `standard` or an available System One controller. The server must advertise and quote that exact choice. |
+| `--compare-with jev` | Separate agent/skill and Jev playthroughs for each chosen persona. Shows their combined quote before purchase. Comparisons run one playthrough at a time. |
+| `--record-video` | Capture the selected web or graphical Godot game locally. Does not upload the video. |
+| `--upload-video` | Record and explicitly upload `session.webm` or `session.mp4` with the reports. |
 | `--upload-transcript` | Upload `transcript.jsonl` as well as the reports. |
 | `--dry-upload` | List every file and byte that would be sent, and send nothing. |
 | `--key-modifier <meta\|control>` | The modifier the browser driver uses for editing shortcuts such as select all. Defaults to this machine's own (Command on macOS, Control elsewhere); set it when the browser's platform is not this machine's. `RAVENSIGHT_KEY_MODIFIER` does the same. |
@@ -780,7 +783,7 @@ to it. The short version:
 
 Uploaded by default: `report.md`, `report.json`, `usage.json`, `changeset.json`,
 the aggregate and capability reports, and `screenshots/`. Uploaded only when you ask:
-`transcript.jsonl` (`--upload-transcript`) and `session.webm`
+`transcript.jsonl` (`--upload-transcript`) and `session.webm` or `session.mp4`
 (`--upload-video`). **Source code never, at any path.** The upload allowlist has
 no entry for it, the CLI refuses it locally, and the server refuses anything not
 on its own copy of the list. `src/upload/allowlist.js` is that copy, and a test
@@ -881,6 +884,59 @@ before registering a paid job. No provider key belongs in the CLI. Selection cos
 share the existing persona-run budget and do not create another wallet charge.
 Resume preserves the controller recorded in the job journal.
 
+### Recorded playthroughs and comparisons
+
+The `0.5.0` candidate adds local recording and explicit comparisons. It requires a
+matching server release for controller plans; an older server is refused before
+purchase. Rune is recognized as a future controller but is unavailable until
+provider access, pricing and acceptance are complete. These commands do not
+activate graphical AWS hosting.
+
+To keep one playthrough's video on your computer:
+
+```sh
+ravensight-playtest run --driver playwright_web --build-url https://your-game.example/ --record-video
+```
+
+To request two reports, one from the agent/skill controller and one from Jev:
+
+```sh
+ravensight-playtest run --driver playwright_web --build-url https://your-game.example/ --personas curious-kid --compare-with jev --record-video
+```
+
+The server quotes every persona/controller run before the CLI asks for payment.
+Each keeps its own existing model budget and settlement rules. Two standard runs
+at the current $1 unit price total $2; the displayed server quote is authoritative.
+No comparison model call or extra recording fee is added. Use `--upload-video`
+instead of `--record-video` only when you want the videos uploaded for private
+dashboard playback. Recording is silent and captures only the selected game,
+not your desktop. Browser popups and navigation to another origin are blocked
+while recording.
+
+Web recording produces WebM. Graphical Godot recording needs an existing FFmpeg
+installation and produces MP4; a headless Godot run or missing encoder is reported
+as unsupported. Capture is bounded to 15 minutes, 200 MiB of video, 1280 by 720
+pixels and 10 frames per second for the Godot encoder. Godot raw frames have a
+separate 256 MiB bound. This is evidence capture, not a promise of smooth video
+at the game's original frame rate. Cancellation, crashes and limits can leave a
+playable partial video; local `recording.json` records that outcome. Neither that
+file nor raw frames are uploaded as artifacts.
+
+With `--upload-video`, completion also sends a bounded capture status, duration
+and stop reason. This is the runner's account of capture, separate from verified
+upload or report success. Without video-upload consent, no capture status is sent.
+Older recordings and later byte-only uploads can show status as unreported.
+Malformed local status is omitted so it cannot prevent an existing report from
+being completed.
+
+Each controller starts a separate game session and produces its own report.
+Saved progress and random events may still differ; this is not a guaranteed
+identical-state benchmark. The aggregate compares evidence-bearing findings and
+marks missing, failed or partial playthroughs as inconclusive. Agreement between
+reports is evidence to review, not confirmation that a bug exists. In the
+dashboard, choose a playthrough and its Recording tab; timestamp links refer only
+to that run's uploaded video. Headless Simulation Runner has no rendered video.
+
 For Jump Bunny, use `https://reality-software-entertainment.github.io/poc-jump-bunny/`
 and the public player label **Ravensight Playtester**. This is a local browser run,
 not a hosted browser service. Human playtesting is still needed for subjective
@@ -892,7 +948,7 @@ Browser and Godot runners share the paid playtest lifecycle with desktop, Unity/
 
 ## Simulation Runner
 
-Available in held local candidate `0.4.0`; not yet published. Simulation Runner requires macOS or Linux (including WSL); native Windows is refused before purchase.
+Available since `0.4.0`. Simulation Runner requires macOS or Linux (including WSL); native Windows is refused before purchase.
 
 Run seeded mechanics tests through a local game adapter, compare explicit strategies and replay selected traces. A batch uses the existing standard Playtest purchase of 5 AI units from prepaid balance, confirmed against the server quote. Only the summary is uploaded; game execution and replay traces stay local. It does not automatically call a model or replace testing the player interface.
 

package/docs/SIMULATION.md

@@ -1,6 +1,6 @@
 # Simulation Runner
 
-Held local candidate: `0.4.0`. This package has not been published; use a reviewed local tarball with the matching held server implementation.
+Available since `0.4.0`, with the matching released Ravensight server.
 
 Simulation Runner exercises game mechanics through a developer-owned local adapter. It can run seeded games across explicit strategies, summarize outcomes and metrics, and save selected replay traces. Use it for questions such as whether an economy supports several viable strategies or whether a progression rule traps a simulated player. It does not judge animation, interface usability, feel or subjective fun.
 

package/docs/godot-driver.md

@@ -0,0 +1,310 @@
+# The Godot driver
+
+`godot_driver` lets a playtest persona observe and control a native Godot game
+by launching the developer's own Godot on their own machine. It has two halves:
+
+- `addons/ravensight_driver/` (GDScript, Godot 4.2+): a TCP server that answers
+  JSON-RPC requests from inside the running game.
+- `src/run/drivers/godot.js` (Node): starts Godot, speaks that protocol, turns
+  the scene tree into an observation, and writes screenshots.
+
+Spec 07 is the contract for both halves. This document is how the two sides fit
+together and what the driver is and is not able to do.
+
+## The driver never reaches players
+
+The addon refuses to start unless every one of these holds:
+
+1. The build is not an export template, **or** it was exported with the custom
+   feature tag `ravensight_driver`. A normal release export never activates it.
+2. **And** either `--ravensight-driver` is in the user args (after the `--`
+   separator) or `RAVENSIGHT_DRIVER=1` is set.
+3. It binds `127.0.0.1` only. There is no code path that binds any other
+   address, and the Node client refuses to dial a non loopback host.
+4. If `RAVENSIGHT_DRIVER_TOKEN` is set, the first request on a connection must
+   be `hello{token}`. A wrong token gets `-32002` and the connection closes.
+5. One client at a time. A second connection is answered with `-32001 busy` and
+   hung up on.
+
+When any of that fails the autoload calls `queue_free()` on itself in
+`_ready()`, so an inactive driver is not a node that idles in `_process`, it is
+not in the tree at all.
+
+On top of refusing to run, `export_plugin.gd` **strips the whole addon** from
+any export whose preset was not built with the `ravensight_driver` feature tag,
+so the usual release build does not contain the driver's code.
+
+One file does not simply vanish. A developer who installs the addon in their own
+project gets `RavensightDriver` written into `project.godot`, and `project.godot`
+ships in the export. Deleting `driver.gd` outright would leave that autoload
+pointing at a file that is not there, and the exported game would greet the
+player with a load error before its first frame. So the strip leaves a four line
+inert stub at `driver.gd`'s path: `extends Node` and a `_ready()` that calls
+`queue_free()`. The autoload resolves, nothing listens, and no driver code
+ships. The alternative, failing the export with a message, was rejected because
+it punishes a developer for having installed the addon at all.
+
+This is covered by a test, not an assertion of intent: `godot-real.test.js`
+exports the same project twice with `--export-pack` (which needs no export
+templates) and checks that the plain pack contains the stub and none of the
+driver's source, while a pack built with the feature tag contains all of it.
+
+## How the addon gets into the project
+
+The CLI never writes to the developer's project. `prepareProjectCopy` copies the
+project to a temp directory (mode `0700`, `.git` and `node_modules` excluded,
+refused above 2 GB), drops the addon into `addons/ravensight_driver/` in the
+copy, and registers the autoload **twice** in that copy:
+
+- in `override.cfg`, which is the mechanism spec 07 describes, and
+- in the copy's own `project.godot`.
+
+Spec 07's open question 1 asks whether an `override.cfg` autoload works for
+every project layout. Rather than leave that unanswered at runtime, both are
+written. They name the same script, so whichever Godot honors, the result is
+identical. A project that already ships an `override.cfg` or already registers
+other autoloads keeps them: only the one `RavensightDriver` key is added.
+
+**Symlinks are the trap here.** `project.godot`, `override.cfg` and `addons/`
+can each be a symlink pointing outside the project, which is what a developer
+gets from a shared settings repo or a vendored addons checkout. Writing to a
+symlink writes to its target, so a naive copy would put the injection into the
+developer's real files, permanently, breaking the one promise spec 07 makes
+about the project directory. Three things prevent it:
+
+- the copy runs with `dereference: true`, so a symlink becomes a real file or
+  directory in the copy rather than a pointer back out of it;
+- every path this module writes to is materialized first, component by
+  component, replacing any surviving link with a real copy of its content;
+- `measureProject` follows symlinked directories when it sizes the project,
+  because sizing that ignored them reported a few hundred bytes for a tree the
+  copy then dereferenced into gigabytes, and the 2 GB cap did nothing.
+
+Dereferencing brings its own failure mode, so the copy filter drops two shapes
+outright. A link resolving to the project directory or any ancestor of it
+(`selfref -> .`, `up -> ..`) is dropped: dereferenced, the copy walks the project
+into itself, finds the link again, and repeats until `PATH_MAX`, which one such
+link turned into 33 nested copies. And a directory already dereferenced once is
+not dereferenced again, because `measureProject` counts a shared target once and
+the two have to agree or the cap stops meaning anything.
+
+Both rules compare **realpaths**, not `path.resolve()` output. On macOS
+`os.tmpdir()` is `/var/folders/...`, a symlink to `/private/var/folders/...`, so
+a project living there never matches its own symlink target unless both sides are
+fully resolved, and the rule quietly does nothing. That is a test that caught it,
+not a hypothetical.
+
+A failed `prepareProjectCopy` deletes the temp root it created, so a project that
+cannot be prepared does not leave a half written copy on disk. A `tempRoot` the
+caller passed in is left alone: it is theirs.
+
+A dangling symlink is dropped from the copy instead of failing the launch:
+`fs.cp` throws on one it cannot dereference, and one broken link is not a reason
+a project cannot be playtested.
+
+## Launch
+
+```
+<godot> --path <temp copy> --resolution 1280x720 -- --ravensight-driver --ravensight-driver-port=<port>
+```
+
+The binary is found the way spec 17 describes: `--godot-path`, then `$GODOT`,
+then `PATH` (`godot`, `godot4`), then the platform's usual install locations.
+When the project declares an engine version in `config/features`, a binary with
+that major.minor wins over a newer one. Anything below 4.2 is refused.
+
+The environment matters as much as the arguments:
+
+| Variable | Read by | Why |
+|---|---|---|
+| `RAVENSIGHT_DRIVER=1` | the addon | activation |
+| `RAVENSIGHT_DRIVER_PORT` | the addon | loopback port, default 47800 |
+| `RAVENSIGHT_DRIVER_TOKEN` | the addon | per run handshake secret |
+| `RAVENSIGHT_PLAYTEST_TOKEN` | the game's own `Ravensight.gd` | playtest session auth |
+| `RAVENSIGHT_PLAYTEST_RUN_ID` | the game's own `Ravensight.gd` | `pt_run` tag, throwaway device id |
+| `RAVENSIGHT_PLAYTEST_JOB_ID` | the game's own `Ravensight.gd` | `pt_job` tag |
+| `RAVENSIGHT_PLAYTEST_PERSONA` | the game's own `Ravensight.gd` | `persona` tag |
+
+Only those, plus a fixed allowlist of variables a process needs to start at all
+(`PATH`, `HOME`, the `XDG_*`, `DISPLAY`, `LD_LIBRARY_PATH` and
+`LIBGL_ALWAYS_SOFTWARE` set that the Xvfb path needs, the Windows equivalents),
+reach Godot. The environment is **not** inherited wholesale: the game is the
+customer's own code running on the developer's machine, and it has no business
+seeing `ANTHROPIC_API_KEY`, `RAVENSIGHT_TOKEN`, cloud credentials or CI secrets.
+`ENV_ALLOWLIST` is the list, and `extraEnv` is the deliberate escape hatch for a
+caller that needs one more variable through.
+
+The last four are the whole reason a driven run shows up correctly in the
+dashboard: the SDK reads them in `_ready()` and every event the game already
+tracks gets `synthetic: true`, `persona`, `pt_job`, `pt_run` and `pt_tier`
+without the developer writing anything. The driver sets them and otherwise
+stays out of the SDK's way. When the SDK autoload is present the driver also
+connects to its `event_tracked` signal and mirrors each tracked event to the
+runner as a `ravensight_event` notification.
+
+The runner then dials `127.0.0.1`, retrying every 250 ms for up to 60 s, sends
+`hello`, and calls `ping`. Godot's stdout and stderr go to `<run dir>/godot.log`
+and are scanned for `SCRIPT ERROR` / `ERROR:` lines, which `consoleErrors()`
+returns with the `at:` source line attached. That is spec 07's stdout fallback:
+the engine's `Logger` hook is not available on every 4.x version the CLI
+supports, so it is not used.
+
+## The driver interface
+
+`src/run/drivers/driver.js` does not exist on this branch, so `godot.js` writes
+the five method shape out in its own header instead of importing it:
+
+```js
+launch({ recordVideo, viewport })  // -> {}
+observe({ vision })                // -> Observation
+act(action)                        // -> { ok, error?, code?, data?, changed }
+screenshot(name)                   // -> { path } | { path: null, skipped }
+stop({ reason })                   // -> { videoPath, recording, frameCount, chapters, logPath, exitCode }
+```
+
+plus the two optional extras spec 03 lists, `consoleErrors()` and
+`chapter(title)`.
+
+`act` maps the spec 03 action kinds onto driver methods:
+
+| Action | Driver calls |
+|---|---|
+| `key` | `key` press, `wait_frames{1}`, `key` release |
+| `type` | one press and release per character, each carrying its unicode, then Enter when `submit` |
+| `click` with a string target | `click{node_path}` (the center of the Control's global rect) |
+| `click` with coordinates | `click{x,y}` |
+| `press_action` | `press_action{action, duration_ms}` |
+| `wait` | `wait_frames`, milliseconds converted at 60 fps |
+| `set_time_scale` | `set_time_scale` |
+| `quit` | `quit` |
+
+`changed` compares the hash of the scene tree outline before and after the
+action. Two details make that answer honest:
+
+- input reaches the game a frame after it is parsed, and a handler that updates
+  a label runs a frame after that, so `act` waits `settleFrames` (default 2)
+  before taking the comparison snapshot;
+- `observe()` always takes its own `get_state`. An earlier version reused the
+  snapshot `act()` had just taken, to save a round trip, and that handed the
+  persona a screen one action out of date, which is the opposite of what
+  `observe()` is for. Two `get_state` calls a turn is the price of the
+  observation being current.
+
+A driver level error is returned, not thrown: an undefined input action comes
+back as `{ ok: false, code: -32012, data: { available_actions } }` so the
+persona can pick a real one instead of the run ending.
+
+## Observations cite files
+
+Every node in a `get_state` snapshot carries its `path`, and also its `scene`
+(`scene_file_path`) and `script` (the script's `resource_path`) when it has
+them. `buildObservation` collects all of those and puts them at the top of the
+observation text:
+
+```
+scene: res://main.tscn
+scene node: /root/Main
+scripts: res://ravensight_stub.gd, res://main.gd
+
+Window root
+  Node2D Main @(0,0) [groups=ravensight_state] {res://main.tscn} <res://main.gd>
+    CanvasLayer HUD
+      Label TrustLabel "Trust: 4" @(24,16)
+      Button BeginButton "Begin" @(600,400) [focus]
+
+game state:
+  /root/Main: {"phase":"HEALING","trust":4}
+```
+
+That is what lets a finding say which scene and which script to open. The
+driver's own autoload is filtered out of the snapshot, so its script path never
+appears in that list.
+
+Subtrees with nothing a player could see or press are dropped, the outline is
+capped at 12k characters, and `needsVision()` asks for a screenshot when the
+remaining outline is under 200 characters, which is the canvas drawn or 3D case.
+
+Spec 07's `extensions/navigation.gd` (`nav_sample`, `nav_path`, `teleport`,
+`raycast`) is **not** part of this addon: those methods exist for spec 06 level
+probing, not for a persona run, and they can be added later through
+`register_method()` without touching the protocol. `list_actions` is the one
+extension method implemented here, because a persona that gets `-32012` needs to
+know what it may press instead.
+
+The author hook is unchanged from the spec: a node in the `ravensight_state`
+group with a `_ravensight_state() -> Dictionary` method. Vectors become arrays,
+recursion stops at 8 levels, each result is capped at 64 KB, and a hook that
+returns something other than a Dictionary gets `-32020` naming the node path.
+A broken hook is recorded as a console error against the game, not treated as a
+driver failure.
+
+## Headless is for smoke tests, not for playtests
+
+Verified against Godot on macOS: in `--headless` mode the engine routes no
+synthesized mouse or GUI input at all. A `click` on a Button whose rect is right
+there does nothing, and `screenshot` returns `-32010` because there is no
+viewport. The same clicks work windowed.
+
+So `--headless` is only useful for a game driven entirely by InputMap actions
+that manages its own focus. `launch()` pushes a console entry saying so when it
+detects a headless run, and the local CLI default stays windowed.
+
+`--record-video` opts into local capture of this game's viewport; it does not
+consent to upload. `--upload-video` still enables both recording and optional
+video upload. Audio capture is off. Godot samples up to ten frames per second,
+records their actual timestamps, and uses optional local FFmpeg to create
+`session.mp4`. Set `RAVENSIGHT_FFMPEG` to an existing executable when it is not
+on PATH. No package or encoder is downloaded automatically.
+
+Recording is capped at 15 minutes, 1280 by 720 pixels, 256 MiB of temporary
+frames and the existing 200 MiB final video limit. Temporary frames are removed
+after encoding. A missing encoder or headless viewport is reported as
+`unsupported` in local `recording.json`; frames are never labeled as video.
+Cancellation, a game crash, or a recording limit can produce a playable partial
+video, explicitly marked with its stop reason. Encoding failure produces no
+uploadable video. Normal stop and concurrent cancellation share one bounded
+finalization. A hard kill of the CLI or machine cannot guarantee finalization.
+
+`recording.json` stays local and includes timestamps, duration, size, status,
+reason and `audio: false`. When optional FFprobe is available, duration is read
+from the actual silent media stream; otherwise Godot uses frame timestamps and
+Playwright uses capture clock time, labeled by `durationSource`. This metadata
+is not part of the upload allowlist. Video upload remains separately opted in,
+with the existing per-run limits and server retention unchanged.
+
+## Verifying a change
+
+The GDScript is parse checked with a real Godot binary, which is the only thing
+that catches an undeclared identifier:
+
+```bash
+/Users/.../game_tracker/scripts/check-gdscript.sh \
+  /Applications/Godot.app/Contents/MacOS/Godot \
+  addons/ravensight_driver/driver.gd \
+  addons/ravensight_driver/plugin.gd \
+  addons/ravensight_driver/export_plugin.gd
+```
+
+It works on this standalone package: `--check-only --script` needs no
+`project.godot`, and `res://` resolves against the working directory, so
+`plugin.gd`'s `preload` of `export_plugin.gd` is checked too. Run it from the
+package root.
+
+Most Node tests (`npm test`) run against `testing/fake-godot.js`, a fake driver
+socket that speaks the same wire protocol and enforces the same single client,
+hello before anything, and keycode resolution rules. No Godot binary is involved
+there, so those tests prove the Node side's shapes and nothing about the
+GDScript.
+
+`godot-real.test.js` covers what only a real engine can: the activation matrix
+(nothing listens without the flag, a wrong token is refused, a second client is
+refused, an unknown method and a bad param get the right codes), the author hook,
+typing a space, and the export strip. It drives the fixture project at
+`test/fixtures/godot-project/` and **skips itself** when no Godot 4.2+ binary is
+found, the same way the Ravensight repo's ClickHouse integration tests skip
+without a server. A skip there is expected, not a failure to fix, but it does
+mean a change to `driver.gd` is unverified until it has run on a machine with
+Godot installed.
+
+The fixture is never written to: every case copies it to a temp directory first,
+so `--import` and `--export-pack` leave no `.godot/` or `out/` behind.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ravensight-playtest",
-  "version": "0.4.0",
+  "version": "0.5.0",
   "description": "Local AI playtests and seeded game simulations with reports in Ravensight.",
   "license": "MIT",
   "homepage": "https://ravensight.io/docs/#playtest",
@@ -29,6 +29,7 @@
     "addons",
     "docs/ADAPTERS.md",
     "docs/NATIVE_RUNNER.md",
+    "docs/godot-driver.md",
     "docs/SIMULATION.md",
     "examples/simulation",
     "README.md",

package/src/api/client.js

@@ -194,9 +194,9 @@ export function createClient(options = {}) {
      * persona run's price; anything else leaves `run_length` out, which the
      * server reads as a standard run.
      */
-    estimate: (gameId, { modules = [], personas = [], runLength = null, simulation } = {}) =>
+    estimate: (gameId, { modules = [], personas = [], runLength = null, simulation, controllers, driver } = {}) =>
       request('POST', `${gameBase(gameId)}/estimate`, {
-        body: { modules, personas, ...(runLength === 'long' ? { run_length: 'long' } : {}), ...(simulation ? { simulation } : {}) }
+        body: { modules, personas, ...(controllers ? { controllers, driver } : {}), ...(runLength === 'long' ? { run_length: 'long' } : {}), ...(simulation ? { simulation } : {}) }
       }).then(answer => answer.body),
 
     /**

package/src/cli.js

@@ -54,7 +54,9 @@ export const RUNNER_HELP = `
 Run options (parsed by the runner, see src/run/args.js):
   --personas <list>         comma separated persona slugs, default one (curious-kid); "personas" lists them
   --driver <name>           playwright_web, godot_driver, desktop_driver, unity_driver, unreal_driver, android_driver, ios_driver or cli_stdio
-  --controller <name>       standard or jev action selection (default standard)
+  --controller <name>       one controller: standard, jev or rune (if enabled)
+  --compare-with <names>    separate standard + jev and/or rune playthroughs; quotes every run
+  --record-video           record this game locally; --upload-video also uploads it
   --build-url <url>         the web build to play
   --godot-project <dir>     the Godot project directory
   --driver-config <file>    local WebDriver or JSON-RPC bridge configuration