ravensight-playtest 0.2.3 → 0.2.4

package/README.md

@@ -465,3 +465,29 @@ import { createClient, packs, uploadRunDir } from 'ravensight-playtest';
 ## License
 
 MIT. Copyright Reality Software Entertainment.
+
+## Ravensight Playtester action selection
+
+The optional `--controller jev` mode uses the existing persona run. A screenshot
+observer proposes actions, Ravensight selects one on the server, and the local
+driver executes only that choice. Ambiguous controls stop the run for review.
+Reports, screenshots and findings use the normal upload and review flow.
+
+```sh
+ravensight-playtest run --driver playwright_web --controller jev --build-url https://your-game.example/ --max-actions 12 --headed
+```
+
+Sign in, initialize the project and save its expectations brief first. The server
+must advertise this controller in its content pack; otherwise the command stops
+before registering a paid job. No provider key belongs in the CLI. Selection costs
+share the existing persona-run budget and do not create another wallet charge.
+Resume preserves the controller recorded in the job journal.
+
+For Jump Bunny, use `https://reality-software-entertainment.github.io/poc-jump-bunny/`
+and the public player label **Ravensight Playtester**. This is a local browser run,
+not a hosted browser service. Human playtesting is still needed for subjective
+feedback and timing-sensitive gameplay.
+
+## Build adapters
+
+Browser and Godot runners share the paid playtest lifecycle with desktop, Unity/Unreal export, Android/iOS and custom JSON-RPC adapters. Native adapters require a locally configured automation server and remain experimental until validated against your build. See [adapter setup and validation boundaries](docs/ADAPTERS.md).

package/docs/ADAPTERS.md

@@ -0,0 +1,77 @@
+# Connect a game to Ravensight Playtester
+
+Run the CLI on the machine that can access your build. Sign in, link the project with `ravensight-playtest init --game <gameId>`, and complete its expectations brief. `run` quotes the existing persona price before registration. Adapters share the same prepaid job, action limits, model budget, cancellation, reports and uploads.
+
+| Adapter | Target | Validation |
+| --- | --- | --- |
+| `playwright_web` | Browser URL, including canvas/WebGL exports | Automated tests and live Jump Bunny gameplay |
+| `godot_driver` | Godot project with the playtest addon | Existing addon protocol and runner tests |
+| `desktop_driver` | Local W3C/Appium session configuration | Protocol tested; experimental on physical builds |
+| `unity_driver`, `unreal_driver` | Desktop exports through that same W3C interface | Engine-neutral screen controls, not editor plugins; experimental |
+| `android_driver`, `ios_driver` | Appium device/emulator/simulator configuration | Protocol tested; hardware/build validation still required |
+| `cli_stdio` | Developer-owned JSON-RPC bridge | Real child-process protocol test |
+
+No adapter promises every game is autonomously solvable. Fast action games, unusual controls, anti-cheat, hardware access and protected platforms may require a game-specific bridge. Console tooling and authorization remain the developer's responsibility. The runner cannot bypass platform access restrictions.
+
+## Browser and Godot
+
+```sh
+ravensight-playtest run --driver playwright_web --build-url https://your-game.example/ --headed
+ravensight-playtest run --driver godot_driver --godot-project ./your-game
+```
+
+`--controller jev` enables structured action selection when the server supports it. Screen interpretation proposes controls and the selector chooses a supplied action. Uncertain decisions stop for review. Provider credentials stay on Ravensight. A completed playtest report is evidence for human review, not proof that a game is bug-free.
+
+## Desktop and mobile
+
+Install and configure an Appium-compatible driver for your platform. Start its server locally. Supply `ravensight-driver.json` with a loopback endpoint and capabilities appropriate to the installed driver:
+
+```json
+{
+  "endpoint": "http://127.0.0.1:4723",
+  "capabilities": {
+    "platformName": "Android",
+    "appium:automationName": "UiAutomator2",
+    "appium:deviceName": "YOUR_LOCAL_DEVICE",
+    "appium:app": "/absolute/path/to/game.apk"
+  }
+}
+```
+
+```sh
+ravensight-playtest run --driver android_driver --driver-config ./ravensight-driver.json
+```
+
+For iOS, use `ios_driver`, `platformName: iOS`, your XCUITest driver and a provisioned device or simulator. For desktop, use the capabilities required by your installed desktop driver. Unity/Unreal exports use `unity_driver`/`unreal_driver`; no engine source is uploaded by this configuration. Configuration files are local and should not contain provider credentials. Do not expose the automation server publicly.
+
+The runner creates a W3C session, reads page source plus PNG screenshots, and executes bounded coordinate clicks, keys, text and single-pointer swipes. It releases input after each action and closes the session when done. One persona uses the device at a time. Native adapters currently capture screenshots, not video or engine logs. Browser navigation is unavailable on these adapters. Model-visible text is limited to 12,000 characters; responses are bounded to 8 MB and commands time out after 30 seconds.
+
+## Custom bridge
+
+Use this for text games or your own engine/hardware integration. `--driver-config` points to:
+
+```json
+{
+  "command": "/absolute/path/to/bridge-executable",
+  "args": ["--game", "/absolute/path/to/build"],
+  "cwd": "/absolute/path/to/project"
+}
+```
+
+The configured executable is trusted local code chosen by the developer. The model cannot change it. It runs without a shell. Standard output must contain newline-delimited JSON-RPC 2.0 responses. Put diagnostics on stderr. Respond with the request ID and `result`, or an `error`:
+
+```json
+{"jsonrpc":"2.0","id":1,"method":"launch","params":{"protocolVersion":1}}
+{"jsonrpc":"2.0","id":1,"result":{"ok":true}}
+```
+
+Required methods:
+
+- `launch`: initialize/reset the game; return `{ "ok": true }`.
+- `observe`: return `{ "text": "visible state" }`, optionally `screenshotBase64` containing PNG data.
+- `act`: accept a `kind` of `key`, `type`, `click` or `wait`; return `{ "ok": true }` after execution. Apply your own game-specific legality checks. Never interpret model text as shell code.
+- `screenshot`: return `{ "pngBase64": "..." }`, or a JSON-RPC error if unavailable. Text-only games can still produce text reports without screenshots.
+
+Requests expire after 30 seconds. Oversized or malformed responses stop the bridge. The runner terminates its child when the session ends; the bridge must clean up any game processes it owns on termination. No custom protocol can establish compatibility with untested hardware by itself.
+
+Protocol references: [W3C WebDriver](https://www.w3.org/TR/webdriver/), [Appium capabilities](https://appium.io/docs/en/3.2/guides/caps/).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ravensight-playtest",
-  "version": "0.2.3",
+  "version": "0.2.4",
   "description": "AI personas play your game and file reports to Ravensight.",
   "license": "MIT",
   "homepage": "https://ravensight.io/docs/#playtest",
@@ -25,6 +25,7 @@
     "bin",
     "src",
     "addons",
+    "docs/ADAPTERS.md",
     "README.md",
     "LICENSE",
     "!src/**/*.test.js",

package/src/cli.js

@@ -53,10 +53,11 @@ async function loadRunner(name, exportName) {
 const RUNNER_HELP = `
 Run options (parsed by the runner, see src/run/args.js):
   --personas <list>         comma separated persona slugs, default the pack's set
-  --driver <name>           playwright_web or godot_driver
+  --driver <name>           playwright_web, godot_driver, desktop_driver, unity_driver, unreal_driver, android_driver, ios_driver or cli_stdio
+  --controller <name>       standard or jev action selection (default standard)
   --build-url <url>         the web build to play
   --godot-project <dir>     the Godot project directory
-  --cli <command>           the command that starts a text build
+  --driver-config <file>    local WebDriver or JSON-RPC bridge configuration
   --max-actions <n>         cap the actions one persona may take
   --concurrency <n>         personas at once, default 2
   --upload-video            upload session.webm as well as the reports

package/src/commands/run.js

@@ -10,6 +10,8 @@ import { createHeartbeat } from '../run/heartbeat.js';
 import { createUsageLedger } from '../run/usage.js';
 import { createModelClient, ModelUnavailableError, ProxyError } from '../run/model.js';
 import { jobDir, runDir as runDirFor, JOB_FILES, RUN_FILES } from '../run/paths.js';
+import { readBridgeConfig } from "../run/drivers/stdio.js";
+import { WEBDRIVER_NAMES, readDriverConfig } from '../run/drivers/webdriver.js';
 import { runPersona } from '../run/personaLoop.js';
 import * as stateFile from '../run/state.js';
 import { emitRunEvents } from '../run/synthetic.js';
@@ -86,10 +88,11 @@ async function run(flags, injected, deps, log) {
   if (setup.code !== undefined) return setup.code;
   const { context, gameId, repoRoot, apiUrl, cliVersion, api } = setup;
 
+  if (options.controller && !['standard', 'jev'].includes(options.controller)) { log('Choose --controller standard or --controller jev.'); return EXIT.ENVIRONMENT; }
   const driverName = options.driver || (context.config && context.config.driver) || 'playwright_web';
   const target = resolveTarget(options, context.config, driverName);
   if (!target) {
-    log('Point the run at a build: --build-url for a web build, or --godot-project for a Godot project.');
+    log('Point the run at a build: --build-url for a web build, --godot-project for Godot, or --driver-config for a native or text bridge.');
     return EXIT.ENVIRONMENT;
   }
 
@@ -97,8 +100,11 @@ async function run(flags, injected, deps, log) {
   // `aggregate.*` steps are spendable from, and it is free (the price list has
   // no unit for it), so asking costs nothing and not asking costs the judged
   // dedup and success-criteria passes.
+  if (WEBDRIVER_NAMES.includes(driverName)) await readDriverConfig(target);
+  if (driverName === "cli_stdio") await readBridgeConfig(target);
   const modules = ['persona_playtest', 'aggregate_report'];
   const pack = await deps.packs.get({ api, gameId, modules });
+  if (options.controller === 'jev' && !pack.routing?.action_controllers?.includes('jev')) { log('This server does not currently support the requested action controller. No job was registered.'); return EXIT.ENVIRONMENT; }
   const personaSlugs = resolvePersonas(options, context.config, pack);
   const personas = personaSlugs.map(slug => {
     const found = (pack.personas || []).find(persona => persona.slug === slug);
@@ -164,7 +170,7 @@ async function run(flags, injected, deps, log) {
       cli_version: cliVersion,
       confirm_price_cents: estimate.estimate_cents,
       brief_version: brief.version ?? null,
-      build: { kind: driverName === 'playwright_web' ? 'url' : 'godot_project', ref: target },
+      build: { kind: driverName === 'playwright_web' ? 'url' : (WEBDRIVER_NAMES.includes(driverName) || driverName === 'cli_stdio') ? 'driver_config' : 'godot_project', ref: target },
       repo: { commit: (context.config && context.config.commit) || '', dirty: Boolean(context.config && context.config.dirty) },
       max_actions: Number.isInteger(options.maxActions) ? options.maxActions : null,
       pack_versions: { pack: pack.pack_version }
@@ -191,6 +197,7 @@ async function run(flags, injected, deps, log) {
     priceCents: registered.price && registered.price.price_cents,
     options: {
       maxActions: options.maxActions ?? null,
+      controller: options.controller || 'standard',
       concurrency: options.concurrency ?? DEFAULT_CONCURRENCY,
       uploadVideo: Boolean(options.uploadVideo),
       uploadTranscript: Boolean(options.uploadTranscript)
@@ -349,7 +356,7 @@ export async function playJob({
 
   const results = [];
   const queue = personaRuns.filter(entry => !stateFile.isRunDone(journal.runs[entry.run_id]));
-  const concurrency = Math.max(1, Math.min(Number(options.concurrency) || DEFAULT_CONCURRENCY, queue.length || 1));
+  const concurrency = WEBDRIVER_NAMES.includes(driverName) ? 1 : Math.max(1, Math.min(Number(options.concurrency) || DEFAULT_CONCURRENCY, queue.length || 1));
 
   /**
    * Runs the process is in the middle of. A Ctrl-C has to move these to
@@ -516,6 +523,7 @@ export async function playPersonaRun({
   try {
     if (inFlight) inFlight.set(runId, { state: 'launching' });
     outcome = await runPersona({
+      controller: options.controller || 'standard',
       client,
       api,
       gameId,
@@ -985,6 +993,7 @@ export function resolveMaxActions(options, persona, pack) {
 }
 
 export function resolveTarget(options, config, driverName) {
+  if (WEBDRIVER_NAMES.includes(driverName) || driverName === 'cli_stdio') return options.driverConfig || (config?.build?.kind === 'driver_config' ? config.build.ref : null);
   const build = (config && config.build) || {};
   if (driverName === 'playwright_web') return options.buildUrl || (build.kind === 'url' ? build.ref : null) || null;
   if (driverName === 'godot_driver') return options.godotProject || (build.kind === 'godot_project' ? build.ref : null) || null;

package/src/run/actionSelector.js

@@ -0,0 +1,47 @@
+import { ProxyError, STEPS } from './model.js';
+export const CONTROLLERS = ['standard', 'jev'];
+export const JEV_CONTROLLER_PROMPT = '\nRavensight Playtester action-selection mode: describe the currently visible screen briefly in text, including each proposed target and its screenshot coordinates, then propose one to three ALTERNATIVE next actions as tool calls. Only one will be executed after a separate selector chooses. These are alternatives, never a sequential plan. Use only observed controls. For a canvas game without accessible DOM controls, include a target label describing the visible control AND screenshot x/y coordinates; the driver uses those coordinates. Never invent selectors such as textbox or button names. When the brief identifies a field as a public player label, use Ravensight Playtester. Credentials or private access codes require human review. When the game is loading or transitioning, propose a bounded wait tool call. Do not place secrets in proposals. After their tool results, observe the resulting state before proposing again.';
+export function describeAction(tool) {
+  const input = tool.input || {};
+  const descriptions = {
+    click: input.target ? `Click the visible ${input.target}${Number.isFinite(input.x) && Number.isFinite(input.y) ? ` at (${input.x}, ${input.y})` : ''}.` : `Click the observed screenshot position (${input.x}, ${input.y}).`,
+    type: `Type the prepared text into the focused field${input.submit ? ' and press Enter' : ''}.`,
+    key: `Press the ${input.key} key.`,
+    wait: `Wait ${input.ms} milliseconds for the screen to finish transitioning.`,
+    snapshot: 'Observe the current screen again.',
+    screenshot: 'Save a screenshot of the current screen for the report.',
+    navigate: `Navigate to the supplied build URL ${input.url}.`
+  };
+  return String(descriptions[tool.name] || `${tool.name}: ${JSON.stringify(input)}`).slice(0, 300);
+}
+export async function selectAction({ api, jobId, runId, persona, observation, toolUses, recentActions = [], intentNotes = [] }) {
+  const proposals = toolUses.slice(0, 8);
+  // Observation does not input anything into the game. Keep it in the existing
+  // bounded driver loop; Jev selects game inputs, not whether a frame may load.
+  if (proposals.length && proposals.every(tool => ['wait', 'snapshot', 'screenshot'].includes(tool.name))) {
+    return { status: 'selected', toolId: proposals[0].id, source: 'observation' };
+  }
+  const body = {
+    objective: ('Play the game according to the supplied persona, exploring visible controls and gathering evidence. Developer scope: ' + intentNotes.map(note => note.text || '').join(' ')).slice(0, 1000),
+    // The observer already applies the full persona. The selector needs its identity,
+    // current visible state and proposed inputs, not a second personality prompt.
+    persona: String(persona.name || persona.slug || 'Playtester').slice(0, 200),
+    observation: String(observation).slice(0, 6000),
+    actions: proposals.map((tool, index) => ({ id: `action_${index}`, description: describeAction(tool) })),
+    recentActions: recentActions.slice(-10).map(value => String(value).slice(0, 100))
+  };
+  let result;
+  try {
+    result = (await api.request('POST', '/api/v1/playtest/model/v1/decisions', {
+      body, idempotent: false,
+      headers: { 'X-Playtest-Job': jobId, 'X-Playtest-Run': runId, 'X-Playtest-Step': STEPS.personaAct }
+    })).body;
+  } catch (error) {
+    if (error.status === 402 || error.status === 409) throw new ProxyError({ status: error.status, code: error.code || 'decision_refused', message: 'The action selector stopped because this run is no longer spendable.' });
+    return { status: 'review', reason: 'decision_unavailable' };
+  }
+  if (result.status !== 'suggested') return result;
+  const index = body.actions.findIndex(action => action.id === result.actionId);
+  if (index < 0 || !Number.isFinite(result.confidence) || result.confidence < 0.75) return { status: 'review', reason: result.actionId === 'stop' ? 'selector_stopped' : 'uncertain_action' };
+  return { status: 'selected', toolId: proposals[index].id, confidence: result.confidence, budget: result.budget };
+}

package/src/run/args.js

@@ -14,6 +14,8 @@
 const VALUED = new Map([
   ['--personas', 'personas'],
   ['--driver', 'driver'],
+  ['--driver-config', 'driverConfig'],
+  ['--controller', 'controller'],
   ['--build-url', 'buildUrl'],
   ['--build-dir', 'buildDir'],
   ['--godot-project', 'godotProject'],

package/src/run/drivers/driver.js

@@ -1,9 +1,9 @@
 import { createHash } from 'node:crypto';
 
 /**
- * The driver interface. One shape, three implementations (spec 03's
+ * The driver interface. One shared shape across implementations (spec 03's
  * "Driver interface (all drivers)"): `playwright_web` here in ./web.js,
- * `godot_driver` in ./godot.js, and `cli_stdio` later.
+ * `godot_driver` in ./godot.js, plus native and custom bridge adapters.
  *
  * A driver is the ONLY thing in this CLI that touches the game. The model
  * never gets a shell, a file write outside the run directory, or a network
@@ -138,7 +138,10 @@ export function assertDriver(driver, name) {
 export function observationHash(observation) {
   const text = typeof observation === 'string' ? observation : String((observation && observation.text) || '');
   const normalized = text.replace(/\s+/g, ' ').trim();
-  return createHash('sha256').update(normalized, 'utf8').digest('hex').slice(0, 32);
+  const hash = createHash('sha256').update(normalized, 'utf8');
+  // Canvas games keep the same DOM text while their visible state changes.
+  if (observation && observation.screenshotBase64) hash.update(observation.screenshotBase64, 'base64');
+  return hash.digest('hex').slice(0, 32);
 }
 
 /**

package/src/run/drivers/index.js

@@ -1,16 +1,18 @@
 import { assertDriver, DriverError } from './driver.js';
 import { createWebDriver } from './web.js';
+import { createWebDriverAdapter, WEBDRIVER_NAMES } from './webdriver.js';
+import { createStdioDriver } from './stdio.js';
 
 /**
  * The driver registry. `playwright_web` is here; `godot_driver` lives in
  * ./godot.js and is imported lazily so this package still loads on a
- * machine with no Godot and no addon, and `cli_stdio` is not in v1.
+ * machine with no Godot and no addon, and native/custom adapters use local developer-owned connections.
  *
  * The names are the server's `PlaytestRun.driver` enum and the report
  * schema's `driver` enum, not ours to invent.
  */
 
-export const DRIVERS = Object.freeze(['playwright_web', 'godot_driver', 'cli_stdio']);
+export const DRIVERS = Object.freeze(['playwright_web', 'godot_driver', 'cli_stdio', ...WEBDRIVER_NAMES]);
 
 /**
  * @param {string} name
@@ -19,6 +21,7 @@ export const DRIVERS = Object.freeze(['playwright_web', 'godot_driver', 'cli_std
  * @returns {Promise<Object>} a driver satisfying ./driver.js
  */
 export async function createDriver(name, options = {}) {
+  if (WEBDRIVER_NAMES.includes(name)) return assertDriver(createWebDriverAdapter({...options,name}),name);
   const load = options.importer || (specifier => import(specifier));
   switch (name) {
     case 'playwright_web':
@@ -40,10 +43,7 @@ export async function createDriver(name, options = {}) {
       return assertDriver(await factory(options), name);
     }
     case 'cli_stdio':
-      throw new DriverError(
-        'The cli_stdio driver is not in this release. In v1 a text build is played through its web front end with --driver playwright_web.',
-        { fatal: true }
-      );
+      return assertDriver(createStdioDriver(options), name);
     default:
       throw new DriverError(`unknown driver "${name}"; one of: ${DRIVERS.join(', ')}`, { fatal: true });
   }

package/src/run/drivers/stdio.js

@@ -0,0 +1,111 @@
+import fs from 'node:fs/promises';
+import path from 'node:path';
+import { spawn } from 'node:child_process';
+import { DriverError, observationHash, screenshotName, truncateObservation } from './driver.js';
+
+// A developer-owned bridge implements newline-delimited JSON-RPC. The model
+// receives game controls only, never command arguments or process access.
+export async function readBridgeConfig(file) {
+  if ((await fs.stat(file)).size > 64000) throw new DriverError('Bridge configuration exceeds 64 KB.');
+  const config = JSON.parse(await fs.readFile(file, 'utf8'));
+  if (typeof config.command !== 'string' || !path.isAbsolute(config.command) ||
+      (config.args !== undefined && (!Array.isArray(config.args) || config.args.some(v => typeof v !== 'string'))) ||
+      (config.cwd !== undefined && !path.isAbsolute(config.cwd))) {
+    throw new DriverError('Bridge config needs an absolute command, string arguments and an optional absolute cwd.');
+  }
+  await fs.access(config.command, fs.constants.X_OK);
+  return config;
+}
+
+export function createStdioDriver({ spawn: spawnImpl = spawn, timeoutMs = 30000 } = {}) {
+  let child, buffer = '', nextId = 0, runDirectory, step = 0, shots = 0, lastHash;
+  const pending = new Map();
+  const fail = message => {
+    for (const entry of pending.values()) { clearTimeout(entry.timer); entry.reject(new DriverError(message, { fatal: true })); }
+    pending.clear();
+  };
+  function request(method, params = {}) {
+    if (!child || child.exitCode !== null || child.killed) return Promise.reject(new DriverError('Game bridge is not running.', { fatal: true }));
+    return new Promise((resolve, reject) => {
+      const id = ++nextId;
+      const timer = setTimeout(() => { pending.delete(id); reject(new DriverError('Game bridge timed out.', { fatal: true })); child.kill(); }, timeoutMs);
+      pending.set(id, { resolve, reject, timer });
+      child.stdin.write(JSON.stringify({ jsonrpc: '2.0', id, method, params }) + '\n', error => {
+        if (error) { clearTimeout(timer); pending.delete(id); reject(new DriverError('Game bridge input failed.')); }
+      });
+    });
+  }
+  async function observe() {
+    const value = await request('observe');
+    if (!value || typeof value.text !== 'string') throw new DriverError('Bridge observation needs text.');
+    const observation = { step: ++step, text: truncateObservation(value.text), elapsedMs: 0 };
+    if (typeof value.screenshotBase64 === 'string' && value.screenshotBase64.startsWith('iVBORw0KGgo')) {
+      observation.screenshotBase64 = value.screenshotBase64;
+      observation.screenshotMediaType = 'image/png';
+    }
+    lastHash = observationHash(observation);
+    return observation;
+  }
+  return {
+    name: 'cli_stdio', supportedActions: ['type', 'key', 'click', 'wait'],
+    async launch({ target, runDir }) {
+      const config = await readBridgeConfig(target);
+      runDirectory = runDir;
+      child = spawnImpl(config.command, config.args || [], { cwd: config.cwd || path.dirname(path.resolve(target)), shell: false, stdio: ['pipe', 'pipe', 'pipe'] });
+      child.stdout.setEncoding('utf8');
+      child.stdout.on('data', chunk => {
+        buffer += chunk;
+        if (Buffer.byteLength(buffer) > 8 * 1024 * 1024) { fail('Game bridge response exceeded 8 MB.'); child.kill(); return; }
+        let end;
+        while ((end = buffer.indexOf('\n')) !== -1) {
+          const line = buffer.slice(0, end); buffer = buffer.slice(end + 1);
+          let value;
+          try { value = JSON.parse(line); } catch { fail('Game bridge returned invalid JSON.'); child.kill(); return; }
+          const entry = pending.get(value?.id);
+          if (!entry) continue;
+          pending.delete(value.id); clearTimeout(entry.timer);
+          if (value.jsonrpc !== '2.0' || value.error) entry.reject(new DriverError('Game bridge command failed.'));
+          else entry.resolve(value.result);
+        }
+      });
+      // Drain diagnostics without copying arbitrary game output into reports.
+      child.stderr.resume();
+      child.stdin.on('error', () => fail('Game bridge input failed.'));
+      child.on('error', () => fail('Game bridge could not start.'));
+      child.on('exit', () => fail('Game bridge exited.'));
+      await fs.mkdir(path.join(runDir, 'screenshots'), { recursive: true });
+      await request('launch', { protocolVersion: 1 });
+    },
+    observe,
+    async act(action) {
+      if (!['type', 'key', 'click', 'wait'].includes(action.kind)) return { ok: false, changed: false, error: 'Unsupported bridge action.' };
+      const before = lastHash;
+      const safe = action.kind === 'type' ? { kind: 'type', text: String(action.text || '').slice(0, 2000), submit: Boolean(action.submit) }
+        : action.kind === 'key' ? { kind: 'key', key: String(action.key || '').slice(0, 64) }
+        : action.kind === 'wait' ? { kind: 'wait', ms: Math.min(5000, Math.max(0, Number(action.ms) || 0)) }
+        : { kind: 'click', target: typeof action.target === 'string' ? action.target.slice(0, 500) : action.target };
+      const answer = await request('act', safe);
+      if (answer?.ok !== true) return { ok: false, changed: false, error: 'Game bridge refused this action.' };
+      return { ok: true, changed: observationHash(await observe()) !== before };
+    },
+    async screenshot(reason) {
+      const image = await request('screenshot');
+      if (typeof image?.pngBase64 !== 'string' || !image.pngBase64.startsWith('iVBORw0KGgo')) throw new DriverError('This bridge did not return a PNG screenshot.');
+      const relativePath = path.join('screenshots', screenshotName(++shots, reason));
+      const file = path.join(runDirectory, relativePath);
+      await fs.writeFile(file, Buffer.from(image.pngBase64, 'base64'));
+      return { path: file, relativePath };
+    },
+    async consoleErrors() { return []; },
+    async stop() {
+      if (child && child.exitCode === null && !child.killed) {
+        child.kill('SIGTERM');
+        const process = child;
+        const timer = setTimeout(() => { if (process.exitCode === null) process.kill('SIGKILL'); }, 1000);
+        timer.unref();
+      }
+      fail('Game bridge stopped.');
+      return { videoPath: null };
+    }
+  };
+}

package/src/run/drivers/web.js

@@ -293,7 +293,13 @@ export function createWebDriver({ importer, headless = true } = {}) {
       try {
         // The same truncation observe() applies, so the two hashes are
         // comparable. Without it a long screen is always "changed".
-        changed = observationHash(truncateObservation(await snapshotText(), OBSERVATION_MAX_CHARS)) !== before;
+        const text = truncateObservation(await snapshotText(), OBSERVATION_MAX_CHARS);
+        const after = { text };
+        if (text.trim().length < THIN_OBSERVATION_CHARS) {
+          const frame = await current.screenshot({ type: 'png', timeout: ACTION_TIMEOUT_MS });
+          after.screenshotBase64 = frame.toString('base64');
+        }
+        changed = observationHash(after) !== before;
       } catch {
         changed = true;
       }

package/src/run/drivers/webdriver.js

@@ -0,0 +1,67 @@
+// Native desktop and mobile adapters use the W3C session/actions protocol.
+// The developer supplies a local Appium/WebDriver server and a reviewed config;
+// game observations cannot choose servers, capabilities, executables or scripts.
+import fs from 'node:fs/promises';
+import path from 'node:path';
+import { DriverError, observationHash, screenshotName, truncateObservation } from './driver.js';
+
+export const WEBDRIVER_NAMES = ['desktop_driver', 'unity_driver', 'unreal_driver', 'android_driver', 'ios_driver'];
+const KEYS = { Enter:'\uE007', Escape:'\uE00C', Space:' ', Tab:'\uE004', Backspace:'\uE003', ArrowUp:'\uE013', ArrowDown:'\uE015', ArrowLeft:'\uE012', ArrowRight:'\uE014' };
+const MAX_BYTES = 8 * 1024 * 1024;
+export async function readDriverConfig(file) {
+  const stat=await fs.stat(file); if(stat.size>64000) throw new DriverError('Driver configuration exceeds 64 KB.',{fatal:true});
+  const config=JSON.parse(await fs.readFile(file,'utf8'));
+  const url=new URL(config.endpoint);
+  if(!['http:','https:'].includes(url.protocol)||!['localhost','127.0.0.1','[::1]'].includes(url.hostname)||url.username||url.password||url.search||url.hash) throw new DriverError('The device server must use a loopback URL without credentials.',{fatal:true});
+  if(!config.capabilities||Array.isArray(config.capabilities)||typeof config.capabilities!=='object'||typeof config.capabilities.platformName!=='string') throw new DriverError('Supply W3C capabilities with platformName.',{fatal:true});
+  return {...config,endpoint:url.href.replace(/\/$/,'')};
+}
+export function createWebDriverAdapter({name='desktop_driver',fetch:fetchImpl=globalThis.fetch,sleep=ms=>new Promise(resolve=>setTimeout(resolve,ms))}={}) {
+ let config,session,runDirectory,step=0,shots=0,lastHash=null;
+ async function request(method,route,body) {
+  const response=await fetchImpl(config.endpoint+route,{method,redirect:'error',signal:AbortSignal.timeout(30000),headers:{'Content-Type':'application/json'},...(body?{body:JSON.stringify(body)}:{})});
+  if(response.status===204) return null;
+  if(!response.body) throw new DriverError('Device response was empty.');
+  const reader=response.body.getReader();const chunks=[];let size=0;
+  try {for(;;){const part=await reader.read();if(part.done)break;size+=part.value.length;if(size>MAX_BYTES){await reader.cancel();throw new DriverError('Device response exceeded the limit.');}chunks.push(Buffer.from(part.value));}}finally{reader.releaseLock();}
+  const data=JSON.parse(Buffer.concat(chunks).toString());
+  if(!response.ok||data.value?.error)throw new DriverError('Device command failed: '+String(data.value?.error||response.status));
+  return data.value;
+ }
+ const route=tail=>{if(!session)throw new DriverError('No device session is active.',{fatal:true});return '/session/'+encodeURIComponent(session)+tail;};
+ async function frame(){const data=await request('GET',route('/screenshot'));if(typeof data!=='string'||!data.startsWith('iVBORw0KGgo'))throw new DriverError('Device screenshot is not a PNG.');return data;}
+ async function observe(){const [source,image]=await Promise.all([request('GET',route('/source')),frame()]);const observation={step:++step,text:truncateObservation(typeof source==='string'?source:JSON.stringify(source)),screenshotBase64:image,screenshotMediaType:'image/png',elapsedMs:0};lastHash=observationHash(observation);return observation;}
+ async function actions(type,values,pointerType='mouse'){await request('POST',route('/actions'),{actions:[{type,id:'ravensight',...(type==='pointer'?{parameters:{pointerType}}:{}),actions:values}]});}
+ const driver={name,supportedActions:['click','type','key','wait','swipe'],
+  async launch({target,runDir}){
+   config=await readDriverConfig(target);runDirectory=runDir;
+   const platform=config.capabilities.platformName.toLowerCase();
+   if(name==='android_driver'&&platform!=='android'||name==='ios_driver'&&platform!=='ios')throw new DriverError('The device platform does not match the selected adapter.',{fatal:true});
+   const answer=await request('POST','/session',{capabilities:{alwaysMatch:config.capabilities,firstMatch:[{}]}});
+   if(!answer||typeof answer.sessionId!=='string')throw new DriverError('The device server did not create a session.',{fatal:true});
+   session=answer.sessionId;await fs.mkdir(path.join(runDir,'screenshots'),{recursive:true});
+  },observe,
+  async act(action){
+   const before=lastHash;const touch=['android_driver','ios_driver'].includes(name)?'touch':'mouse';
+   const point=value=>{if(!value||!Number.isFinite(value.x)||!Number.isFinite(value.y)||value.x<0||value.y<0||value.x>32768||value.y>32768)throw new DriverError('Supply bounded screenshot coordinates.');return {x:Math.round(value.x),y:Math.round(value.y)};};
+   try {
+    if(action.kind==='wait')await sleep(Math.min(5000,Math.max(0,Number(action.ms)||0)));
+    else if(action.kind==='click'){
+     const p=point(action.target);await actions('pointer',[{type:'pointerMove',duration:0,origin:'viewport',...p},{type:'pointerDown',button:0},{type:'pointerUp',button:0}],touch);
+    }else if(action.kind==='swipe'){
+     const from=point(action.from),to=point(action.to);await actions('pointer',[{type:'pointerMove',duration:0,origin:'viewport',...from},{type:'pointerDown',button:0},{type:'pointerMove',duration:Math.min(2000,Math.max(100,Number(action.durationMs)||500)),origin:'viewport',...to},{type:'pointerUp',button:0}],touch);
+    }else if(action.kind==='key'||action.kind==='type'){
+     const chars=action.kind==='type'?Array.from(String(action.text||'').slice(0,2000)):[KEYS[action.key]||action.key];
+     if(chars.some(key=>typeof key!=='string'||Array.from(key).length!==1))throw new DriverError('Unsupported device key.');
+     if(action.submit)chars.push(KEYS.Enter);
+     await actions('key',chars.flatMap(value=>[{type:'keyDown',value},{type:'pause',duration:Math.min(2000,Math.max(0,Number(action.durationMs)||0))},{type:'keyUp',value}]));
+    }else throw new DriverError('This device adapter does not support that action.');
+    return {ok:true,changed:observationHash(await observe())!==before};
+   }catch(error){return {ok:false,changed:false,error:error instanceof DriverError?error.message:'Device input failed.'};}
+   finally {if(session)await request('DELETE',route('/actions')).catch(()=>{});}
+  },
+  async screenshot(reason){const relativePath=path.join('screenshots',screenshotName(++shots,reason));const file=path.join(runDirectory,relativePath);await fs.writeFile(file,Buffer.from(await frame(),'base64'));return {path:file,relativePath,reason};},
+  async consoleErrors(){return [];},
+  async stop(){if(session){const current=route('');session=null;await request('DELETE',current);}return {videoPath:null};}
+ };return driver;
+}

package/src/run/fixLoop.js

@@ -51,6 +51,55 @@ export const PLAN_MAX_TURNS = 6;
  */
 export const NUDGE_EVERY = 8;
 
+/** One breakpoint, frozen so no caller can hand a mutated one to the next turn. */
+const CACHE_BREAKPOINT = Object.freeze({ type: 'ephemeral' });
+
+/**
+ * Move a cache breakpoint onto the newest turn of a conversation.
+ *
+ * `fixSystem` already marks the end of the system prompt, but a breakpoint
+ * there only ever caches the skill and the repository facts. The tokens on
+ * this path are in `messages`: a `read_file` result is a whole source file,
+ * and the apply loop accumulates dozens of them over up to sixty turns, every
+ * one of which re-sends the whole conversation. Without a breakpoint in
+ * `messages` that conversation is processed at full price on every turn, which
+ * is quadratic in the number of turns and is where `fix.apply`'s bill comes
+ * from.
+ *
+ * The prefix written on turn N is exactly the prefix read on turn N+1, because
+ * these two loops only ever APPEND to `messages`. Nothing rewrites an earlier
+ * turn, so the bytes ahead of the marker are identical from one call to the
+ * next. (`personaLoop` deliberately does not do this: `prepareMessages`
+ * rewrites older observations into stubs as a run goes on, so its prefix is not
+ * stable and a breakpoint in its messages would pay the write premium instead
+ * of earning a read.)
+ *
+ * Applied to a COPY, leaving the stored conversation unmarked, for two
+ * reasons. A request may carry at most four breakpoints, so a marker left
+ * behind on each turn would be refused by the fifth. And the bytes this call
+ * sends for an earlier turn have to be the bytes the last call sent, which a
+ * marker written into the history would change. A marker that moves is not an
+ * invalidator; one that accumulates is.
+ *
+ * No breakpoint can leak anything across runs or across customers. A cache
+ * read requires the caller to present the identical prefix bytes, so the only
+ * thing an entry can ever serve back is a prompt whoever asked for it already
+ * held. The proxy passes `messages` through by reference, so nothing about
+ * this needs the server's field allowlist widened.
+ *
+ * @param {Array<Object>} messages the conversation so far, unmarked
+ * @returns {Array<Object>} the conversation to send, with one moving marker
+ */
+export function withRollingCacheBreakpoint(messages) {
+  const last = messages.at(-1);
+  if (!last || !Array.isArray(last.content) || last.content.length === 0) return messages;
+  const block = last.content.at(-1);
+  if (!block || typeof block !== 'object') return messages;
+  const content = [...last.content];
+  content[content.length - 1] = { ...block, cache_control: CACHE_BREAKPOINT };
+  return [...messages.slice(0, -1), { ...last, content }];
+}
+
 /** The text of a tool result, whichever of the two content shapes it uses. */
 function resultText(result) {
   if (!result) return '';
@@ -213,7 +262,7 @@ export async function planFix({
       runId,
       step: STEPS.fixPlan,
       system,
-      messages,
+      messages: withRollingCacheBreakpoint(messages),
       tools: repo ? repo.tools : undefined,
       maxTokens: PLAN_MAX_TOKENS
     });
@@ -337,7 +386,7 @@ export async function applyFix({
         runId,
         step: STEPS.fixApply,
         system,
-        messages,
+        messages: withRollingCacheBreakpoint(messages),
         tools: repo.tools,
         maxTokens: APPLY_MAX_TOKENS
       });
@@ -486,4 +535,4 @@ export function clamp(value, limit) {
   return text.length <= limit ? text : text.slice(0, limit);
 }
 
-export default { planFix, applyFix, summariseFix, resolveFixMaxActions, fixSystem, planContent, normalisePlan };
+export default { planFix, applyFix, summariseFix, resolveFixMaxActions, fixSystem, planContent, normalisePlan, withRollingCacheBreakpoint };

package/src/run/personaLoop.js

@@ -1,3 +1,4 @@
+import { selectAction, JEV_CONTROLLER_PROMPT } from './actionSelector.js';
 import { mkdir, writeFile } from 'node:fs/promises';
 import path from 'node:path';
 import { prepareMessages, MAX_IMAGES_IN_CONTEXT, KEEP_OBSERVATIONS, shrinkImages } from './context.js';
@@ -6,7 +7,7 @@ import { callStepWithRetry, ProxyError, STEPS } from './model.js';
 import { RUN_FILES } from './paths.js';
 import { checkEvidence, createReportValidator, lintMarkdownSections, repairInstruction, REQUIRED_MARKDOWN_SECTIONS, stampReport, writeReportFiles } from './report.js';
 import { assertNoSecrets } from './secretScan.js';
-import { ACTION_TOOLS, PLAY_TOOLS, REPORT_TOOLS, executeTool } from './tools.js';
+import { ACTION_TOOLS, PLAY_TOOLS, SWIPE_TOOL, REPORT_TOOLS, executeTool } from './tools.js';
 import { createTranscript } from './transcript.js';
 import { createUsageLedger, writeUsageFile } from './usage.js';
 
@@ -114,6 +115,7 @@ export async function runPersona(args) {
     job_id: jobId,
     persona: persona.slug,
     driver: driverName,
+    controller: args.controller || 'standard',
     target,
     prompt_version: promptVersionOf(skill),
     brief_id: brief ? brief.brief_id || null : null,
@@ -248,6 +250,8 @@ export async function runPersona(args) {
  * @returns {Promise<Object>}
  */
 async function playLoop({
+  api,
+  controller = 'standard',
   client,
   jobId,
   runId,
@@ -270,6 +274,25 @@ async function playLoop({
   log = () => {}
 }) {
   const system = buildSystem({ skill, persona, brief, intentNotes, driverName, target, maxActions });
+  if (controller === 'jev') system.push({ type: 'text', text: JEV_CONTROLLER_PROMPT });
+  const availableTools = driver.supportedActions
+    ? [...PLAY_TOOLS, ...(driver.supportedActions.includes('swipe') ? [SWIPE_TOOL] : [])].filter(tool => !ACTION_TOOLS.includes(tool.name) || driver.supportedActions.includes(tool.name))
+    : PLAY_TOOLS;
+  const playTools = controller === 'jev' ? availableTools.map(tool => ({
+    ...tool,
+    input_schema: {
+      ...tool.input_schema,
+      required: [...(tool.input_schema.required || []), 'visible_state', ...(tool.name === 'click' ? ['target'] : [])],
+      properties: {
+        ...tool.input_schema.properties,
+        visible_state: { type: 'string', description: 'Describe what is visibly on the current screenshot, including this action target and its coordinates. This observation is required even when the assistant text is empty. Do not claim unobserved state.' },
+        ...(tool.name === 'click' ? { target: { type: 'string', description: 'Label of the visible control. For canvas also supply x and y; the driver uses those coordinates.' } } : {})
+      }
+    }
+  })) : availableTools;
+  const recentActions = [];
+  let latestObservation = '';
+  let visualObservation = false;
   const messages = [];
   let actions = 0;
   let turns = 0;
@@ -283,6 +306,8 @@ async function playLoop({
   const maxTurns = maxActions * 3 + 20;
 
   const first = await driver.observe();
+  latestObservation = first.text;
+  visualObservation = Boolean(first.screenshotBase64);
   transcript.write({ kind: 'observation', step: first.step, text: first.text, url: first.url });
   const landing = await safeScreenshot(driver, 'landing', screenshots, transcript);
   messages.push({
@@ -317,7 +342,7 @@ async function playLoop({
         step: STEPS.personaAct,
         system,
         messages,
-        tools: PLAY_TOOLS,
+        tools: playTools,
         maxTokens: ACT_MAX_TOKENS,
         escalate: shouldEscalate(persona),
         imageBudget,
@@ -370,9 +395,23 @@ async function playLoop({
       break;
     }
 
+    let selection = null;
+    if (controller === 'jev') {
+      try {
+        selection = await selectAction({ api, jobId, runId, persona, observation: `${visualObservation ? '' : latestObservation}\nScreen interpretation: ${answer.toolUses.map(tool => tool.input?.visible_state || '').filter(Boolean).join('\n') || answer.text}`, toolUses: answer.toolUses, recentActions, intentNotes });
+      } catch (error) {
+        if (error instanceof ProxyError && error.isBudget) { budgetExceeded = true; quitReason = 'cost_cap'; quitDetail = error.message; break; }
+        if (error instanceof ProxyError && error.isClosed) { canceled = true; quitDetail = error.message; break; }
+        throw error;
+      }
+      transcript.write({ kind: 'action_selection', source: selection.source || 'jev', status: selection.status, confidence: selection.confidence, selected_tool_id: selection.toolId, reason: selection.reason });
+      if (selection.status !== 'selected') { quitReason = 'persona_quit'; quitDetail = 'Ravensight Playtester stopped for review: ' + selection.reason; break; }
+      if (selection.budget === 'warn') { budgetWarned = true; wrapUpLeft = Math.min(wrapUpLeft, WRAP_UP_TURNS); }
+    }
     messages.push({ role: 'assistant', content: assistantContent(answer) });
     const results = [];
     for (const toolUse of answer.toolUses) {
+      if (selection && toolUse.id !== selection.toolId) { results.push({ type: 'tool_result', tool_use_id: toolUse.id, content: 'Alternative not selected. No action executed.' }); continue; }
       const executed = await executeTool({
         driver,
         toolUse,
@@ -402,6 +441,10 @@ async function playLoop({
         actions
       });
       if (executed.observation) {
+        latestObservation = executed.observation.text;
+        visualObservation = Boolean(executed.observation.screenshotBase64);
+        recentActions.push(`${toolUse.name}: ${executed.changed ? 'screen changed' : 'no observed change'}`);
+        if (recentActions.length > 10) recentActions.shift();
         transcript.write({ kind: 'observation', step: executed.observation.step, text: executed.observation.text, url: executed.observation.url });
         // Only an input to the game can tell us anything about being stuck. A
         // `snapshot` or a `screenshot` never changes the screen by design, so

package/src/run/resume.js

@@ -153,6 +153,7 @@ export async function resumeJob({ api, gameId, jobId, job, runs, state, repoRoot
 
   const resumeOptions = {
     ...options,
+    controller: journal.options?.controller || options.controller || 'standard',
     maxActions: (journal.options && journal.options.maxActions) ?? options.maxActions,
     concurrency: options.concurrency ?? (journal.options && journal.options.concurrency),
     uploadVideo: options.uploadVideo ?? (journal.options && journal.options.uploadVideo),

package/src/run/tools.js

@@ -29,7 +29,16 @@ export const NOTE_KINDS = Object.freeze(['bug', 'confusing', 'good', 'suggestion
 export const SEVERITIES = Object.freeze(['blocker', 'major', 'minor', 'cosmetic', 'praise']);
 
 /** Tools that spend an action. */
-export const ACTION_TOOLS = Object.freeze(['click', 'type', 'key', 'navigate']);
+export const ACTION_TOOLS = Object.freeze(['click', 'type', 'key', 'navigate', 'swipe']);
+
+export const SWIPE_TOOL = {
+  name: 'swipe',
+  description: 'Drag from one screenshot coordinate to another. Spends one action.',
+  input_schema: {
+    type: 'object', additionalProperties: false, required: ['x', 'y', 'toX', 'toY'],
+    properties: Object.fromEntries(['x', 'y', 'toX', 'toY', 'durationMs'].map(key => [key, { type: 'number', minimum: 0, maximum: key === 'durationMs' ? 2000 : 32768 }]))
+  }
+};
 
 const thought = {
   type: 'string',
@@ -185,6 +194,8 @@ export function actionFor(toolUse) {
       return { kind: 'navigate', url: String(input.url || '') };
     case 'wait':
       return { kind: 'wait', ms: Number(input.ms) || 0 };
+    case 'swipe':
+      return { kind: 'swipe', from: { x: input.x, y: input.y }, to: { x: input.toX, y: input.toY }, durationMs: input.durationMs };
     default:
       return null;
   }

package/src/states.js

@@ -26,7 +26,10 @@ export const DRIVERS = Object.freeze([
   'godot_driver',
   'cli_stdio',
   'unity_driver',
-  'unreal_driver'
+  'unreal_driver',
+  'desktop_driver',
+  'android_driver',
+  'ios_driver'
 ]);
 
 export const JOB_STATES = Object.freeze([