ravensight-playtest 0.2.7 → 0.2.9

package/README.md

@@ -206,6 +206,7 @@ splitting the job in two is what gets everything played.
 | `--upload-video` | Upload `session.webm` as well as the reports. |
 | `--upload-transcript` | Upload `transcript.jsonl` as well as the reports. |
 | `--dry-upload` | List every file and byte that would be sent, and send nothing. |
+| `--key-modifier <meta\|control>` | The modifier the browser driver uses for editing shortcuts such as select all. Defaults to this machine's own (Command on macOS, Control elsewhere); set it when the browser's platform is not this machine's. `RAVENSIGHT_KEY_MODIFIER` does the same. |
 | `--headed`, `--headless` | Show or hide the browser. |
 | `--verbose` | Print each turn. |
 
@@ -253,10 +254,56 @@ How the report step reads what happened:
 - Every note carries `after`, the URL and title of the screen that followed
   it. A blocker or major finding has to cite the observation where the failure
   is visible and must not be contradicted by that next screen or by a later
-  persona note.
+  persona note. It also has to cite the screenshot taken at or after that
+  observation (the prompt says which observation each screenshot followed),
+  and that screenshot must not show what the finding says is missing.
+- Every evidence entry is `{"type": ..., "ref": ...}`. A screenshot path the
+  model put under `path` or `file` is moved to `ref` before the report is
+  checked; the file still has to exist in the run directory. If the server
+  still refuses `report.json` at `complete`, the CLI prints every validation
+  message it sent, the run is not counted as succeeded or in the aggregate,
+  and the job exits 1.
 - The model is told never to write calendar dates. `report.md` carries the
   CLI's own `Generated at` timestamp under its title, and the aggregate report's
   header uses its `generated_at`.
+- A template slot the model copied instead of filling (`**Date:** <run date>`,
+  `<driver>`, `<N actions>`) is filled from the CLI's own facts, or "not
+  recorded" when it has none, in `report.md`, the prose of `report.json` and
+  the aggregate. Other angle brackets are left as written: keys such as
+  `<Enter>`, elements such as `<form>`, types such as `Array<string>`, links,
+  code and quoted game text.
+- The server keeps a finding only when one of its evidence references points
+  at a file the run uploaded. After delivery the CLI marks the findings it
+  dropped (by the server's `dropped_finding_ids`, or by applying the same rule
+  to the files it uploaded when its count matches): `dropped: true` in
+  `report.json`, out of the summary in `report.md` and into a section headed
+  "Dropped by the server (no usable evidence)", and out of the aggregate and
+  the synthetic telemetry, so the local report counts what the dashboard
+  counts. When it cannot tell which, `report.md` says how many and that the
+  dashboard is the record. The rewritten files are journalled as local edits,
+  and `upload <job>` never re-sends a completed run's report files.
+
+### What personas do well
+
+A persona plays like a first time player at human speed: menus, onboarding,
+the first few minutes, unclear text, screens that do not respond, and whether
+it would come back. On the web, when a persona replaces a field's value (the
+`type` tool with `replace`), the driver clears it with `fill`, the same way on
+macOS, Windows and Linux; ordinary typing is keystrokes. An editing shortcut
+such as select all uses the platform's own modifier (Command on a Mac, Control
+elsewhere, or `--key-modifier`).
+
+On a long web page the persona's text view of the screen is capped at 4,000
+characters. Over the cap, buttons, links and fields are kept first, a long run
+of look-alike items (a list of 249 cards) is shown as its first two and "and
+247 more like it", and what changed since the last look comes next, so a form
+after a long list still reaches the persona.
+
+Each persona starts at the game's URL as a fresh player with no account, so a
+sign-up form with validation can use up a whole unit before the game itself is
+reached. Point `--build-url` at a guest or demo path, or say where to start in
+the brief's `how_to_start` note (for example "choose Play as guest"), when the
+sign-up is not what you want tested.
 
 ### fix
 
@@ -407,6 +454,10 @@ lost to it. The CLI lints locally first and feeds any miss back into one repair
 turn, which costs a model turn; a report written with the four headings from the
 start avoids that.
 
+The repository's tests check rendered reports against `test/fixtures/server/`, a
+byte copy of the server's report schema from game_tracker commit 864027e,
+which has to be refreshed by hand whenever the server's schema changes.
+
 ## Exit codes
 
 | Code | Meaning |
@@ -429,6 +480,7 @@ CI can treat 2 as a success with a warning.
 | `RAVENSIGHT_API_URL` | The API base URL. Defaults to `https://api.ravensight.io`. |
 | `RAVENSIGHT_DASHBOARD_URL` | Where the dashboard is, if it is not derivable from the API URL. |
 | `RAVENSIGHT_GAME_ID` | The game, when there is no linked repo. |
+| `RAVENSIGHT_KEY_MODIFIER` | `meta` or `control`: the browser driver's modifier for editing shortcuts, as `--key-modifier`. The flag wins when both are set. |
 | `RAVENSIGHT_INGEST_KEY` | The game's ingest key, so a web run can emit its synthetic session into your event stream. `ingest_key` in `.ravensight/config.json` does the same. Without either the run still reports; it just sends no telemetry. |
 | `RAVENSIGHT_GODOT` | Path to the Godot binary. Tried first, then `$GODOT`, then `$GODOT_PATH`, then the PATH, then the platform's usual install locations. `--godot-path` beats all of them. |
 | `RAVENSIGHT_FFMPEG` | Path to ffmpeg. |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ravensight-playtest",
-  "version": "0.2.7",
+  "version": "0.2.9",
   "description": "AI personas play your game and file reports to Ravensight.",
   "license": "MIT",
   "homepage": "https://ravensight.io/docs/#playtest",

package/src/cli.js

@@ -63,6 +63,7 @@ Run options (parsed by the runner, see src/run/args.js):
   --upload-video            upload session.webm as well as the reports
   --upload-transcript       upload transcript.jsonl as well as the reports
   --dry-upload              list every file and byte that would be sent, send nothing
+  --key-modifier <name>     meta or control: the modifier for editing shortcuts in the browser, default the platform's own
   --headed                  show the browser
   --headless                hide it
   --verbose                 print each turn

package/src/commands/run.js

@@ -13,6 +13,7 @@ import { jobDir, runDir as runDirFor, JOB_FILES, RUN_FILES } from '../run/paths.
 import { readBridgeConfig } from "../run/drivers/stdio.js";
 import { WEBDRIVER_NAMES, readDriverConfig } from '../run/drivers/webdriver.js';
 import { runPersona } from '../run/personaLoop.js';
+import { reconcileDroppedFindings } from '../run/dropped.js';
 import * as stateFile from '../run/state.js';
 import { emitRunEvents } from '../run/synthetic.js';
 
@@ -90,6 +91,8 @@ async function run(flags, injected, deps, log) {
   const { context, gameId, repoRoot, apiUrl, cliVersion, api } = setup;
 
   if (options.controller && !['standard', 'jev'].includes(options.controller)) { log('Choose --controller standard or --controller jev.'); return EXIT.ENVIRONMENT; }
+  const keyModifier = resolveKeyModifier(options.keyModifier);
+  if (keyModifier.error) { log(keyModifier.error); return EXIT.ENVIRONMENT; }
   const driverName = options.driver || (context.config && context.config.driver) || 'playwright_web';
   const target = resolveTarget(options, context.config, driverName);
   if (!target) {
@@ -442,7 +445,10 @@ export async function playJob({
   const reportBudgetRefused = results.some(result => result.reportStepBudget === 'refused');
   const reportBudgetDelivered = results.some(result => result.reportStepBudget === 'delivered' || result.reportStepBudget === 'dry_upload');
   const canceled = results.some(result => result.state === 'canceled');
-  const anySucceeded = results.some(result => result.state === 'succeeded');
+  // A run whose report the server refused played, but the server never
+  // completed it: it is not a success, and CI should hear about it.
+  const anySucceeded = results.some(result => result.state === 'succeeded' && !result.reportRefused);
+  const reportRefused = results.some(result => result.reportRefused);
 
   if (finish) {
     const finished = await api.jobs.finish(gameId, jobId, {
@@ -456,7 +462,7 @@ export async function playJob({
   log(`Reports: ${directory}`);
 
   if (canceled) return EXIT.CANCELED;
-  if (reportBudgetRefused) return EXIT.FAILED;
+  if (reportBudgetRefused || reportRefused) return EXIT.FAILED;
   if (playBudgetHit) return EXIT.BUDGET;
   return anySucceeded || reportBudgetDelivered ? EXIT.OK : EXIT.FAILED;
 }
@@ -497,7 +503,7 @@ export async function playPersonaRun({
   // Headless is right for a browser and wrong for Godot: a headless Godot
   // routes no GUI input, so it is a smoke-test mode rather than a way to play.
   const headless = driverName === 'godot_driver' ? options.headless === true : options.headed !== true;
-  const driver = await makeDriver(driverName, driverOptions({ headless, driverName, target, directory, entry, jobId }));
+  const driver = await makeDriver(driverName, driverOptions({ headless, driverName, target, directory, entry, jobId, keyModifier: options.keyModifier }));
   // The ledger is created here rather than inside the loop so the heartbeat
   // can feed it the server's own figure for what this run has spent. Prices
   // are server side only, so that answer is the only real one.
@@ -563,7 +569,7 @@ export async function playPersonaRun({
     const failed = error.outcome || { state: 'failed', quitReason: 'error', actionsTaken: 0, report: null, reportLint: [], reachedState: 'playing' };
     await stateFile.patchRun(repoRoot, journal, runId, { state: failed.state, error: String(error && error.message), quit_reason: failed.quitReason });
     const delivery = {};
-    const answer = await deliver({ deps, api, gameId, jobId, runId, directory, outcome: failed, options, log, delivery });
+    const answer = await deliver({ deps, api, gameId, jobId, runId, directory, outcome: failed, options, log, repoRoot, delivery });
     if (isReportStepBudget(failed)) {
       // The money ran out while the report was being written. The partial
       // report went up and the run is completed as budget_exceeded from
@@ -617,7 +623,15 @@ export async function playPersonaRun({
     budget_warned: outcome.budgetWarned
   });
 
-  await deliver({ deps, api, gameId, jobId, runId, directory, outcome, options, log, journal, repoRoot });
+  const delivered = {};
+  await deliver({ deps, api, gameId, jobId, runId, directory, outcome, options, log, journal, repoRoot, delivery: delivered });
+
+  if (delivered.reportInvalid) {
+    // No synthetic telemetry for findings the server does not hold, and no
+    // "succeeded" line for a run the server never completed.
+    log(`run ${runId} (${persona.slug}) played ${outcome.actionsTaken} actions (${outcome.quitReason}), but the server refused its report, so the run is not complete.`);
+    return { runId, persona: persona.slug, state: outcome.state, quitReason: outcome.quitReason, reportRefused: true };
+  }
 
   if (outcome.report) {
     const emitted = await emitRunEvents({
@@ -710,9 +724,10 @@ export async function deliver({ deps, api, gameId, jobId, runId, directory, outc
       const { report: ignored, report_md: alsoIgnored, ...withoutReport } = body;
       answer = await api.runs.complete(gameId, jobId, runId, withoutReport);
     }
-    if (journal) await stateFile.patchRun(repoRoot, journal, runId, { uploaded: true, completed: true });
+    if (journal) await stateFile.patchRun(repoRoot, journal, runId, { uploaded: true, completed: true, report_refused: false });
     if (answer && answer.dropped_findings > 0) {
       log(`run ${runId}: the server dropped ${answer.dropped_findings} finding(s) with no usable evidence.`);
+      await markDropped({ directory, jobId, runId, repoRoot, outcome, droppedCount: answer.dropped_findings, droppedIds: answer.dropped_finding_ids, options, log });
     }
     if (answer && Array.isArray(answer.report_lint) && answer.report_lint.length > 0) {
       log(`run ${runId}: report lint: ${answer.report_lint.join('; ')}`);
@@ -724,11 +739,58 @@ export async function deliver({ deps, api, gameId, jobId, runId, directory, outc
       delivery.status = error && error.status;
       delivery.code = error && error.code;
     }
+    if (error && error.status === 422 && error.code === 'invalid_report') {
+      // Every validation message first, one per line, before anything says
+      // the run did not complete: the summary line alone ("does not match
+      // the playtest report schema") is not something anyone can act on.
+      const details = refusalDetails(error);
+      log(`run ${runId}: the server refused report.json (${details.length} problem(s)):\n${details.length > 0 ? details.map(line => `  - ${line}`).join('\n') : '  - (the server named no problem)'}`);
+      if (delivery) delivery.reportInvalid = true;
+      if (journal) await stateFile.patchRun(repoRoot, journal, runId, { report_refused: true });
+      log(`run ${runId}: not completed, so the server keeps no findings from it; whether its unit is refunded is decided when the job settles. report.json and the screenshots stay in ${directory}.`);
+      return null;
+    }
     log(`run ${runId}: upload or complete was refused (${String(error && error.message)}). The artifacts are in the run directory and "ravensight-playtest resume" can retry it.`);
     return null;
   }
 }
 
+/**
+ * The validation messages a 422 carried, as `pointer: message` lines. Read
+ * from `details` (the API client's error) or `body.details` (the envelope).
+ *
+ * @param {any} error
+ * @returns {Array<string>}
+ */
+export function refusalDetails(error) {
+  const list = Array.isArray(error && error.details) && error.details.length > 0
+    ? error.details
+    : (error && error.body && Array.isArray(error.body.details) ? error.body.details : []);
+  return list.map(detail => `${(detail && detail.instancePath) || '/'}: ${(detail && detail.message) || 'is invalid'}`);
+}
+
+/**
+ * Bring the local report into line with what the server kept, so report.md,
+ * report.json, the aggregate and the telemetry count what the dashboard
+ * counts. A failure here never fails the delivery: the run is already
+ * complete on the server.
+ */
+async function markDropped({ directory, jobId, runId, repoRoot, outcome, droppedCount, droppedIds, options, log }) {
+  try {
+    const result = await reconcileDroppedFindings({ directory, droppedCount, droppedIds, options, journal: { jobId, runId, repoRoot } });
+    if (!result) return;
+    if (outcome) {
+      outcome.report = result.report;
+      outcome.reportMarkdown = result.markdown;
+    }
+    log(result.identified
+      ? `run ${runId}: marked ${result.dropped.join(', ')} as dropped in report.md and report.json, under "Dropped by the server".`
+      : `run ${runId}: could not tell which finding(s) the server dropped, so report.md says so and the dashboard is the record.`);
+  } catch (error) {
+    log(`run ${runId}: could not mark the dropped finding(s) in the local report (${String(error && error.message)}).`);
+  }
+}
+
 /**
  * Whether a run ran out of money in the report step: it is budget_exceeded
  * and the server had already accepted `reporting` (or `uploading`) for it.
@@ -817,7 +879,7 @@ async function aggregatePhase({
 
   // The aggregate reads reports off disk, so a resumed job whose earlier runs
   // were played by a previous process is covered too.
-  const reports = await loadRunReports(repoRoot, jobId, personaRuns);
+  const reports = await loadRunReports(repoRoot, jobId, personaRuns, journal);
   let aggregate;
   try {
     aggregate = await buildAggregate({
@@ -1040,9 +1102,12 @@ export async function resolveSetup(deps, options, log) {
  *
  * @returns {Object}
  */
-export function driverOptions({ headless, driverName, target, directory, entry, jobId }) {
+export function driverOptions({ headless, driverName, target, directory, entry, jobId, keyModifier }) {
+  // Only the browser driver sends key chords with a modifier.
+  const modifier = driverName === 'playwright_web' ? resolveKeyModifier(keyModifier).modifier : null;
   return {
     headless,
+    ...(modifier ? { modifier } : {}),
     playtest: {
       token: entry.playtest_token || null,
       runId: entry.run_id,
@@ -1054,14 +1119,40 @@ export function driverOptions({ headless, driverName, target, directory, entry,
   };
 }
 
+/**
+ * The modifier the web driver uses for editing shortcuts, from
+ * `--key-modifier meta|control`, else `RAVENSIGHT_KEY_MODIFIER`, else the
+ * platform's own (Command on macOS, Control elsewhere, decided in the driver).
+ * For a remote or containerised browser whose platform is not this machine's.
+ *
+ * @param {string} [value] the flag's value
+ * @param {Object} [env]
+ * @returns {{modifier: 'Meta'|'Control'|null, error?: string}}
+ */
+export function resolveKeyModifier(value, env = process.env) {
+  const raw = value !== undefined && value !== null && value !== '' ? value : env.RAVENSIGHT_KEY_MODIFIER;
+  if (raw === undefined || raw === null || raw === '') return { modifier: null };
+  const key = String(raw).trim().toLowerCase();
+  if (['meta', 'cmd', 'command'].includes(key)) return { modifier: 'Meta' };
+  if (['control', 'ctrl'].includes(key)) return { modifier: 'Control' };
+  return { modifier: null, error: `Choose --key-modifier meta or --key-modifier control (or set RAVENSIGHT_KEY_MODIFIER), not "${raw}".` };
+}
+
 /**
  * The persona reports on disk, for the aggregate. A partial report (written
  * from the notes when the money ran out) is loaded as a `budget_exceeded`
  * run: its findings count, and the aggregate says the job is partial.
  */
-export async function loadRunReports(repoRoot, jobId, personaRuns) {
+export async function loadRunReports(repoRoot, jobId, personaRuns, journal = null) {
   const out = [];
   for (const entry of personaRuns) {
+    // A report the server refused is on disk but not in the record: the
+    // summary must not count findings the dashboard does not have.
+    const recorded = journal && journal.runs ? journal.runs[entry.run_id] : null;
+    if (recorded && recorded.report_refused === true) {
+      out.push({ runId: entry.run_id, persona: entry.persona, state: 'failed', report: null, reportRefused: true });
+      continue;
+    }
     const file = path.join(runDirFor(repoRoot, jobId, entry.run_id), RUN_FILES.reportJson);
     try {
       const report = JSON.parse(await readFile(file, 'utf8'));

package/src/commands/upload.js

@@ -8,6 +8,9 @@ import { pending } from '../upload/queue.js';
 import { uploadRunDir } from '../upload/index.js';
 import { JOB_UPLOAD_PATHS } from '../upload/allowlist.js';
 import { ui } from '../ui/index.js';
+
+/** Report files never re-sent for a run the journal marks completed. */
+const COMPLETED_RUN_REPORTS = Object.freeze(['report.md', 'report.json']);
 import { CliError, ExitCode } from '../errors.js';
 
 /**
@@ -73,11 +76,18 @@ async function drainJob(context, jobId, flags) {
   const results = [];
 
   for (const runId of runIds) {
+    // A completed run's report went to the server with `complete`, which
+    // validated it; the copy on disk may since have been marked locally
+    // (dropped findings), and sending it again would replace the validated
+    // artifact, or be refused once the job is closed.
+    const completed = Boolean(state && state.runs && state.runs[runId] && state.runs[runId].completed);
     const result = await uploadRunDir(paths.runDir(jobId, runId, { repoRoot }), {
       api: context.api,
       gameId,
       jobId,
       runId,
+      exclude: completed ? COMPLETED_RUN_REPORTS : null,
+      excludeReason: 'the run is complete; its report was delivered with complete',
       includeVideo,
       includeTranscript,
       dryRun: Boolean(flags.dryRun),

package/src/run/aggregate.js

@@ -1,4 +1,5 @@
 import { createHash } from 'node:crypto';
+import { fillPlaceholders } from './placeholders.js';
 import { writeFile } from 'node:fs/promises';
 import path from 'node:path';
 import { parseJsonArray, parseJsonObject } from './json.js';
@@ -331,7 +332,10 @@ export async function buildAggregate({
     persona: run.persona,
     state: run.state,
     report: run.report,
-    findings: run.report.findings || []
+    // A finding the server dropped for want of evidence is marked on disk
+    // after delivery (./dropped.js) and kept out of the aggregate, so the
+    // job report counts what the dashboard counts.
+    findings: (run.report.findings || []).filter(finding => !(finding && finding.dropped === true))
   }));
   const personasRun = new Set(inputs.map(run => run.persona).filter(Boolean)).size;
 
@@ -624,7 +628,9 @@ export function renderAggregateMarkdown(json) {
     ? `Observations dropped for lack of evidence:\n${json.observations.map(o => `- ${o.persona}: ${o.text}`).join('\n')}`
     : 'No dropped observations.');
   lines.push('');
-  return lines.join('\n');
+  // The synthesis writes the TL;DR, and a model can copy a template slot into
+  // it as easily as into a persona report.
+  return fillPlaceholders(lines.join('\n'), { generatedAt: json.generated_at, game: json.game_id });
 }
 
 /**

package/src/run/args.js

@@ -22,7 +22,8 @@ const VALUED = new Map([
   ['--cli', 'cli'],
   ['--max-actions', 'maxActions'],
   ['--concurrency', 'concurrency'],
-  ['--quality', 'quality']
+  ['--quality', 'quality'],
+  ['--key-modifier', 'keyModifier']
 ]);
 
 /** Options that are just present or absent. */

package/src/run/drivers/godot.js

@@ -360,6 +360,11 @@ export class GodotDriver {
     if (!action || typeof action.kind !== 'string') {
       return { ok: false, error: 'action needs a kind', changed: false };
     }
+    // Typing more characters is not replacing a value, and claiming it was
+    // would join the old text to the new one without saying so.
+    if (action.kind === 'type' && (action.replace || action.target)) {
+      return { ok: false, error: 'replace (empty a field, then type) and a typing target are only available on playwright_web; this driver can only type more text into what has focus', changed: false };
+    }
     try {
       await this._perform(action);
     } catch (error) {

package/src/run/drivers/stdio.js

@@ -79,6 +79,7 @@ export function createStdioDriver({ spawn: spawnImpl = spawn, timeoutMs = 30000
     observe,
     async act(action) {
       if (!['type', 'key', 'click', 'wait'].includes(action.kind)) return { ok: false, changed: false, error: 'Unsupported bridge action.' };
+      if (action.kind === 'type' && (action.replace || action.target)) return { ok: false, changed: false, error: 'replace (empty a field, then type) and a typing target are only available on playwright_web; this driver can only type more text into what has focus' };
       const before = lastHash;
       const safe = action.kind === 'type' ? { kind: 'type', text: String(action.text || '').slice(0, 2000), submit: Boolean(action.submit) }
         : action.kind === 'key' ? { kind: 'key', key: String(action.key || '').slice(0, 64) }