ravensight-playtest 0.1.0 → 0.2.0

package/README.md

@@ -47,7 +47,7 @@ In CI, set `RAVENSIGHT_TOKEN` instead and skip `login` entirely.
 
 ## Commands
 
-Eleven commands. Flags below are the ones the code accepts, not the ones the
+Twelve commands. Flags below are the ones the code accepts, not the ones the
 prose elsewhere describes; `--help` on any command prints the same list.
 
 Every command takes these four:
@@ -171,6 +171,73 @@ silently ignored: `--budget-usd` and `--quality` (the budget and the model tier
 are the server's, enforced at the proxy), `--offline-upload` (uploads already
 queue) and `--build-dir` (serve the build and pass `--build-url`).
 
+### fix
+
+Takes one finding, attempts the change on a branch of its own, replays the
+finding to see whether the change worked, and prints the pull request command
+for you to run.
+
+```bash
+ravensight-playtest fix pt_job_1/F002 --build-url http://localhost:8080/
+ravensight-playtest fix fnd_7g3k --no-verify
+```
+
+The target is either `<jobId>/<label>`, where the label is the one the report
+gives the finding, or a bare finding id.
+
+| Flag | Effect |
+|---|---|
+| `--game <gameId>` | The game, when this repo is not linked. |
+| `--branch <name>` | The branch to create. Defaults to `ravensight/fix-<label>-<6 hex>`. |
+| `--no-verify` | Do not replay the finding afterwards. The changeset then carries no verdict. |
+| `--allow-dirty` | Fix on top of uncommitted changes instead of refusing. Your changes end up in the diff too. |
+| `--max-actions <n>` | Cap the turns the fix may spend. Capped at 60 whatever you ask for. |
+| `--build-url <url>`, `--godot-project <dir>`, `--driver <name>` | Where the verify run plays, resolved exactly as `run` resolves them. |
+| `--json` | Print the changeset instead of the summary. |
+
+What happens, in order:
+
+1. The finding and its run report are read, with up to two of its screenshots.
+2. Git is checked: a directory that is not a repository, or a machine with no
+   git, stops here with one sentence, and so does a working tree with
+   uncommitted changes unless you passed `--allow-dirty`.
+3. The price is printed and confirmed. Nothing is charged and no branch is
+   created until you say yes.
+4. The server is asked whether the finding is a bug at all. A finding it reads
+   as design intent with confidence 0.7 or more stops the attempt before
+   anything is edited.
+5. The change is planned, then made with a write tool that can only write text
+   files inside your repository, at most 200 kB and at most 20 files.
+6. The verify run replays the finding on the build you pointed at.
+7. `changeset.json` and `report.md` are written, uploaded and attached to the
+   fix run, and the pull request command is printed.
+
+**It never pushes and never commits.** There is no `git commit`, no `git push`
+and no forge client anywhere in the command. You are left on the branch with
+the change in your working tree, and the last thing printed is the command to
+open the pull request once you have read the diff.
+
+A fix attempt is one unit. It is refunded when the finding turns out to be
+design intent, and refunded when the attempt changes nothing, in both cases
+without a model turn being spent on the edit itself. The verify run is included
+in the price.
+
+**A verify can only mean something against a build this machine serves.** The
+change exists in your working tree and nowhere else, so a `--build-url`
+pointing at a hosted build would answer about the last upload instead. Any
+non-local URL is refused unless you pass `--no-verify`, because a confident
+"fixed" from the wrong build is worse than no answer. Rebuild and serve the
+branch locally before the verify, or skip it.
+
+The verdict is printed in words: fixed below 0.35, still reproduced above 0.65,
+unknown in between, and "not verified" when there was no verify run.
+
+**A fix cannot be resumed.** Ctrl-C, or cancelling the job from the dashboard,
+abandons the attempt and gives the unit back: whatever was written stays on the
+branch for you to read or delete, and nothing is committed. That is different
+from `run`, which `resume` picks up where it left off, because a half-applied
+edit is not a playthrough to carry on with.
+
 ### resume
 
 With no job id, lists the jobs this repo has a journal for, newest first. With
@@ -299,9 +366,17 @@ In your repo:
       session.webm            only when you opt in
 ```
 
+A `fix` job's run directory holds `changeset.json`, `report.md` (the pull
+request body) and `usage.json`, plus the screenshots it was shown as evidence.
+
 Add `.ravensight/jobs/` to your `.gitignore`; `init` does it for you when you
 already have one.
 
+`fix` also leaves a branch on your working copy, checked out, with the change in
+your working tree and nothing committed. It is the only thing in this CLI that
+writes to your source at all, and it never pushes: no commit, no push, no forge
+client exists in it. Delete the branch and you are back where you started.
+
 ## What leaves your machine
 
 The dashboard has the full answer, artifact by artifact, at
@@ -314,8 +389,8 @@ which is the page to trust: it is rendered from the server's own allowlist, so
 it cannot drift from what is actually accepted. `ravensight-playtest open` links
 to it. The short version:
 
-Uploaded by default: `report.md`, `report.json`, `usage.json`, the aggregate and
-capability reports, and `screenshots/`. Uploaded only when you ask:
+Uploaded by default: `report.md`, `report.json`, `usage.json`, `changeset.json`,
+the aggregate and capability reports, and `screenshots/`. Uploaded only when you ask:
 `transcript.jsonl` (`--upload-transcript`) and `session.webm`
 (`--upload-video`). **Source code never, at any path.** The upload allowlist has
 no entry for it, the CLI refuses it locally, and the server refuses anything not
@@ -325,6 +400,15 @@ pins it equal to the server's file when a checkout is on the same machine.
 Findings cite a file and a line range. The source text behind that line is not
 uploaded.
 
+A `fix` uploads `changeset.json` and `report.md`. The changeset carries the
+finding id, the branch name, the commit it branched from, the file paths it
+changed with their added and removed line counts, the verify verdict, and the
+pull request title and body. **Not the diff, and not the file contents.** What
+the model read while working is what goes over the wire as part of its own
+reasoning, exactly as `profile` already does, and the same guards apply: a file
+that trips the secret scan is withheld rather than read, and both files a fix
+writes are scanned again before they are uploaded.
+
 Run `upload --dry-upload` or `run --dry-upload` to see the exact file list and
 byte count before anything leaves the machine.
 

package/package.json

@@ -1,8 +1,19 @@
 {
   "name": "ravensight-playtest",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "AI personas play your game and file reports to Ravensight.",
   "license": "MIT",
+  "homepage": "https://ravensight.io/docs/#playtest",
+  "bugs": {
+    "email": "support@realityse.com"
+  },
+  "keywords": [
+    "godot",
+    "playtest",
+    "game-analytics",
+    "ai-personas",
+    "ravensight"
+  ],
   "type": "module",
   "bin": {
     "ravensight-playtest": "bin/ravensight-playtest.js"

package/src/api/client.js

@@ -152,9 +152,14 @@ export function createClient(options = {}) {
       cacheable(`${PACKS_BASE}/skills`, { query: { modules: packModules(modules) }, etag }),
     schemas: ({ etag } = {}) => cacheable(`${PACKS_BASE}/schemas`, { etag }),
     routing: ({ etag } = {}) => cacheable(`${PACKS_BASE}/routing`, { etag }),
-    bundle: ({ gameId, modules, etag } = {}) =>
+    /**
+     * `jobId` asks for the bundle in the context of one job, which is the only
+     * way `graph_context` comes back: without it that field is null, because
+     * the graph is assembled from the job's own findings.
+     */
+    bundle: ({ gameId, modules, jobId, etag } = {}) =>
       cacheable(`${PACKS_BASE}/bundle`, {
-        query: { game_id: gameId, modules: packModules(modules) },
+        query: { game_id: gameId, modules: packModules(modules), job_id: jobId },
         etag
       })
   };
@@ -243,6 +248,18 @@ export function createClient(options = {}) {
       request('POST', `${gameBase(gameId)}/jobs/${encodeURIComponent(jobId)}/aggregate/complete`, { body })
         .then(answer => answer.body),
 
+    /**
+     * Ask whether a finding is a bug at all before spending a fix on it.
+     *
+     * Answers `{ verdict: 'bug' | 'design_intent' | 'content' | null,
+     * confidence, source }`. The verdict is the server's, from the brief's
+     * design intent notes and whatever the dashboard has been told, which is
+     * why it is a route rather than a judgement the CLI makes locally.
+     */
+    fixTriage: (gameId, jobId, body) =>
+      request('POST', `${gameBase(gameId)}/jobs/${encodeURIComponent(jobId)}/fix/triage`, { body })
+        .then(answer => answer.body),
+
     artifacts: (gameId, jobId) =>
       get(`${gameBase(gameId)}/jobs/${encodeURIComponent(jobId)}/artifacts`),
 
@@ -271,6 +288,12 @@ export function createClient(options = {}) {
   const runs = {
     list: (gameId, jobId) => get(`${gameBase(gameId)}/jobs/${encodeURIComponent(jobId)}/runs`),
 
+    /**
+     * One run. `fix` reads it for `verify_verdict`, which the server attaches
+     * to a verify run once it has completed and which nothing else answers.
+     */
+    get: (gameId, jobId, runId) => get(runPath(gameId, jobId, runId)),
+
     add: (gameId, jobId, personas) =>
       request('POST', `${gameBase(gameId)}/jobs/${encodeURIComponent(jobId)}/runs`, {
         body: { personas }

package/src/cli.js

@@ -19,7 +19,8 @@ import { ApiError } from './api/errors.js';
 /** Where each runner backed command lives, for the message below. */
 const RUNNER_COMMANDS = {
   run: 'src/run/index.js',
-  profile: 'src/run/index.js'
+  profile: 'src/run/index.js',
+  fix: 'src/run/index.js'
 };
 
 async function loadRunner(name, exportName) {
@@ -108,6 +109,19 @@ function flagsFrom(options, extra = {}) {
 
 const list = value => value.split(',').map(part => part.trim()).filter(Boolean);
 
+/**
+ * A copy of `values` with the first occurrence of `value` removed.
+ * @param {Array<string>} values
+ * @param {string} value
+ * @returns {Array<string>}
+ */
+function withoutFirst(values, value) {
+  const copy = [...values];
+  const at = copy.indexOf(value);
+  if (at !== -1) copy.splice(at, 1);
+  return copy;
+}
+
 /**
  * Build the program. Exported rather than executed so a test can parse an
  * argument vector without spawning a process or exiting it.
@@ -241,6 +255,31 @@ export function buildProgram() {
       return profile(flagsFrom(options, { game: options.game, argv: command.args }));
     }));
 
+  common(program.command('fix').description('attempt a fix for one finding, on a new branch'))
+    .argument('<target>', 'the finding: <jobId>/<label>, or a finding id')
+    .option('--game <gameId>', 'the game, when this repo is not linked')
+    .option('--branch <name>', 'the branch to create (default ravensight/fix-<label>-<hex>)')
+    .option('--no-verify', 'do not replay the finding after the fix')
+    .option('--allow-dirty', 'fix on top of uncommitted changes instead of refusing')
+    .allowUnknownOption()
+    // Unknown options' values land in command.args; without this commander
+    // counts them as excess positional arguments and refuses the command.
+    .allowExcessArguments()
+    .addHelpText('after', RUNNER_HELP)
+    .action(capture(slot, async (target, options, command) => {
+      const fix = await loadRunner('fix', 'fixCommand');
+      return fix(target, flagsFrom(options, {
+        game: options.game,
+        branch: options.branch,
+        verify: options.verify,
+        allowDirty: Boolean(options.allowDirty),
+        // Commander puts operands first in `command.args` and appends the
+        // options it did not recognise, so the target is in there too and only
+        // the first copy of it is the operand.
+        argv: withoutFirst(command.args, target)
+      }));
+    }));
+
   common(program.command('run').description('run a playtest'))
     .option('--game <gameId>', 'the game, when this repo is not linked')
     .allowUnknownOption()

package/src/commands/fakeCore.js

@@ -16,6 +16,7 @@
 
 const SKILL = ['---', 'name: persona-playtest', 'prompt_version: persona-playtest@0.1.0', '---', '# Persona Playtest'].join('\n');
 const PROFILE_SKILL = ['---', 'name: game-profile', 'prompt_version: game-profile@0.1.0', '---', '# Game Profile'].join('\n');
+const FIX_SKILL = ['---', 'name: fix', 'prompt_version: fix@0.1.0', '---', '# Fix'].join('\n');
 
 /** The server's table, verbatim. */
 export const RUN_TRANSITIONS = Object.freeze({
@@ -102,7 +103,8 @@ export function fakePack(overrides = {}) {
     ],
     skills: [
       { module: 'persona_playtest', name: 'persona-playtest', content: SKILL },
-      { module: 'game_profile', name: 'game-profile', content: PROFILE_SKILL }
+      { module: 'game_profile', name: 'game-profile', content: PROFILE_SKILL },
+      { module: 'fix', name: 'fix', content: FIX_SKILL }
     ],
     schemas: [
       { name: 'playtest-report', content: REPORT_SCHEMA },
@@ -186,6 +188,36 @@ export const REPORT_MD = [
   ''
 ].join('\n');
 
+/**
+ * The two runs a `fix` job buys: the fix itself, and the persona run that
+ * replays the finding afterwards. There is no aggregate run, because a fix
+ * job has nothing to aggregate.
+ */
+export const FIX_RUNS = Object.freeze([
+  { run_id: 'pt_run_fix', module: 'fix', persona: null, playtest_token: 'tok_fix', budget_usd: 1.5 },
+  { run_id: 'pt_run_verify', module: 'persona_playtest', persona: 'curious-kid', playtest_token: 'tok_verify', budget_usd: 0.6 }
+]);
+
+/**
+ * @param {Object} [overrides]
+ * @returns {Object} one finding, the shape `GET /findings` answers with
+ */
+export function fakeFinding(overrides = {}) {
+  return {
+    finding_id: 'fnd_1',
+    label: 'F002',
+    job_id: 'pt_job_1',
+    run_id: 'pt_run_a',
+    persona: 'curious-kid',
+    title: 'The pause menu cannot be closed',
+    severity: 'major',
+    kind: 'bug',
+    repro_steps: ['press start', 'press escape', 'press escape again'],
+    persona_quote: 'I am stuck in this menu and nothing I press gets me out.',
+    ...overrides
+  };
+}
+
 /**
  * @param {Object} [options]
  * @returns {Object} deps, with a `calls` log and a `runStates` map
@@ -198,7 +230,11 @@ export function fakeCore({
   publish = true,
   aggregateFinishes = false,
   brief = fakeBrief(),
-  pack = fakePack()
+  pack = fakePack(),
+  findings = [fakeFinding()],
+  triage = { verdict: 'bug', confidence: 0.9, source: 'brief' },
+  graphContext = { levels: [{ name: 'title', exits: ['level_1'] }] },
+  verifyVerdict = { reproduced: 0.1, source: 'judge' }
 } = {}) {
   const calls = [];
   const record = (name, payload) => calls.push({ name, ...payload });
@@ -265,6 +301,21 @@ export function fakeCore({
         record('uploads', { jobId, body });
         return { urls: body.files.map(file => ({ path: file.path, put_url: 'https://s3.test/x', tagging: null })) };
       },
+      async findings(gameId, query) {
+        record('findings', { gameId, query });
+        // The server narrows by `job` when it is given; the fake does the same
+        // so a test cannot pass by matching a finding from another job.
+        const wanted = query && query.job;
+        return { findings: wanted ? findings.filter(entry => entry.job_id === wanted) : findings };
+      },
+      async fixTriage(gameId, jobId, body) {
+        record('fixTriage', { jobId, body });
+        return { ...triage };
+      },
+      async artifacts(gameId, jobId) {
+        record('artifacts', { jobId });
+        return { artifacts: [] };
+      },
       async aggregateComplete(gameId, jobId, body) {
         record('aggregateComplete', { jobId, body });
         return aggregateFinishes
@@ -302,6 +353,15 @@ export function fakeCore({
       async list(gameId, jobId) {
         record('list', { jobId });
         return { runs: plan.map(entry => ({ ...entry, state: runStates.get(entry.run_id), actions_taken: 0 })) };
+      },
+      async get(gameId, jobId, runId) {
+        record('runGet', { jobId, runId });
+        const entry = plan.find(candidate => candidate.run_id === runId) || { run_id: runId };
+        const state = runStates.get(runId) || 'queued';
+        // `verify_verdict` exists only on a completed verify run, which is the
+        // one thing the fix command reads this route for.
+        const verdict = entry.module === 'persona_playtest' && state === 'succeeded' ? verifyVerdict : undefined;
+        return { run: { ...entry, state, ...(verdict === undefined ? {} : { verify_verdict: verdict }) } };
       }
     }
   };
@@ -325,7 +385,12 @@ export function fakeCore({
     packs: {
       async get(options) {
         record('packs', { options });
-        return pack;
+        // `graph_context` is populated only in a bundle asked for with a job
+        // id, because the server assembles it from that job's findings. A fake
+        // that answered it either way would hide a missing refetch.
+        return options && options.jobId
+          ? { ...pack, graph_context: pack.graph_context ?? graphContext }
+          : { ...pack, graph_context: null };
       }
     },
     async finalizeRun(dir, options) {