ravensight-playtest 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Reality Software Entertainment
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,380 @@
+# ravensight-playtest
+
+AI personas play your game, then file a report you can read, mark up and argue
+with. The personas run on your machine, against your build; Ravensight stores
+the reports and renders them.
+
+You need a Ravensight account and Node 22 or newer. You do not need an
+Anthropic API key: model calls go through Ravensight under the same token you
+log in with, so there is one credential on your machine and you can revoke it
+from the dashboard.
+
+## Install
+
+Run it without installing anything:
+
+```bash
+npx ravensight-playtest login
+```
+
+Or install it globally:
+
+```bash
+npm install -g ravensight-playtest
+ravensight-playtest login
+```
+
+Nothing is downloaded until you need it. Playwright is an optional dependency,
+so `login`, `check`, `init` and `brief` work without a browser installed;
+`run --driver playwright_web` is what asks for one. `keytar` is optional too,
+which is why a machine with no keychain still installs cleanly.
+
+## First run
+
+```bash
+ravensight-playtest login                  # device code, approved in the browser
+cd path/to/your/game
+ravensight-playtest init --game <gameId>   # links this repo
+ravensight-playtest check                  # confirms the machine and the balance
+ravensight-playtest brief                  # four questions, two minutes
+ravensight-playtest run                    # prints the price, asks, then plays
+```
+
+`login` stores the token in your OS keychain (macOS Keychain, Windows
+Credential Manager, libsecret on Linux). With no keychain available it falls
+back to a file with 0600 permissions under your home directory, and says so.
+In CI, set `RAVENSIGHT_TOKEN` instead and skip `login` entirely.
+
+## Commands
+
+Eleven commands. Flags below are the ones the code accepts, not the ones the
+prose elsewhere describes; `--help` on any command prints the same list.
+
+Every command takes these four:
+
+| Flag | Effect |
+|---|---|
+| `--api-url <url>` | Point at another Ravensight deployment. Defaults to `RAVENSIGHT_API_URL`. |
+| `--repo <dir>` | Work in a different directory. |
+| `--json` | Machine readable output, and nothing else on stdout. Stable for CI. |
+| `-y, --yes` | Answer every confirmation with yes. Required for CI, since a prompt with no terminal is treated as no. |
+
+### login
+
+Device code flow. Prints a code, waits for you to approve it in the dashboard,
+stores the token.
+
+| Flag | Effect |
+|---|---|
+| `--no-browser` | Do not try to open a browser. `RAVENSIGHT_NO_BROWSER=1` does the same thing. |
+
+The token goes in your OS keychain (macOS Keychain, Windows Credential Manager,
+libsecret on Linux). With no keychain available it falls back to a file with 0600
+permissions under `RAVENSIGHT_HOME`, and says which store it used. In CI, set
+`RAVENSIGHT_TOKEN` and skip `login` entirely.
+
+### logout
+
+Revokes the token on the server, then forgets it locally. No flags of its own. A
+token from `RAVENSIGHT_TOKEN` is left alone: it is probably a CI secret shared
+with other jobs.
+
+### check
+
+The doctor. Node version, server reachable, CLI new enough, credential valid and
+which store it came from, game visible, balance against the next job's price,
+content pack verified by sha256, disk, recommended concurrency, and the two
+optional tools. Exit code 3 when any row fails; a missing optional tool is a
+warning and never a failure, so CI does not break on it.
+
+Past `--personas` in `POST /jobs`, requesting more than 100 personas or the same
+slug twice is refused with 400 `too_many_personas`, and a slug this game does
+not have is 400 `unknown_persona`; `run` checks the count against the same
+ceiling before it asks the server anything. Past 8 personas the job is still
+allowed, but a warning row explains that the later runs may not get to play
+inside the job's wall clock; any run that does not play is refunded
+automatically. `check` shows that same warning as a row when it applies.
+
+| Flag | Effect |
+|---|---|
+| `--game <gameId>` | Check against this game instead of the linked one. |
+| `--modules <list>` | Price this module list, comma separated. |
+| `--personas <list>` | Price this persona list. Without it the balance row prices no personas, so the quote is lower than a real job's. |
+| `--godot-path <path>` | Where Godot is, if it is not on the PATH. |
+
+### init
+
+Links this repo to a game and writes `.ravensight/config.json`. Refuses a game
+your token cannot see, so the first job does not fail with a 404. Adds
+`.ravensight/jobs/` to an existing `.gitignore`.
+
+| Flag | Effect |
+|---|---|
+| `--game <gameId>` | The game to link. With one visible game it is optional. |
+| `--force` | Relink a repo that is already linked. |
+
+### brief
+
+Writes the expectations brief: your goals, what success looks like, how the game
+is controlled, where a persona starts. Then prints what a job would cost,
+whether your balance covers it, and, past 8 personas, the same wall-clock
+warning `run` shows. A brief is required before `run`; the server answers 409
+`brief_required` without one.
+
+| Flag | Effect |
+|---|---|
+| `--show` | Print the current brief and stop. |
+| `--open` | Open the brief form in the dashboard instead of asking here. |
+| `--draft` | Save as a draft even when it looks complete. |
+| `--from <file>` | Read the fields from a JSON file instead of asking. |
+| `--game <gameId>` | The game, when this repo is not linked. |
+| `--modules <list>`, `--personas <list>` | Price this module and persona list in the estimate it prints. |
+
+### profile
+
+Builds the game profile and the capability report: what your game is, what can
+drive it, what a persona will be able to do. Takes `--game <gameId>` and all the
+run options below.
+
+### run
+
+Registers a job, plays it, writes the reports, uploads them, finishes the job.
+Prints the price and asks before spending.
+
+There is no product limit on how many personas one job can name; an unplayed
+run is refunded automatically when the job settles, so a large job never costs
+more than what actually got played. `--personas` past 100, or the same slug
+twice, is refused before anything is charged, checked locally against the
+content pack so the round trip to the server is never spent finding that out.
+Past 8 personas the job is still allowed; `run` prints a warning above the
+price prompt saying the later runs may not fit inside the job's wall clock, and
+splitting the job in two is what gets everything played.
+
+| Flag | Effect |
+|---|---|
+| `--game <gameId>` | The game, when this repo is not linked. |
+| `--personas <list>` | Comma separated persona slugs. Defaults to the pack's set. |
+| `--driver <name>` | `playwright_web` or `godot_driver`. |
+| `--build-url <url>` | The web build to play. |
+| `--godot-project <dir>` | The Godot project directory. |
+| `--cli <command>` | The command that starts a text build. |
+| `--max-actions <n>` | Cap the actions one persona may take. |
+| `--concurrency <n>` | Personas at once. Default 2; `check` recommends a number for this machine. |
+| `--upload-video` | Upload `session.webm` as well as the reports. |
+| `--upload-transcript` | Upload `transcript.jsonl` as well as the reports. |
+| `--dry-upload` | List every file and byte that would be sent, and send nothing. |
+| `--headed`, `--headless` | Show or hide the browser. |
+| `--verbose` | Print each turn. |
+
+Four flags are accepted and reported as not applied by this release, rather than
+silently ignored: `--budget-usd` and `--quality` (the budget and the model tier
+are the server's, enforced at the proxy), `--offline-upload` (uploads already
+queue) and `--build-dir` (serve the build and pass `--build-url`).
+
+### resume
+
+With no job id, lists the jobs this repo has a journal for, newest first. With
+one, keeps finished runs, restarts or replays the rest, and never charges twice:
+the register is replayed under the journal's own idempotency key, which answers
+the stored job without pricing anything.
+
+| Flag | Effect |
+|---|---|
+| `--game <gameId>` | The game, when the job journal is missing. |
+| `--status` | Report and stop, without restarting anything. |
+| `--restart-interrupted` | Replay an interrupted run from the start. By default a run that got far enough is finalised from the transcript it already has, which spends no model turns; this forces the replay, which spends the action budget again. |
+
+### upload
+
+Finishes the uploads a crashed or offline run still owes. Safe to run
+repeatedly: files already sent are skipped by checksum.
+
+| Flag | Effect |
+|---|---|
+| `--game <gameId>` | The game, when the job journal is missing. |
+| `--dry-upload` | List what would be sent and send nothing. |
+| `--video` | Include the session video. |
+| `--transcript` | Include the transcript. |
+
+### review
+
+Opens the post job review form. Your marks on the findings are what tunes the
+personas, so this is the useful half of the loop. Takes `--game <gameId>` and an
+optional job id.
+
+### open
+
+Opens the dashboard page for a job, or for the game. Takes `--game <gameId>` and
+an optional job id.
+
+## Playing a Godot game
+
+`--driver godot_driver` launches your own Godot binary against your own project,
+with a small driver addon injected so the persona can read the scene tree and
+send input.
+
+**Your project directory is never written to.** The project is copied to a temp
+directory and the addon is injected into the copy, through an `override.cfg` and
+the copy's own `project.godot`. Nothing about the copy is written back. The copy
+excludes `.git` and `node_modules` and is refused above 2 GB, and its temp root
+is `0700`, because a project directory can contain a `.env`.
+
+The addon only ever activates deliberately: it needs `--ravensight-driver` on the
+command line or `RAVENSIGHT_DRIVER=1` in the environment, it binds `127.0.0.1`
+and nothing else, it requires a per run token in a `hello` call before it will
+answer any other method, it accepts one client at a time, and it frees itself the
+moment any of that fails. On a release export it is stripped from the build
+unless the preset carries the `ravensight_driver` feature tag.
+
+**Headless Godot routes no synthesized mouse or GUI input.** A click on a button
+does nothing under `--headless` and works windowed, and nothing takes keyboard
+focus without a click first. So `--headless` is a smoke test mode, not a playtest
+mode, for anything with a user interface: on a machine with no display, use Xvfb
+and a windowed Godot rather than `--headless`. The driver prints this in the
+run's console log when it launches headless.
+
+Video from the Godot driver is a `frames/` sequence at 4 fps rather than a
+container, so stitching needs ffmpeg and ffmpeg stays optional.
+
+`docs/godot-driver.md` has the protocol, the method set and the error codes.
+
+## What a report has to contain
+
+`report.json` is validated against the pack schema before it is sent, and
+`report.md` is linted against the four sections the server expects:
+
+- **What worked**
+- **Top issues**
+- **What this run didn't cover**
+- **Suggested next step**
+
+A missing section is a lint finding rather than a rejection, so a report is never
+lost to it. The CLI lints locally first and feeds any miss back into one repair
+turn, which costs a model turn; a report written with the four headings from the
+start avoids that.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| 0 | Success |
+| 1 | Failure |
+| 2 | Budget exceeded. Partial results were written and uploaded. |
+| 3 | Environment check failed |
+| 4 | Authentication failed |
+| 5 | Model API error after retries |
+| 10 | Canceled |
+
+CI can treat 2 as a success with a warning.
+
+## Environment variables
+
+| Variable | Effect |
+|---|---|
+| `RAVENSIGHT_TOKEN` | Use this token instead of the keychain. The CI path. |
+| `RAVENSIGHT_API_URL` | The API base URL. Defaults to `https://api.ravensight.io`. |
+| `RAVENSIGHT_DASHBOARD_URL` | Where the dashboard is, if it is not derivable from the API URL. |
+| `RAVENSIGHT_GAME_ID` | The game, when there is no linked repo. |
+| `RAVENSIGHT_INGEST_KEY` | The game's ingest key, so a web run can emit its synthetic session into your event stream. `ingest_key` in `.ravensight/config.json` does the same. Without either the run still reports; it just sends no telemetry. |
+| `RAVENSIGHT_GODOT` | Path to the Godot binary. Tried first, then `$GODOT`, then `$GODOT_PATH`, then the PATH, then the platform's usual install locations. `--godot-path` beats all of them. |
+| `RAVENSIGHT_FFMPEG` | Path to ffmpeg. |
+| `RAVENSIGHT_HOME` | Where per user state lives. Defaults to `~/.ravensight-playtest`. |
+| `RAVENSIGHT_CACHE_DIR` | Where the content pack cache lives. Defaults to the OS cache directory. |
+| `RAVENSIGHT_NO_BROWSER` | Set to 1 to never try to open a browser. |
+
+## What ends up on disk
+
+In your repo:
+
+```
+.ravensight/
+  config.json                 the game link and your defaults
+  jobs/<jobId>/
+    state.json                the resumable journal
+    upload-queue.jsonl        what was uploaded, and what is still owed
+    runs/<runId>/
+      report.md  report.json  usage.json
+      screenshots/NN-slug.png
+      transcript.jsonl        only when you opt in
+      session.webm            only when you opt in
+```
+
+Add `.ravensight/jobs/` to your `.gitignore`; `init` does it for you when you
+already have one.
+
+## What leaves your machine
+
+The dashboard has the full answer, artifact by artifact, at
+
+```
+https://app.ravensight.io/games/<gameId>/playtest/what-leaves-your-machine
+```
+
+which is the page to trust: it is rendered from the server's own allowlist, so
+it cannot drift from what is actually accepted. `ravensight-playtest open` links
+to it. The short version:
+
+Uploaded by default: `report.md`, `report.json`, `usage.json`, the aggregate and
+capability reports, and `screenshots/`. Uploaded only when you ask:
+`transcript.jsonl` (`--upload-transcript`) and `session.webm`
+(`--upload-video`). **Source code never, at any path.** The upload allowlist has
+no entry for it, the CLI refuses it locally, and the server refuses anything not
+on its own copy of the list. `src/upload/allowlist.js` is that copy, and a test
+pins it equal to the server's file when a checkout is on the same machine.
+
+Findings cite a file and a line range. The source text behind that line is not
+uploaded.
+
+Run `upload --dry-upload` or `run --dry-upload` to see the exact file list and
+byte count before anything leaves the machine.
+
+Model calls go to Ravensight's proxy rather than to Anthropic, under the token
+you logged in with. No Anthropic key exists on your machine, and the CLI builds
+its client with `apiKey` explicitly null, so an `ANTHROPIC_API_KEY` already in
+your environment is never picked up and never sent anywhere.
+
+Before anything is written to a run directory, the transcript and `usage.json`
+are scanned for secrets. A hit is replaced by a line naming the shape that
+matched, never the value.
+
+### The 90 day artifacts
+
+The session video and the transcript expire after 90 days; everything else lives
+for the life of the game. That is enforced as an S3 object tag rather than as a
+filename rule, because a lifecycle rule can filter on a tag or a key prefix and
+the filename is at the end of the key.
+
+The tag is a contract between three parties, and the CLI is one of them: the
+presign response hands back the exact `x-amz-tagging` value that was signed, and
+the CLI sends it **verbatim** on the PUT. Changing it, or dropping it, is a
+`SignatureDoesNotMatch` rather than an untagged object, which is the failure you
+want: an untagged 200 MB video would sit in the bucket forever and nothing would
+say so. The CLI never computes the value itself.
+
+## Offline
+
+After one online fetch the content pack is cached and verified by sha256, so a
+run works with the network down except for the model calls themselves. Uploads
+and progress reports queue up and flush when the server is reachable again;
+`upload <jobId>` drains them by hand.
+
+## Embedding
+
+`src/api/README.md` documents the one module other code may import, which is
+also this package's entry point:
+
+```js
+import { createClient, packs, uploadRunDir } from 'ravensight-playtest';
+```
+
+## Reference
+
+| File | What is in it |
+|---|---|
+| `docs/api-contract.md` | Every server route this CLI calls, with the headers it sends and the status codes it handles. The list to check a server change against. |
+| `docs/godot-driver.md` | The Godot driver protocol, its method set, its error codes and its safety rules. |
+| `src/api/README.md` | The module other code may import, and what the runner half expects from it. |
+
+## License
+
+MIT. Copyright Reality Software Entertainment.