tinybeeai 0.0.0-stage → 0.16.1

package/docs/SANDBOX.md

@@ -0,0 +1,320 @@
+# Test runners
+
+A dot runs your Playwright tests after a merge, on a PR, and when you ask it in chat. The dot's `runner` field says where the tests run:
+
+```json
+{
+  "mode": "local",
+  "setup": "npm ci --no-audit --no-fund\nnpx playwright install chromium\nnpm run build",
+  "command": "npx playwright test --grep @smoke",
+  "smokeGrep": "@smoke"
+}
+```
+
+| Field | Default | Purpose |
+|---|---|---|
+| `mode` | | `local`, `docker`, or `demo` |
+| `setup` | unset | Install and build steps that run through the shell before the tests, see [Setup steps](#setup-steps) |
+| `command` | `npx playwright test` | Test command. It runs without a shell, see [Test command](#test-command) |
+| `workdir` | unset | A checkout on the localdost host. Unset: localdost clones the commit under test into a temp folder |
+| `workingDirectory` | unset | A folder inside the checkout, such as `apps/web`, where setup and the test command run. Set from a CI workflow's `working-directory`. No absolute path and no `..` |
+| `copyFiles` | unset | Files of the `workdir`, relative paths such as `.env`, copied into the temporary worktree of a run before the setup steps, see [Copy files](#copy-files) |
+| `passEnv` | unset | Host variables, each starting with `LOCALDOST_TEST_`, that carry the sign-in of the app under test. The tests see them as `TEST_<name>`, see [Test login](#test-login) |
+| `image` | `mcr.microsoft.com/playwright:v<version>-noble` | Docker image, with `<version>` from the checkout's `@playwright/test` |
+| `smokeGrep` | `@smoke` | Text that marks a spec as a smoke spec |
+| `trustForks` | `false` | Runs tests for fork PRs too, see [Fork safety](#fork-safety) |
+
+`testBranches` is a field of the dot itself, not of `runner`: a list of branches besides the default branch that run post-merge tests, see [Test branches](#test-branches). The dot dialog has it under **Test branches** in the Test runner section.
+
+localdost appends `--reporter=json`, the spec files, and `--output=<temp folder>` to the command, and passes `BASE_URL` from the dot's `baseUrl` or the PR's preview URL. Your Playwright config should read `process.env.BASE_URL`. A run times out after 10 minutes.
+
+## Setup steps
+
+`setup` holds the steps that prepare the checkout: install dependencies, install browsers, build the app. Write one step per line, or join them with `&&`. Each step runs through `sh -c` in the workdir, in order, and the first step that fails stops the run before the tests start. Pipes, redirects, and `||` work inside a step.
+
+The output of every step streams to the run's live terminal, and its tail is kept in the report next to the test output. Setup has its own time budget of 15 minutes (`LOCALDOST_SETUP_TIMEOUT_MS`), on top of the 10 minutes for the tests.
+
+An install step that only reads the lockfile (`npm ci`, `npm install` or `npm i` with flags only, `pnpm install`, `yarn`, `yarn install`, `bun install`) is cached. After a successful install, localdost copies `node_modules` to `<LOCALDOST_DATA_DIR>/runner-cache/<key>/`, where the key hashes the lockfile, the step text, the platform, and the Node major version. A later run with the same key copies `node_modules` back and skips the install, and a workdir whose `node_modules` already carry the key skips the copy too. On macOS and Linux the copy uses copy-on-write where the file system has it. The four most recently used entries are kept. Every other step, such as a build, runs on every attempt. A step that adds a package (`npm install left-pad`) or holds a redirect is never cached.
+
+Setup runs with the same filtered environment as the tests (see [Local](#local)), so a private registry token in `NPM_TOKEN` does not reach `npm ci`. Use a registry config the host user already has, outside the repo.
+
+## Test command
+
+`command` runs without a shell. localdost splits it into words with shell quoting rules: single quotes, double quotes, and backslash escapes work, and leading `NAME=value` words set environment variables. It then appends `--reporter=json`, the spec files, and `--output=<temp folder>`.
+
+A shell operator in the command (`&&`, `||`, `|`, `;`, `&`, `<`, `>`, a backtick, or `$(`) is an error. The dot dialog and `PATCH /api/dots/:id` refuse it with a message that points to Setup steps, and a dot saved before this check gets the same message as a "tests did not run" result. Move install and build steps into `setup` and keep only the test command here.
+
+## Tests that did not run
+
+When setup fails, the command is not valid, or the command ends without a Playwright JSON report, the tests did not run. The report carries `notRun` with the stage (`setup`, `command`, or `report`), one sentence on why, the exit code, and the command output. It counts no failed test and is not retried.
+
+The verdict is `environment_failure`. Its headline reads "Tests did not run: a setup step failed" (or the command or report reason), and the summary says that the result says nothing about whether the change works, then gives the reason, the fix, and the redacted tail of the output. The model does not write this verdict, so a PR comment never reads a missing report as a product or test failure. The GitHub check concludes neutral, and the QA review section reads "tests: did not run".
+
+## Run tests per repo
+
+A repo of a dot has a **Run tests** switch, on by default for an app repo (`runTests: false` on the repo entry turns it off, KAT-361). Turn it off for a repo that has no Playwright suite, such as a Python or Java back end that shares a dot with a front end. A PR or a merge of that repo still gets the review, the report, and the check, and no test command starts: no setup step, no AI-Vision case, no generated spec. The run step reads "Tests skipped: Run tests is off for acme/api", the report has `notRun.stage` `skipped`, and the verdict is a pass with no test counts, the same as a docs-only change. It is never "Tests did not run": the run page shows "Tests skipped" with the reason in the summary card and in the Tests section, with no warning and no runner advice. The review still reads the results of the PR's own CI jobs for that repo. An Issues only repo is not watched at all, and a Tests repo is the tests, so neither has the switch. The switch decides about the app repo's own suite only: a Tests repo of the dot still runs for a PR or a merge of that repo, see [Tests repo](#tests-repo).
+
+The switch is on the repo form of the dot dialog (Sources, under the PR scope) and on each App card of the Repos step. A dot with several app repos has one switch for each in the dot dialog, named by its repo, and the dialog saves them through `repos`; the note at the top of the Test runner section says its settings are idle only when no app repo is tested. A dot with several app repos picks the repo from the event, so a front end with the switch on is tested and a back end with it off is not. Try it and a rerun asked for by a person ignore the switch: they run what you ask for. `PATCH /api/dots/:id` takes `repo.runTests` or `repos[].runTests`, answers 400 for a value that is not a boolean or for a test or docs repo, and stores only `false`.
+
+## Tests repo
+
+A team often keeps its end-to-end suite in a repo of its own, and its app repos have no Playwright suite. A repo of a dot with the role **Tests** holds that suite. When a PR of an app repo is checked, or an app repo is tested after a merge, the dot runs the suite of the Tests repo after the app repo's own tests, against the same app, and puts the result in the same report. The app repo's Run tests switch does not hold it back: an app repo with Run tests off still gets the Tests repo run (KAT-86). The setup side of this is in [SETUP.md](SETUP.md#tests-repo); this section is what a run does on the localdost host.
+
+```json
+{
+  "repo": "acme/shop-e2e",
+  "role": "test",
+  "test": { "framework": "playwright", "dir": "e2e", "setup": "npm ci", "label": "Shop", "passEnv": ["LOCALDOST_TEST_USER"] }
+}
+```
+
+| `test` field | Default | Purpose |
+|---|---|---|
+| `framework` | | `playwright` runs. `cypress`, `webdriverio`, `pytest`, `junit` and `katalon` are accepted and not run yet: the report says "which localdost cannot run yet" |
+| `command` | `npx playwright test` | Test command, without a shell, like the dot's own. localdost appends the reporter, the spec files, the title filter and `--output` |
+| `dir` | the repo root | The folder with the Playwright project, such as `e2e`. No absolute path and no `..` |
+| `setup` | the default install | Install and build steps for the Tests repo, run through `sh -c` in `dir` before the tests. The app repo's `runner.setup` never runs for a Tests repo. Without `setup`, a checkout with a `package.json` gets the default install and the install cache, as a clone does |
+| `ref` | `default` | Which version of the Tests repo runs: `default`, `match_branch`, or `{ "fixed": "<branch or tag>" }`, see [Which version of the tests](SETUP.md#tests-repo) |
+| `baseUrlEnv` | `["BASE_URL"]` | Variables that carry the URL of the app under test. At most 5 names in capital letters that end in `_URL` or are `BASE_URL` or `BASEURL`; `PATH`, `NODE_OPTIONS` and names that look like a credential are refused. An admin setting |
+| `passEnv` | the dot's `runner.passEnv` | Host variables that carry the test login, `LOCALDOST_TEST_` names only, passed as `TEST_<name>`, see [Test login](#test-login). With none listed, the dot's own list is used |
+| `label` | unset | The product the tests cover, up to 40 characters, for the report line "Ran 24 True Platform tests from true-dogfood-tests" |
+| `areas` | unset | Which tests cover which area of a change, see [SETUP.md](SETUP.md#tests-repo) |
+| `smokeGrep` | `@smoke` | The text of the quick tests that run with a narrowed selection |
+| `on` | `["pr_check", "post_merge"]` | The jobs that run the Tests repo |
+
+**The run.** localdost asks GitHub for the commit of the chosen ref (the event never names a commit), fetches that commit into a cache folder, `<data folder>/test-repos/<owner>__<name>`, and adds a git worktree of it at `<data folder>/worktrees/<run id>-a<attempt>-tests`, the folder the app repo's worktrees use. A worktree is a clean checkout: none of the files of another folder are copied in, and `copyFiles` of the dot does not apply. The runner then checks that the worktree is on the commit GitHub named, and a count is shown only then. The app repo is never cloned for this run: only its URL crosses over, the PR's preview or the dot's base URL. The worktree is removed when the run ends, after a failure, a cancel or a timeout, and the server removes leftovers of a crash at start with the other worktrees. A fetch that cannot reach GitHub fails the job, which runs again after a short wait, like a clone of the app repo. Runs of one Tests repo share its cache folder, and git allows one shallow fetch into a folder at a time, so localdost lets them take turns: a second run waits for the first fetch and checkout instead of failing.
+
+The run uses the dot's mode (local or Docker), image and video setting, and the 10 minute test timeout and 15 minute setup timeout. The report of one run holds the cases of the app repo's own tests and of each Tests repo. Each case of a Tests repo names its file `owner/name:path`, with the repo in lower case and the path in the repo (Playwright reports a spec by its path under its `testDir`; localdost rewrites it to the path in the repo, so the link in a defect and the file a retry filters on are the real ones). The same spec file of two repos is two cases, with their own results, flaky memory and bug signatures. A failing spec is retried twice, in the Tests repo, as an app spec is. Screenshots, video and trace of each test are artifacts of the run like any other.
+
+**The login and the environment.** The suite gets the environment of a local run (the host environment minus every credential of the server, see [Local](#local)) or of a Docker run (nothing but the test login and `BASE_URL`), plus the test login from `passEnv`, renamed to `TEST_<name>`. A `test.passEnv` name must start with `LOCALDOST_TEST_`, as `runner.passEnv` names must, and a name that does not is refused at save and dropped at every run. `GITHUB_TOKEN` and a model key cannot reach the suite this way.
+
+**When the Tests repo cannot be read.** A read of the Tests repo that fails does not stop the app repo's review: the batch shows as not run, with one sentence. A 401 says GitHub refused the credentials. A 403 says GitHub denied access and asks for read access for the App or token. A 403 that GitHub sends for a spent rate limit (a `Retry-After` header, or the words rate limit or secondary limit) and a 429 say that GitHub is throttling requests and that localdost retries on the next run: no permission is missing. A 5xx or a network error says GitHub did not answer.
+
+**Fork safety.** A Tests repo run for a fork PR never gets the environment of this machine. A fork PR does not run the Tests repo in local mode: the batch is skipped with the step "Tests repo skipped" and a sentence in the report, and nothing is fetched. In Docker mode it runs only when no test login would go with it: when the Tests repo's login list (or, without one, the dot's) names a variable, the batch is skipped as well, with a sentence that says so, because the page under test is the fork's preview and its author controls it. A Docker run for a fork that has no login list gets nothing from this machine: the container holds only the report paths and `BASE_URL`. The description of a fork PR and the name of its branch never pick the ref of the Tests repo; only the fixed ref or the default branch of the settings apply. A PR of the repo itself is not a fork.
+
+**Runs of the Tests repo itself.** A PR or a merge of the Tests repo does not start this run yet: such an event gets CI failure triage only. See [MULTI-REPO.md](MULTI-REPO.md#5-build-tasks) task T5.
+
+## Try it
+
+The Test runner section of the dot dialog has a **Try it** button. It runs the setup steps and the test command once with the settings in the form, saved or not, and shows the outcome: tests passed, tests failed, or tests did not run and why, plus whether the install came from the cache and the command output. A try is not a run: it posts nothing, keeps no artifacts, and makes no verdict.
+
+The API is `POST /api/runner/try` with `{ dotId?, runner, repo?, baseUrl? }`, which answers `202 { id }`, then `GET /api/runner/try/:id` for the state, the output lines so far, and the result. One try runs at a time; a second request gets `409`. A try without a workdir clones the repo's default branch.
+
+## Which specs run
+
+For a merge or a PR, the dot picks specs from three sources: specs whose file name contains a changed area (`src/checkout/...` picks `checkout.spec.ts`), specs the diff itself touched, and every spec that contains the smoke text. When the PR review names an existing test in its What to test list, that spec runs too. When nothing matches, the full suite runs.
+
+Failing specs are retried up to two times. A spec that passes on a retry counts as flaky, and the dot saves it to memory, so a later verdict can tell a flaky test from a real break.
+
+## Modes
+
+### Local
+
+The setup steps and the command run as child processes on the localdost host. With `workdir` set, it runs in that folder: the dot tests what is checked out there. A run for a commit (a PR head or a merge) counts only if that folder is on the commit; see Tests for the PR head below. Without `workdir`, localdost clones the repo at the commit under test into a temp folder and deletes it after the run. A local run in a clone needs dependencies and browsers. Without `setup`, a clone that has a `package.json` first runs the default install, `npm ci --no-audit --no-fund || npm install --no-audit --no-fund`, with browser downloads off and dev dependencies on, and the install cache makes the second run fast. The localdost `runner` image already has the browsers. On a host without them, put `npx playwright install chromium` in `setup`; setting `setup` replaces the default install, so list `npm ci` there too. A `workdir` is your own checkout and gets no default install.
+
+The child process, and every setup step, gets the host environment minus every variable that holds a credential of the server (KAT-361): names that contain `TOKEN`, `SECRET`, `PASSWORD`, `PRIVATE`, `API_KEY`, or `COOKIE`, names that end in `_KEY` or `_HEADERS`, and names that start with `AWS_`, `GITHUB_`, `JIRA_`, `LINEAR_`, `ANTHROPIC_`, `NTFY_`, `OPENAI_`, `SLACK_`, `ELEVENLABS_`, or `LOCALDOST_`. The header variable of each connection goes too, whatever it is called: the one a preset names (`KATALON_MCP_HEADERS` for Connect True Platform) and the `headersSecretRef` of every saved connection. The AI-Vision sign-in variables go too, under the default names (`AIVISION_INTERNAL_TOKEN`, `VEESAN_SESSION_COOKIE`, `VEESAN_USER`, `VEESAN_PASSWORD`) and under the names any dot gives in `runner.aivision` (`tokenEnv`, `cookieEnv`, `userEnv`, `passwordEnv`), archived dots included, because those names can be anything under `AIVISION_` or `VEESAN_`. `src/runner/host-env.ts` lists the names the server reads, and `test/runner-env.test.ts` starts a child process and checks it sees none of them. The AI-Vision CLI gets the same list. Only variables are filtered. The test process still has `HOME` and `PATH`, so PR code can use a login that lives in a file of the host user: the GitHub CLI login (`gh auth token`), or the Claude or Codex CLI login that bills a personal account. Test code runs with the host user's file access, so use local mode only for repos you trust, and see [Test login](#test-login) for the one set of variables a dot may pass on.
+
+`PORT` and `HOST` of the localdost server are not passed on, because a test project that reads `PORT` would start its app on the port of localdost. When the dot's base URL is on this machine and names a port (`http://localhost:3000`), the run sets `PORT=3000` and `HOST=localhost`, so a project that starts its own app finds it where the base URL points. A `PORT=` or `HOST=` word at the start of the command wins.
+
+When the command is `npm run <script>`, localdost adds the `--` npm needs to pass its options on only if the script ends in `playwright test`. Any other script runs with the arguments you wrote.
+
+### Docker
+
+The test runs in a container from the Playwright image, through the host's Docker daemon:
+
+- The checkout is mounted read-only at `/src` and copied into the container without `node_modules`, `.git`, and credential files (`.env*`, `.npmrc`, `.netrc`, `.aws`, `.ssh`, `*.pem`, `*.key`, `credentials.json`, and more).
+- The container runs the `setup` steps, each through `bash -c`, then the test command. Without `setup`, it runs `npm ci`, falling back to `npm install`. Docker runs do not use the install cache.
+- Only `CI`, `BASE_URL`, the report paths, and the [test login](#test-login) cross into the container. No other host variable and no localdost credential.
+- Limits: 4 GB of memory and 2 CPUs.
+- `host.docker.internal` points to the host, so tests can reach an app that runs there.
+- Docker covers your own specs and the test command. A test the model writes for a new scenario (the Playwright e2e engine, see [Generated Playwright specs](#generated-playwright-specs)) runs on the localdost host in every mode, not in the container. The dot form says so when Docker is picked.
+
+The container has network access, because `npm ci` needs the registry and tests need the app. A private registry must come from the image, since `.npmrc` stays behind.
+
+localdost needs to reach a Docker daemon. The Docker runner row on the Setup page detects it: when it shows Docker as not reachable, start Docker Desktop or the daemon and press **Check again**. On your machine that is Docker Desktop or the local socket. In Docker Compose, mount `/var/run/docker.sock` into the app container, as the comment in `compose.yml` shows. The socket gives the container root-equivalent control of the host, so mount it only on a machine dedicated to localdost. Railway has no Docker daemon.
+
+### Demo
+
+The run uses the bundled sample shop in `demo/sample-app` and its Playwright specs. A simulated merge turns on a seeded bug first, so one spec fails. The demo dot uses this mode.
+
+## Test login
+
+Tests for an app behind a sign-in need a user. Put the user's variables in the environment of the localdost server, and name them in `runner.passEnv`:
+
+```json
+{ "mode": "local", "passEnv": ["LOCALDOST_TEST_USER", "LOCALDOST_TEST_PASSWORD"] }
+```
+
+Each name must start with `LOCALDOST_TEST_` and use capital letters, digits, and underscores, so a dot cannot forward `GITHUB_TOKEN`, a model key, or a connection header (KAT-361). The server checks it when you save and again on every run. A local run and its setup steps, and a Docker container, get each name that is set on the host with `LOCALDOST_TEST_` replaced by `TEST_`: `LOCALDOST_TEST_USER` reaches the test as `TEST_USER`. A name that is not set is not passed, and a `NAME=value` word at the start of the test command wins. At most 20 names.
+
+The dot dialog has **Test login variables** in the Test runner section. It lists each name with what the test sees it as and whether the server has it set (`GET /api/runner/pass-env?names=A,B`). The answer holds names and a flag, never a value. After you set a variable on the host, restart the server: a process does not see a later change of its environment.
+
+The values are masked as `[redacted]` in the report, the live output, and the error text, because Playwright's call log of a failed `fill()` prints the text it typed. A value shorter than 4 characters is not masked. A variable named for a user or an email (`TEST_USER`, `TEST_ADMIN_EMAIL`) is not masked either, so a user called `admin` does not blank out every `/admin/users` in an error line; a name that also says `PASS`, `SECRET`, `TOKEN`, or `KEY` is masked, and so is any name this rule does not know. Traces and videos of a failed test can still show what the page showed, so make the test user a low-privilege account made for this. Generated Playwright specs get no host variables at all. An AI-Vision case gets no test login either: `userEnv` and `passwordEnv` sign localdost in to the AI-Vision server, never to the app under test, so a case opens the app signed out unless the AI-Vision profile it runs under already holds a session.
+
+## Live view
+
+While a local or demo run is in progress, the run page shows each test step as it happens. localdost adds a step reporter and writes a wrapper config `.localdost-run-<run id>-<attempt>.config.ts` next to your Playwright config. The wrapper imports your config and turns on screenshots, traces, and video. Your config file stays as it is. Video is kept for tests that did not pass; to keep the video of every test, set **Keep video of** to Every test in the dot's Test runner settings (`runner.video: "on"`). A video mode your own config asks for stays. Docker runs keep no video. It is deleted when the run ends. Add `.localdost-run-*` to your repo's `.gitignore` in case the server stops mid-run.
+
+With browser frames on, the wrapper also opens a DevTools port on Chromium projects and runs one worker, and the run page shows the page as the test drives it, about two frames per second. Frames are on by default for demo runs. Set `LOCALDOST_LIVE_FRAMES=1` for local runs. Docker runs show their log when the run ends and stream no frames.
+
+## Artifacts
+
+After each attempt, localdost copies screenshots, traces, videos, and the JSON report into `<LOCALDOST_DATA_DIR>/artifacts/<run id>/`. The runner records a trace of every test (`--trace=on`) and reads the pages each test visited from it into `observations.jsonl`, with one screenshot per page per test, for the Traceability map ([TRACE-MAP.md](TRACE-MAP.md#capture)). It then deletes the traces of tests that passed, so the run keeps traces only of tests that did not pass. The run page lists them, and `GET /api/runs/:id/artifacts` returns the list. Files are served from `/artifacts/<run id>/<file>` behind the same bearer token as the API, with a content security policy that blocks scripts. Open a trace in `npx playwright show-trace` or at trace.playwright.dev.
+
+### Evidence per test
+
+The report names the files of each test in `cases[].evidence`: `screenshots` (`{ name, step }`, in the order taken), `video`, and `trace`, all artifact names. On the run page, each test is a row under Steps. Open it to see its video in a player, its screenshots, and its trace. The Browser tab is a timeline of the frames of all tests; pick a frame to see it, and choose **Show test in Steps** to jump to the row of its test. Artifact files support byte ranges (`Range`), so the video player can seek.
+
+localdost does not delete artifacts on its own. Remove old run folders when the disk fills.
+
+## Cancel and rerun
+
+`POST /api/runs/:id/cancel` stops a running test process. `POST /api/runs/:id/rerun` queues the run again. A run whose server stopped mid-run ends as interrupted, and a rerun picks it up (the app shows Retry).
+
+On SIGTERM or SIGINT the server stops claiming jobs, marks every running run interrupted, and kills the process group of every git clone, setup step, test command, AI-Vision CLI, and model CLI it started, then exits. The process is gone within 10 seconds whatever is still running, and a second signal exits at once. Runs a crashed process left running are marked interrupted at the next boot. Queued jobs wait until the connections, the GitHub client, the model, and the HTTP server have loaded, so a run that starts during a restart does not fail its setup. A setup step that fails in under a second with no output at all is retried once; the failure report carries the exit code, signal, or spawn error and the last lines the step printed.
+
+## Tests for the PR head
+
+A dot with a `workdir` that is a git checkout of the PR's repository (one of its remotes names the repo) runs a PR check of that repository in a temporary git worktree at the PR head commit, so the test count belongs to the PR and not to whatever the folder has checked out (KAT-284). localdost fetches the commit, or `refs/pull/<n>/head`, when the folder does not have it. The worktree is a clean checkout: it gets the same install as a fresh clone (your `setup` steps, else the default install, with the install cache) and none of the untracked or ignored files of your folder, such as `.env` or a build output, so put what the tests need in `setup`, and list the config files that git does not carry in `copyFiles` ([Copy files](#copy-files)). A `workdir` inside the repo maps to the same sub folder of the worktree.
+
+A merge run (a push to the base branch, or a merged PR) works the same way: the merge commit is on the base branch of the repo itself, so it is trusted, and its tests run in a worktree at that commit. localdost fetches the commit, or the base branch, when the folder does not have it. A range a person asked for on demand can name any commit, so it keeps the rule below: the folder itself must be on the commit.
+
+The worktree lives in `<data folder>/worktrees/<run id>-a<attempt>`, never in your repo folder, and is removed when the run ends, also after a failure, a cancel, or a timeout. The checkout, branch, index, and files of your folder do not change; git only adds and removes a worktree entry under its `.git/worktrees`. The artifact cap, the disk watch, and the install cache never delete a worktree a run holds, and the server removes leftovers of a crash at start.
+
+After the worktree is made, the runner checks that it is on the head commit, and a count is reported only then. When the folder is not a checkout of the repo, the run needs the folder itself to be on the commit. When the commit cannot be fetched, no tests run for it and the report says so.
+
+A PR from a fork gets no own run (next section): the report says that no tests have run for the commit, and the results come from the CI of the PR.
+
+## Copy files
+
+A worktree has only what git tracks, so an untracked `.env` of your working folder is not there, and tests that read it fail. List such files in the dot's `runner.copyFiles` (**Copy files** in the Test runner section of the dot dialog, comma separated):
+
+```json
+{ "mode": "local", "workdir": "/home/me/shop-web", "copyFiles": [".env", "config/.env.ci"] }
+```
+
+Before the first setup step, localdost copies each listed file from the `workdir` to the same place in the worktree. The paths are relative to the `workdir`; for a `workdir` inside the repo, the worktree has the same sub folder. Nothing else of your folder is copied, and your folder is not changed. The rules, which the API and the dialog check on save (a bad value is a 400) and the runner checks again on every run:
+
+- Only relative paths of files. An absolute path, a drive letter, `~`, `..`, a backslash, a path inside `.git` (in any letter case, as `.GIT` is the same folder on a default macOS volume), and a folder are refused. At most 20 files, and a file larger than 1 MB is skipped.
+- The real path of the file, after symlinks, must be inside the `workdir`. A symlink that leads out of the folder is not followed and the file is skipped; a symlink that stays inside the folder is copied as the file it points to.
+- The copy never writes through a symlink of the worktree. If the commit has a symlink where a listed file or one of its folders should go, that file is skipped. This includes the folders from the worktree root down to the `workdir` (`apps/web`): a symlink on that path skips every file, and the real path of the destination must stay inside the worktree.
+- A file that does not exist is skipped, and the run goes on. Each copied or skipped file has one line in the setup output of the run ("copyFiles: copied .env into the worktree.", "copyFiles: .env.test is not in the working folder, skipped."), never its contents.
+- An existing file of the worktree with the same path is replaced by the file of your folder.
+
+Only runs in a worktree copy files: PR checks and merge runs of a `workdir` that is a checkout of the repo. A run in a fresh clone has no working folder to copy from. Docker mode copies the worktree into the container without credential files (`.env*` and more, see [Docker](#docker)), so a `.env` still does not reach the container; pass what the tests need in `setup` or the image.
+
+When a run in a worktree did not pass (a test failed, or the tests did not run) and the `workdir` has untracked dotfiles that are not in `copyFiles` (`.env`, `.env.local`, ignored files included), the report carries a `hint` that says untracked files were not copied and points to `runner.copyFiles`. The `hint` holds no file name, so the "tests did not run" comment and the PR comment can repeat it on a public repo. The names (`hintFiles`) are shown on the run page under Tests only. A passing run has no hint, and neither has a folder without such dotfiles.
+
+## Test branches
+
+A merge or push runs post-merge tests only on the default branch of the repository, as GitHub reports it. To test release branches too, list them in the dot's `testBranches` (`["release/1.0", "release/2.0"]`, or comma separated in the dot dialog). Empty or unset means the default branch only. A name is a plain branch name: no `refs/heads/` prefix and no wildcards, at most 20 names, and names are case sensitive. `POST /api/dots` and `PATCH /api/dots/:id` answer 400 for anything else.
+
+A merge or push to a listed branch is picked up like one to the default branch (a webhook, or the poller, which then also reads the new commits of each listed branch). The run is checked before anything is checked out:
+
+- The branch the event names counts only when it is listed (or is the default branch). The event is not trusted for anything else, so a forged event cannot name a branch of its own.
+- The commit must be an ancestor of that branch as fetched from the repository. With a `workdir`, localdost fetches the branch into a private ref and asks git; without one, it asks GitHub to compare the commit with the branch. A commit of another branch, an unpushed commit, or a branch that cannot be fetched is not run and no count is shown.
+- A branch that is not listed keeps the refusal. The commit is still checked against the default branch only, and the "tests did not run" reason names the branch and the setting, for example "It was sent for branch release/1, which is not the default branch (main) and is not listed in the dot's Test branches setting (testBranches), so only the default branch is tested."
+
+Everything else follows the default branch run: the worktree, the setup steps, the install cache, and the report. Generated specs that the dot proposes as a pull request after a merge target the listed branch the merge went to.
+
+## Fork safety
+
+A PR from a fork carries code from someone outside your repo. When a dot would clone and run that code, it skips the test run and records "Tests did not run: fork pull request" on the run. The PR comment and the check say "Tests did not run: this PR comes from a fork. Ask a maintainer to check it again." The rule for the Tests repo run of a fork PR is in [Tests repo](#tests-repo). This applies in local and docker modes without a `workdir`. With a `workdir`, the runner refuses a fork PR as well: the tests would run in your folder, and a count for other code is never reported. The report has the same line. Set `runner.trustForks: true` on the dot to run fork PRs anyway, for example when only your team can open PRs.
+
+The PR review still runs for fork PRs, because it only reads files and runs nothing (KAT-384). The report is not a pass: it says the tests did not run, and the results of the CI of the PR show in it once that CI has run.
+
+## AI-Vision cases
+
+A dot can also run natural-language browser cases next to Playwright, on an AI-Vision server (Veesan) or the open-source Midscene CLI. The cases come from the PR review: each What to test item of type e2e or exploratory that has no existing test becomes one case, and test ideas from a ticket review of a ticket the PR names (in its title, branch, or body) fill the rest. Each case is one YAML file with a start URL and `ai`, `aiAssert`, and `aiQuery` steps that describe what the user does and sees. The start URL is the PR preview URL or the dot's `baseUrl`, joined with the first route of the item's impact area. The cases run in `pr_check` and `post_merge` jobs, and their results merge into the same report and verdict as the Playwright results, as `aivision/<item id>.yaml`.
+
+```json
+{ "mode": "local", "workdir": "/home/me/shop-web", "aivision": { "backend": "server", "serverUrl": "http://localhost:5880", "profile": "smoke" } }
+```
+
+| Field | Default | Purpose |
+|---|---|---|
+| `backend` | | `server`, `cli`, or `midscene` |
+| `enabled` | `true` | Turns the cases off without removing the settings |
+| `on` | `["pr_check", "post_merge"]` | Jobs that run the cases |
+| `maxCases` | `5` | Upper bound on cases per run |
+| `timeoutMs` | 15 minutes | Per run |
+| `serverUrl` | the AI-Vision server on the Setup page (`AIVISION_SERVER_URL`) | `server`: base URL of the AI-Vision server |
+| `profile` | unset | `server`: run profile to bind |
+| `tokenEnv`, `cookieEnv` | `AIVISION_INTERNAL_TOKEN`, `VEESAN_SESSION_COOKIE` | `server`: names of the env vars that hold an internal token or a signed-in session cookie |
+| `userEnv`, `passwordEnv` | `VEESAN_USER`, `VEESAN_PASSWORD` | `server`: names of the env vars for localdost' own sign-in to the AI-Vision server on a 401. They never sign in to the app under test |
+| `checkout` | unset | `cli`: AI-Vision checkout; the CLI is `packages/cli/bin/veesan` in it. `VEESAN_CLI` wins |
+| `midsceneVersion` | `1.14.0` | `midscene`: exact `@midscene/cli` version for `npx` |
+| `casesDir` | temp folder | `cli` and `midscene`: where the YAML is written |
+| `passEnv` | `[]` | `cli` and `midscene`: extra env var names passed to the CLI |
+
+The dot stores only env var names, never a token, cookie, or password. The four `server` env refs must name a variable that starts with `AIVISION_` or `VEESAN_`; any other name fails the run, so a dot config cannot send `GITHUB_TOKEN` or a model key to the server URL it names. When an AI-Vision server is connected on the Setup page, the default refs go only to that server; a dot that points at another server must name its own refs to send credentials there.
+
+### Server backend
+
+localdost writes each case into the server's tests repo under the `localdost/` category, with a file name that stays the same for the same dot and content, so a rerun overwrites it. Then it starts one run over all cases, polls it every 3 seconds, and streams each finished step and its screenshot to the run page. A failed case carries the failed step, its error, and a link to the server's report. Each HTTP request times out after 60 seconds, and screenshots over 10 MB, or past 200 MB for one run, are skipped. When the server stops a run to ask a question, localdost stops the run and reports the cases as skipped, since nobody is there to answer. Cancel on the run page stops the server run.
+
+### CLI and Midscene backends
+
+localdost writes the YAML into a temp folder, or `casesDir`, and runs the CLI once: `<cli> --summary <file> --continue-on-error --files <yaml...>`. It reads the summary JSON for the verdicts and keeps each case's HTML report as a run artifact. The CLI output streams to the run log.
+
+- `cli` runs the AI-Vision CLI from `VEESAN_CLI` or `<checkout>/packages/cli/bin/veesan` with the Node that runs localdost.
+- `midscene` runs `npx --yes @midscene/cli@<version>`. Set the model with the Midscene variables on the localdost host, for example `MIDSCENE_MODEL_NAME`, `MIDSCENE_MODEL_BASE_URL`, and `MIDSCENE_MODEL_API_KEY`. Puppeteer needs a Chrome; set `PUPPETEER_EXECUTABLE_PATH` to use an installed one.
+
+The CLI gets the host environment minus credentials, by the same rule and the same list as a local Playwright run (see [Local](#local)). Variables that start with `MIDSCENE_` or `VEESAN_MODEL_`, `PUPPETEER_EXECUTABLE_PATH`, and the names in `passEnv` are passed on purpose, model keys included.
+
+### Prompt safety
+
+Review items come from a model that read the PR, so PR text can reach a prompt. The YAML engines expand `${NAME}` to an environment variable, evaluate `{{expression}}` as JavaScript, and resolve `<<KEY>>` to a vault secret. localdost turns every brace in a prompt into a parenthesis and splits doubled angle brackets, so none of the three can form, and it percent-encodes the same characters in the start URL. Every value is written as a quoted YAML string. Only http and https start URLs without credentials are used; a case without one is reported as skipped.
+
+## Generated Playwright specs
+
+In the app this engine is called **Playwright e2e**. A dot can write real Playwright specs for What to test items that no existing spec covers, run them, and report them next to the repo specs and AI-Vision cases. Each item then shows the engine that covered it: an existing spec, a generated Playwright spec, or an AI-Vision case.
+
+The engine is a per-dot setting, `runner.engine`, also in the dot dialog, under the Change reviewer stage, as **Engine for new scenarios**. Existing Playwright specs in the repo always run through the test command; the engine covers new scenarios only. When a dot sets none, it uses the workspace default from the Setup page, see [Test engines](SETUP.md#test-engines):
+
+| Value | Label in the app | What runs items without an existing spec |
+|---|---|---|
+| unset | Uses the workspace default | The workspace default (`LOCALDOST_TEST_ENGINE`, saved on Setup). While none is chosen it is `aivision`, so a dot behaves as before |
+| `auto` | Playwright e2e, then AI-Vision | A generated Playwright spec first; AI-Vision runs the items no kept spec covered |
+| `playwright` | Playwright e2e only | Generated Playwright specs only |
+| `aivision` | AI-Vision only | AI-Vision cases only, when `runner.aivision` is set |
+| `both` | Both on every item | Both engines on every item. The worse result stands |
+
+```json
+{ "mode": "local", "workdir": "/home/me/shop-web", "engine": "auto", "pwgen": { "maxSpecs": 3 } }
+```
+
+| `runner.pwgen` field | Default | Purpose |
+|---|---|---|
+| `on` | `["pr_check", "post_merge"]` | Jobs that generate specs |
+| `maxSpecs` | `3` | Upper bound on generated specs per run, at most 10 |
+| `timeoutMs` | 15 minutes | The whole generation stage per run. Exploration, model calls, and spec runs stop at it |
+| `proposePr` | `true` | Propose kept passing specs as a pull request through the `open_pr` rule |
+
+The loop follows the Playwright test agents (planner, generator, healer; `npx playwright init-agents`), run inside localdost with its own model backend and no agent tools. One difference is on purpose: the Playwright healer may change expected values until a test passes, and the localdost healer may not, so a product bug cannot be healed away.
+
+1. Plan. The PR review is the plan: the item's scenario, steps, expected result, and test data. Items of type e2e or exploratory without an existing spec qualify, must before should before could.
+2. Explore. Headless Chromium opens the preview URL, or the dot's `baseUrl`, at `/`, the item's area routes, paths written in its steps (for example `/member?tier=gold`), and links from the first page, up to six pages. It records each page's ARIA snapshot, its `data-testid` elements, and its links. Links that read like log out, delete, or reset are never followed.
+3. Generate. The model writes one `.spec.ts` with one `test()` from the item and the page snapshots, into `data/artifacts/<run id>/pwgen/<item id>/`.
+4. Run. Playwright Test (the copy localdost ships) runs it once against the target URL with `BASE_URL` set, Chromium, no retries, and a trace and screenshot on failure.
+5. Heal. When the spec fails for its own reason (a locator that matched nothing or several elements, a timeout, a syntax error), the model repairs it once from the error and the page snapshot Playwright wrote on failure. A repair that changes an expected value (in a matcher, or a value such as `'$6.40'` in the located element), drops an assertion, or turns one into a check for absence is rejected.
+6. Keep or discard. A spec is kept when it passes, or when a value assertion found its element and read another value (a product failure). A value failure goes to the healer once as a check: when it leaves the spec as it is, or its repair fails the same way, the product failure stands; when its repair passes, the item gets no generated result, since the repair may read another element that shows the expected value. A spec that still fails for its own reason on a preview runs once on the base URL: when it passes there, the flow broke in the change, and the spec is kept as failed. Any other spec is discarded, and its item shows why.
+
+Results land in the run report as `pwgen/<item id>.spec.ts`, and the final source of each spec is a run artifact (`pwgen-<item id>.spec.ts`), with screenshots and traces next to it. The sticky comment marks each item, for example `passed (checked by a Playwright e2e test)`.
+
+Generation needs a model that writes text: the Claude CLI, the Codex CLI, or an Anthropic key. With the mock backend, every item is discarded with that reason, and `auto` sends all of them to AI-Vision.
+
+### Proposing specs
+
+When kept specs passed, the run drafts a pull request that adds them under `<spec folder>/localdost/`, where the spec folder is the top folder most existing e2e specs live in (else `tests`). On a pull request the proposal targets the PR branch, so the specs land with the change they test; after a merge it targets the default branch. The PR opens as a draft. Approving again after a failed send moves the same branch and reuses an open PR for it. The draft goes through the `open_pr` rule: with the default `act_when_asked` it is recorded on the run and skipped, with `ask` it waits in the Inbox for approval, and with `act` it opens after the auto-review. Fork pull requests get no proposal.
+
+### Spec safety
+
+A model wrote the spec from text a PR author can shape, and it runs on the localdost host, so a spec must pass these checks before it runs. A spec that fails them is sent back once with the problems listed, then discarded:
+
+- One import, from `@playwright/test`. No `require`, dynamic import, or `import.meta`.
+- Names come from an allowlist: the spec's own `const` and `let` names and arrow parameters, `test`, `expect`, `page`, and a few plain helpers (`Math`, `Number`, `String`). Properties come from an allowlist too: Playwright page, locator, keyboard, and assertion calls, `test.step`, and string, number, and array helpers. So `evaluate`, `route`, `context`, `request`, `constructor`, `test.fail`, and `test.skip` all fail the check, and host names such as `process`, `globalThis`, `Object`, and `eval` are refused by name. Words inside strings, regexes, and comments do not count.
+- No bracket property access other than a number index, and no absolute URLs. `page.goto()` takes one quoted path that starts with a single `/`.
+- Exactly one `test()` and at least one `expect()`.
+
+The spec process gets only `PATH`, `TMPDIR`, `LANG`, `TZ`, `BASE_URL`, `CI`, and `PLAYWRIGHT_BROWSERS_PATH`, never a token or key, and its `HOME` is the scratch folder. These checks narrow what a spec can do; they are not a sandbox, and generated specs run on the host in every runner mode. Keep the engine unset for repos where people you do not trust write PR text, or run localdost as a user whose files hold no credentials.