@asrunjs/asrun 0.2.0 → 0.5.3

package/docs/install.md

@@ -0,0 +1,174 @@
+# Installing AsRun
+
+AsRun is a normal devDependency. A **workspace** is the folder that owns `profiles.yaml`, `cases/` and `tests/`;
+everything AsRun needs at run time is read through it. The framework itself lives in `node_modules/@asrunjs/asrun`
+and is never edited: a project commits only its own test content.
+
+## Requirements
+
+- **Node.js 22 or newer** and **pnpm**.
+- **ffmpeg** joins the recordings of one test (a new browser tab has its own video) into one video for the report.
+  `pnpm go` installs it with Homebrew on macOS or winget on Windows; on Linux run `sudo apt install ffmpeg`. When it
+  is missing, the optional `ffmpeg-static` package is used instead; if neither works, the report lists the videos
+  separately. Not fatal.
+- **Claude Code or OpenCode** for the AI steps (make cases, explore, generate, heal). Everything else works without
+  them; `pnpm go` warns when neither command is found.
+
+## Install in a project
+
+```bash
+pnpm dlx @asrunjs/asrun init --install
+pnpm go
+```
+
+`init --install` scaffolds the workspace and then adds the dependency. Running `init` first matters on pnpm 12: it
+writes `pnpm-workspace.yaml` with the `allowBuilds` entry (`esbuild` through `tsx`, `ffmpeg-static`), and without it
+`pnpm add` exits with `ERR_PNPM_IGNORED_BUILDS`. An existing `pnpm-workspace.yaml` that lacks the entry gets a note
+instead: add the block by hand (see "Manual install").
+
+`pnpm go` then does the rest and is the command to remember:
+
+1. checks Node (22+) and ffmpeg, and warns when no AI CLI is installed;
+2. installs the dependencies and the Chromium browser (`pnpm install` + `playwright install chromium`);
+3. runs `asrun init` (workspace files and `.env`) and `asrun doctor`;
+4. starts the web UI at http://127.0.0.1:4173 and opens it.
+
+Run it again on any day: the install step then takes seconds, and if the UI is already running it only opens it.
+`pnpm go --no-open` does not open a browser. Set `ASRUN_UI_PORT` to change the port.
+
+The UI is optional: all commands in [commands.md](./commands.md) work without it.
+
+### Manual install
+
+Same result, step by step:
+
+```bash
+pnpm add -D @asrunjs/asrun
+pnpm asrun init
+pnpm go
+```
+
+If pnpm refuses the build scripts of `esbuild` / `ffmpeg-static`, add this to `pnpm-workspace.yaml` and run the
+install again:
+
+```yaml
+allowBuilds:
+  esbuild: true
+  ffmpeg-static: true
+```
+
+To pin a version:
+
+```bash
+pnpm add -D @asrunjs/asrun@<version>
+```
+
+`pnpm asrun init` is safe to run many times: it never overwrites a file the project owns.
+
+## What `init` writes
+
+| Kind | Files | On `--update` |
+|---|---|---|
+| Generated | `playwright.config.ts`, `tests/seed.spec.ts`, `.claude/commands/*`, `.opencode/commands/*` | rewritten |
+| Once | `AGENTS.md`, `CLAUDE.md`, `.mcp.json`, `opencode.jsonc`, `.claude/settings.json`, `tsconfig.json`, `pnpm-workspace.yaml`, `profiles.yaml`, `cases/_template.md`, `package.json` (scripts and devDependencies merged in) | kept |
+
+It also writes the asrun block in `.gitignore`, the `ASRUN_ROOT` line in `.env`, and `.asrun.json` (the marker with
+the framework version and the generated file list).
+
+Flags:
+
+| Flag | Meaning |
+|---|---|
+| `--root <dir>` | initialize that folder instead of the current one |
+| `--update` | refresh only the generated files |
+| `--force <path>` | overwrite one file init owns (repeatable) |
+| `--reset-rules` | allow `--force` on `AGENTS.md`, `CLAUDE.md` or `.claude/skills/**` |
+| `--no-env` | do not write `.env` |
+| `--install [spec]` | also add the dependency (default `@asrunjs/asrun`) |
+
+`AGENTS.md`, `CLAUDE.md` and `.claude/skills/**` belong to the project. `init --force` refuses them without
+`--reset-rules`, so a framework update can never replace a project's rules.
+
+## Update
+
+```bash
+pnpm add -D @asrunjs/asrun@latest
+pnpm asrun init --update
+pnpm asrun doctor
+```
+
+`init --update` rewrites only the generated files and leaves project rules, case files, values and baselines alone.
+`doctor` tells you when the workspace is behind the installed version.
+
+There is no folder to move: the framework lives in `node_modules`, which the package manager owns. To change the
+framework itself, `pnpm patch @asrunjs/asrun` (a committed patch).
+
+## `pnpm asrun doctor`
+
+Run it after changing profiles, rules or overrides, and after an update. It checks:
+
+- every entry in `profiles.yaml` (ids, environments, URLs, session check settings);
+- `tsconfig.json`, `playwright.config.ts` (its re-export of the framework config) and the generated files;
+- that `.gitignore` ignores `.auth/` and `reports/`;
+- whether the installed version is behind the package's templates, and whether `dist/` is out of date;
+- the project rule table in `AGENTS.md` §2 (unknown ids, a missing `Effect`, a non-negotiable rule);
+- the framework skills and the project overrides under `.claude/skills/<skill>.md`, and lists the ones it sees.
+
+## Which workspace a command runs against
+
+The order is:
+
+1. a real environment variable `ASRUN_ROOT` (it wins over everything);
+2. `ASRUN_ROOT` in the `.env` next to the command, or in the framework package's own `.env` (relative to that file);
+3. otherwise the nearest folder at or above the working directory that has `profiles.yaml` or `.asrun.json`;
+4. in the framework repository, the framework's own `.env` points at `examples/demo`.
+
+A real environment variable always wins, so one checkout can point at another project for a single command:
+
+```bash
+ASRUN_ROOT=../my-real-project pnpm asrun --env staging --read-only
+```
+
+## Workspace layout
+
+```
+profiles.yaml       one entry per environment: URL, account label, tags, environment (no expected values)
+cases/              human-written cases (+ _template.md); optional group folders: cases/<group>/<name>.md
+tests/              generated specs (same folders as cases/) and *-snapshots/ baselines per profile
+playwright.config.ts  re-exports node_modules/@asrunjs/asrun/dist/playwright.config.js
+tsconfig.json       plain compiler options (specs import the framework by package name)
+AGENTS.md           project rules: they point at the framework baseline; §2 says what differs
+CLAUDE.md           imports AGENTS.md for Claude Code
+.claude/commands/   /make-case, /explore-case, /gen-spec, /heal-spec (thin shells that load the framework skill)
+.claude/skills/     optional project overrides of a skill: <skill>.md
+.opencode/commands/ the same commands for OpenCode (it does not read .claude/commands)
+.mcp.json           the playwright-test MCP server for Claude Code
+opencode.jsonc      the same for OpenCode, plus an optional model
+.asrun.json         marker: framework version and the generated file list
+package.json        the workspace scripts (`asrun`, `signin`, ...) and devDependencies
+.env                ASRUN_ROOT (which workspace the commands use) and local settings
+node_modules/@asrunjs/asrun/   the framework (installed dependency; never edited)
+.auth/              saved logins, one per profile
+reports/            one folder per run; reports/latest points at the newest
+.tmp/ .explore/     job logs, hints, reference screenshots (local state)
+```
+
+The framework repository's own layout is in [agents/framework-dev.md](./agents/framework-dev.md).
+
+## What to commit
+
+Commit `profiles.yaml`, `cases/`, `tests/` (with `*-snapshots/`), `AGENTS.md`, `CLAUDE.md`, `.claude/`,
+`.opencode/commands/`, `.mcp.json`, `opencode.jsonc`, `.asrun.json`, the generated files, `package.json`, `tsconfig.json`
+and `pnpm-workspace.yaml`.
+
+Never commit or share `.auth/` (one saved login per profile, as sensitive as a password), `reports/`, `.tmp/`,
+`.explore/`, `.env` or `node_modules/`. The asrun block of `.gitignore` covers them.
+
+## Uninstall
+
+```bash
+pnpm remove @asrunjs/asrun
+```
+
+Then delete the workspace files you no longer want (`profiles.yaml`, `cases/`, `tests/`, the generated commands and
+configs, `.asrun.json`). They are ordinary files: keep them if another tool still uses them.

package/docs/profiles.md

@@ -0,0 +1,139 @@
+# Profiles and environments
+
+A **profile** is one environment of the system under test: a URL, an account label, optional tags and the environment
+name. Each entry in `profiles.yaml` becomes one Playwright project named after its `id`, and every spec runs once per
+profile. Two profiles of the same site give you two independent views, logins and baselines.
+
+```yaml
+profiles:
+  - id: example-staging
+    name: Example (staging)
+    env: staging
+    baseURL: https://staging.example.com
+    account: qa@example.com
+    tags: [core, us]
+    email: false
+    startPath: /app/home
+    sessionCheckPath: /settings
+    sessionCheck:
+      loggedOutUrl: /login
+      loggedInSelector: "input[name='username']"
+      timeoutMs: 5000
+```
+
+## Fields
+
+| Field | Meaning |
+|---|---|
+| `id` | Lowercase words with single dashes. It names the Playwright project, the saved session (`.auth/<id>.json`) and the baseline files. |
+| `name` | What people see in the UI and the reports. |
+| `env` | The environment this profile points at (`production`, `staging`, ...). `--env <env>` selects the profiles of one environment. Any environment except `staging` protects writing: a case that writes there needs a person's confirmation on every run. |
+| `baseURL` | The site address, without a path. Must be a full `http(s)` URL. |
+| `account` | A label for humans (a mailbox or username). Never a password: logins live in `.auth/`. |
+| `tags` | Optional lowercase words, for grouping and `pnpm asrun --tag <tag>`. Duplicates are refused. |
+| `email` | `true` only when a real mailbox is connected, so tests may send email. Defaults to `false`; `true` is refused for `production`. |
+| `startPath` | The page "Visit" opens after a login, for example `/app/home`. Empty (the default) opens the site root. A path on the site: one leading `/`, no spaces, never another host. |
+| `sessionCheckPath` | A page a signed-in visitor sees (default `/account/`). "Verify" and the pre-run check use it to tell whether a saved login still works. |
+| `sessionCheck` | How that page shows a missing login. See below. |
+
+Expected values are **not** here. They live in each case's `## Values` block: `profiles.yaml` holds connection
+settings only, so adding a profile never touches a case and two profiles can never fight over one case's values.
+
+## There is no default environment
+
+`pnpm asrun` and every AI command require `--env <env>`; the Run tests page requires one radio button. A run selects
+the profiles whose `env` matches.
+
+`pnpm signin <profile-id>` and `pnpm explore <profile-id>` take no `--env`: the profile already names its environment.
+
+## Logins
+
+Sessions are saved per profile in `.auth/<profile-id>.json`, one file per profile, and are as sensitive as passwords.
+
+```bash
+pnpm signin example-staging
+```
+
+A normal browser opens with a small card in its bottom-left corner. Log in yourself, including 2FA, wait for the
+logged-in page, then press **Done, save the session** on the card (or press Enter in the terminal, which does the
+same). The session is saved and verified. Nothing about 2FA is automated.
+
+`pnpm status` shows the login state of every profile; `pnpm status --env staging` only of one environment.
+
+Before every run, `pnpm asrun` checks the session of each profile in the chosen environment. A profile without a
+saved session, or with an expired one, fails right away (no retries) and the run prints what to do:
+
+```
+Not logged in (tests for these profiles will FAIL):
+  example-production           expired  -> pnpm signin example-production
+Or skip them for this run:  pnpm asrun --env production --skip-profile example-production
+```
+
+Either log in or leave the profile out (`--skip-profile`, or `--logged-in` to keep only the working ones). If a
+session expires in the middle of a run, the failed test is marked "session expired".
+
+Login itself is not a test case. Never write a spec for the login page or the login flow.
+
+### Making the session check fit your site
+
+The defaults fit a site that redirects a signed-out visitor to `/login`. Sites differ: a single-page app redirects
+after it has started, some answer `401` or `403`, some show the login form at the same address. `sessionCheck` names
+what your site does:
+
+| Key | What it says |
+|---|---|
+| `loggedOutUrl` | Where a signed-out visitor ends up (default `/login`): a path on the site or a full URL for single sign-on. A `401`/`403` response counts too. |
+| `loggedInSelector` | A selector that only exists while signed in, for example `"input[name='username']"`. |
+| `loggedOutSelector` | A selector that proves the visitor is signed out, for example the password field of a login form shown at the same address. |
+| `timeoutMs` | How long to wait for one of those signals (default 3000, 500–60000). |
+
+Without a `loggedInSelector`, "not sent to the login page" counts as signed in, which can mislead on a slow
+single-page app.
+
+## Visit and explore
+
+**Visit** (Profiles page, or `pnpm explore <profile-id> [path]`) opens a logged-in browser window at the profile's
+`startPath`, or at the site root when it is empty. Files downloaded in that window go to `~/Downloads` (an existing
+file is never overwritten; a ` (1)` is added) and the saved path is printed in the terminal or the job log.
+
+Use it when you need to see what a logged-in person sees, without writing anything. The exploring agent uses the same
+mechanism, with the Playwright MCP server and the profile's project.
+
+## Adding a profile
+
+1. Add an entry to `profiles.yaml` (or use **Add a new profile** on the Profiles page, which appends it for you).
+2. `pnpm signin <id>` and log in.
+3. Add the values for that profile to the cases it runs: a column in each `## Values` block, or nothing when
+   `default` already fits.
+
+Deleting a profile removes its entry and its saved login; screenshot baselines and old reports keep the id in their
+file names and stay on disk. Duplicating one copies the settings under a new id and does **not** copy the saved login.
+
+## How a profile becomes a test
+
+The profile id is the Playwright project name. The `profile` fixture exposes the running profile to a spec:
+
+```ts
+// case: cases/example.md
+import {caseTags} from '@asrunjs/asrun/cases';
+import {configureCase, expect, test} from '@asrunjs/asrun/fixtures';
+import {requireValues} from '@asrunjs/asrun/values';
+
+test.describe('Example', {tag: caseTags('example')}, () => {
+  configureCase('example');
+  test('one outcome', async ({page, profile}) => {
+    // profile.id, profile.env, profile.baseURL, profile.account, profile.tags, profile.email
+    const expected = requireValues<{homeTitle: string}>('example', profile);
+    await test.step('1. Open the home page', async () => {
+      await page.goto('/');
+      await expect(page).toHaveTitle(expected.homeTitle);
+    });
+  });
+});
+```
+
+Expected values come from the case document through `requireValues(caseId, profile)`: its `## Values` block with
+`default` deep-merged under the profile's own entry. A case with no values for the running profile, or with empty or
+`TODO` values, fails with a plain-language message instead of a crash.
+
+See [cases.md](./cases.md) for the document format and [commands.md](./commands.md) for the run filters.