ccqa 0.10.1 → 0.11.0

package/README.md

@@ -4,10 +4,8 @@
 
 ccqa turns Claude Code into a browser test recorder. Write a spec in YAML, declare in the spec whether it should run **deterministic** or **live**, then `ccqa run` does the right thing per spec:
 
-- **Deterministic** (`mode: deterministic`, default): record once with `ccqa record`. Claude drives the browser, ccqa compiles every action into a `test.spec.ts` you can replay in CI under vitest — no LLM at run time. Cheapest and most stable.
-- **Live** (`mode: live`): no codegen. `ccqa run` sends each step to Claude every time, Claude drives `agent-browser` directly, judges pass/fail against the step's `expected`, and saves a before/after screenshot. More flexible for fragile UIs.
-
-A single project mixes both: each spec.yaml picks its own mode, and `ccqa run` reads the field and dispatches. The HTML report covers both in one page.
+- **Deterministic** (`mode: deterministic`, default): record once with `ccqa record` — Claude drives the browser and ccqa compiles every action into a `test.spec.ts` you replay in CI under vitest, no LLM at run time. Cheapest and most stable.
+- **Live** (`mode: live`): no codegen. `ccqa run` sends each step to Claude every time; Claude drives `agent-browser`, judges pass/fail against the step's `expected`, and saves a before/after screenshot. More flexible for fragile UIs.
 
 No extra API key. Just `claude`.
 
@@ -15,6 +13,8 @@ No extra API key. Just `claude`.
 
 ## How it works
 
+Each spec picks its own mode; `ccqa run` reads the field and dispatches. One project mixes both, and one report covers both.
+
 ```mermaid
 flowchart LR
     A["Write spec\n(spec.yaml + mode:)"] --> B{mode}
@@ -23,10 +23,6 @@ flowchart LR
     B -- live --> E["ccqa run\n(Claude drives every time,\nper-step pass/fail)"]
 ```
 
-For deterministic specs, `record` invokes Claude Code with your spec, Claude drives the browser step by step, every action is recorded, and a vitest-compatible script is generated. `run` then replays it without involving an LLM.
-
-For live specs, `record` is not needed. `run` directly sends each step to Claude, which drives the browser through `agent-browser`, judges whether the step's `expected` clause holds, and writes a PNG before and after each step. Useful when codegen is fragile (timing-dependent UIs, rich-text editors, dynamic selectors).
-
 ## Install
 
 ```bash
@@ -69,11 +65,9 @@ ccqa run tasks/create-and-complete      # vitest replays test.spec.ts; no LLM
 ccqa run tasks/create-and-complete      # Claude drives the browser every time
 ```
 
-Live specs can start already-signed-in by pointing `statePath:` at a saved agent-browser state file (cookies + localStorage). Run an interactive login locally once, save the state with `agent-browser state save .ccqa/sessions/<name>.json`, then commit the path (not the file) — see [Pre-authenticated state](#pre-authenticated-state-statepath) below for the local bootstrap and the CI restore pattern.
-
-By default deterministic runs write step-boundary screenshots and metadata to `ccqa-report/evidence/<feature>/<spec>/` so a reviewer can confirm a passing spec actually reached the states its `expected` clauses describe. Disable with `--no-evidence`.
+Live specs can start already-signed-in by naming a saved session with `session:` — see [Saved sessions](./docs/sessions.md). Deterministic runs also write step-boundary screenshots to `ccqa-report/evidence/` (disable with `--no-evidence`).
 
-In CI you can opt in to an HTML run report by passing `--report` — every failing spec gets a drift audit plus a root-cause call (TEST_DRIFT / SPEC_CHANGE / PRODUCT_BUG) using the branch's git diff as context, and the report lets a human grade those calls to measure their accuracy. Requires `ANTHROPIC_API_KEY` or a local Claude login for the analysis part. Opt out with `--no-failure-analysis` (which also implicitly skips the drift audit — the audit is rendered as evidence under the classification, so without the classification the cost has nowhere to land). Use `--no-drift-audit` to keep the classification but skip the audit. See [Run report](./docs/report.md).
+In CI, add `--report` to write a `report.json` (+ evidence PNGs) you push to the [ccqa hub](./docs/hub.md) and view in its UI; each failing spec gets a root-cause call (TEST_DRIFT / SPEC_CHANGE / PRODUCT_BUG) you can grade for accuracy. Needs `ANTHROPIC_API_KEY` or a local Claude login. See [Run report](./docs/report.md).
 
 ```bash
 ccqa run tasks/create-and-complete --report --base origin/main
@@ -85,6 +79,8 @@ ccqa run --changed --report                    # only specs whose relatedPaths t
 | Feature | Docs |
 |---|---|
 | Write specs interactively with Claude | [Draft](./docs/draft.md) |
+| Run specs live (no codegen), with per-project guidance | [Live specs](./docs/live.md) |
+| Restore a saved login to skip device-trust gates | [Saved sessions](./docs/sessions.md) |
 | Reuse login and other shared step sequences | [Blocks](./docs/blocks.md) |
 | Drive `<input type="file">` without an OS picker | [File upload](./docs/file-upload.md) |
 | Assertion helper functions | [Assertions](./docs/assertions.md) |
@@ -93,6 +89,7 @@ ccqa run --changed --report                    # only specs whose relatedPaths t
 | HTML run report with failure root-cause calls | [Run report](./docs/report.md) |
 | Inventory existing test coverage | [Perspectives](./docs/perspectives.md) |
 | Architecture decision records (why it is built this way) | [ADR](./docs/adr/README.md) |
+| Aggregate CI run results, sessions, and variables on a shared server | [Hub](./docs/hub.md) |
 
 ## Commands
 
@@ -105,63 +102,44 @@ ccqa run [feature/spec...]         Execute specs. Per spec, the spec.yaml `mode:
                                    (vitest replay) or live (Claude drives every time). One run can mix both;
                                    `--report` writes one unified HTML. Pass multiple targets space-separated.
 ccqa drift [feature/spec]          Standalone spec ↔ codebase static audit (for PR checks)
+ccqa serve                         Start a hub: a control-plane HTTP server that aggregates run results,
+                                   sessions, and variables (it does not execute tests)
+ccqa hub push                      Push a finished run's report to a hub
+ccqa hub session|var <cmd>         Manage sessions/variables stored on a hub
 ```
 
-`ccqa run` flags:
-
-- `--report [dir]` — write a self-contained HTML run report (default dir: `ccqa-report/`)
-- `--profile <name>` — load `.ccqa/profiles/<name>.env` into the environment before resolving spec `${VAR}` references, so one spec targets dev/stg/prd without per-environment copies. See [Profiles](#profiles---profile).
-- `--changed` — restrict execution to specs whose `relatedPaths` intersect `git diff <base>...HEAD`. Mutually exclusive with explicit spec targets.
-- `--concurrency <n>` — run up to N specs in parallel **within each mode** (deterministic specs run as one phase, live specs as the next; parallelism is within a phase, not across). Default `1` (sequential, identical to before). Above 1, each spec's output is buffered and flushed as a labelled block so parallel logs stay legible. Live specs each launch their own headed Chrome, so high values spawn many browser instances.
-- `--base <ref>` — base ref for the git diff (default: `$GITHUB_BASE_REF`, then `origin/main`)
-- `--no-failure-analysis` — skip the per-failure root-cause classification (also skips the drift audit, since the audit only shows under the classification)
-- `--no-drift-audit` — skip the spec ↔ code drift audit while keeping the classification
-- `--no-evidence` — (deterministic specs only) skip step-boundary PNG capture
-- `--retry <n>` — (live specs only) retry each failing step up to N more times
-- `--format <fmt>` — `text` (default), `json` (report.json), `github` (Actions annotations)
-- `--out <dir>` — (live specs only, single-spec invocations) override the per-run artifact directory
-- `--update-agent-prompt` — (live specs only) after the run, summarise it back to Claude and rewrite `.ccqa/prompts/live.agent.md` so the next run inherits the lessons learned. `ccqa record` ships the same flag, refreshing `record.agent.md` from the trace summary.
+Key `ccqa run` flags (see `ccqa run --help` for the rest):
 
-All Claude-driven commands accept `-m, --model <name>` (alias `sonnet` | `opus` | `haiku`, or a full model ID). The flag overrides `CCQA_MODEL`; when both are unset, the Claude Code CLI default is used. They also accept `--language <bcp47>` (e.g. `ja`, `en`) to set the language of human-readable output; the default `auto` follows the language of the spec/codebase. `--cwd <path>` works on `record` / `run` / `drift` so you can target a subpackage inside a monorepo from the repo root. Interactive commands authenticate via your local Claude Code login; commands that talk to Claude in CI (`ccqa run --report`, `ccqa drift`) additionally honor `ANTHROPIC_API_KEY`.
+- `--report [dir]` — write the run report (report.json + evidence PNGs) for `ccqa hub push` (default dir: `ccqa-report/`)
+- `--profile <name>` — load `.ccqa/profiles/<name>.env` before resolving spec `${VAR}` references, so one spec targets dev/stg/prd. See [Profiles](#profiles---profile).
+- `--changed` — restrict execution to specs whose `relatedPaths` intersect `git diff <base>...HEAD`
+- `--concurrency <n>` — run up to N specs in parallel **within each mode** (deterministic phase, then live phase — parallelism is per-phase, not across phases). Default `1` (sequential, same behavior as before).
+- `--no-failure-analysis` / `--no-drift-audit` — `--no-failure-analysis` skips failure classification (and its drift audit, since drift is the classification's evidence); `--no-drift-audit` keeps classification but skips just the audit.
+- `--format <fmt>` — `text` (default), `json` (report.json), `github` (Actions annotations)
 
-`<feature/spec>` is a 2-segment alias for the on-disk path `.ccqa/features/<feature>/test-cases/<spec>/`. `ccqa run` accepts several targets space-separated (each a `<feature>/<spec>`, a bare `<feature>` for all its specs, or omitted for everything); duplicates are de-duped and `--changed` cannot be combined with explicit targets.
+`<feature/spec>` is a 2-segment alias for `.ccqa/features/<feature>/test-cases/<spec>/`; `ccqa run` takes several space-separated targets (a `<feature>/<spec>`, a bare `<feature>`, or none for everything). Claude-driven commands accept `-m/--model`, `--language`, and `--cwd` (for monorepos); they use your local Claude login, and CI commands (`ccqa run --report`, `ccqa drift`) also honor `ANTHROPIC_API_KEY`.
 
 ## File structure
 
 ```
 .ccqa/
-  perspectives.yaml              # Inventory of existing coverage (machine-readable, canonical)
-  perspectives.md                # Category index, regenerated from the YAML
-  profiles/                      # `--profile <name>` env files
-    stg.env                      # URLs + credential refs; commit if it uses secret-manager refs, gitignore if it holds plaintext secrets
-    prd.env
-  prompts/                       # Run `ccqa init` to scaffold these
-    record.user.md               # Human-maintained guidance appended to `ccqa record` (trace phase)
-    record.agent.md              # Auto-updated by `ccqa record --update-agent-prompt`
-    live.user.md                 # Human-maintained guidance appended to `ccqa run` (live specs)
-    live.agent.md                # Auto-updated by `ccqa run --update-agent-prompt`
+  perspectives.yaml              # Inventory of existing coverage (canonical)
+  profiles/                      # `--profile <name>` env files (stg.env, prd.env, ...)
+  prompts/                       # `ccqa init` scaffolds these; *.user.md human-maintained, *.agent.md auto-updated
+    record.user.md  record.agent.md  live.user.md  live.agent.md
   blocks/
-    login/
-      spec.yaml                  # Reusable block (params + steps)
+    login/spec.yaml              # Reusable block (params + steps)
   features/
     tasks/
-      perspectives.md            # Per-category detail tables (one per case)
       test-cases/
         create-and-complete/
           spec.yaml              # Test definition, with `mode: deterministic | live`
-          actions.json           # (deterministic only) Recorded actions from `ccqa record`
-          test.spec.ts           # (deterministic only) Generated vitest script
-          runs/
-            2026-06-14T10-00-00-000Z/  # (live only) one `ccqa run` invocation
-              run.json                  # Machine-readable summary
-              run.md                    # Human-readable per-step log
-              steps/
-                step-01.before.png      # Before-step screenshot
-                step-01.after.png       # After-step screenshot
-                step-01.log.txt         # Claude's full transcript for the step
+          actions.json           # (deterministic) recorded actions
+          test.spec.ts           # (deterministic) generated vitest script
+          runs/<timestamp>/      # (live) one `ccqa run` — run.json/run.md + steps/*.png
 ```
 
-Add `.ccqa/features/*/test-cases/*/runs/` to `.gitignore` — these are per-run artefacts that should not be committed. Likewise `ccqa-report*/`.
+Gitignore the per-run artefacts: `.ccqa/features/*/test-cases/*/runs/` and `ccqa-report*/`.
 
 ## Profiles (`--profile`)
 
@@ -177,11 +155,10 @@ TEST_USER_PASSWORD=...
 ccqa run auth/login --profile stg    # same spec, stg values
 ```
 
-- Name is free-form (`stg`/`prd` are conventions); a path separator / `..` / leading dot is rejected, and a missing profile exits 2. Only the name is logged, never values.
-- Format is a small `.env` subset (`KEY=value`, `#` comments, `export`, quotes). Profile values **override** the inherited environment.
-- Without `--profile`, ccqa auto-loads `<cwd>/.env` if present (like dotenv); with neither, `${VAR}` resolves against the existing `process.env` (e.g. `direnv`).
-
-**Secrets:** gitignore any profile that holds plaintext secrets. ccqa only parses `.env` files — it doesn't resolve secret-manager references — so to keep secrets off disk, drop `--profile` and run ccqa under your secret manager instead (e.g. `op run --env-file=.ccqa/profiles/stg.env -- ccqa run ...`), which injects the resolved values into `process.env` for ccqa to read.
+- Format is a small `.env` subset (`KEY=value`, `#` comments, `export`, quotes); profile values **override** the inherited environment.
+- Name is free-form (`stg`/`prd` by convention); path separators, `..`, and leading dots are rejected, and an unknown name exits 2. Only the name is logged, never values.
+- Without `--profile`, ccqa auto-loads `<cwd>/.env` if present; with neither, `${VAR}` resolves against the existing `process.env`.
+- **Secrets:** gitignore any profile holding plaintext secrets. To keep secrets off disk entirely, drop `--profile` and run ccqa under your secret manager (e.g. `op run --env-file=... -- ccqa run ...`), which injects resolved values into `process.env`.
 
 ## Live specs (`mode: live`)
 
@@ -191,110 +168,29 @@ For specs declared `mode: live` in their spec.yaml, `ccqa run` skips codegen ent
 - the codegen output for a spec is fragile (heavily timing-dependent UIs, rich-text editors, dynamic selectors)
 - you want a visual audit trail of what the page looked like at every step
 
-```bash
-# Run a single live spec
-ccqa run tasks/create-and-complete
+Constraints on selectors / `agent-browser` subcommands that apply during `ccqa record` (no `eval`, no `@ref`, no bare-tag positional `find`, no chained agent-browser calls) are relaxed for live specs, since there is no replay contract to honour.
 
-# Run every spec under a feature (mixes deterministic + live as declared)
-ccqa run tasks
+See [Live specs](./docs/live.md) for usage examples and per-project guidance.
 
-# Run every spec in the project, into a unified HTML report
-ccqa run --report
+### Saved sessions (`session:`)
 
-# Retry each failing step up to 2 more times (live specs only)
-ccqa run --retry 2 tasks/create-and-complete
-```
-
-Constraints on selectors / `agent-browser` subcommands that apply during `ccqa record` (no `eval`, no `@ref`, no bare-tag positional `find`, no chained agent-browser calls) are **relaxed** for live specs — Claude can use any subcommand and any selector style because there is no replay contract to honour.
-
-### Pre-authenticated state (`statePath:`)
-
-By default each `ccqa run` of a live spec spins up a fresh `agent-browser` session and starts signed-out. That keeps runs hermetic but forces every device-trust gate (Slack "we don't recognize this browser", Google's unfamiliar-device prompt, MFA challenges, …) to fire on every run.
-
-To skip them, save an authenticated browser state to a JSON file once locally and point the spec at it:
+Some providers gate every fresh browser with a device-trust check (an "unrecognized device" e-mail code, an MFA prompt) that a human has to clear by hand — impractical to repeat on every run, impossible in CI. `session:` restores a saved, signed-in browser state instead.
 
 ```yaml
-title: Slack App Home — non-admin access denied
+title: Admin can open the settings page
 mode: live
-statePath: .ccqa/sessions/slack-stg.json   # cookies + localStorage to restore
+session: admin            # restore the saved "admin" session before step 1
 steps:
-  - ...
+  - ...                   # no login steps — the spec starts signed-in
 ```
 
-ccqa resolves the path against the project root and passes `--state <path>` to every `agent-browser` invocation in the run (including ccqa's own screenshot calls). The file is **read-only** — `--state` loads it but never writes back to it. Re-running locally or in CI does not mutate it.
-
-Bootstrap once locally:
-
-```bash
-# 1. Log in interactively in a headed browser.
-agent-browser --headed open https://app.slack.com
-# …complete login + device-trust prompts by hand…
-
-# 2. Snapshot cookies + localStorage to the path the spec references.
-mkdir -p .ccqa/sessions
-agent-browser state save .ccqa/sessions/slack-stg.json
-agent-browser close
-
-# 3. ccqa run reuses the saved state — no login prompt.
-ccqa run slack/app-home-non-admin-access-denied
-```
-
-Add `.ccqa/sessions/` to `.gitignore` — these files contain live auth cookies and must never be committed.
-
-#### CI: bring the state file with you
-
-`statePath:` lives entirely inside `.ccqa/` and never touches `~/`. CI re-uses the state by writing the file into the same path the spec already references:
-
-```bash
-# Locally, after the interactive bootstrap above:
-base64 -i .ccqa/sessions/slack-stg.json | pbcopy
-# paste into your CI secret store as CCQA_SLACK_STG_STATE_B64
-```
-
-```yaml
-# .github/workflows/ccqa.yml (sketch)
-- name: Restore agent-browser state
-  env:
-    CCQA_SLACK_STG_STATE_B64: ${{ secrets.CCQA_SLACK_STG_STATE_B64 }}
-  run: |
-    mkdir -p .ccqa/sessions
-    printf '%s' "$CCQA_SLACK_STG_STATE_B64" | base64 -d \
-      > .ccqa/sessions/slack-stg.json
-
-- name: Run live specs
-  env:
-    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-  run: pnpm ccqa run --report
-```
-
-Caveats:
-
-- **Expiry.** Whatever the upstream service's "remember this device" window is (Slack ≈ 30 days, others vary), the cookies in the state file eventually expire and CI starts failing on the device-trust gate again. Re-bootstrap locally and rotate the secret.
-- **Treat the file as a credential.** It contains live auth cookies. Store it in your CI secret manager (GitHub Actions encrypted secrets, Vault, …) and never commit it.
-- **Deterministic specs ignore `statePath:`.** Today it only affects `mode: live`; vitest-replayed specs always run isolated.
-
-### Per-project guidance (`.ccqa/prompts/live.user.md` + `live.agent.md`)
-
-ccqa's live-mode system prompt is deliberately product-agnostic. Anything specific to **your** project — staging URLs, login flow quirks, rich-editor types, common access-denied wording — belongs in two sibling files (run `ccqa init` to scaffold both):
-
-- `.ccqa/prompts/live.user.md` — human-maintained stable guidance.
-- `.ccqa/prompts/live.agent.md` — auto-updated by `ccqa run --update-agent-prompt` from each run's summary. You can hand-edit it, but the next `--update-agent-prompt` run may rewrite the whole file; durable rules should live in `live.user.md`.
-
-Both files (when present) are read once per invocation and appended to the system prompt under "Project-specific guidance". The `ccqa record` (trace) side has the same split: `record.user.md` + `record.agent.md`, refreshed by `ccqa record --update-agent-prompt`.
-
-Keep them short. A page or two of focused notes beats a long handbook — Claude has the spec's `expected` text to work from, these files are for the *non-obvious* product knowledge that isn't in any single spec. Examples of what's useful here:
-
-- "the rich text editor is `[contenteditable='true']` — use `fill`, not keystrokes"
-- "login redirects through an IDP service-selection screen; you can skip it by opening the destination URL directly"
-- "access-denied is signalled by a specific in-app message string — name it here so the model asserts on it"
+ccqa does not manage authentication — `session` is purely an optional restore of cookies + localStorage.
 
-Examples of what does **not** belong:
+See [Saved sessions](./docs/sessions.md) for how to bootstrap a session and use it in CI.
 
-- per-spec details (those belong in the spec's `instruction` / `expected`)
-- restating the STEP_RESULT contract (already in the system prompt)
-- copy-pasted style guidelines from `record.user.md` (the relaxed-constraint mode doesn't need them)
+## CI result aggregation (hub)
 
-The combined bundle is capped at 32 KiB; anything beyond that is truncated with a warning.
+`ccqa serve` starts a hub — a control-plane HTTP server that aggregates CI run results, saved sessions, and variables on a shared server. It does not execute tests: a CI job (or a laptop) runs `ccqa run --report` as usual, fetching whatever sessions/variables/prompts it needs from the hub directly at run time, then `ccqa hub push` uploads the resulting report to the hub. Sessions and variables are registered on the hub ahead of time (`ccqa hub session push`, `ccqa hub var set`, or the bundled UI's Secrets tab), so a CI job needs only one secret: `CCQA_HUB_TOKEN`. One hub manages many projects — secrets are scoped per project/profile, and `--project` defaults to the current directory's name. See [docs/hub.md](./docs/hub.md) for setup and usage.
 
 ## License