kaloko 0.3.0

package/skills/kaloko/reference/collaboration.md

@@ -0,0 +1,218 @@
+# Collaboration: share, feedback, inbox
+
+A run becomes useful when the people who decide can see it. The loop that
+works for teams using Kaloko:
+
+1. **Share the run** (`kaloko share`, tester token, after asking). The
+   command prints the canvas URL.
+2. **Put the link where the work is tracked.** When the branch has a pull
+   request and `gh` is signed in, `kaloko share --pr` posts one comment per
+   scenario on it (summary line, canvas link, criteria that need attention
+   with evidence) and updates that same comment on every later run, so the
+   PR never fills up with Kaloko noise. Otherwise write the comment yourself:
+   what was tested (scenario, environment, locales), the summary line
+   (`n/m steps passed`), every failing or `needs_review` criterion with
+   evidence, and the canvas URL. Reviewers accept, return or comment on the
+   canvas.
+   When the organization connected the Kaloko GitHub App (Settings →
+   Integrations), the same share also creates a check on the pull request
+   that waits for review and turns green or red with the verdict; do not
+   report it as passed before a person accepted the run.
+3. **Read feedback before continuing.** `kaloko feedback` lists the run
+   verdict, open rejections and comments, and failed manual criteria.
+   `kaloko inbox` lists what needs your attention across runs: mentions,
+   returned runs, rejected steps in your runs, new runs of scenarios you
+   reviewed, and expiry warnings. Both take `--json`; `kaloko inbox --read`
+   marks the shown items read.
+4. **Work through the comments, then close them.** See *Processing
+   feedback* below.
+5. **A returned run means fix, then a new run.** Runs are immutable; never
+   patch a shared run. Start a new one on the same scenario, share it and
+   reply in the tracker with the new link. The reviewer who returned it is
+   told about the new run automatically.
+
+## Processing feedback
+
+Comments are threads in three places:
+
+- **A step:** optionally pinned to a position on the screenshot
+  (`pin: {x, y}` as fractions, per viewport) or to an element of a live
+  design step (`anchor`).
+- **The whole run:** `node_id` is `''`.
+- **The whole flow:** these stay across runs, e.g. "add the empty-cart
+  state".
+
+`kaloko feedback` (or MCP `get_feedback` / `list_comments`) shows the open
+ones with their ids and replies. A comment written by an agent carries the
+agent's name (`KALOKO_AGENT`, e.g. `claude-code`), so people can tell who
+said what.
+
+The loop, for every open thread:
+
+1. **Understand it.** Open the canvas link when a pin or an element
+   matters; the position is on the screenshot of that viewport and locale.
+2. **Fix it** in the code or the scenario. When the comment asks for a
+   change of intent (criteria, scope), ask the person first.
+3. **Answer and close.** Use
+   `kaloko comment --resolve <id> --note "Fixed in <commit>: <what changed>"`
+   (add `--flow` for flow comments; MCP `resolve_comment` takes the same
+   `note`). The closing note is a reply in the thread, and the author is
+   told in their inbox.
+4. **When you disagree or need a decision,** reply with
+   `kaloko comment --reply <id> --note "…"` and leave the thread open.
+   Mention the person with `@name` when they must act; mentions are mailed
+   at once.
+5. **Share a new run** once the fixes are in, and link it in the tracker.
+   Resolved threads stay visible under "Resolved" on the canvas.
+
+Rules:
+
+- Never resolve a thread you did not address.
+- Never delete other people's comments; the service refuses anyway.
+- Your own comment may be edited (`--edit <id>`) or deleted (`--delete <id>`).
+- A rejected step (`status: rejected`) is feedback like any other: fix it,
+  resolve it, share again.
+
+People see open counts next to flows and runs in the app, on map nodes and
+index rows, and in the canvas's *Comments* panel (`#comments=all|run|flow`
+in a link). Mails:
+
+- **Right away:** mentions, rejected steps, returned runs and runs waiting
+  for approval.
+- **In the digest:** new comments, replies, resolutions and new runs, as a
+  daily or weekly mail grouped by flow.
+
+## Presenting a run
+
+For stakeholders who do not read canvases, open the run in **Replay**
+(button in the top bar, key `R`, or `#replay=1` in the URL): one step at a
+time with its screenshot, purpose and criteria, a single *Next* action, and a
+choice of branch at every decision node. `#replay=1&step=<id>&path=a,b,c`
+deep-links a specific path, so an issue comment can point at "the sign-in
+branch, step 3". Replay reads the manifest only; nothing is uploaded.
+
+For people who work through a list rather than a flow (QA leads, content
+and SEO teams, big runs), switch the canvas to **Index**. Use *Map | Index*
+in the top bar, the key `I`, or `#mode=index` in the URL.
+
+- **Rows:** every step in flow order, grouped by section. Each row shows
+  a thumbnail, status, passed criteria, failing criteria, HTTP status and
+  console errors.
+- **Filtering:** by status, plus search over titles, paths and criteria.
+  The search also reaches the pages inside stacks.
+- **Stacks:** a stack step is a folder. Opening it shows its page index
+  with a breadcrumb back to the run.
+- **Navigation:** the sidebar selects rows, and the lightbox's previous and
+  next follow the filtered list.
+- **Export:** CSV of the rows.
+- **Links:** `#mode=index&step=<stack>` links straight into a folder.
+
+## Public runs
+
+An admin can make one run per scenario **public** (canvas report → *Make
+public*, or `kaloko public --run <id>`): readable without sign-in, reviewer
+names hidden, embeddable in an iframe. Use it for demos, status pages or
+sharing with people outside the organization, and only for scenarios that
+show no personal data. A tester token may later move the flag to a newer run
+of that scenario, which is how a scheduled demo refresh works.
+
+## Staying informed without the app
+
+Members choose in Settings → Profile a daily or weekly e-mail digest of
+their unread inbox and whether to get an immediate mail when their run is
+returned or a step rejected; every mail has an unsubscribe link. Admins can
+connect one Slack channel (Settings → Integrations, "Add to Slack") that
+receives new shared runs, rejected steps and run verdicts. None of this
+changes what agents do: share the run, post the link, read the feedback.
+
+## Mentions
+
+`@name` in a comment or verdict note notifies that colleague in the app
+(the part of their e-mail before `@`, e.g. `@anna` or `@anna.novak`; a full
+e-mail works too). Use it when a step needs a decision from a specific person
+(design, legal, product owner) instead of asking in chat.
+
+## Verdicts and baselines
+
+- Reviewers set a **run verdict**: accepted, returned (with a note) or
+  pending. Testers see the note in `kaloko feedback` and in the inbox.
+- An accepted run may be the scenario's **baseline**. Later runs compare
+  against it; report differences against the baseline, not only against the
+  previous run.
+- `kaloko compare [--run <id>] [--baseline <id>] [--json]` prints the text
+  diff (step statuses, criterion verdicts, scores, evidence) of the current
+  run against the baseline, or the previous run when there is no baseline.
+  Put the relevant lines in the issue comment. On the canvas the same
+  comparison is the **Compare** selector in the top bar (or `#cmp=<run>` in
+  the URL): pixel differences per step, changed criteria, and a side-by-side
+  or diff view in the lightbox. Pixel diffs are computed in the browser;
+  nothing is uploaded.
+- **Manual criteria** are decided on the canvas by a reviewer; treat a
+  reviewer's `fail` like a failing deterministic check.
+- Admins may name **required approvers** per scenario (Settings). Such a
+  run stays `pending` until every one of them approved; `kaloko feedback`
+  prints `awaiting approval from: …`, so do not treat a single approval as
+  acceptance there.
+
+## Cleaning up runs
+
+Re-running a scenario after every fix is expected, and it fills the list: the
+scenario list (the organization home) shows one row per scenario, and `/runs`
+shows the newest run per scenario with "+ N older" unless a filter is set.
+Clean up so the history stays readable; everything goes to the trash, which
+can be restored until it is purged.
+
+```bash
+npx kaloko runs                              # newest run per scenario, "+ N older"
+npx kaloko runs --scenario TASK-1273-cart    # every run of one scenario
+npx kaloko trash <run-id>...                 # soft delete (uploader or admin); kaloko restore <run-id>
+npx kaloko prune --scenario TASK-1273-cart --keep 3            # dry run: lists what would go
+npx kaloko prune --scenario TASK-1273-cart --keep 3 --yes      # ask first
+npx kaloko keep <run-id>                     # never expires, never pruned (--off to undo)
+npx kaloko archive TASK-1273-klub            # retired scenario, ask first (admin); --undo
+```
+
+- **Without asking:** trash runs you uploaded in this session that were
+  superseded before anyone looked at them (debug rounds without comments or a
+  verdict), and a dry-run `prune`.
+- **Ask first:** `prune --yes`, trashing someone else's run, archiving a
+  scenario.
+- **Never:** trash an accepted run, a baseline, a run with review comments, or a
+  run linked from an open PR or issue comment. `prune` already keeps kept,
+  accepted, baseline, design-reference and commented runs, and without the admin
+  role it trashes only your own runs.
+- After splitting or renaming a scenario, propose archiving the old one
+  (`archived: true` in its file, or `kaloko archive <id>`). `kaloko validate`
+  warns when the service still lists a scenario whose file is gone.
+- Report clean-ups in the tracker next to the run links: which runs went to the
+  trash and why.
+
+An organization admin can turn on automatic clean-up after each share in
+Settings → Storage (keep the newest N per scenario and environment).
+
+## API for agents and CI
+
+Everything above is available over the API with an organization token:
+
+- `GET /api/v1/orgs/<org>/feedback?run=<id|latest>[&scenario=<id>]`
+- `GET|POST /api/v1/orgs/<org>/runs/<id>/comments` (a step: `node_id`;
+  the whole run: `node_id: ""`), `GET|POST …/scenarios/<id>/comments`
+  (the whole flow), and per comment `PATCH`, `DELETE`, `POST …/<cid>/resolve`
+  (with `note`) and `…/<cid>/reopen`; `GET /api/v1/orgs/<org>/comments?state=open|all&scenario=&run=&mine=1`
+  across the organization. Lists of runs and scenarios carry `open_comments`.
+- `GET /api/v1/orgs/<org>/notifications[?unread=1]`, `POST` with
+  `{"read": "all"}` or `{"read": [ids]}`
+- `GET /api/v1/orgs/<org>/runs[?project=&scenario=&kind=&verdict=&author=&environment=&branch=&q=][&latest=1]`
+  (`latest=1`: the newest run per scenario with `older`), `GET /api/v1/orgs/<org>/runs/<id>`
+- Clean-up: `DELETE …/runs/<id>` (trash), `POST …/runs/<id>/restore`,
+  `POST …/runs/<id>/keep` with `{"keep": true|false}`,
+  `POST /api/v1/orgs/<org>/runs/prune` with `{ scenario?, project?, environment?, keep_latest: 3, older_than_days?, dry_run: true }`
+  (answers `trash` and `kept` with reasons), `POST …/scenarios/<id>/archive`
+  and `…/unarchive` (admin)
+- `GET /api/v1/orgs/<org>/runs/<id>/export.zip` (`kaloko export`): the whole
+  run as a zip before it expires; keep it with the issue when the evidence
+  must outlive the retention period.
+- MCP server for agents: see [mcp.md](mcp.md).
+
+Read-only tokens may list but not change anything; keep CI tokens read-only
+unless the job publishes runs.

package/skills/kaloko/reference/design.md

@@ -0,0 +1,143 @@
+# Design with Kaloko
+
+Kaloko hosts designs as live HTML steps of the same scenario that development and QA use. You design with
+whatever skill, model and style the person uses; Kaloko keeps the draft, the versions, the comments and the
+design reference, and checks the implementation against it. This reference says how to fit your work into
+that. It says nothing about how a design should look.
+
+## Design environments
+
+A design environment is a folder the CLI serves while it walks:
+
+```yaml
+# kaloko.config.yml
+environments:
+  design: { kind: design, serve: design }
+```
+
+`kaloko start --scenario qa/flows/checkout.yml --env design` starts a run of it. The scenario is the one the
+implementation uses: same step ids, same criteria. Design steps meet the acceptance criteria too (headings,
+labels, semantic questions), so a design is accepted before anyone builds it. Checks against the design
+reference (`design_pixels`, `design_structure`) are left out on a design environment.
+
+## The bundle
+
+```
+design/
+  tokens.json                  token set (W3C DTCG), optional
+  _shared/app.css              shared files: fonts, styles, images, scripts
+  cart/index.html              one folder per step id
+  cart/cs/index.html           optional override for a locale
+  payment/index.html
+```
+
+- Plain HTML, CSS, JS and assets only (`qawalk.bundle.v1`: html, css, js, mjs, json, svg, woff2, woff, png,
+  jpg, webp, avif, gif, mp4, webm, txt). No build step on the service side: build locally, keep the output.
+- A step's revision is its page, the files next to it and everything its page references (stylesheets,
+  images, fonts, module imports, `fetch('./fixtures/x.json')`). Links to other steps are not files it needs.
+- Live steps run sandboxed on their own origin with `connect-src 'self'`: no real APIs. Use JSON fixtures in
+  the bundle.
+- Prototype links: `<a data-qawalk-go="payment">` moves the canvas to that step, whatever the `href` says.
+- Reference token values as CSS custom properties (`var(--color-accent)`), so the inspector and the token
+  check see where values come from.
+
+## Sync and publish
+
+- `kaloko walk` renders only revisions whose captures are missing (same files, same capture).
+- `kaloko status` shows each step: synced, changed here, changed in the draft, conflict; and its step status.
+- `kaloko sync` saves your changed steps into the shared draft and takes steps others changed. A step changed
+  on both sides is a conflict: name it with `--mine <step>` or `--theirs <step>`; never delete to resolve.
+- `kaloko share` on a design environment publishes a version of the flow (`v4 · 3 changed, 17 unchanged`).
+  A version counts as a shared run. Publish when the person asks, not after every change.
+- Step status (`in progress`, `ready for review`, `ready for dev`) is set in the canvas or with MCP
+  `set_step_status`; `ready for dev` notifies followers.
+- Set `KALOKO_AGENT=<your name>` so people see your writes as yours in presence, history and audit.
+
+## The loop
+
+1. Design or change steps in the folder with the person's design skill.
+2. `kaloko walk` (and `kaloko evaluate` if the step has semantic criteria), `kaloko sync`.
+3. Read `kaloko feedback` on the design environment: open comments, pinned ones with their element
+   (`on main > label`). People comment in the canvas; they may be looking right now.
+4. Change what the comments ask, sync again, then answer in the comment thread (MCP `add_draft_comment`).
+5. When the person is happy, `kaloko share --name "…"` publishes a version.
+
+## References
+
+- The design reference is an accepted version: `kaloko reference --design --version 4` (a person accepts the
+  version first; ask before setting a reference).
+- The baseline stays the last good implementation run: `kaloko reference --baseline --run <id>`.
+- Changing the design reference changes what implementation runs are checked against: tell the team.
+
+## Handoff
+
+Development builds from the design reference:
+
+- `kaloko export --live [--version n] --out design/` writes the live files into a folder.
+- MCP `get_step` (scenario, step, `ref: design`) returns the file list, a signed entry URL, the token set, the
+  page structure and open comments; `list_step_changes` names the steps that changed since the last reference.
+- Translate to the project's stack with the project's own skills; keep the step ids and the token names.
+
+## Tokens
+
+- The set is W3C DTCG. Both value styles are read: strings (`"#2E6BFF"`, `"16px"`) and the current format's
+  objects (`{ "colorSpace": "srgb", "components": [...], "hex": "#2e6bff" }`, `{ "value": 16, "unit": "px" }`),
+  composite types (typography, shadow, border, transition, gradient), aliases `{group.token}` and
+  `{ "$ref": "#/json/pointer" }`, `$extends`, `$root` and `$deprecated`. New sets: prefer the object values.
+- Modes (light, dark): a DTCG resolver `design/tokens.resolver.json` (sets plus a `theme` modifier whose
+  contexts point at files) wins over `tokens.json`; `push` inlines its files. Older sets with top-level
+  `light`/`dark` groups still work. `tokens_mode` in the environment picks the mode the checks use.
+- `kaloko tokens validate` reports names with `{`, `}` or `.`, references that point nowhere and values that
+  cannot be read; `push` refuses a set with problems, so does the service.
+- `kaloko tokens push` stores the set as the project's next token set version; `pull` gets it.
+- `kaloko tokens check` lists raw colors, fonts, sizes, spacing and radii in the design folder that are not in
+  the set, with the nearest token; a value that only a deprecated token has is named with its note.
+
+## Checking implementation
+
+Turn on `packs: [design]` (usually `[design, a11y]`) for the steps in the implementation scenario:
+
+| Id | Checks |
+|---|---|
+| AC801 | pixels: under 3 % differ from the design reference (same algorithm as the canvas) |
+| AC802 | structure: headings, landmarks, actions and form fields as in the reference |
+| AC803 | tokens: authored colors, fonts, sizes, spacing and radii come from the token set |
+
+Contrast and the other accessibility basics come from the `a11y` pack: `packs: [design, a11y]`.
+
+`kaloko evaluate` fetches the design reference (of the design branch linked to the git branch, if any) and
+fills the checks. Findings name what differs and the token to use; fix the implementation, not the reference.
+
+## Canvas edits
+
+People can change a step's text and assign tokens in the canvas inspector. Each edit is a new revision; a file
+shared by several locales changes for all of them. `kaloko sync` pulls the edit into the folder as a
+`<style data-qawalk-edits>` block and changed text: fold the rules into the real stylesheet when you next touch
+the step, then sync.
+
+## Branches
+
+- `kaloko branch create redesign --git feat/redesign` starts a branch of the draft; `--from v4` starts from a
+  version. Work on it with `--branch redesign` on `sync`, `status` and `share`.
+- The folder mirrors one branch at a time; switching (`kaloko sync --branch main`) takes that branch's steps
+  and never pushes the other branch into it.
+- Merging is the person's decision: `kaloko branch merge redesign`; steps changed on both sides need
+  `--take-branch <step>` or `--keep-main <step>`.
+
+## HTML view
+
+Experimental, Business plan. `html_view: true` on an app environment keeps a static, script-free copy of each
+captured page (secrets and hidden values removed) that opens in the canvas like a live step, so the inspector
+works on the real product. It is refused on production pages behind a sign-in. `kaloko export --live --views
+--run <id> --out design/` turns those copies into design steps: a redesign that starts from the product.
+
+## Bring your own design skill
+
+Any skill works if it writes the bundle layout above:
+
+- a design skill of Claude or another agent: ask it to write `design/<step>/index.html` per scenario step and
+  shared files into `design/_shared/`;
+- a brand or design-system skill: have it produce `tokens.json` and the shared CSS;
+- a Figma export through the person's own tooling (for example a Figma MCP server): convert frames to HTML
+  per step, then walk and sync;
+- hand-written HTML is fine.

package/skills/kaloko/reference/documentation.md

@@ -0,0 +1,72 @@
+# Documentation from runs
+
+A run is also a library of reference captures: every step, locale and
+viewport has a screenshot and the HTML. On the Business plan the service
+gives each step a stable address, lists what changed, and calls you when a
+run is accepted, so a manual, a course or product documentation can be
+rebuilt from the newest accepted run instead of screenshotting by hand.
+
+## Prepare the run
+
+- Walk the scenario as usual and let a person accept the run on the canvas;
+  documentation should come from accepted (or baseline) runs only.
+- Tag the release (`git tag v2.4.0`) before `kaloko start`, or pass
+  `--app-version 2.4.0`; the version lands in the manifest as `app_version` and in every
+  latest-capture answer, so the documentation can say which version it shows.
+- Accepted and baseline runs do not expire on Business and Enterprise.
+
+## Stable URLs
+
+With an organization token (`Authorization: Bearer qwk_…`, a read-only
+viewer token is enough):
+
+```
+GET /api/v1/orgs/<org>/scenarios/<scenario>/steps                      step index with links
+GET /api/v1/orgs/<org>/scenarios/<scenario>/steps/<step>/latest.json   metadata (run with app_version, captured_at, page URL, size)
+GET /api/v1/orgs/<org>/scenarios/<scenario>/steps/<step>/latest.image  the screenshot (latest.webp or latest.png when you know the format)
+GET /api/v1/orgs/<org>/scenarios/<scenario>/steps/<step>/latest.html   the captured HTML (served sandboxed)
+GET /api/v1/orgs/<org>/scenarios/<scenario>/changes                    steps changed since the previous accepted run
+```
+
+Add `?locale=cs&viewport=desktop` (defaults: the run's first locale and
+viewport) and `&accepted=1` to use the newest accepted run instead of the
+newest run. Files answer with an `ETag`; send `If-None-Match` and a 304 means
+nothing to download.
+
+## Only rebuild what changed
+
+`changes` (MCP `list_step_changes`) compares the newest run with the previous
+accepted one, or with `?since=<run id>`. A step is listed when it is new, its
+image bytes differ, or its status changed; `removed` lists steps that are
+gone. Timestamps or animations make a page count as changed, so keep
+captures stable (fixed test data, no clocks on screen) for documentation
+scenarios.
+
+## Webhooks
+
+Settings → Integrations → Webhooks (admin): an https URL and the events
+`run.accepted` and/or `run.shared`. Each call is a JSON POST
+`{ id, type, created_at, org, data: { run, steps, steps_url, changes_url } }`
+with `QAwalk-Signature: t=<unix time>,v1=<hex HMAC-SHA256 of "<t>.<body>">`
+signed with the webhook's secret (copy it from Settings). Verify the
+signature and the time (reject older than 5 minutes), answer 2xx within 5
+seconds, and do the work asynchronously. Also over the API:
+`GET|POST /api/v1/orgs/<org>/webhooks`, `DELETE …/webhooks/<id>`.
+
+## Export into a folder
+
+```bash
+npx kaloko export --scenario checkout --accepted --out docs/assets/checkout
+```
+
+writes `steps/<step>/<locale>/<viewport>.<ext>` and `.html` plus
+`index.json` (run with app_version, captured_at, page URL per file). Run it again and
+files from an unchanged run are kept, so a documentation repository sees only
+real changes. `examples/docs-pipeline.mjs` in the Kaloko repository shows a
+complete loop that rebuilds a Markdown manual and touches only changed steps.
+
+## Done means
+
+The manual or course references the exported files (or the stable URLs), says
+which application version it shows, and a re-run after a new accepted run
+changes only the pages of steps listed by `changes`.

package/skills/kaloko/reference/environments.md

@@ -0,0 +1,140 @@
+# Environments
+
+Profiles live in `kaloko.config.yml` under `environments`. The CLI enforces
+them; do not work around a block.
+
+- `local`, `review`, `staging`: you may create the test data a scenario
+  needs (users with roles, carts, orders in test payment mode).
+- `production`: always readonly, regardless of config. Only scenarios with
+  `readonly: true`, as an ordinary visitor or customer. No admin, no sign-in
+  to back offices, no data creation, no orders. Typical: SEO checks, landing
+  pages, checkout up to payment.
+
+Test data preparation for review and staging belongs to the project's CI
+prepare step; assume it exists and create only what the scenario itself needs.
+
+## Who does what
+
+Kaloko needs evidence in its format: captures of screens and e-mails in a run.
+How you get to them depends on the project's stack, and that part is yours.
+
+- **The CLI handles the standards**: HTTP Basic and header tokens in front of
+  an environment, client certificates, a saved browser session, the readonly
+  policy, Mailpit, and e-mails as `.eml`, `.html`, `.json` or `.txt`.
+- **You arrange the rest for this project** with its own tools and common
+  practice: how test users are created and signed in, where the application's
+  e-mails end up, how a review app's URL is known. Write that down once in the
+  project (config, step scripts, a mail adapter, a project skill) so the next
+  run does not rediscover it.
+- **You ask the person** for secret values and for anything only a person can
+  do (an SSO or MFA sign-in, VPN access, a new test account in a system you
+  cannot reach).
+
+## Access to an environment
+
+Three layers, each in its own place:
+
+| Layer | Examples | Where |
+|---|---|---|
+| Gate in front of the whole environment | HTTP Basic, Cloudflare Access service token, Vercel protection bypass, a custom header, mutual TLS | `environments.<env>.access` |
+| Sign-in to the application | form login, SSO/OAuth, MFA/TOTP | scenario `sessions` + `environments.<env>.accounts` + step scripts |
+| Network | VPN, IP allowlist | not config: the machine running the agent must have access; say so when it does not |
+
+```yaml
+environments:
+  staging:
+    base_url: https://staging.example.com
+    access:
+      origins: [https://staging.example.com, https://api.staging.example.com]  # default: base_url origins
+      basic: { user_env: STAGING_USER, password_env: STAGING_PASSWORD }
+      headers:
+        CF-Access-Client-Id: { env: CF_ACCESS_CLIENT_ID }
+        CF-Access-Client-Secret: { env: CF_ACCESS_CLIENT_SECRET }
+      client_certificate: { cert_file: certs/qa.pem, key_env: QA_CLIENT_KEY }
+    accounts:
+      member: { user: member@example.test, password_env: STAGING_MEMBER_PASSWORD }
+      admin: { user_env: STAGING_ADMIN_USER, password_env: STAGING_ADMIN_PASSWORD, totp_env: STAGING_ADMIN_TOTP }
+      sso: {}
+  review:
+    base_url: https://${REVIEW_HOST}      # review apps: CI provides the variable
+```
+
+Rules:
+
+- Secrets never go into `kaloko.config.yml`, a scenario or a step script.
+  Every `*_env` and `{ env: NAME }` names a variable; the values live in the
+  environment, the project `.env` (ignored by git) or `~/.config/kaloko/.env`.
+  Choose names that fit the project; ask the person for the values.
+- The gate is sent only to `access.origins`, never to third parties (analytics,
+  CDNs, payment providers). List an API on another host explicitly;
+  `https://*.example.com` covers subdomains.
+- The CLI applies the gate and the readonly policy to every browser context it
+  creates: the shared browser, `walk` sessions and one-off captures. A context
+  you create yourself over CDP does not get them; work in the shared browser's
+  default context, or use `walk`.
+- Values the CLI knows as secrets are replaced by `[redacted]` in captured HTML,
+  page text and console entries. Screenshots are not redacted: do not capture
+  a screen that shows a secret.
+- `kaloko doctor` checks that the variables are set, that the gate lets you in
+  (HTTP 401/403 is a failure) and that each account can sign in or has a
+  saved session.
+
+## Accounts and sign-in
+
+A scenario session's `account` is a key; `environments.<env>.accounts.<key>`
+maps it to credentials in that environment. Step scripts receive them as
+`session.credentials` (`user`, `password`, `totp()`, `storage_state`) and sign
+in the way the application does it:
+
+```js
+export default async ({ page, url, session }) => {
+  const { user, password } = session.credentials
+  await page.goto(url('sign-in'))
+  await page.fill('input[type=email]', user)
+  await page.fill('input[type=password]', password)
+  await page.click('button[type=submit]')
+  if (await page.locator('input[autocomplete=one-time-code]').isVisible()) await page.fill('input[autocomplete=one-time-code]', session.credentials.totp())
+}
+```
+
+SSO or MFA that a script cannot pass: ask the person to run
+`kaloko auth save --env <env> --account <key>` once. They sign in in a
+visible browser; the session is saved under the output directory (owner-only,
+git-ignored, never uploaded) and `walk` loads it into that account's sessions.
+When the application asks for sign-in again, the session expired: ask for a
+new `auth save`. `kaloko auth list` shows the saved sessions and their age.
+
+Prefer, in this order: a test account the project already has for QA; a
+seeded or API-created account on non-production environments; a saved
+session. Never use a real person's account.
+
+## E-mails
+
+Flows that send e-mails need the message as an `email` step. Where the
+message is depends on the stack:
+
+- **Mailpit** (built in): `mail: { provider: mailpit, url, auth, address }`.
+  `auth` takes `basic` and `headers` like `access`. `address` with `{run}`
+  gives every run its own recipient, so shared mailboxes stay apart.
+- **Anything else** (letter_opener, Mailosaur, MailSlurp, IMAP, a log, the
+  application's own API): write a small adapter and set
+  `mail: { provider: script, script: qa/mail/inbox.mjs, options: {...} }`.
+  Its default export is `async ({ to, subject, since, options, root, env })`
+  and returns the newest matching message
+  `{ id, subject, from, to: [address], date, html, text }` or `null`;
+  `kaloko mail` and `walk` poll it. The package ships an example to copy:
+  `node_modules/kaloko/templates/mail/letter-opener.mjs`.
+- **One-off or unusual**: get the message yourself and hand it over:
+  `kaloko mail --step <id> --file message.eml` (also `.html`, `.json` with the
+  message shape, `.txt`).
+
+`kaloko mail` and `walk` take only messages received after the run started
+(adapters get it as `since`), so a staging mailbox that collects everyone's
+mail cannot hand over an old message silently. `--since <ISO date>` moves the
+boundary, `--any-time` drops it, and `--id <message id>` takes one exact
+message (subjects repeat). The canvas shows when the message arrived and warns
+when it predates the run.
+
+Pick what fits the project and keep it in the project; do not change how the
+application sends mail just for QA unless the person agrees (switching a Rails
+app from letter_opener to SMTP locally, for example).

package/skills/kaloko/reference/evaluation.md

@@ -0,0 +1,62 @@
+# Evaluation
+
+`kaloko walk` sends the semantic questions of each step as soon as it is captured and ends with the
+verdicts: what fails and what a person should look at, per step, locale and viewport. Use that list to fix
+things right away. Steps captured later with `kaloko capture` need `kaloko evaluate`; it reuses every cached
+answer, so running it again is cheap.
+
+`kaloko evaluate` combines, per capture:
+
+- deterministic results computed in the live page at capture time (with
+  evidence: selector snippet, measured value);
+- the semantic evaluator for semantic criteria. It is a replaceable provider;
+  the default is JEV by TypeSafe (`TYPESAFE_API_KEY`), which returns only a
+  probability per question (`score`, 0–1). Verdicts: `pass` ≥ 0.85,
+  `fail` ≤ 0.15, otherwise `needs_review` (thresholds in config);
+- `manual` criteria stay `not_evaluated` until a reviewer decides.
+
+Where the evaluator runs: with `TYPESAFE_API_KEY` set locally, your own key; without it, the organization's
+service evaluates with its key (Team plan and up, a monthly allowance of questions; `kaloko evaluate` prints
+"via Kaloko (340 of 1000 questions this month)"). Answers are cached by content locally and in the service, so
+evaluating the same capture again costs nothing. `evaluators.jev.hosted: false` keeps it to your own key.
+When neither is available, or the allowance is used up, semantic criteria stay `not_evaluated`; say so in the
+report with the reason `kaloko evaluate` gives.
+
+Results are cached by content hash, so re-running is cheap. The default
+evaluator is text-only and most accurate in English; it is weak at numbers,
+which is why those are deterministic.
+
+When a criterion comes back `needs_review`, read the reduced state the
+evaluator saw (`steps/<node>/<locale>/<viewport>.capture.json`, key `state`)
+and the screenshot. Either the page is genuinely unclear (report it), the
+reducer missed something (report a Kaloko bug), or the question is too broad
+(narrow it with `scope`, `true`/`false`; this changes mechanics, not intent).
+Check evidence of passing deterministic checks too: a match in the wrong
+region (e.g. a shipping threshold instead of the product price) means the
+selector needs a scope.
+
+When the check itself was wrong (selector, pattern too broad) and the step
+cannot be repeated (data created, an e-mail sent once, a state that moved
+on), fix the scenario and run `kaloko evaluate --recheck [--steps a,b]`: the
+deterministic checks run again against the stored HTML with page scripts off
+and are marked `rechecked`; console errors, overflow and performance keep
+their live result. When the page content itself is in question, capture again.
+
+## Quick signals
+
+Beside the criteria, every captured screen gets three cheap questions in the same evaluator request: does the
+page do what the step's `purpose` says, is its text in the expected language, does it look broken (an error,
+placeholder text, a blank page). They never change a verdict; the canvas marks a screen with ⚠ when one is
+off (purpose or language at 30 % or less, broken at 70 % or more). Mention them in the report when they
+disagree with passing criteria: it usually means a criterion is missing. `evaluators.jev.signals: false`
+turns them off; they do not count against the hosted allowance.
+
+## Reporting
+
+After `kaloko build`, report: the summary line, each failing or
+`needs_review` criterion with step, locale, viewport and evidence, blocked
+requests if any, and the preview path. Do not claim a criterion passed when
+it was not evaluated.
+
+`needs_review` is not a failure: look at the screenshot and state your
+reading, or leave it to the reviewer.