kaloko 0.4.9 → 0.4.10

package/README.md

@@ -27,8 +27,11 @@ npx kaloko doctor
 ```
 
 `init` creates `kaloko.config.yml` and copies the `kaloko` skill to
-`.claude/skills/kaloko` (Claude Code) or `skills/kaloko` (Codex). Run it again
-after `npm update kaloko` to refresh the skill.
+`.claude/skills/kaloko` (Claude Code) or `skills/kaloko` (Codex). It reads the
+project first: the framework (Next.js, Nuxt, SvelteKit, Remix, Astro, Rails and
+others), the dev server's command and URL, the locales of the i18n config and
+the main routes, and drafts `qa/flows/main-pages.yml` so the first walk works
+without editing. Run it again after `npm update kaloko` to refresh the skill.
 
 ## Then ask your agent
 
@@ -61,7 +64,65 @@ npx kaloko preview                    # local canvas
 npx kaloko share --pr                 # upload to kaloko.app, comment on the PR
 ```
 
-`npx kaloko help` lists every command. When something does not work (Chrome,
+After a fix, walk only what the change reaches: `npx kaloko affected` explains
+which steps the changed files touch (routes, imports, `affects:` globs in the
+scenario), and `npx kaloko walk --affected` walks them. Several locales walk at
+once with `--concurrency 3`. When a step's selector no longer exists, the walk
+proposes one from the page and prints the patch; `--heal` applies it once the
+step passes.
+
+`npx kaloko help` lists every command.
+
+## Pull request previews in CI
+
+```bash
+npx kaloko ci github      # or: npx kaloko ci gitlab
+```
+
+writes a workflow that waits for the pull request's preview deployment
+(Vercel, Netlify, Cloudflare Pages, any review app), walks the steps the change
+affects and puts one comment per scenario on the pull request. Add a
+`KALOKO_TOKEN` secret (tester role) and commit it. In short, the GitHub
+workflow does this:
+
+```yaml
+on:
+  deployment_status:
+jobs:
+  walk:
+    if: github.event.deployment_status.state == 'success'
+    runs-on: ubuntu-latest
+    env:
+      KALOKO_TOKEN: ${{ secrets.KALOKO_TOKEN }}
+      GH_TOKEN: ${{ github.token }}
+      KALOKO_BROWSER_CHANNEL: chromium
+    steps:
+      - uses: actions/checkout@v4
+        with: { fetch-depth: 0 }
+      - uses: actions/setup-node@v4
+        with: { node-version: 22 }
+      - run: npm ci
+      - run: npx kaloko browser install --with-deps
+      - run: npx kaloko ci preview-url --wait 600 --github-env
+      - run: |
+          npx kaloko start --scenario qa/flows/checkout.yml --env preview
+          npx kaloko walk --ephemeral --affected
+          npx kaloko share --pr
+```
+
+Uploads go by content hash: captures that did not change since an earlier run
+are not sent again, and an interrupted `share` picks up where it stopped.
+
+## Screenshots you already have
+
+```bash
+npx kaloko import playwright-report/        # or test-results/, a trace.zip, results.json
+npx kaloko import walkthrough/ --adhoc      # a Cursor or Antigravity walkthrough, a folder of screenshots
+```
+
+lays screenshots, Playwright reports and traces out as a run: matched to the
+scenario's steps (by step id in `steps.json`, URL or title), or as an ad-hoc
+flow when there is no scenario. Build and share it like any run. When something does not work (Chrome,
 token, environment URL, evaluator key), `npx kaloko doctor` says what to fix.
 
 ## Review only what changed
@@ -169,7 +230,11 @@ captured HTML, page text and console output.
 each environment lets you in.
 
 Set `browser.channel: chrome` to use an installed Chrome, or run
-`npx playwright install chromium`.
+`npx kaloko browser install` for the Chromium of kaloko's Playwright
+(`KALOKO_BROWSER_CHANNEL=chromium` then uses it, which is how CI runs).
+In CI, `KALOKO_PREVIEW_URL`, `KALOKO_PR` and `KALOKO_BASE` come from
+`npx kaloko ci preview-url`; `KALOKO_GITLAB_TOKEN` lets `share --pr` comment on
+GitLab merge requests.
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "kaloko",
-  "version": "0.4.9",
+  "version": "0.4.10",
   "description": "See what your AI agents built: every screen, language and device walked, assessed and approved on one canvas.",
   "keywords": [
     "qa",

package/schemas/config.v1.json

@@ -74,6 +74,13 @@
         }
       }
     },
+    "walk": {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "concurrency": { "type": "integer", "minimum": 1, "maximum": 16, "description": "Locales walked at the same time, each in its own browser context (kaloko walk --concurrency). Default: by CPU in an own browser, one at a time on the shared browser." }
+      }
+    },
     "share": {
       "type": "object",
       "additionalProperties": false,

package/schemas/scenario.v1.json

@@ -243,6 +243,13 @@
           "type": "string",
           "description": "Key from `sessions`; which person performs this step."
         },
+        "affects": {
+          "oneOf": [
+            { "const": "always" },
+            { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }
+          ],
+          "description": "For `kaloko walk --affected`: globs of repository files (relative to the config) that shape this step, next to the routes and imports Kaloko finds itself; `always` walks the step on every affected walk (sign-in, consent)."
+        },
         "packs": {
           "type": "array",
           "items": {

package/skills/kaloko/SKILL.md

@@ -48,7 +48,9 @@ the canvas, storage and collaboration are the service.
 Start the shared browser, walk a scenario on a non-production environment,
 create test data there (users, roles, carts), capture screens and e-mails,
 evaluate, build the preview, and fix scenario mechanics (selectors, scripts)
-that do not change what is being accepted. On a design environment: change
+that do not change what is being accepted, including applying a selector
+proposal with `kaloko walk --heal` when it names the same element. Walk only
+the steps a change reaches (`--affected`) and import screenshots into a run. On a design environment: change
 the design folder, walk, and sync the draft (including taking others' steps
 with `--theirs`). Move to the trash runs you uploaded in this session that
 were superseded before anyone looked at them (debug rounds: no comments, no
@@ -56,7 +58,8 @@ verdict), and run `kaloko prune` without `--yes` to see what a clean-up would do
 
 ## Ask first
 
-Changing acceptance criteria or scope, any production run, uploading with
+Changing acceptance criteria or scope, any production run, writing or
+replacing a CI workflow (`kaloko ci github|gitlab`), uploading with
 `kaloko share` (on a design environment: publishing a version), setting a
 design reference or baseline, merging a design branch, keeping your version of
 a step in a sync conflict (`--mine`), turning on `html_view`, a real clean-up
@@ -85,6 +88,14 @@ they can do; never write secrets into the config, a scenario or a script.
   browser with your capabilities: [reference/execution.md](reference/execution.md)
 - Evaluation, thresholds, the vision evaluator and its reasons, reading
   results: [reference/evaluation.md](reference/evaluation.md)
+  browser with your capabilities, walking only the steps a change reaches
+  (`--affected`), several locales at once, and selectors that moved
+  (`--heal`): [reference/execution.md](reference/execution.md)
+- Screenshots, walkthroughs, Playwright reports and traces that an agent or a
+  tool already produced, laid out as a run: [reference/import.md](reference/import.md)
+- CI: a workflow that walks each pull request's preview deployment and
+  comments on it (`kaloko ci github|gitlab`): [reference/ci.md](reference/ci.md)
+- Evaluation, JEV thresholds and reading results: [reference/evaluation.md](reference/evaluation.md)
 - Environments, access (gates, accounts, sign-in), e-mails, readonly and
   production rules, and what is yours to arrange for the project's stack:
   [reference/environments.md](reference/environments.md)

package/skills/kaloko/reference/acceptance.md

@@ -20,6 +20,16 @@ unclear, ask one focused question instead of guessing.
 When intent changes, update `ACCEPTANCE.md` first, add a change-log row with
 date and commit, bump `version` in the scenario, then adjust the scenario.
 
+In a project without scenarios, `kaloko init` already drafted
+`qa/flows/main-pages.yml` from the route files: one screen per main route in
+the locales of the i18n config, with the dev server's URL as `local`. It walks
+as it is; sharpen the purposes and criteria and add the flows that matter.
+
+Screenshots an agent or a test tool already took (a Cursor or Antigravity
+walkthrough, a Playwright report or trace) can start a scenario too:
+`kaloko import <folder> --adhoc` writes one manual step per screen; see
+[import.md](import.md).
+
 For public pages of a site (landings, marketing, documentation) start from a
 draft: `npx kaloko draft https://example.com/ https://example.com/pricing`
 fetches the pages and writes a read-only production scenario and its plan
@@ -133,6 +143,12 @@ criteria whose verdict differs between runs on the same commit as flaky;
 treat a flaky criterion as a selector or timing problem to fix in the
 scenario, not as a verdict.
 
+When a step is shaped by files no route reaches (a payment library, a shared
+price component), name them on the step with `affects: [lib/payments/**]`, so
+`kaloko walk --affected` walks it when they change; `affects: always` marks
+steps every walk needs (sign-in, consent). See
+[execution.md](execution.md#only-what-changed).
+
 ## Naming and linkage
 
 When the work has a repository and an issue number, make them visible:

package/skills/kaloko/reference/ci.md

@@ -0,0 +1,70 @@
+# CI and pull request previews
+
+Kaloko runs in CI with the same commands as on a laptop. `kaloko ci` writes the
+workflow; you only add a token. Writing or replacing a CI workflow changes the
+repository's pipeline: ask the person first.
+
+```bash
+kaloko ci github [--scenario qa/flows/checkout.yml,qa/flows/signup.yml]   # .github/workflows/kaloko.yml
+kaloko ci gitlab                                                       # kaloko.gitlab-ci.yml (include it)
+kaloko ci github --print                                               # show it, write nothing
+```
+
+What the generated workflow does on every preview deployment of a pull
+request (Vercel, Netlify and Cloudflare Pages report their previews as GitHub
+deployments; `--trigger pull_request` runs on every push instead and waits for
+the deployment):
+
+1. installs the project (`npm ci`, or pnpm, yarn, bun by the lockfile) and the
+   Chromium of kaloko's Playwright (`kaloko browser install --with-deps`);
+   `KALOKO_BROWSER_CHANNEL=chromium` makes the run use it, so no separate CI
+   config is needed;
+2. `kaloko ci preview-url --wait 600 --github-env` finds the preview URL, the
+   pull request and its base branch and puts them into `KALOKO_PREVIEW_URL`,
+   `KALOKO_PR`, `KALOKO_BASE`;
+3. for each scenario: `kaloko affected --check` skips it when the change does
+   not reach any of its steps, otherwise `start --env preview`,
+   `walk --ephemeral --affected` and `share --pr` (one comment per scenario,
+   updated on later pushes).
+
+`kaloko ci` also adds the `preview` environment to `kaloko.config.yml` with
+`base_url: ${KALOKO_PREVIEW_URL}`. Scenarios that list `environments:` must
+include `preview`, or `start` refuses them; the command says which.
+
+## Secrets
+
+- `KALOKO_TOKEN`: a token with the tester role and the upload scope (Settings →
+  API tokens; on Enterprise a service account's token, which no person owns).
+- `TYPESAFE_API_KEY` only when semantic criteria should use your own evaluator
+  key instead of the organization's allowance.
+- Vercel deployment protection: a `VERCEL_AUTOMATION_BYPASS_SECRET` secret and
+  the commented `access.headers` block of the preview environment.
+- GitLab: `KALOKO_GITLAB_TOKEN` (project access token, `api` scope) for the
+  merge request note; set `KALOKO_PREVIEW_URL` in the job when the review app
+  URL is known (`https://$CI_COMMIT_REF_SLUG.review.example.com`), or rely on
+  `CI_ENVIRONMENT_URL`.
+
+## Where the preview URL comes from
+
+`kaloko ci preview-url` takes the first of: `KALOKO_PREVIEW_URL`; the
+`deployment_status` event; `VERCEL_BRANCH_URL`/`VERCEL_URL`,
+`DEPLOY_PRIME_URL`/`DEPLOY_URL` (Netlify), `CF_PAGES_URL`,
+`CI_ENVIRONMENT_URL` (GitLab), `RENDER_EXTERNAL_URL`; then the commit's GitHub
+deployments, skipping production, polled every 10 s up to `--wait` seconds.
+Only the URL goes to stdout, so `export KALOKO_PREVIEW_URL="$(kaloko ci preview-url)"`
+works in any CI.
+
+## Affected steps in CI
+
+`walk --affected` needs the history: check out with `fetch-depth: 0` (GitHub)
+or `GIT_DEPTH: "0"` (GitLab). The base is `--base`, else `KALOKO_BASE`,
+`GITHUB_BASE_REF`, the merge request target, then `origin/main`. URLs captured
+in the newest shared run of the scenario count too, so a fresh CI checkout
+maps routes as well as a laptop with earlier runs. See
+[execution.md](execution.md#only-what-changed).
+
+## Done means
+
+The workflow file and the config change are committed after the person agreed,
+the secret is named for the person to add, and you report the first run's PR
+comment or why it did not appear (no preview deployment, missing token).

package/skills/kaloko/reference/collaboration.md

@@ -12,7 +12,9 @@ loop teams use:
    request and `gh` is signed in, `kaloko share --pr` posts one comment per
    scenario on it (summary line, canvas link, criteria that need attention
    with evidence) and updates that same comment on every later run, so the
-   PR never fills up with Kaloko noise. Otherwise write the comment yourself:
+   PR never fills up with Kaloko noise. In CI `KALOKO_PR` names the pull
+   request of a detached checkout, and on GitLab merge requests the comment is a
+   note on the MR (`KALOKO_GITLAB_TOKEN`). Otherwise write the comment yourself:
    what was tested (scenario, environment, locales), the summary line
    (`n/m steps passed`), every failing or `needs_review` criterion with
    evidence, and the canvas URL. Reviewers accept, return or comment on the
@@ -255,6 +257,12 @@ Everything above is available over the API with an organization token:
 - `GET /api/v1/orgs/<org>/runs/<id>/export.zip` (`kaloko export`): the whole
   run as a zip before it expires; keep it with the issue when the evidence
   must outlive the retention period.
+- Uploads by content hash: `POST …/blobs/missing` with `{"hashes": [sha256…]}`
+  answers what the organization does not store yet, `PUT …/blobs/<sha256>`
+  (header `X-QAwalk-Type: png|webp|html|json`) stores one, and the run's file
+  plan `POST …/runs/<id>/files` takes `{ path, bytes, blob }` entries that
+  point at stored blobs. `kaloko share` does this: unchanged captures are not
+  sent again and a share that broke off resumes when run again.
 - MCP server for agents: see [mcp.md](mcp.md).
 
 Read-only tokens may list but not change anything; keep CI tokens read-only

package/skills/kaloko/reference/environments.md

@@ -13,6 +13,11 @@ them; do not work around a block.
 Test data preparation for review and staging belongs to the project's CI
 prepare step; assume it exists and create only what the scenario itself needs.
 
+Pull request previews (Vercel, Netlify, Cloudflare Pages, review apps) are an
+environment whose URL comes from CI: `preview: { base_url: ${KALOKO_PREVIEW_URL} }`,
+which `kaloko ci github|gitlab` adds and `kaloko ci preview-url` fills; see
+[ci.md](ci.md).
+
 ## Who does what
 
 Kaloko needs evidence in its format: captures of screens and e-mails in a run.

package/skills/kaloko/reference/execution.md

@@ -91,3 +91,59 @@ screenshots before applying; a region that changed because the page really
 changed is not noise. `--same-commit` compares only runs of the same commit,
 `--runs a,b` names them, `--last N` sets how many runs are compared. Capture
 the steps again after applying: the capture records the ignored regions.
+
+## Only what changed
+
+After a fix, walk the steps the change reaches instead of the whole flow:
+
+```bash
+kaloko affected                      # which steps, and why (against origin/main; --base <ref>)
+kaloko walk --affected [--carry]     # walk them; --carry copies the other steps from the previous local run
+```
+
+A changed file reaches a step when it is a route file whose route matches the
+step's path or a URL captured for it earlier (Next.js `app/` and `pages/`,
+Nuxt, SvelteKit, Remix, Astro, Rails views and controllers), when it is
+imported by such a file, when it is global (layouts, global styles,
+dependencies; i18n messages of one locale keep the walk in that locale), when
+it is the step's script or the step changed in the scenario, or when it
+matches the step's `affects:` globs. Steps on the main path before a selected
+step are walked first (`--no-upstream` skips them); steps with
+`affects: always` (sign-in, consent) are always walked. Files nothing maps are
+listed as unmapped: add `affects:` to the steps they shape, or pass
+`--if-unmapped all` to walk everything then. Changes to tests, docs and CI are
+ignored.
+
+```yaml
+- id: checkout
+  path: /checkout
+  affects: [lib/payments/**, src/components/PriceSummary.tsx]
+- id: consent
+  affects: always
+```
+
+Report the selection with the run: which steps were walked and why, and that
+the rest was not walked (or carried from which run).
+
+## Several locales at once
+
+Locales are independent, so `walk --concurrency N` walks N of them at the same
+time, each in its own browser context (a new visitor per locale). In an own
+browser (`--ephemeral`, CI) the default follows the CPU (up to 4); on the
+shared browser it stays one at a time unless you ask, because people may rely
+on its signed-in tabs. Each locale's lines are printed together, in locale
+order. Set `walk.concurrency` in the config for the project's default.
+Viewports of one step are still captured one after another on the same page.
+
+## Selectors that moved
+
+When a step script waits in vain for a selector (the button lost its id, the
+class was renamed), the walk proposes a replacement from the page as it is
+now: a test id, then role and accessible name, then a stable id, then the text,
+preferring the element the old selector matched in the step's previous
+capture. It prints the reason and a patch of the step script. Review it; if it
+is the same element, `kaloko walk --steps <id> --heal` runs the step with the
+patched script and writes the patch only when the step passes. Applying a heal
+is fixing scenario mechanics; if the proposal points at a different element,
+the product changed: report it, do not heal it. `--no-heal` skips the
+proposals.

package/skills/kaloko/reference/import.md

@@ -0,0 +1,66 @@
+# Importing screenshots, walkthroughs and traces
+
+Agents and test tools already leave screenshots behind: Cursor and
+Antigravity walkthroughs, Claude Code screenshots, Playwright reports and
+traces. `kaloko import` lays them out as a run, so they are reviewed and
+signed off on the canvas like a walk.
+
+```bash
+kaloko import shots/                         # into the current run (kaloko start first)
+kaloko import playwright-report/ --scenario qa/flows/checkout.yml --env staging
+kaloko import test-results/checkout/trace.zip --map "click Pay=payment"
+kaloko import walkthrough.md --adhoc         # no scenario: an ad-hoc flow of the screens
+kaloko evaluate && kaloko preview            # then kaloko share, as usual
+```
+
+Where the screens go: the current run, or a new run of `--scenario`, or with no
+run (or `--adhoc`) an ad-hoc flow written to `qa/flows/import-<title>.yml`: one
+manual screen per image, in order, read-only.
+
+## Sources
+
+- **A folder with `steps.json`** (also `walkthrough.json`, `kaloko-import.json`):
+  ```json
+  { "title": "Checkout", "tool": "cursor",
+    "steps": [
+      { "title": "Cart", "image": "01-cart.png", "step": "cart", "url": "https://shop.test/cart",
+        "notes": "Two items, total 41 €", "locale": "en", "viewport": "desktop" },
+      { "title": "Payment", "image": "02-payment.png" }
+    ] }
+  ```
+  Only `image` is required. `step` names the scenario step; `viewport` and
+  `locale` must be in the run's scope or are chosen for you.
+- **A Markdown walkthrough** (`walkthrough.md`, `README.md`, any `.md` in the
+  folder): every image `![alt](file.png)` in order, titled by the heading above
+  it, with the text between as notes. Links to `.webm`/`.mp4` are recordings.
+- **A plain folder of images**: in file name order (`01-…`, `02-…`), titled by
+  the name. A folder of folders (Playwright `test-results/`) reads each
+  sub-folder as one test.
+- **Playwright**: the JSON reporter's file (`--reporter=json`), an HTML report
+  folder (`playwright-report/`) or a `trace.zip`. Screenshot attachments become
+  screens; a test without them contributes its trace: one screen after each
+  page action (goto, click, fill, press …).
+
+`--format folder|markdown|playwright-json|playwright-html|trace` overrides the
+detection.
+
+## Mapping to steps
+
+In this order: the item's `step`, `--map "Title=step,…"` (a title, group or
+file name containing the label), the URL path against step paths, then the
+title against step ids and titles. A second screen for the same step, locale
+and viewport becomes an attachment of the step. Unmatched screens are listed
+with the command that maps them; nothing is dropped silently. The viewport is
+the run viewport closest to the image width (2× and 3× screenshots count by
+their CSS width).
+
+## What an imported screen is
+
+The image, encoded like a walked capture, with the tool and the source file.
+There is no page behind it: deterministic criteria stay "not evaluated" until
+the step is walked, and semantic questions are not asked without page text, so
+a person decides on the canvas. Recordings are kept in the run folder
+(`imported/`) and are not uploaded yet.
+
+Report it as imported evidence, not as a walk: name the tool and say which
+steps still need a real walk.

package/skills/kaloko/reference/mcp.md

@@ -35,6 +35,7 @@ file.
 | `list_step_changes` | documentation layer: steps whose capture or status changed since the previous accepted run or `since` (`scenario`, `since`, `accepted`); see [documentation.md](documentation.md) |
 | `get_started` | where the organization stands (`empty`, `sample`, `active`), its sample run, the next steps and a prompt; call it in a new project, see [first-run.md](first-run.md) |
 | `add_sample_run` | adds the sample run (or returns the one there is); needs an upload or full token and the Developer / Tester role |
+| `get_affected_steps` | which steps of a scenario changed files reach, with the reason per step (`scenario`, `changed_files`, `run_id`, `upstream`); route files are matched against the step paths and URLs of the newest run. Imports and `affects:` globs need the repository: `kaloko affected` follows them |
 
 ## Comment tools