kaloko 0.4.8 → 0.4.10

package/skills/kaloko/reference/evaluation.md

@@ -10,9 +10,11 @@ answer, so running it again is cheap.
 - deterministic results computed in the live page at capture time (with
   evidence: selector snippet, measured value);
 - the semantic evaluator for semantic criteria. It is a replaceable provider;
-  the default is JEV by TypeSafe (`TYPESAFE_API_KEY`), which returns only a
-  probability per question (`score`, 0–1). Verdicts: `pass` ≥ 0.85,
-  `fail` ≤ 0.15, otherwise `needs_review` (thresholds in config);
+  the default is JEV by TypeSafe (`TYPESAFE_API_KEY`), which reads the
+  reduced page state (no images) and returns only a probability per question
+  (`score`, 0–1). Verdicts: `pass` ≥ 0.85, `fail` ≤ 0.15, otherwise
+  `needs_review` (thresholds in config);
+- a vision evaluator for a second look with the screenshot (below);
 - `manual` criteria stay `not_evaluated` until a reviewer decides.
 
 Where the evaluator runs: with `TYPESAFE_API_KEY` set locally, your own key; without it, the organization's
@@ -25,6 +27,47 @@ report with the reason `kaloko evaluate` gives.
 The default evaluator is text-only and most accurate in English. It is weak
 at numbers, which is why those are checked deterministically.
 
+## Vision and reasons
+
+Every semantic verdict carries one sentence, `reason`, shown as "Why" next to
+the score on the canvas, in `kaloko build`, in the walk summary and in
+`get_run_summary`. With only the text evaluator the sentence says what it
+read and how sure it was (JEV gives no explanation). The vision evaluator
+gives a real one: it sees the screenshot, or a crop of the criterion's
+`scope` (or `highlight`), and answers with a probability and what in the
+image decided it. It is asked for:
+
+- criteria with `vision: true` on the check (layout, imagery, "looks like the
+  brand", anything the text cannot show);
+- text answers in the uncertain band (0.15–0.85 by default), which would
+  otherwise go to a person.
+
+Its answer replaces the score (the text score stays as `text_score` in the
+evaluation). Cost stays bounded: only those criteria, one request per
+capture, images scaled to 1000 px wide and at most two tiles of a full page,
+at most 40 uncached requests per evaluation, and answers cached by content
+like the text ones. The provider is Anthropic's Messages API
+(`ANTHROPIC_API_KEY`, model from `evaluators.vision.model`, default
+`claude-opus-5-5`). Without the key it goes through the organization's
+service, where one vision question counts as five of the monthly allowance;
+when the text evaluator runs with your own `TYPESAFE_API_KEY`, the vision
+evaluator also waits for your own key (`evaluators.vision.hosted: true`
+lets it use the service anyway).
+Configure or switch it off in `kaloko.config.yml`:
+
+```yaml
+evaluators:
+  vision:
+    when: uncertain      # flagged (only vision: true) | always | off
+    uncertain: [0.15, 0.85]
+    max_calls: 40
+    reason_language: en  # cs for Czech reviewers
+  # vision: false        # text evaluator only
+```
+
+Report the reason with each `needs_review` or failing criterion; when the
+evaluator saw the screenshot, say so.
+
 When a criterion comes back `needs_review`, read the reduced state the
 evaluator saw (`steps/<node>/<locale>/<viewport>.capture.json`, key `state`)
 and the screenshot. Either the page is genuinely unclear (report it), the

package/skills/kaloko/reference/execution.md

@@ -73,3 +73,77 @@ The environment policy and gate apply to every context the CLI creates (the
 shared browser, `walk` sessions, one-off captures): blocked paths and, when
 readonly, non-GET requests outside `allowed_post_paths` fail and are listed as
 blocked requests in the capture; the gate's credentials go to its origins only.
+
+## Noisy regions (`--stability`, `kaloko stability`)
+
+When pixel diffs between runs keep flagging the same clock, ad or carousel:
+
+```bash
+kaloko walk --stability 3        # 3 screenshots per capture, a moment apart; what moved is noted
+kaloko stability                 # this run against the 3 previous local runs of the scenario
+kaloko stability --apply         # adds the suggested ignore: to each step (block-style YAML only)
+```
+
+`kaloko stability` writes `stability.json` and `stability.yml` into the run
+and prints the suggestions per step: a selector when an element of the stored
+page covers the region, otherwise a rectangle for one viewport. Look at the
+screenshots before applying; a region that changed because the page really
+changed is not noise. `--same-commit` compares only runs of the same commit,
+`--runs a,b` names them, `--last N` sets how many runs are compared. Capture
+the steps again after applying: the capture records the ignored regions.
+
+## Only what changed
+
+After a fix, walk the steps the change reaches instead of the whole flow:
+
+```bash
+kaloko affected                      # which steps, and why (against origin/main; --base <ref>)
+kaloko walk --affected [--carry]     # walk them; --carry copies the other steps from the previous local run
+```
+
+A changed file reaches a step when it is a route file whose route matches the
+step's path or a URL captured for it earlier (Next.js `app/` and `pages/`,
+Nuxt, SvelteKit, Remix, Astro, Rails views and controllers), when it is
+imported by such a file, when it is global (layouts, global styles,
+dependencies; i18n messages of one locale keep the walk in that locale), when
+it is the step's script or the step changed in the scenario, or when it
+matches the step's `affects:` globs. Steps on the main path before a selected
+step are walked first (`--no-upstream` skips them); steps with
+`affects: always` (sign-in, consent) are always walked. Files nothing maps are
+listed as unmapped: add `affects:` to the steps they shape, or pass
+`--if-unmapped all` to walk everything then. Changes to tests, docs and CI are
+ignored.
+
+```yaml
+- id: checkout
+  path: /checkout
+  affects: [lib/payments/**, src/components/PriceSummary.tsx]
+- id: consent
+  affects: always
+```
+
+Report the selection with the run: which steps were walked and why, and that
+the rest was not walked (or carried from which run).
+
+## Several locales at once
+
+Locales are independent, so `walk --concurrency N` walks N of them at the same
+time, each in its own browser context (a new visitor per locale). In an own
+browser (`--ephemeral`, CI) the default follows the CPU (up to 4); on the
+shared browser it stays one at a time unless you ask, because people may rely
+on its signed-in tabs. Each locale's lines are printed together, in locale
+order. Set `walk.concurrency` in the config for the project's default.
+Viewports of one step are still captured one after another on the same page.
+
+## Selectors that moved
+
+When a step script waits in vain for a selector (the button lost its id, the
+class was renamed), the walk proposes a replacement from the page as it is
+now: a test id, then role and accessible name, then a stable id, then the text,
+preferring the element the old selector matched in the step's previous
+capture. It prints the reason and a patch of the step script. Review it; if it
+is the same element, `kaloko walk --steps <id> --heal` runs the step with the
+patched script and writes the patch only when the step passes. Applying a heal
+is fixing scenario mechanics; if the proposal points at a different element,
+the product changed: report it, do not heal it. `--no-heal` skips the
+proposals.

package/skills/kaloko/reference/import.md

@@ -0,0 +1,66 @@
+# Importing screenshots, walkthroughs and traces
+
+Agents and test tools already leave screenshots behind: Cursor and
+Antigravity walkthroughs, Claude Code screenshots, Playwright reports and
+traces. `kaloko import` lays them out as a run, so they are reviewed and
+signed off on the canvas like a walk.
+
+```bash
+kaloko import shots/                         # into the current run (kaloko start first)
+kaloko import playwright-report/ --scenario qa/flows/checkout.yml --env staging
+kaloko import test-results/checkout/trace.zip --map "click Pay=payment"
+kaloko import walkthrough.md --adhoc         # no scenario: an ad-hoc flow of the screens
+kaloko evaluate && kaloko preview            # then kaloko share, as usual
+```
+
+Where the screens go: the current run, or a new run of `--scenario`, or with no
+run (or `--adhoc`) an ad-hoc flow written to `qa/flows/import-<title>.yml`: one
+manual screen per image, in order, read-only.
+
+## Sources
+
+- **A folder with `steps.json`** (also `walkthrough.json`, `kaloko-import.json`):
+  ```json
+  { "title": "Checkout", "tool": "cursor",
+    "steps": [
+      { "title": "Cart", "image": "01-cart.png", "step": "cart", "url": "https://shop.test/cart",
+        "notes": "Two items, total 41 €", "locale": "en", "viewport": "desktop" },
+      { "title": "Payment", "image": "02-payment.png" }
+    ] }
+  ```
+  Only `image` is required. `step` names the scenario step; `viewport` and
+  `locale` must be in the run's scope or are chosen for you.
+- **A Markdown walkthrough** (`walkthrough.md`, `README.md`, any `.md` in the
+  folder): every image `![alt](file.png)` in order, titled by the heading above
+  it, with the text between as notes. Links to `.webm`/`.mp4` are recordings.
+- **A plain folder of images**: in file name order (`01-…`, `02-…`), titled by
+  the name. A folder of folders (Playwright `test-results/`) reads each
+  sub-folder as one test.
+- **Playwright**: the JSON reporter's file (`--reporter=json`), an HTML report
+  folder (`playwright-report/`) or a `trace.zip`. Screenshot attachments become
+  screens; a test without them contributes its trace: one screen after each
+  page action (goto, click, fill, press …).
+
+`--format folder|markdown|playwright-json|playwright-html|trace` overrides the
+detection.
+
+## Mapping to steps
+
+In this order: the item's `step`, `--map "Title=step,…"` (a title, group or
+file name containing the label), the URL path against step paths, then the
+title against step ids and titles. A second screen for the same step, locale
+and viewport becomes an attachment of the step. Unmatched screens are listed
+with the command that maps them; nothing is dropped silently. The viewport is
+the run viewport closest to the image width (2× and 3× screenshots count by
+their CSS width).
+
+## What an imported screen is
+
+The image, encoded like a walked capture, with the tool and the source file.
+There is no page behind it: deterministic criteria stay "not evaluated" until
+the step is walked, and semantic questions are not asked without page text, so
+a person decides on the canvas. Recordings are kept in the run folder
+(`imported/`) and are not uploaded yet.
+
+Report it as imported evidence, not as a walk: name the tool and say which
+steps still need a real walk.

package/skills/kaloko/reference/mcp.md

@@ -23,7 +23,7 @@ file.
 | Tool | Use it for |
 |---|---|
 | `list_runs` | published runs, newest first; filters `scenario`, `project`, `verdict`, `author`, `environment`, `branch`, `limit` |
-| `get_run_summary` | one run: statuses per step and locale, criteria that need attention with evidence, screenshot URLs, verdict, baseline, canvas URL, open feedback |
+| `get_run_summary` | one run: statuses per step and locale, criteria that need attention with evidence (semantic ones with score and the evaluator's one-sentence `reason` per locale and viewport), regions ignored in pixel diffs and regions seen moving (`ignored_regions`, `unstable_regions`), screenshot URLs, verdict, baseline, canvas URL, open feedback |
 | `get_feedback` | run verdict with note, approved steps, unresolved rejections and comments, failed manual criteria (`run_id`, or `scenario` for its latest run) |
 | `list_notifications` | the token owner's inbox (`unread_only`, `limit`) |
 | `list_projects` | projects of the organization with counts; the slug is the `project` filter of `list_runs` and the value for `project:` in a scenario |
@@ -35,6 +35,7 @@ file.
 | `list_step_changes` | documentation layer: steps whose capture or status changed since the previous accepted run or `since` (`scenario`, `since`, `accepted`); see [documentation.md](documentation.md) |
 | `get_started` | where the organization stands (`empty`, `sample`, `active`), its sample run, the next steps and a prompt; call it in a new project, see [first-run.md](first-run.md) |
 | `add_sample_run` | adds the sample run (or returns the one there is); needs an upload or full token and the Developer / Tester role |
+| `get_affected_steps` | which steps of a scenario changed files reach, with the reason per step (`scenario`, `changed_files`, `run_id`, `upstream`); route files are matched against the step paths and URLs of the newest run. Imports and `affects:` globs need the repository: `kaloko affected` follows them |
 
 ## Comment tools
 
@@ -65,6 +66,8 @@ The same data and permissions as the REST API; each tool is its REST route.
 |---|---|
 | `get_draft` | which revision each step shows, step status, who changed it (person or agent) |
 | `get_step` | the handoff: live files, signed entry URL, token set, page structure, open comments (`ref`: draft, design, vN, run id) |
+| `get_step_spec` | what to build for a step as approved: elements with role, name and selector, styles mapped to tokens, states, contrast (`ref`: design by default, draft, vN, run id; `format` text or json) |
+| `get_step_drift` | how an implementation run (`run`: id or `latest`) differs from the design reference, per element: expected token against the actual value |
 | `get_step_history` | versions, and revisions of a step with authors |
 | `list_step_changes` | steps that differ between two references (default design reference → draft) |
 | `get_references`, `get_tokens` | design reference and baseline; the project's token set |