kaloko 0.4.9 → 0.4.11

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/README.md +88 -4
  2. package/package.json +1 -1
  3. package/schemas/config.v1.json +35 -0
  4. package/schemas/scenario.v1.json +30 -1
  5. package/skills/kaloko/SKILL.md +22 -4
  6. package/skills/kaloko/reference/acceptance.md +16 -0
  7. package/skills/kaloko/reference/ci.md +70 -0
  8. package/skills/kaloko/reference/collaboration.md +9 -1
  9. package/skills/kaloko/reference/environments.md +5 -0
  10. package/skills/kaloko/reference/execution.md +56 -0
  11. package/skills/kaloko/reference/import.md +66 -0
  12. package/skills/kaloko/reference/mcp.md +2 -0
  13. package/skills/kaloko/reference/recording.md +95 -0
  14. package/src/cli.mjs +467 -100
  15. package/src/lib/affected-core.mjs +242 -0
  16. package/src/lib/affected.mjs +233 -0
  17. package/src/lib/anim.mjs +125 -0
  18. package/src/lib/apps/recording-native.mjs +212 -0
  19. package/src/lib/build.mjs +60 -4
  20. package/src/lib/ci.mjs +207 -0
  21. package/src/lib/config.mjs +2 -0
  22. package/src/lib/doctor.mjs +5 -1
  23. package/src/lib/evaluate.mjs +4 -2
  24. package/src/lib/git.mjs +6 -2
  25. package/src/lib/heal.mjs +207 -0
  26. package/src/lib/import.mjs +374 -0
  27. package/src/lib/init-detect.mjs +219 -0
  28. package/src/lib/onboarding.mjs +15 -8
  29. package/src/lib/pr.mjs +24 -2
  30. package/src/lib/recording.mjs +373 -0
  31. package/src/lib/scenario.mjs +2 -1
  32. package/src/lib/service.mjs +7 -1
  33. package/src/lib/trace.mjs +109 -0
  34. package/src/lib/unzip.mjs +31 -0
  35. package/src/lib/upload.mjs +114 -0
  36. package/src/lib/video-cli.mjs +209 -0
  37. package/src/lib/video.mjs +427 -0
  38. package/src/lib/webm.mjs +144 -0
  39. package/viewer/index.html +186 -7
package/README.md CHANGED
@@ -27,8 +27,11 @@ npx kaloko doctor
27
27
  ```
28
28
 
29
29
  `init` creates `kaloko.config.yml` and copies the `kaloko` skill to
30
- `.claude/skills/kaloko` (Claude Code) or `skills/kaloko` (Codex). Run it again
31
- after `npm update kaloko` to refresh the skill.
30
+ `.claude/skills/kaloko` (Claude Code) or `skills/kaloko` (Codex). It reads the
31
+ project first: the framework (Next.js, Nuxt, SvelteKit, Remix, Astro, Rails and
32
+ others), the dev server's command and URL, the locales of the i18n config and
33
+ the main routes, and drafts `qa/flows/main-pages.yml` so the first walk works
34
+ without editing. Run it again after `npm update kaloko` to refresh the skill.
32
35
 
33
36
  ## Then ask your agent
34
37
 
@@ -61,7 +64,83 @@ npx kaloko preview # local canvas
61
64
  npx kaloko share --pr # upload to kaloko.app, comment on the PR
62
65
  ```
63
66
 
64
- `npx kaloko help` lists every command. When something does not work (Chrome,
67
+ After a fix, walk only what the change reaches: `npx kaloko affected` explains
68
+ which steps the changed files touch (routes, imports, `affects:` globs in the
69
+ scenario), and `npx kaloko walk --affected` walks them. Several locales walk at
70
+ once with `--concurrency 3`. When a step's selector no longer exists, the walk
71
+ proposes one from the page and prints the patch; `--heal` applies it once the
72
+ step passes.
73
+
74
+ `npx kaloko help` lists every command.
75
+
76
+ ## Pull request previews in CI
77
+
78
+ ```bash
79
+ npx kaloko ci github # or: npx kaloko ci gitlab
80
+ ```
81
+
82
+ writes a workflow that waits for the pull request's preview deployment
83
+ (Vercel, Netlify, Cloudflare Pages, any review app), walks the steps the change
84
+ affects and puts one comment per scenario on the pull request. Add a
85
+ `KALOKO_TOKEN` secret (tester role) and commit it. In short, the GitHub
86
+ workflow does this:
87
+
88
+ ```yaml
89
+ on:
90
+ deployment_status:
91
+ jobs:
92
+ walk:
93
+ if: github.event.deployment_status.state == 'success'
94
+ runs-on: ubuntu-latest
95
+ env:
96
+ KALOKO_TOKEN: ${{ secrets.KALOKO_TOKEN }}
97
+ GH_TOKEN: ${{ github.token }}
98
+ KALOKO_BROWSER_CHANNEL: chromium
99
+ steps:
100
+ - uses: actions/checkout@v4
101
+ with: { fetch-depth: 0 }
102
+ - uses: actions/setup-node@v4
103
+ with: { node-version: 22 }
104
+ - run: npm ci
105
+ - run: npx kaloko browser install --with-deps
106
+ - run: npx kaloko ci preview-url --wait 600 --github-env
107
+ - run: |
108
+ npx kaloko start --scenario qa/flows/checkout.yml --env preview
109
+ npx kaloko walk --ephemeral --affected
110
+ npx kaloko share --pr
111
+ ```
112
+
113
+ Uploads go by content hash: captures that did not change since an earlier run
114
+ are not sent again, and an interrupted `share` picks up where it stopped.
115
+
116
+ ## When a screenshot is not enough
117
+
118
+ ```bash
119
+ npx kaloko walk --record # each step on video, from the previous screen to its capture
120
+ npx kaloko walk --record failed # only the steps that fail (flaky ones, mostly)
121
+ npx kaloko walk --record flow # the whole walk per language, a chapter per step
122
+ npx kaloko video export --format mp4 # the run as one video, a title card and verdict before each step
123
+ ```
124
+
125
+ The canvas plays a step's video next to its screenshot, and the Technical tab
126
+ shows its trace: clicks, typing, requests with status and timing, console
127
+ messages. Click a row and the video jumps there. Web recordings need nothing
128
+ but Chrome (WebM with VP9); with ffmpeg installed you also get MP4 for Safari,
129
+ app recordings (Android `screenrecord`, iOS Simulator, macOS) and the
130
+ walkthrough export. `npx kaloko video tools` says what this machine has.
131
+ Recordings stay in the local preview on Free and Team; Business and Enterprise
132
+ share them with the run.
133
+
134
+ ## Screenshots you already have
135
+
136
+ ```bash
137
+ npx kaloko import playwright-report/ # or test-results/, a trace.zip, results.json
138
+ npx kaloko import walkthrough/ --adhoc # a Cursor or Antigravity walkthrough, a folder of screenshots
139
+ ```
140
+
141
+ lays screenshots, Playwright reports and traces out as a run: matched to the
142
+ scenario's steps (by step id in `steps.json`, URL or title), or as an ad-hoc
143
+ flow when there is no scenario. Build and share it like any run. When something does not work (Chrome,
65
144
  token, environment URL, evaluator key), `npx kaloko doctor` says what to fix.
66
145
 
67
146
  ## Review only what changed
@@ -132,6 +211,7 @@ Nothing in Kaloko is stored in a format only Kaloko can read. Every export opens
132
211
  | Live files (handoff) | `npx kaloko export --live [--version n] --out design/` | the design folder as written |
133
212
  | Shared run | `npx kaloko export --run <id>` | zip with `run.json` (`qawalk.run.v1`), screenshots and HTML snapshots, live HTML of design steps, comments and verdicts, and `index.html`: the canvas, offline |
134
213
  | Documentation captures | `npx kaloko export --scenario <id>` | newest image and HTML of every step, `index.json` |
214
+ | Walkthrough video | `npx kaloko video export [--format mp4\|webm\|gif]` | one video of the run's recordings with title cards; step recordings are WebM (VP9) and MP4 (H.264), traces JSON (`qawalk.trace.v1`) |
135
215
 
136
216
  The schemas of these formats ship with the package (`schemas/`).
137
217
 
@@ -169,7 +249,11 @@ captured HTML, page text and console output.
169
249
  each environment lets you in.
170
250
 
171
251
  Set `browser.channel: chrome` to use an installed Chrome, or run
172
- `npx playwright install chromium`.
252
+ `npx kaloko browser install` for the Chromium of kaloko's Playwright
253
+ (`KALOKO_BROWSER_CHANNEL=chromium` then uses it, which is how CI runs).
254
+ In CI, `KALOKO_PREVIEW_URL`, `KALOKO_PR` and `KALOKO_BASE` come from
255
+ `npx kaloko ci preview-url`; `KALOKO_GITLAB_TOKEN` lets `share --pr` comment on
256
+ GitLab merge requests.
173
257
 
174
258
  ## License
175
259
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "kaloko",
3
- "version": "0.4.9",
3
+ "version": "0.4.11",
4
4
  "description": "See what your AI agents built: every screen, language and device walked, assessed and approved on one canvas.",
5
5
  "keywords": [
6
6
  "qa",
@@ -74,6 +74,22 @@
74
74
  }
75
75
  }
76
76
  },
77
+ "walk": {
78
+ "type": "object",
79
+ "additionalProperties": false,
80
+ "properties": {
81
+ "concurrency": { "type": "integer", "minimum": 1, "maximum": 16, "description": "Locales walked at the same time, each in its own browser context (kaloko walk --concurrency). Default: by CPU in an own browser, one at a time on the shared browser." }
82
+ }
83
+ },
84
+ "capture": {
85
+ "type": "object",
86
+ "additionalProperties": false,
87
+ "description": "Defaults for every scenario of the project; a scenario's capture block wins.",
88
+ "properties": {
89
+ "record": { "$ref": "#/$defs/record" },
90
+ "trace": { "type": "boolean", "default": true, "description": "Keep a trace per walked step (actions, network requests, console) shown under Technical; false or walk --no-trace turns it off." }
91
+ }
92
+ },
77
93
  "share": {
78
94
  "type": "object",
79
95
  "additionalProperties": false,
@@ -85,6 +101,25 @@
85
101
  }
86
102
  },
87
103
  "$defs": {
104
+ "record": {
105
+ "description": "Recordings of walked steps (opt-in): true or step records each step from the previous screen to its capture; flow records the whole walk per locale with a chapter per step; failed keeps only steps that fail; false or off turns it off. Uploaded on Business and Enterprise.",
106
+ "oneOf": [
107
+ { "type": "boolean" },
108
+ { "enum": ["step", "flow", "failed", "off"] },
109
+ {
110
+ "type": "object",
111
+ "additionalProperties": false,
112
+ "properties": {
113
+ "mode": { "enum": ["step", "flow", "failed", "off"], "default": "step" },
114
+ "quality": { "enum": ["low", "medium", "high"], "default": "medium", "description": "low 8 fps up to 960 px, medium 12 fps up to 1280 px, high 24 fps up to 1920 px" },
115
+ "max_seconds": { "type": "integer", "minimum": 1, "maximum": 3600, "description": "Longest kept recording; a longer one keeps its end (default 60 for a step, 900 for a flow)" },
116
+ "codec": { "enum": ["auto", "vp9", "av1", "h264"], "default": "auto", "description": "auto: WebM VP9 (VP8 fallback) from the browser, plus MP4 H.264 when ffmpeg is installed; av1 needs ffmpeg" },
117
+ "mp4": { "oneOf": [{ "type": "boolean" }, { "const": "auto" }], "default": "auto", "description": "MP4 H.264 next to the WebM (Safari before iOS 17.4); needs ffmpeg" },
118
+ "cursor": { "type": "boolean", "default": true, "description": "Draw the pointer and clicks into web recordings" }
119
+ }
120
+ }
121
+ ]
122
+ },
88
123
  "environment": {
89
124
  "type": "object",
90
125
  "additionalProperties": false,
@@ -163,6 +163,25 @@
163
163
  }
164
164
  },
165
165
  "$defs": {
166
+ "record": {
167
+ "description": "Recordings of walked steps (opt-in): true or step records each step from the previous screen to its capture; flow records the whole walk per locale with a chapter per step; failed keeps only steps that fail; false or off turns it off. Uploaded on Business and Enterprise.",
168
+ "oneOf": [
169
+ { "type": "boolean" },
170
+ { "enum": ["step", "flow", "failed", "off"] },
171
+ {
172
+ "type": "object",
173
+ "additionalProperties": false,
174
+ "properties": {
175
+ "mode": { "enum": ["step", "flow", "failed", "off"], "default": "step" },
176
+ "quality": { "enum": ["low", "medium", "high"], "default": "medium", "description": "low 8 fps up to 960 px, medium 12 fps up to 1280 px, high 24 fps up to 1920 px" },
177
+ "max_seconds": { "type": "integer", "minimum": 1, "maximum": 3600, "description": "Longest kept recording; a longer one keeps its end (default 60 for a step, 900 for a flow)" },
178
+ "codec": { "enum": ["auto", "vp9", "av1", "h264"], "default": "auto", "description": "auto: WebM VP9 (VP8 fallback) from the browser, plus MP4 H.264 when ffmpeg is installed; av1 needs ffmpeg" },
179
+ "mp4": { "oneOf": [{ "type": "boolean" }, { "const": "auto" }], "default": "auto", "description": "MP4 H.264 next to the WebM (Safari before iOS 17.4); needs ffmpeg" },
180
+ "cursor": { "type": "boolean", "default": true, "description": "Draw the pointer and clicks into web recordings" }
181
+ }
182
+ }
183
+ ]
184
+ },
166
185
  "slug": {
167
186
  "type": "string",
168
187
  "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$"
@@ -243,6 +262,13 @@
243
262
  "type": "string",
244
263
  "description": "Key from `sessions`; which person performs this step."
245
264
  },
265
+ "affects": {
266
+ "oneOf": [
267
+ { "const": "always" },
268
+ { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }
269
+ ],
270
+ "description": "For `kaloko walk --affected`: globs of repository files (relative to the config) that shape this step, next to the routes and imports Kaloko finds itself; `always` walks the step on every affected walk (sign-in, consent)."
271
+ },
246
272
  "packs": {
247
273
  "type": "array",
248
274
  "items": {
@@ -262,6 +288,7 @@
262
288
  "capture": {
263
289
  "$ref": "#/$defs/capture"
264
290
  },
291
+ "record": { "$ref": "#/$defs/record", "description": "Record this step even when the walk does not (true), or never (false)." },
265
292
  "ignore": {
266
293
  "type": "array",
267
294
  "description": "Regions left out of pixel comparisons (clocks, ads, carousels): a selector, or a rectangle in page pixels of one viewport. kaloko stability suggests them.",
@@ -703,7 +730,9 @@
703
730
  "full_page": {
704
731
  "type": "boolean",
705
732
  "description": "Screenshot of the whole page (default true; false for stack pages: the first screen only)"
706
- }
733
+ },
734
+ "record": { "$ref": "#/$defs/record" },
735
+ "trace": { "type": "boolean", "default": true, "description": "Keep a trace per walked step (actions, network requests, console) shown under Technical; false or walk --no-trace turns it off." }
707
736
  }
708
737
  }
709
738
  }
@@ -13,7 +13,8 @@ description: >
13
13
  design environment (live HTML steps, drafts, versions, the design
14
14
  reference, checking implementation against it), and when implementing a
15
15
  screen that has an approved design (read the step spec, build with its
16
- tokens, check drift).
16
+ tokens, check drift), and when a step should be recorded on video or its
17
+ trace read (actions, requests, console).
17
18
  ---
18
19
 
19
20
  # Kaloko
@@ -48,7 +49,11 @@ the canvas, storage and collaboration are the service.
48
49
  Start the shared browser, walk a scenario on a non-production environment,
49
50
  create test data there (users, roles, carts), capture screens and e-mails,
50
51
  evaluate, build the preview, and fix scenario mechanics (selectors, scripts)
51
- that do not change what is being accepted. On a design environment: change
52
+ that do not change what is being accepted, including applying a selector
53
+ proposal with `kaloko walk --heal` when it names the same element. Walk only
54
+ the steps a change reaches (`--affected`) and import screenshots into a run. Record steps locally
55
+ (`walk --record`) when a screenshot cannot show what a person has to judge, read traces, and
56
+ export a walkthrough video into the run folder. On a design environment: change
52
57
  the design folder, walk, and sync the draft (including taking others' steps
53
58
  with `--theirs`). Move to the trash runs you uploaded in this session that
54
59
  were superseded before anyone looked at them (debug rounds: no comments, no
@@ -56,10 +61,12 @@ verdict), and run `kaloko prune` without `--yes` to see what a clean-up would do
56
61
 
57
62
  ## Ask first
58
63
 
59
- Changing acceptance criteria or scope, any production run, uploading with
64
+ Changing acceptance criteria or scope, any production run, writing or
65
+ replacing a CI workflow (`kaloko ci github|gitlab`), uploading with
60
66
  `kaloko share` (on a design environment: publishing a version), setting a
61
67
  design reference or baseline, merging a design branch, keeping your version of
62
- a step in a sync conflict (`--mine`), turning on `html_view`, a real clean-up
68
+ a step in a sync conflict (`--mine`), turning on `html_view`, turning on recordings for
69
+ a whole project (`capture.record` in the config: storage), a real clean-up
63
70
  (`kaloko prune --yes`), trashing someone else's run, archiving a scenario, and
64
71
  anything the environment profile blocks. Never trash an accepted run, a
65
72
  baseline, a run with review comments, or a run linked from an open PR or issue
@@ -85,6 +92,17 @@ they can do; never write secrets into the config, a scenario or a script.
85
92
  browser with your capabilities: [reference/execution.md](reference/execution.md)
86
93
  - Evaluation, thresholds, the vision evaluator and its reasons, reading
87
94
  results: [reference/evaluation.md](reference/evaluation.md)
95
+ browser with your capabilities, walking only the steps a change reaches
96
+ (`--affected`), several locales at once, and selectors that moved
97
+ (`--heal`): [reference/execution.md](reference/execution.md)
98
+ - Screenshots, walkthroughs, Playwright reports and traces that an agent or a
99
+ tool already produced, laid out as a run: [reference/import.md](reference/import.md)
100
+ - Video of a step or a whole flow, traces (actions, requests, console) and the
101
+ walkthrough video; when recording is worth its storage:
102
+ [reference/recording.md](reference/recording.md)
103
+ - CI: a workflow that walks each pull request's preview deployment and
104
+ comments on it (`kaloko ci github|gitlab`): [reference/ci.md](reference/ci.md)
105
+ - Evaluation, JEV thresholds and reading results: [reference/evaluation.md](reference/evaluation.md)
88
106
  - Environments, access (gates, accounts, sign-in), e-mails, readonly and
89
107
  production rules, and what is yours to arrange for the project's stack:
90
108
  [reference/environments.md](reference/environments.md)
@@ -20,6 +20,16 @@ unclear, ask one focused question instead of guessing.
20
20
  When intent changes, update `ACCEPTANCE.md` first, add a change-log row with
21
21
  date and commit, bump `version` in the scenario, then adjust the scenario.
22
22
 
23
+ In a project without scenarios, `kaloko init` already drafted
24
+ `qa/flows/main-pages.yml` from the route files: one screen per main route in
25
+ the locales of the i18n config, with the dev server's URL as `local`. It walks
26
+ as it is; sharpen the purposes and criteria and add the flows that matter.
27
+
28
+ Screenshots an agent or a test tool already took (a Cursor or Antigravity
29
+ walkthrough, a Playwright report or trace) can start a scenario too:
30
+ `kaloko import <folder> --adhoc` writes one manual step per screen; see
31
+ [import.md](import.md).
32
+
23
33
  For public pages of a site (landings, marketing, documentation) start from a
24
34
  draft: `npx kaloko draft https://example.com/ https://example.com/pricing`
25
35
  fetches the pages and writes a read-only production scenario and its plan
@@ -133,6 +143,12 @@ criteria whose verdict differs between runs on the same commit as flaky;
133
143
  treat a flaky criterion as a selector or timing problem to fix in the
134
144
  scenario, not as a verdict.
135
145
 
146
+ When a step is shaped by files no route reaches (a payment library, a shared
147
+ price component), name them on the step with `affects: [lib/payments/**]`, so
148
+ `kaloko walk --affected` walks it when they change; `affects: always` marks
149
+ steps every walk needs (sign-in, consent). See
150
+ [execution.md](execution.md#only-what-changed).
151
+
136
152
  ## Naming and linkage
137
153
 
138
154
  When the work has a repository and an issue number, make them visible:
@@ -0,0 +1,70 @@
1
+ # CI and pull request previews
2
+
3
+ Kaloko runs in CI with the same commands as on a laptop. `kaloko ci` writes the
4
+ workflow; you only add a token. Writing or replacing a CI workflow changes the
5
+ repository's pipeline: ask the person first.
6
+
7
+ ```bash
8
+ kaloko ci github [--scenario qa/flows/checkout.yml,qa/flows/signup.yml] # .github/workflows/kaloko.yml
9
+ kaloko ci gitlab # kaloko.gitlab-ci.yml (include it)
10
+ kaloko ci github --print # show it, write nothing
11
+ ```
12
+
13
+ What the generated workflow does on every preview deployment of a pull
14
+ request (Vercel, Netlify and Cloudflare Pages report their previews as GitHub
15
+ deployments; `--trigger pull_request` runs on every push instead and waits for
16
+ the deployment):
17
+
18
+ 1. installs the project (`npm ci`, or pnpm, yarn, bun by the lockfile) and the
19
+ Chromium of kaloko's Playwright (`kaloko browser install --with-deps`);
20
+ `KALOKO_BROWSER_CHANNEL=chromium` makes the run use it, so no separate CI
21
+ config is needed;
22
+ 2. `kaloko ci preview-url --wait 600 --github-env` finds the preview URL, the
23
+ pull request and its base branch and puts them into `KALOKO_PREVIEW_URL`,
24
+ `KALOKO_PR`, `KALOKO_BASE`;
25
+ 3. for each scenario: `kaloko affected --check` skips it when the change does
26
+ not reach any of its steps, otherwise `start --env preview`,
27
+ `walk --ephemeral --affected` and `share --pr` (one comment per scenario,
28
+ updated on later pushes).
29
+
30
+ `kaloko ci` also adds the `preview` environment to `kaloko.config.yml` with
31
+ `base_url: ${KALOKO_PREVIEW_URL}`. Scenarios that list `environments:` must
32
+ include `preview`, or `start` refuses them; the command says which.
33
+
34
+ ## Secrets
35
+
36
+ - `KALOKO_TOKEN`: a token with the tester role and the upload scope (Settings →
37
+ API tokens; on Enterprise a service account's token, which no person owns).
38
+ - `TYPESAFE_API_KEY` only when semantic criteria should use your own evaluator
39
+ key instead of the organization's allowance.
40
+ - Vercel deployment protection: a `VERCEL_AUTOMATION_BYPASS_SECRET` secret and
41
+ the commented `access.headers` block of the preview environment.
42
+ - GitLab: `KALOKO_GITLAB_TOKEN` (project access token, `api` scope) for the
43
+ merge request note; set `KALOKO_PREVIEW_URL` in the job when the review app
44
+ URL is known (`https://$CI_COMMIT_REF_SLUG.review.example.com`), or rely on
45
+ `CI_ENVIRONMENT_URL`.
46
+
47
+ ## Where the preview URL comes from
48
+
49
+ `kaloko ci preview-url` takes the first of: `KALOKO_PREVIEW_URL`; the
50
+ `deployment_status` event; `VERCEL_BRANCH_URL`/`VERCEL_URL`,
51
+ `DEPLOY_PRIME_URL`/`DEPLOY_URL` (Netlify), `CF_PAGES_URL`,
52
+ `CI_ENVIRONMENT_URL` (GitLab), `RENDER_EXTERNAL_URL`; then the commit's GitHub
53
+ deployments, skipping production, polled every 10 s up to `--wait` seconds.
54
+ Only the URL goes to stdout, so `export KALOKO_PREVIEW_URL="$(kaloko ci preview-url)"`
55
+ works in any CI.
56
+
57
+ ## Affected steps in CI
58
+
59
+ `walk --affected` needs the history: check out with `fetch-depth: 0` (GitHub)
60
+ or `GIT_DEPTH: "0"` (GitLab). The base is `--base`, else `KALOKO_BASE`,
61
+ `GITHUB_BASE_REF`, the merge request target, then `origin/main`. URLs captured
62
+ in the newest shared run of the scenario count too, so a fresh CI checkout
63
+ maps routes as well as a laptop with earlier runs. See
64
+ [execution.md](execution.md#only-what-changed).
65
+
66
+ ## Done means
67
+
68
+ The workflow file and the config change are committed after the person agreed,
69
+ the secret is named for the person to add, and you report the first run's PR
70
+ comment or why it did not appear (no preview deployment, missing token).
@@ -12,7 +12,9 @@ loop teams use:
12
12
  request and `gh` is signed in, `kaloko share --pr` posts one comment per
13
13
  scenario on it (summary line, canvas link, criteria that need attention
14
14
  with evidence) and updates that same comment on every later run, so the
15
- PR never fills up with Kaloko noise. Otherwise write the comment yourself:
15
+ PR never fills up with Kaloko noise. In CI `KALOKO_PR` names the pull
16
+ request of a detached checkout, and on GitLab merge requests the comment is a
17
+ note on the MR (`KALOKO_GITLAB_TOKEN`). Otherwise write the comment yourself:
16
18
  what was tested (scenario, environment, locales), the summary line
17
19
  (`n/m steps passed`), every failing or `needs_review` criterion with
18
20
  evidence, and the canvas URL. Reviewers accept, return or comment on the
@@ -255,6 +257,12 @@ Everything above is available over the API with an organization token:
255
257
  - `GET /api/v1/orgs/<org>/runs/<id>/export.zip` (`kaloko export`): the whole
256
258
  run as a zip before it expires; keep it with the issue when the evidence
257
259
  must outlive the retention period.
260
+ - Uploads by content hash: `POST …/blobs/missing` with `{"hashes": [sha256…]}`
261
+ answers what the organization does not store yet, `PUT …/blobs/<sha256>`
262
+ (header `X-QAwalk-Type: png|webp|html|json`) stores one, and the run's file
263
+ plan `POST …/runs/<id>/files` takes `{ path, bytes, blob }` entries that
264
+ point at stored blobs. `kaloko share` does this: unchanged captures are not
265
+ sent again and a share that broke off resumes when run again.
258
266
  - MCP server for agents: see [mcp.md](mcp.md).
259
267
 
260
268
  Read-only tokens may list but not change anything; keep CI tokens read-only
@@ -13,6 +13,11 @@ them; do not work around a block.
13
13
  Test data preparation for review and staging belongs to the project's CI
14
14
  prepare step; assume it exists and create only what the scenario itself needs.
15
15
 
16
+ Pull request previews (Vercel, Netlify, Cloudflare Pages, review apps) are an
17
+ environment whose URL comes from CI: `preview: { base_url: ${KALOKO_PREVIEW_URL} }`,
18
+ which `kaloko ci github|gitlab` adds and `kaloko ci preview-url` fills; see
19
+ [ci.md](ci.md).
20
+
16
21
  ## Who does what
17
22
 
18
23
  Kaloko needs evidence in its format: captures of screens and e-mails in a run.
@@ -91,3 +91,59 @@ screenshots before applying; a region that changed because the page really
91
91
  changed is not noise. `--same-commit` compares only runs of the same commit,
92
92
  `--runs a,b` names them, `--last N` sets how many runs are compared. Capture
93
93
  the steps again after applying: the capture records the ignored regions.
94
+
95
+ ## Only what changed
96
+
97
+ After a fix, walk the steps the change reaches instead of the whole flow:
98
+
99
+ ```bash
100
+ kaloko affected # which steps, and why (against origin/main; --base <ref>)
101
+ kaloko walk --affected [--carry] # walk them; --carry copies the other steps from the previous local run
102
+ ```
103
+
104
+ A changed file reaches a step when it is a route file whose route matches the
105
+ step's path or a URL captured for it earlier (Next.js `app/` and `pages/`,
106
+ Nuxt, SvelteKit, Remix, Astro, Rails views and controllers), when it is
107
+ imported by such a file, when it is global (layouts, global styles,
108
+ dependencies; i18n messages of one locale keep the walk in that locale), when
109
+ it is the step's script or the step changed in the scenario, or when it
110
+ matches the step's `affects:` globs. Steps on the main path before a selected
111
+ step are walked first (`--no-upstream` skips them); steps with
112
+ `affects: always` (sign-in, consent) are always walked. Files nothing maps are
113
+ listed as unmapped: add `affects:` to the steps they shape, or pass
114
+ `--if-unmapped all` to walk everything then. Changes to tests, docs and CI are
115
+ ignored.
116
+
117
+ ```yaml
118
+ - id: checkout
119
+ path: /checkout
120
+ affects: [lib/payments/**, src/components/PriceSummary.tsx]
121
+ - id: consent
122
+ affects: always
123
+ ```
124
+
125
+ Report the selection with the run: which steps were walked and why, and that
126
+ the rest was not walked (or carried from which run).
127
+
128
+ ## Several locales at once
129
+
130
+ Locales are independent, so `walk --concurrency N` walks N of them at the same
131
+ time, each in its own browser context (a new visitor per locale). In an own
132
+ browser (`--ephemeral`, CI) the default follows the CPU (up to 4); on the
133
+ shared browser it stays one at a time unless you ask, because people may rely
134
+ on its signed-in tabs. Each locale's lines are printed together, in locale
135
+ order. Set `walk.concurrency` in the config for the project's default.
136
+ Viewports of one step are still captured one after another on the same page.
137
+
138
+ ## Selectors that moved
139
+
140
+ When a step script waits in vain for a selector (the button lost its id, the
141
+ class was renamed), the walk proposes a replacement from the page as it is
142
+ now: a test id, then role and accessible name, then a stable id, then the text,
143
+ preferring the element the old selector matched in the step's previous
144
+ capture. It prints the reason and a patch of the step script. Review it; if it
145
+ is the same element, `kaloko walk --steps <id> --heal` runs the step with the
146
+ patched script and writes the patch only when the step passes. Applying a heal
147
+ is fixing scenario mechanics; if the proposal points at a different element,
148
+ the product changed: report it, do not heal it. `--no-heal` skips the
149
+ proposals.
@@ -0,0 +1,66 @@
1
+ # Importing screenshots, walkthroughs and traces
2
+
3
+ Agents and test tools already leave screenshots behind: Cursor and
4
+ Antigravity walkthroughs, Claude Code screenshots, Playwright reports and
5
+ traces. `kaloko import` lays them out as a run, so they are reviewed and
6
+ signed off on the canvas like a walk.
7
+
8
+ ```bash
9
+ kaloko import shots/ # into the current run (kaloko start first)
10
+ kaloko import playwright-report/ --scenario qa/flows/checkout.yml --env staging
11
+ kaloko import test-results/checkout/trace.zip --map "click Pay=payment"
12
+ kaloko import walkthrough.md --adhoc # no scenario: an ad-hoc flow of the screens
13
+ kaloko evaluate && kaloko preview # then kaloko share, as usual
14
+ ```
15
+
16
+ Where the screens go: the current run, or a new run of `--scenario`, or with no
17
+ run (or `--adhoc`) an ad-hoc flow written to `qa/flows/import-<title>.yml`: one
18
+ manual screen per image, in order, read-only.
19
+
20
+ ## Sources
21
+
22
+ - **A folder with `steps.json`** (also `walkthrough.json`, `kaloko-import.json`):
23
+ ```json
24
+ { "title": "Checkout", "tool": "cursor",
25
+ "steps": [
26
+ { "title": "Cart", "image": "01-cart.png", "step": "cart", "url": "https://shop.test/cart",
27
+ "notes": "Two items, total 41 €", "locale": "en", "viewport": "desktop" },
28
+ { "title": "Payment", "image": "02-payment.png" }
29
+ ] }
30
+ ```
31
+ Only `image` is required. `step` names the scenario step; `viewport` and
32
+ `locale` must be in the run's scope or are chosen for you.
33
+ - **A Markdown walkthrough** (`walkthrough.md`, `README.md`, any `.md` in the
34
+ folder): every image `![alt](file.png)` in order, titled by the heading above
35
+ it, with the text between as notes. Links to `.webm`/`.mp4` are recordings.
36
+ - **A plain folder of images**: in file name order (`01-…`, `02-…`), titled by
37
+ the name. A folder of folders (Playwright `test-results/`) reads each
38
+ sub-folder as one test.
39
+ - **Playwright**: the JSON reporter's file (`--reporter=json`), an HTML report
40
+ folder (`playwright-report/`) or a `trace.zip`. Screenshot attachments become
41
+ screens; a test without them contributes its trace: one screen after each
42
+ page action (goto, click, fill, press …).
43
+
44
+ `--format folder|markdown|playwright-json|playwright-html|trace` overrides the
45
+ detection.
46
+
47
+ ## Mapping to steps
48
+
49
+ In this order: the item's `step`, `--map "Title=step,…"` (a title, group or
50
+ file name containing the label), the URL path against step paths, then the
51
+ title against step ids and titles. A second screen for the same step, locale
52
+ and viewport becomes an attachment of the step. Unmatched screens are listed
53
+ with the command that maps them; nothing is dropped silently. The viewport is
54
+ the run viewport closest to the image width (2× and 3× screenshots count by
55
+ their CSS width).
56
+
57
+ ## What an imported screen is
58
+
59
+ The image, encoded like a walked capture, with the tool and the source file.
60
+ There is no page behind it: deterministic criteria stay "not evaluated" until
61
+ the step is walked, and semantic questions are not asked without page text, so
62
+ a person decides on the canvas. Recordings are kept in the run folder
63
+ (`imported/`) and are not uploaded yet.
64
+
65
+ Report it as imported evidence, not as a walk: name the tool and say which
66
+ steps still need a real walk.
@@ -28,6 +28,7 @@ file.
28
28
  | `list_notifications` | the token owner's inbox (`unread_only`, `limit`) |
29
29
  | `list_projects` | projects of the organization with counts; the slug is the `project` filter of `list_runs` and the value for `project:` in a scenario |
30
30
  | `get_criteria_verdicts` | reviewer verdicts on manual or overridden criteria of a run |
31
+ | `get_step_recording` | a step's recording and trace (`run_id`, `step`, `locale`, `viewport`, `trace: true` for the full trace): video URLs, duration, its chapter of a flow recording, failed attempts, the trace summary (failed requests, console errors, slowest request); see [recording.md](recording.md). `get_run_summary` lists trace summaries per step too |
31
32
  | `get_review_queue` | what a person still has to decide in a run (`run_id`, or `scenario` / nothing for the newest run): steps in review order with reasons, decisions, approvals carried over unchanged steps with their origin, counts and `review_url` to give the person |
32
33
  | `decide_step` | records your (the agent's) decision on a step: `approve`, or `return` with a `note`; marked as an agent's and not counted as a person's. Never use it to approve for a person |
33
34
  | `revoke_carried_approval` | revokes an approval carried over to a step that looks unchanged but should be looked at again (`run_id`, `step`) |
@@ -35,6 +36,7 @@ file.
35
36
  | `list_step_changes` | documentation layer: steps whose capture or status changed since the previous accepted run or `since` (`scenario`, `since`, `accepted`); see [documentation.md](documentation.md) |
36
37
  | `get_started` | where the organization stands (`empty`, `sample`, `active`), its sample run, the next steps and a prompt; call it in a new project, see [first-run.md](first-run.md) |
37
38
  | `add_sample_run` | adds the sample run (or returns the one there is); needs an upload or full token and the Developer / Tester role |
39
+ | `get_affected_steps` | which steps of a scenario changed files reach, with the reason per step (`scenario`, `changed_files`, `run_id`, `upstream`); route files are matched against the step paths and URLs of the newest run. Imports and `affects:` globs need the repository: `kaloko affected` follows them |
38
40
 
39
41
  ## Comment tools
40
42