kaloko 0.4.9 → 0.4.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +88 -4
- package/package.json +1 -1
- package/schemas/config.v1.json +35 -0
- package/schemas/scenario.v1.json +30 -1
- package/skills/kaloko/SKILL.md +22 -4
- package/skills/kaloko/reference/acceptance.md +16 -0
- package/skills/kaloko/reference/ci.md +70 -0
- package/skills/kaloko/reference/collaboration.md +9 -1
- package/skills/kaloko/reference/environments.md +5 -0
- package/skills/kaloko/reference/execution.md +56 -0
- package/skills/kaloko/reference/import.md +66 -0
- package/skills/kaloko/reference/mcp.md +2 -0
- package/skills/kaloko/reference/recording.md +95 -0
- package/src/cli.mjs +467 -100
- package/src/lib/affected-core.mjs +242 -0
- package/src/lib/affected.mjs +233 -0
- package/src/lib/anim.mjs +125 -0
- package/src/lib/apps/recording-native.mjs +212 -0
- package/src/lib/build.mjs +60 -4
- package/src/lib/ci.mjs +207 -0
- package/src/lib/config.mjs +2 -0
- package/src/lib/doctor.mjs +5 -1
- package/src/lib/evaluate.mjs +4 -2
- package/src/lib/git.mjs +6 -2
- package/src/lib/heal.mjs +207 -0
- package/src/lib/import.mjs +374 -0
- package/src/lib/init-detect.mjs +219 -0
- package/src/lib/onboarding.mjs +15 -8
- package/src/lib/pr.mjs +24 -2
- package/src/lib/recording.mjs +373 -0
- package/src/lib/scenario.mjs +2 -1
- package/src/lib/service.mjs +7 -1
- package/src/lib/trace.mjs +109 -0
- package/src/lib/unzip.mjs +31 -0
- package/src/lib/upload.mjs +114 -0
- package/src/lib/video-cli.mjs +209 -0
- package/src/lib/video.mjs +427 -0
- package/src/lib/webm.mjs +144 -0
- package/viewer/index.html +186 -7
package/README.md
CHANGED
|
@@ -27,8 +27,11 @@ npx kaloko doctor
|
|
|
27
27
|
```
|
|
28
28
|
|
|
29
29
|
`init` creates `kaloko.config.yml` and copies the `kaloko` skill to
|
|
30
|
-
`.claude/skills/kaloko` (Claude Code) or `skills/kaloko` (Codex).
|
|
31
|
-
|
|
30
|
+
`.claude/skills/kaloko` (Claude Code) or `skills/kaloko` (Codex). It reads the
|
|
31
|
+
project first: the framework (Next.js, Nuxt, SvelteKit, Remix, Astro, Rails and
|
|
32
|
+
others), the dev server's command and URL, the locales of the i18n config and
|
|
33
|
+
the main routes, and drafts `qa/flows/main-pages.yml` so the first walk works
|
|
34
|
+
without editing. Run it again after `npm update kaloko` to refresh the skill.
|
|
32
35
|
|
|
33
36
|
## Then ask your agent
|
|
34
37
|
|
|
@@ -61,7 +64,83 @@ npx kaloko preview # local canvas
|
|
|
61
64
|
npx kaloko share --pr # upload to kaloko.app, comment on the PR
|
|
62
65
|
```
|
|
63
66
|
|
|
64
|
-
|
|
67
|
+
After a fix, walk only what the change reaches: `npx kaloko affected` explains
|
|
68
|
+
which steps the changed files touch (routes, imports, `affects:` globs in the
|
|
69
|
+
scenario), and `npx kaloko walk --affected` walks them. Several locales walk at
|
|
70
|
+
once with `--concurrency 3`. When a step's selector no longer exists, the walk
|
|
71
|
+
proposes one from the page and prints the patch; `--heal` applies it once the
|
|
72
|
+
step passes.
|
|
73
|
+
|
|
74
|
+
`npx kaloko help` lists every command.
|
|
75
|
+
|
|
76
|
+
## Pull request previews in CI
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
npx kaloko ci github # or: npx kaloko ci gitlab
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
writes a workflow that waits for the pull request's preview deployment
|
|
83
|
+
(Vercel, Netlify, Cloudflare Pages, any review app), walks the steps the change
|
|
84
|
+
affects and puts one comment per scenario on the pull request. Add a
|
|
85
|
+
`KALOKO_TOKEN` secret (tester role) and commit it. In short, the GitHub
|
|
86
|
+
workflow does this:
|
|
87
|
+
|
|
88
|
+
```yaml
|
|
89
|
+
on:
|
|
90
|
+
deployment_status:
|
|
91
|
+
jobs:
|
|
92
|
+
walk:
|
|
93
|
+
if: github.event.deployment_status.state == 'success'
|
|
94
|
+
runs-on: ubuntu-latest
|
|
95
|
+
env:
|
|
96
|
+
KALOKO_TOKEN: ${{ secrets.KALOKO_TOKEN }}
|
|
97
|
+
GH_TOKEN: ${{ github.token }}
|
|
98
|
+
KALOKO_BROWSER_CHANNEL: chromium
|
|
99
|
+
steps:
|
|
100
|
+
- uses: actions/checkout@v4
|
|
101
|
+
with: { fetch-depth: 0 }
|
|
102
|
+
- uses: actions/setup-node@v4
|
|
103
|
+
with: { node-version: 22 }
|
|
104
|
+
- run: npm ci
|
|
105
|
+
- run: npx kaloko browser install --with-deps
|
|
106
|
+
- run: npx kaloko ci preview-url --wait 600 --github-env
|
|
107
|
+
- run: |
|
|
108
|
+
npx kaloko start --scenario qa/flows/checkout.yml --env preview
|
|
109
|
+
npx kaloko walk --ephemeral --affected
|
|
110
|
+
npx kaloko share --pr
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Uploads go by content hash: captures that did not change since an earlier run
|
|
114
|
+
are not sent again, and an interrupted `share` picks up where it stopped.
|
|
115
|
+
|
|
116
|
+
## When a screenshot is not enough
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
npx kaloko walk --record # each step on video, from the previous screen to its capture
|
|
120
|
+
npx kaloko walk --record failed # only the steps that fail (flaky ones, mostly)
|
|
121
|
+
npx kaloko walk --record flow # the whole walk per language, a chapter per step
|
|
122
|
+
npx kaloko video export --format mp4 # the run as one video, a title card and verdict before each step
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
The canvas plays a step's video next to its screenshot, and the Technical tab
|
|
126
|
+
shows its trace: clicks, typing, requests with status and timing, console
|
|
127
|
+
messages. Click a row and the video jumps there. Web recordings need nothing
|
|
128
|
+
but Chrome (WebM with VP9); with ffmpeg installed you also get MP4 for Safari,
|
|
129
|
+
app recordings (Android `screenrecord`, iOS Simulator, macOS) and the
|
|
130
|
+
walkthrough export. `npx kaloko video tools` says what this machine has.
|
|
131
|
+
Recordings stay in the local preview on Free and Team; Business and Enterprise
|
|
132
|
+
share them with the run.
|
|
133
|
+
|
|
134
|
+
## Screenshots you already have
|
|
135
|
+
|
|
136
|
+
```bash
|
|
137
|
+
npx kaloko import playwright-report/ # or test-results/, a trace.zip, results.json
|
|
138
|
+
npx kaloko import walkthrough/ --adhoc # a Cursor or Antigravity walkthrough, a folder of screenshots
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
lays screenshots, Playwright reports and traces out as a run: matched to the
|
|
142
|
+
scenario's steps (by step id in `steps.json`, URL or title), or as an ad-hoc
|
|
143
|
+
flow when there is no scenario. Build and share it like any run. When something does not work (Chrome,
|
|
65
144
|
token, environment URL, evaluator key), `npx kaloko doctor` says what to fix.
|
|
66
145
|
|
|
67
146
|
## Review only what changed
|
|
@@ -132,6 +211,7 @@ Nothing in Kaloko is stored in a format only Kaloko can read. Every export opens
|
|
|
132
211
|
| Live files (handoff) | `npx kaloko export --live [--version n] --out design/` | the design folder as written |
|
|
133
212
|
| Shared run | `npx kaloko export --run <id>` | zip with `run.json` (`qawalk.run.v1`), screenshots and HTML snapshots, live HTML of design steps, comments and verdicts, and `index.html`: the canvas, offline |
|
|
134
213
|
| Documentation captures | `npx kaloko export --scenario <id>` | newest image and HTML of every step, `index.json` |
|
|
214
|
+
| Walkthrough video | `npx kaloko video export [--format mp4\|webm\|gif]` | one video of the run's recordings with title cards; step recordings are WebM (VP9) and MP4 (H.264), traces JSON (`qawalk.trace.v1`) |
|
|
135
215
|
|
|
136
216
|
The schemas of these formats ship with the package (`schemas/`).
|
|
137
217
|
|
|
@@ -169,7 +249,11 @@ captured HTML, page text and console output.
|
|
|
169
249
|
each environment lets you in.
|
|
170
250
|
|
|
171
251
|
Set `browser.channel: chrome` to use an installed Chrome, or run
|
|
172
|
-
`npx
|
|
252
|
+
`npx kaloko browser install` for the Chromium of kaloko's Playwright
|
|
253
|
+
(`KALOKO_BROWSER_CHANNEL=chromium` then uses it, which is how CI runs).
|
|
254
|
+
In CI, `KALOKO_PREVIEW_URL`, `KALOKO_PR` and `KALOKO_BASE` come from
|
|
255
|
+
`npx kaloko ci preview-url`; `KALOKO_GITLAB_TOKEN` lets `share --pr` comment on
|
|
256
|
+
GitLab merge requests.
|
|
173
257
|
|
|
174
258
|
## License
|
|
175
259
|
|
package/package.json
CHANGED
package/schemas/config.v1.json
CHANGED
|
@@ -74,6 +74,22 @@
|
|
|
74
74
|
}
|
|
75
75
|
}
|
|
76
76
|
},
|
|
77
|
+
"walk": {
|
|
78
|
+
"type": "object",
|
|
79
|
+
"additionalProperties": false,
|
|
80
|
+
"properties": {
|
|
81
|
+
"concurrency": { "type": "integer", "minimum": 1, "maximum": 16, "description": "Locales walked at the same time, each in its own browser context (kaloko walk --concurrency). Default: by CPU in an own browser, one at a time on the shared browser." }
|
|
82
|
+
}
|
|
83
|
+
},
|
|
84
|
+
"capture": {
|
|
85
|
+
"type": "object",
|
|
86
|
+
"additionalProperties": false,
|
|
87
|
+
"description": "Defaults for every scenario of the project; a scenario's capture block wins.",
|
|
88
|
+
"properties": {
|
|
89
|
+
"record": { "$ref": "#/$defs/record" },
|
|
90
|
+
"trace": { "type": "boolean", "default": true, "description": "Keep a trace per walked step (actions, network requests, console) shown under Technical; false or walk --no-trace turns it off." }
|
|
91
|
+
}
|
|
92
|
+
},
|
|
77
93
|
"share": {
|
|
78
94
|
"type": "object",
|
|
79
95
|
"additionalProperties": false,
|
|
@@ -85,6 +101,25 @@
|
|
|
85
101
|
}
|
|
86
102
|
},
|
|
87
103
|
"$defs": {
|
|
104
|
+
"record": {
|
|
105
|
+
"description": "Recordings of walked steps (opt-in): true or step records each step from the previous screen to its capture; flow records the whole walk per locale with a chapter per step; failed keeps only steps that fail; false or off turns it off. Uploaded on Business and Enterprise.",
|
|
106
|
+
"oneOf": [
|
|
107
|
+
{ "type": "boolean" },
|
|
108
|
+
{ "enum": ["step", "flow", "failed", "off"] },
|
|
109
|
+
{
|
|
110
|
+
"type": "object",
|
|
111
|
+
"additionalProperties": false,
|
|
112
|
+
"properties": {
|
|
113
|
+
"mode": { "enum": ["step", "flow", "failed", "off"], "default": "step" },
|
|
114
|
+
"quality": { "enum": ["low", "medium", "high"], "default": "medium", "description": "low 8 fps up to 960 px, medium 12 fps up to 1280 px, high 24 fps up to 1920 px" },
|
|
115
|
+
"max_seconds": { "type": "integer", "minimum": 1, "maximum": 3600, "description": "Longest kept recording; a longer one keeps its end (default 60 for a step, 900 for a flow)" },
|
|
116
|
+
"codec": { "enum": ["auto", "vp9", "av1", "h264"], "default": "auto", "description": "auto: WebM VP9 (VP8 fallback) from the browser, plus MP4 H.264 when ffmpeg is installed; av1 needs ffmpeg" },
|
|
117
|
+
"mp4": { "oneOf": [{ "type": "boolean" }, { "const": "auto" }], "default": "auto", "description": "MP4 H.264 next to the WebM (Safari before iOS 17.4); needs ffmpeg" },
|
|
118
|
+
"cursor": { "type": "boolean", "default": true, "description": "Draw the pointer and clicks into web recordings" }
|
|
119
|
+
}
|
|
120
|
+
}
|
|
121
|
+
]
|
|
122
|
+
},
|
|
88
123
|
"environment": {
|
|
89
124
|
"type": "object",
|
|
90
125
|
"additionalProperties": false,
|
package/schemas/scenario.v1.json
CHANGED
|
@@ -163,6 +163,25 @@
|
|
|
163
163
|
}
|
|
164
164
|
},
|
|
165
165
|
"$defs": {
|
|
166
|
+
"record": {
|
|
167
|
+
"description": "Recordings of walked steps (opt-in): true or step records each step from the previous screen to its capture; flow records the whole walk per locale with a chapter per step; failed keeps only steps that fail; false or off turns it off. Uploaded on Business and Enterprise.",
|
|
168
|
+
"oneOf": [
|
|
169
|
+
{ "type": "boolean" },
|
|
170
|
+
{ "enum": ["step", "flow", "failed", "off"] },
|
|
171
|
+
{
|
|
172
|
+
"type": "object",
|
|
173
|
+
"additionalProperties": false,
|
|
174
|
+
"properties": {
|
|
175
|
+
"mode": { "enum": ["step", "flow", "failed", "off"], "default": "step" },
|
|
176
|
+
"quality": { "enum": ["low", "medium", "high"], "default": "medium", "description": "low 8 fps up to 960 px, medium 12 fps up to 1280 px, high 24 fps up to 1920 px" },
|
|
177
|
+
"max_seconds": { "type": "integer", "minimum": 1, "maximum": 3600, "description": "Longest kept recording; a longer one keeps its end (default 60 for a step, 900 for a flow)" },
|
|
178
|
+
"codec": { "enum": ["auto", "vp9", "av1", "h264"], "default": "auto", "description": "auto: WebM VP9 (VP8 fallback) from the browser, plus MP4 H.264 when ffmpeg is installed; av1 needs ffmpeg" },
|
|
179
|
+
"mp4": { "oneOf": [{ "type": "boolean" }, { "const": "auto" }], "default": "auto", "description": "MP4 H.264 next to the WebM (Safari before iOS 17.4); needs ffmpeg" },
|
|
180
|
+
"cursor": { "type": "boolean", "default": true, "description": "Draw the pointer and clicks into web recordings" }
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
]
|
|
184
|
+
},
|
|
166
185
|
"slug": {
|
|
167
186
|
"type": "string",
|
|
168
187
|
"pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$"
|
|
@@ -243,6 +262,13 @@
|
|
|
243
262
|
"type": "string",
|
|
244
263
|
"description": "Key from `sessions`; which person performs this step."
|
|
245
264
|
},
|
|
265
|
+
"affects": {
|
|
266
|
+
"oneOf": [
|
|
267
|
+
{ "const": "always" },
|
|
268
|
+
{ "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }
|
|
269
|
+
],
|
|
270
|
+
"description": "For `kaloko walk --affected`: globs of repository files (relative to the config) that shape this step, next to the routes and imports Kaloko finds itself; `always` walks the step on every affected walk (sign-in, consent)."
|
|
271
|
+
},
|
|
246
272
|
"packs": {
|
|
247
273
|
"type": "array",
|
|
248
274
|
"items": {
|
|
@@ -262,6 +288,7 @@
|
|
|
262
288
|
"capture": {
|
|
263
289
|
"$ref": "#/$defs/capture"
|
|
264
290
|
},
|
|
291
|
+
"record": { "$ref": "#/$defs/record", "description": "Record this step even when the walk does not (true), or never (false)." },
|
|
265
292
|
"ignore": {
|
|
266
293
|
"type": "array",
|
|
267
294
|
"description": "Regions left out of pixel comparisons (clocks, ads, carousels): a selector, or a rectangle in page pixels of one viewport. kaloko stability suggests them.",
|
|
@@ -703,7 +730,9 @@
|
|
|
703
730
|
"full_page": {
|
|
704
731
|
"type": "boolean",
|
|
705
732
|
"description": "Screenshot of the whole page (default true; false for stack pages: the first screen only)"
|
|
706
|
-
}
|
|
733
|
+
},
|
|
734
|
+
"record": { "$ref": "#/$defs/record" },
|
|
735
|
+
"trace": { "type": "boolean", "default": true, "description": "Keep a trace per walked step (actions, network requests, console) shown under Technical; false or walk --no-trace turns it off." }
|
|
707
736
|
}
|
|
708
737
|
}
|
|
709
738
|
}
|
package/skills/kaloko/SKILL.md
CHANGED
|
@@ -13,7 +13,8 @@ description: >
|
|
|
13
13
|
design environment (live HTML steps, drafts, versions, the design
|
|
14
14
|
reference, checking implementation against it), and when implementing a
|
|
15
15
|
screen that has an approved design (read the step spec, build with its
|
|
16
|
-
tokens, check drift)
|
|
16
|
+
tokens, check drift), and when a step should be recorded on video or its
|
|
17
|
+
trace read (actions, requests, console).
|
|
17
18
|
---
|
|
18
19
|
|
|
19
20
|
# Kaloko
|
|
@@ -48,7 +49,11 @@ the canvas, storage and collaboration are the service.
|
|
|
48
49
|
Start the shared browser, walk a scenario on a non-production environment,
|
|
49
50
|
create test data there (users, roles, carts), capture screens and e-mails,
|
|
50
51
|
evaluate, build the preview, and fix scenario mechanics (selectors, scripts)
|
|
51
|
-
that do not change what is being accepted
|
|
52
|
+
that do not change what is being accepted, including applying a selector
|
|
53
|
+
proposal with `kaloko walk --heal` when it names the same element. Walk only
|
|
54
|
+
the steps a change reaches (`--affected`) and import screenshots into a run. Record steps locally
|
|
55
|
+
(`walk --record`) when a screenshot cannot show what a person has to judge, read traces, and
|
|
56
|
+
export a walkthrough video into the run folder. On a design environment: change
|
|
52
57
|
the design folder, walk, and sync the draft (including taking others' steps
|
|
53
58
|
with `--theirs`). Move to the trash runs you uploaded in this session that
|
|
54
59
|
were superseded before anyone looked at them (debug rounds: no comments, no
|
|
@@ -56,10 +61,12 @@ verdict), and run `kaloko prune` without `--yes` to see what a clean-up would do
|
|
|
56
61
|
|
|
57
62
|
## Ask first
|
|
58
63
|
|
|
59
|
-
Changing acceptance criteria or scope, any production run,
|
|
64
|
+
Changing acceptance criteria or scope, any production run, writing or
|
|
65
|
+
replacing a CI workflow (`kaloko ci github|gitlab`), uploading with
|
|
60
66
|
`kaloko share` (on a design environment: publishing a version), setting a
|
|
61
67
|
design reference or baseline, merging a design branch, keeping your version of
|
|
62
|
-
a step in a sync conflict (`--mine`), turning on `html_view`,
|
|
68
|
+
a step in a sync conflict (`--mine`), turning on `html_view`, turning on recordings for
|
|
69
|
+
a whole project (`capture.record` in the config: storage), a real clean-up
|
|
63
70
|
(`kaloko prune --yes`), trashing someone else's run, archiving a scenario, and
|
|
64
71
|
anything the environment profile blocks. Never trash an accepted run, a
|
|
65
72
|
baseline, a run with review comments, or a run linked from an open PR or issue
|
|
@@ -85,6 +92,17 @@ they can do; never write secrets into the config, a scenario or a script.
|
|
|
85
92
|
browser with your capabilities: [reference/execution.md](reference/execution.md)
|
|
86
93
|
- Evaluation, thresholds, the vision evaluator and its reasons, reading
|
|
87
94
|
results: [reference/evaluation.md](reference/evaluation.md)
|
|
95
|
+
browser with your capabilities, walking only the steps a change reaches
|
|
96
|
+
(`--affected`), several locales at once, and selectors that moved
|
|
97
|
+
(`--heal`): [reference/execution.md](reference/execution.md)
|
|
98
|
+
- Screenshots, walkthroughs, Playwright reports and traces that an agent or a
|
|
99
|
+
tool already produced, laid out as a run: [reference/import.md](reference/import.md)
|
|
100
|
+
- Video of a step or a whole flow, traces (actions, requests, console) and the
|
|
101
|
+
walkthrough video; when recording is worth its storage:
|
|
102
|
+
[reference/recording.md](reference/recording.md)
|
|
103
|
+
- CI: a workflow that walks each pull request's preview deployment and
|
|
104
|
+
comments on it (`kaloko ci github|gitlab`): [reference/ci.md](reference/ci.md)
|
|
105
|
+
- Evaluation, JEV thresholds and reading results: [reference/evaluation.md](reference/evaluation.md)
|
|
88
106
|
- Environments, access (gates, accounts, sign-in), e-mails, readonly and
|
|
89
107
|
production rules, and what is yours to arrange for the project's stack:
|
|
90
108
|
[reference/environments.md](reference/environments.md)
|
|
@@ -20,6 +20,16 @@ unclear, ask one focused question instead of guessing.
|
|
|
20
20
|
When intent changes, update `ACCEPTANCE.md` first, add a change-log row with
|
|
21
21
|
date and commit, bump `version` in the scenario, then adjust the scenario.
|
|
22
22
|
|
|
23
|
+
In a project without scenarios, `kaloko init` already drafted
|
|
24
|
+
`qa/flows/main-pages.yml` from the route files: one screen per main route in
|
|
25
|
+
the locales of the i18n config, with the dev server's URL as `local`. It walks
|
|
26
|
+
as it is; sharpen the purposes and criteria and add the flows that matter.
|
|
27
|
+
|
|
28
|
+
Screenshots an agent or a test tool already took (a Cursor or Antigravity
|
|
29
|
+
walkthrough, a Playwright report or trace) can start a scenario too:
|
|
30
|
+
`kaloko import <folder> --adhoc` writes one manual step per screen; see
|
|
31
|
+
[import.md](import.md).
|
|
32
|
+
|
|
23
33
|
For public pages of a site (landings, marketing, documentation) start from a
|
|
24
34
|
draft: `npx kaloko draft https://example.com/ https://example.com/pricing`
|
|
25
35
|
fetches the pages and writes a read-only production scenario and its plan
|
|
@@ -133,6 +143,12 @@ criteria whose verdict differs between runs on the same commit as flaky;
|
|
|
133
143
|
treat a flaky criterion as a selector or timing problem to fix in the
|
|
134
144
|
scenario, not as a verdict.
|
|
135
145
|
|
|
146
|
+
When a step is shaped by files no route reaches (a payment library, a shared
|
|
147
|
+
price component), name them on the step with `affects: [lib/payments/**]`, so
|
|
148
|
+
`kaloko walk --affected` walks it when they change; `affects: always` marks
|
|
149
|
+
steps every walk needs (sign-in, consent). See
|
|
150
|
+
[execution.md](execution.md#only-what-changed).
|
|
151
|
+
|
|
136
152
|
## Naming and linkage
|
|
137
153
|
|
|
138
154
|
When the work has a repository and an issue number, make them visible:
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# CI and pull request previews
|
|
2
|
+
|
|
3
|
+
Kaloko runs in CI with the same commands as on a laptop. `kaloko ci` writes the
|
|
4
|
+
workflow; you only add a token. Writing or replacing a CI workflow changes the
|
|
5
|
+
repository's pipeline: ask the person first.
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
kaloko ci github [--scenario qa/flows/checkout.yml,qa/flows/signup.yml] # .github/workflows/kaloko.yml
|
|
9
|
+
kaloko ci gitlab # kaloko.gitlab-ci.yml (include it)
|
|
10
|
+
kaloko ci github --print # show it, write nothing
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
What the generated workflow does on every preview deployment of a pull
|
|
14
|
+
request (Vercel, Netlify and Cloudflare Pages report their previews as GitHub
|
|
15
|
+
deployments; `--trigger pull_request` runs on every push instead and waits for
|
|
16
|
+
the deployment):
|
|
17
|
+
|
|
18
|
+
1. installs the project (`npm ci`, or pnpm, yarn, bun by the lockfile) and the
|
|
19
|
+
Chromium of kaloko's Playwright (`kaloko browser install --with-deps`);
|
|
20
|
+
`KALOKO_BROWSER_CHANNEL=chromium` makes the run use it, so no separate CI
|
|
21
|
+
config is needed;
|
|
22
|
+
2. `kaloko ci preview-url --wait 600 --github-env` finds the preview URL, the
|
|
23
|
+
pull request and its base branch and puts them into `KALOKO_PREVIEW_URL`,
|
|
24
|
+
`KALOKO_PR`, `KALOKO_BASE`;
|
|
25
|
+
3. for each scenario: `kaloko affected --check` skips it when the change does
|
|
26
|
+
not reach any of its steps, otherwise `start --env preview`,
|
|
27
|
+
`walk --ephemeral --affected` and `share --pr` (one comment per scenario,
|
|
28
|
+
updated on later pushes).
|
|
29
|
+
|
|
30
|
+
`kaloko ci` also adds the `preview` environment to `kaloko.config.yml` with
|
|
31
|
+
`base_url: ${KALOKO_PREVIEW_URL}`. Scenarios that list `environments:` must
|
|
32
|
+
include `preview`, or `start` refuses them; the command says which.
|
|
33
|
+
|
|
34
|
+
## Secrets
|
|
35
|
+
|
|
36
|
+
- `KALOKO_TOKEN`: a token with the tester role and the upload scope (Settings →
|
|
37
|
+
API tokens; on Enterprise a service account's token, which no person owns).
|
|
38
|
+
- `TYPESAFE_API_KEY` only when semantic criteria should use your own evaluator
|
|
39
|
+
key instead of the organization's allowance.
|
|
40
|
+
- Vercel deployment protection: a `VERCEL_AUTOMATION_BYPASS_SECRET` secret and
|
|
41
|
+
the commented `access.headers` block of the preview environment.
|
|
42
|
+
- GitLab: `KALOKO_GITLAB_TOKEN` (project access token, `api` scope) for the
|
|
43
|
+
merge request note; set `KALOKO_PREVIEW_URL` in the job when the review app
|
|
44
|
+
URL is known (`https://$CI_COMMIT_REF_SLUG.review.example.com`), or rely on
|
|
45
|
+
`CI_ENVIRONMENT_URL`.
|
|
46
|
+
|
|
47
|
+
## Where the preview URL comes from
|
|
48
|
+
|
|
49
|
+
`kaloko ci preview-url` takes the first of: `KALOKO_PREVIEW_URL`; the
|
|
50
|
+
`deployment_status` event; `VERCEL_BRANCH_URL`/`VERCEL_URL`,
|
|
51
|
+
`DEPLOY_PRIME_URL`/`DEPLOY_URL` (Netlify), `CF_PAGES_URL`,
|
|
52
|
+
`CI_ENVIRONMENT_URL` (GitLab), `RENDER_EXTERNAL_URL`; then the commit's GitHub
|
|
53
|
+
deployments, skipping production, polled every 10 s up to `--wait` seconds.
|
|
54
|
+
Only the URL goes to stdout, so `export KALOKO_PREVIEW_URL="$(kaloko ci preview-url)"`
|
|
55
|
+
works in any CI.
|
|
56
|
+
|
|
57
|
+
## Affected steps in CI
|
|
58
|
+
|
|
59
|
+
`walk --affected` needs the history: check out with `fetch-depth: 0` (GitHub)
|
|
60
|
+
or `GIT_DEPTH: "0"` (GitLab). The base is `--base`, else `KALOKO_BASE`,
|
|
61
|
+
`GITHUB_BASE_REF`, the merge request target, then `origin/main`. URLs captured
|
|
62
|
+
in the newest shared run of the scenario count too, so a fresh CI checkout
|
|
63
|
+
maps routes as well as a laptop with earlier runs. See
|
|
64
|
+
[execution.md](execution.md#only-what-changed).
|
|
65
|
+
|
|
66
|
+
## Done means
|
|
67
|
+
|
|
68
|
+
The workflow file and the config change are committed after the person agreed,
|
|
69
|
+
the secret is named for the person to add, and you report the first run's PR
|
|
70
|
+
comment or why it did not appear (no preview deployment, missing token).
|
|
@@ -12,7 +12,9 @@ loop teams use:
|
|
|
12
12
|
request and `gh` is signed in, `kaloko share --pr` posts one comment per
|
|
13
13
|
scenario on it (summary line, canvas link, criteria that need attention
|
|
14
14
|
with evidence) and updates that same comment on every later run, so the
|
|
15
|
-
PR never fills up with Kaloko noise.
|
|
15
|
+
PR never fills up with Kaloko noise. In CI `KALOKO_PR` names the pull
|
|
16
|
+
request of a detached checkout, and on GitLab merge requests the comment is a
|
|
17
|
+
note on the MR (`KALOKO_GITLAB_TOKEN`). Otherwise write the comment yourself:
|
|
16
18
|
what was tested (scenario, environment, locales), the summary line
|
|
17
19
|
(`n/m steps passed`), every failing or `needs_review` criterion with
|
|
18
20
|
evidence, and the canvas URL. Reviewers accept, return or comment on the
|
|
@@ -255,6 +257,12 @@ Everything above is available over the API with an organization token:
|
|
|
255
257
|
- `GET /api/v1/orgs/<org>/runs/<id>/export.zip` (`kaloko export`): the whole
|
|
256
258
|
run as a zip before it expires; keep it with the issue when the evidence
|
|
257
259
|
must outlive the retention period.
|
|
260
|
+
- Uploads by content hash: `POST …/blobs/missing` with `{"hashes": [sha256…]}`
|
|
261
|
+
answers what the organization does not store yet, `PUT …/blobs/<sha256>`
|
|
262
|
+
(header `X-QAwalk-Type: png|webp|html|json`) stores one, and the run's file
|
|
263
|
+
plan `POST …/runs/<id>/files` takes `{ path, bytes, blob }` entries that
|
|
264
|
+
point at stored blobs. `kaloko share` does this: unchanged captures are not
|
|
265
|
+
sent again and a share that broke off resumes when run again.
|
|
258
266
|
- MCP server for agents: see [mcp.md](mcp.md).
|
|
259
267
|
|
|
260
268
|
Read-only tokens may list but not change anything; keep CI tokens read-only
|
|
@@ -13,6 +13,11 @@ them; do not work around a block.
|
|
|
13
13
|
Test data preparation for review and staging belongs to the project's CI
|
|
14
14
|
prepare step; assume it exists and create only what the scenario itself needs.
|
|
15
15
|
|
|
16
|
+
Pull request previews (Vercel, Netlify, Cloudflare Pages, review apps) are an
|
|
17
|
+
environment whose URL comes from CI: `preview: { base_url: ${KALOKO_PREVIEW_URL} }`,
|
|
18
|
+
which `kaloko ci github|gitlab` adds and `kaloko ci preview-url` fills; see
|
|
19
|
+
[ci.md](ci.md).
|
|
20
|
+
|
|
16
21
|
## Who does what
|
|
17
22
|
|
|
18
23
|
Kaloko needs evidence in its format: captures of screens and e-mails in a run.
|
|
@@ -91,3 +91,59 @@ screenshots before applying; a region that changed because the page really
|
|
|
91
91
|
changed is not noise. `--same-commit` compares only runs of the same commit,
|
|
92
92
|
`--runs a,b` names them, `--last N` sets how many runs are compared. Capture
|
|
93
93
|
the steps again after applying: the capture records the ignored regions.
|
|
94
|
+
|
|
95
|
+
## Only what changed
|
|
96
|
+
|
|
97
|
+
After a fix, walk the steps the change reaches instead of the whole flow:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
kaloko affected # which steps, and why (against origin/main; --base <ref>)
|
|
101
|
+
kaloko walk --affected [--carry] # walk them; --carry copies the other steps from the previous local run
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
A changed file reaches a step when it is a route file whose route matches the
|
|
105
|
+
step's path or a URL captured for it earlier (Next.js `app/` and `pages/`,
|
|
106
|
+
Nuxt, SvelteKit, Remix, Astro, Rails views and controllers), when it is
|
|
107
|
+
imported by such a file, when it is global (layouts, global styles,
|
|
108
|
+
dependencies; i18n messages of one locale keep the walk in that locale), when
|
|
109
|
+
it is the step's script or the step changed in the scenario, or when it
|
|
110
|
+
matches the step's `affects:` globs. Steps on the main path before a selected
|
|
111
|
+
step are walked first (`--no-upstream` skips them); steps with
|
|
112
|
+
`affects: always` (sign-in, consent) are always walked. Files nothing maps are
|
|
113
|
+
listed as unmapped: add `affects:` to the steps they shape, or pass
|
|
114
|
+
`--if-unmapped all` to walk everything then. Changes to tests, docs and CI are
|
|
115
|
+
ignored.
|
|
116
|
+
|
|
117
|
+
```yaml
|
|
118
|
+
- id: checkout
|
|
119
|
+
path: /checkout
|
|
120
|
+
affects: [lib/payments/**, src/components/PriceSummary.tsx]
|
|
121
|
+
- id: consent
|
|
122
|
+
affects: always
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Report the selection with the run: which steps were walked and why, and that
|
|
126
|
+
the rest was not walked (or carried from which run).
|
|
127
|
+
|
|
128
|
+
## Several locales at once
|
|
129
|
+
|
|
130
|
+
Locales are independent, so `walk --concurrency N` walks N of them at the same
|
|
131
|
+
time, each in its own browser context (a new visitor per locale). In an own
|
|
132
|
+
browser (`--ephemeral`, CI) the default follows the CPU (up to 4); on the
|
|
133
|
+
shared browser it stays one at a time unless you ask, because people may rely
|
|
134
|
+
on its signed-in tabs. Each locale's lines are printed together, in locale
|
|
135
|
+
order. Set `walk.concurrency` in the config for the project's default.
|
|
136
|
+
Viewports of one step are still captured one after another on the same page.
|
|
137
|
+
|
|
138
|
+
## Selectors that moved
|
|
139
|
+
|
|
140
|
+
When a step script waits in vain for a selector (the button lost its id, the
|
|
141
|
+
class was renamed), the walk proposes a replacement from the page as it is
|
|
142
|
+
now: a test id, then role and accessible name, then a stable id, then the text,
|
|
143
|
+
preferring the element the old selector matched in the step's previous
|
|
144
|
+
capture. It prints the reason and a patch of the step script. Review it; if it
|
|
145
|
+
is the same element, `kaloko walk --steps <id> --heal` runs the step with the
|
|
146
|
+
patched script and writes the patch only when the step passes. Applying a heal
|
|
147
|
+
is fixing scenario mechanics; if the proposal points at a different element,
|
|
148
|
+
the product changed: report it, do not heal it. `--no-heal` skips the
|
|
149
|
+
proposals.
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Importing screenshots, walkthroughs and traces
|
|
2
|
+
|
|
3
|
+
Agents and test tools already leave screenshots behind: Cursor and
|
|
4
|
+
Antigravity walkthroughs, Claude Code screenshots, Playwright reports and
|
|
5
|
+
traces. `kaloko import` lays them out as a run, so they are reviewed and
|
|
6
|
+
signed off on the canvas like a walk.
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
kaloko import shots/ # into the current run (kaloko start first)
|
|
10
|
+
kaloko import playwright-report/ --scenario qa/flows/checkout.yml --env staging
|
|
11
|
+
kaloko import test-results/checkout/trace.zip --map "click Pay=payment"
|
|
12
|
+
kaloko import walkthrough.md --adhoc # no scenario: an ad-hoc flow of the screens
|
|
13
|
+
kaloko evaluate && kaloko preview # then kaloko share, as usual
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
Where the screens go: the current run, or a new run of `--scenario`, or with no
|
|
17
|
+
run (or `--adhoc`) an ad-hoc flow written to `qa/flows/import-<title>.yml`: one
|
|
18
|
+
manual screen per image, in order, read-only.
|
|
19
|
+
|
|
20
|
+
## Sources
|
|
21
|
+
|
|
22
|
+
- **A folder with `steps.json`** (also `walkthrough.json`, `kaloko-import.json`):
|
|
23
|
+
```json
|
|
24
|
+
{ "title": "Checkout", "tool": "cursor",
|
|
25
|
+
"steps": [
|
|
26
|
+
{ "title": "Cart", "image": "01-cart.png", "step": "cart", "url": "https://shop.test/cart",
|
|
27
|
+
"notes": "Two items, total 41 €", "locale": "en", "viewport": "desktop" },
|
|
28
|
+
{ "title": "Payment", "image": "02-payment.png" }
|
|
29
|
+
] }
|
|
30
|
+
```
|
|
31
|
+
Only `image` is required. `step` names the scenario step; `viewport` and
|
|
32
|
+
`locale` must be in the run's scope or are chosen for you.
|
|
33
|
+
- **A Markdown walkthrough** (`walkthrough.md`, `README.md`, any `.md` in the
|
|
34
|
+
folder): every image `` in order, titled by the heading above
|
|
35
|
+
it, with the text between as notes. Links to `.webm`/`.mp4` are recordings.
|
|
36
|
+
- **A plain folder of images**: in file name order (`01-…`, `02-…`), titled by
|
|
37
|
+
the name. A folder of folders (Playwright `test-results/`) reads each
|
|
38
|
+
sub-folder as one test.
|
|
39
|
+
- **Playwright**: the JSON reporter's file (`--reporter=json`), an HTML report
|
|
40
|
+
folder (`playwright-report/`) or a `trace.zip`. Screenshot attachments become
|
|
41
|
+
screens; a test without them contributes its trace: one screen after each
|
|
42
|
+
page action (goto, click, fill, press …).
|
|
43
|
+
|
|
44
|
+
`--format folder|markdown|playwright-json|playwright-html|trace` overrides the
|
|
45
|
+
detection.
|
|
46
|
+
|
|
47
|
+
## Mapping to steps
|
|
48
|
+
|
|
49
|
+
In this order: the item's `step`, `--map "Title=step,…"` (a title, group or
|
|
50
|
+
file name containing the label), the URL path against step paths, then the
|
|
51
|
+
title against step ids and titles. A second screen for the same step, locale
|
|
52
|
+
and viewport becomes an attachment of the step. Unmatched screens are listed
|
|
53
|
+
with the command that maps them; nothing is dropped silently. The viewport is
|
|
54
|
+
the run viewport closest to the image width (2× and 3× screenshots count by
|
|
55
|
+
their CSS width).
|
|
56
|
+
|
|
57
|
+
## What an imported screen is
|
|
58
|
+
|
|
59
|
+
The image, encoded like a walked capture, with the tool and the source file.
|
|
60
|
+
There is no page behind it: deterministic criteria stay "not evaluated" until
|
|
61
|
+
the step is walked, and semantic questions are not asked without page text, so
|
|
62
|
+
a person decides on the canvas. Recordings are kept in the run folder
|
|
63
|
+
(`imported/`) and are not uploaded yet.
|
|
64
|
+
|
|
65
|
+
Report it as imported evidence, not as a walk: name the tool and say which
|
|
66
|
+
steps still need a real walk.
|
|
@@ -28,6 +28,7 @@ file.
|
|
|
28
28
|
| `list_notifications` | the token owner's inbox (`unread_only`, `limit`) |
|
|
29
29
|
| `list_projects` | projects of the organization with counts; the slug is the `project` filter of `list_runs` and the value for `project:` in a scenario |
|
|
30
30
|
| `get_criteria_verdicts` | reviewer verdicts on manual or overridden criteria of a run |
|
|
31
|
+
| `get_step_recording` | a step's recording and trace (`run_id`, `step`, `locale`, `viewport`, `trace: true` for the full trace): video URLs, duration, its chapter of a flow recording, failed attempts, the trace summary (failed requests, console errors, slowest request); see [recording.md](recording.md). `get_run_summary` lists trace summaries per step too |
|
|
31
32
|
| `get_review_queue` | what a person still has to decide in a run (`run_id`, or `scenario` / nothing for the newest run): steps in review order with reasons, decisions, approvals carried over unchanged steps with their origin, counts and `review_url` to give the person |
|
|
32
33
|
| `decide_step` | records your (the agent's) decision on a step: `approve`, or `return` with a `note`; marked as an agent's and not counted as a person's. Never use it to approve for a person |
|
|
33
34
|
| `revoke_carried_approval` | revokes an approval carried over to a step that looks unchanged but should be looked at again (`run_id`, `step`) |
|
|
@@ -35,6 +36,7 @@ file.
|
|
|
35
36
|
| `list_step_changes` | documentation layer: steps whose capture or status changed since the previous accepted run or `since` (`scenario`, `since`, `accepted`); see [documentation.md](documentation.md) |
|
|
36
37
|
| `get_started` | where the organization stands (`empty`, `sample`, `active`), its sample run, the next steps and a prompt; call it in a new project, see [first-run.md](first-run.md) |
|
|
37
38
|
| `add_sample_run` | adds the sample run (or returns the one there is); needs an upload or full token and the Developer / Tester role |
|
|
39
|
+
| `get_affected_steps` | which steps of a scenario changed files reach, with the reason per step (`scenario`, `changed_files`, `run_id`, `upstream`); route files are matched against the step paths and URLs of the newest run. Imports and `affects:` globs need the repository: `kaloko affected` follows them |
|
|
38
40
|
|
|
39
41
|
## Comment tools
|
|
40
42
|
|