kaloko 0.4.8 → 0.4.10

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (51) hide show
  1. package/README.md +92 -5
  2. package/package.json +1 -1
  3. package/schemas/config.v1.json +33 -0
  4. package/schemas/scenario.v1.json +39 -0
  5. package/schemas/spec.v1.json +60 -0
  6. package/skills/kaloko/SKILL.md +24 -3
  7. package/skills/kaloko/reference/acceptance.md +50 -2
  8. package/skills/kaloko/reference/ci.md +70 -0
  9. package/skills/kaloko/reference/collaboration.md +9 -1
  10. package/skills/kaloko/reference/design.md +35 -2
  11. package/skills/kaloko/reference/environments.md +5 -0
  12. package/skills/kaloko/reference/evaluation.md +46 -3
  13. package/skills/kaloko/reference/execution.md +74 -0
  14. package/skills/kaloko/reference/import.md +66 -0
  15. package/skills/kaloko/reference/mcp.md +4 -1
  16. package/src/cli.mjs +428 -99
  17. package/src/lib/affected-core.mjs +242 -0
  18. package/src/lib/affected.mjs +233 -0
  19. package/src/lib/build.mjs +30 -5
  20. package/src/lib/capture.mjs +43 -1
  21. package/src/lib/ci.mjs +207 -0
  22. package/src/lib/config.mjs +2 -0
  23. package/src/lib/design-cli.mjs +7 -3
  24. package/src/lib/devspec.mjs +821 -0
  25. package/src/lib/doctor.mjs +7 -0
  26. package/src/lib/drift.mjs +16 -8
  27. package/src/lib/evaluate.mjs +98 -14
  28. package/src/lib/git.mjs +6 -2
  29. package/src/lib/heal.mjs +207 -0
  30. package/src/lib/import.mjs +374 -0
  31. package/src/lib/init-detect.mjs +219 -0
  32. package/src/lib/keyboard.mjs +261 -0
  33. package/src/lib/manifest-core.mjs +16 -5
  34. package/src/lib/onboarding.mjs +15 -8
  35. package/src/lib/pack-help.mjs +7 -0
  36. package/src/lib/packs.mjs +9 -2
  37. package/src/lib/png.mjs +61 -0
  38. package/src/lib/pr.mjs +24 -2
  39. package/src/lib/probe.mjs +57 -0
  40. package/src/lib/recheck.mjs +3 -1
  41. package/src/lib/scenario.mjs +2 -1
  42. package/src/lib/service.mjs +7 -1
  43. package/src/lib/spec-cli.mjs +90 -0
  44. package/src/lib/stability-core.mjs +114 -0
  45. package/src/lib/stability.mjs +249 -0
  46. package/src/lib/unzip.mjs +31 -0
  47. package/src/lib/upload.mjs +114 -0
  48. package/src/lib/vision-core.mjs +132 -0
  49. package/src/lib/vision.mjs +127 -0
  50. package/templates/kaloko.config.yml +6 -0
  51. package/viewer/index.html +500 -44
package/README.md CHANGED
@@ -27,8 +27,11 @@ npx kaloko doctor
27
27
  ```
28
28
 
29
29
  `init` creates `kaloko.config.yml` and copies the `kaloko` skill to
30
- `.claude/skills/kaloko` (Claude Code) or `skills/kaloko` (Codex). Run it again
31
- after `npm update kaloko` to refresh the skill.
30
+ `.claude/skills/kaloko` (Claude Code) or `skills/kaloko` (Codex). It reads the
31
+ project first: the framework (Next.js, Nuxt, SvelteKit, Remix, Astro, Rails and
32
+ others), the dev server's command and URL, the locales of the i18n config and
33
+ the main routes, and drafts `qa/flows/main-pages.yml` so the first walk works
34
+ without editing. Run it again after `npm update kaloko` to refresh the skill.
32
35
 
33
36
  ## Then ask your agent
34
37
 
@@ -61,7 +64,65 @@ npx kaloko preview # local canvas
61
64
  npx kaloko share --pr # upload to kaloko.app, comment on the PR
62
65
  ```
63
66
 
64
- `npx kaloko help` lists every command. When something does not work (Chrome,
67
+ After a fix, walk only what the change reaches: `npx kaloko affected` explains
68
+ which steps the changed files touch (routes, imports, `affects:` globs in the
69
+ scenario), and `npx kaloko walk --affected` walks them. Several locales walk at
70
+ once with `--concurrency 3`. When a step's selector no longer exists, the walk
71
+ proposes one from the page and prints the patch; `--heal` applies it once the
72
+ step passes.
73
+
74
+ `npx kaloko help` lists every command.
75
+
76
+ ## Pull request previews in CI
77
+
78
+ ```bash
79
+ npx kaloko ci github # or: npx kaloko ci gitlab
80
+ ```
81
+
82
+ writes a workflow that waits for the pull request's preview deployment
83
+ (Vercel, Netlify, Cloudflare Pages, any review app), walks the steps the change
84
+ affects and puts one comment per scenario on the pull request. Add a
85
+ `KALOKO_TOKEN` secret (tester role) and commit it. In short, the GitHub
86
+ workflow does this:
87
+
88
+ ```yaml
89
+ on:
90
+ deployment_status:
91
+ jobs:
92
+ walk:
93
+ if: github.event.deployment_status.state == 'success'
94
+ runs-on: ubuntu-latest
95
+ env:
96
+ KALOKO_TOKEN: ${{ secrets.KALOKO_TOKEN }}
97
+ GH_TOKEN: ${{ github.token }}
98
+ KALOKO_BROWSER_CHANNEL: chromium
99
+ steps:
100
+ - uses: actions/checkout@v4
101
+ with: { fetch-depth: 0 }
102
+ - uses: actions/setup-node@v4
103
+ with: { node-version: 22 }
104
+ - run: npm ci
105
+ - run: npx kaloko browser install --with-deps
106
+ - run: npx kaloko ci preview-url --wait 600 --github-env
107
+ - run: |
108
+ npx kaloko start --scenario qa/flows/checkout.yml --env preview
109
+ npx kaloko walk --ephemeral --affected
110
+ npx kaloko share --pr
111
+ ```
112
+
113
+ Uploads go by content hash: captures that did not change since an earlier run
114
+ are not sent again, and an interrupted `share` picks up where it stopped.
115
+
116
+ ## Screenshots you already have
117
+
118
+ ```bash
119
+ npx kaloko import playwright-report/ # or test-results/, a trace.zip, results.json
120
+ npx kaloko import walkthrough/ --adhoc # a Cursor or Antigravity walkthrough, a folder of screenshots
121
+ ```
122
+
123
+ lays screenshots, Playwright reports and traces out as a run: matched to the
124
+ scenario's steps (by step id in `steps.json`, URL or title), or as an ad-hoc
125
+ flow when there is no scenario. Build and share it like any run. When something does not work (Chrome,
65
126
  token, environment URL, evaluator key), `npx kaloko doctor` says what to fix.
66
127
 
67
128
  ## Review only what changed
@@ -81,6 +142,15 @@ what still waits for a person, so it can pass the link on. Agents never
81
142
  approve for people: their decisions are marked and do not count as a
82
143
  person's.
83
144
 
145
+ Every AI verdict comes with one sentence saying why. Criteria marked
146
+ `vision: true`, and answers the text evaluator is unsure about, get a second
147
+ look at the screenshot from a vision model (`ANTHROPIC_API_KEY`, or through
148
+ Kaloko); `kaloko build` lists what to act on with the score and the reason.
149
+ The `a11y` pack walks the page with Tab (order, visible focus, traps, skip
150
+ link) and reads names from the accessibility tree. Pixel diffs skip the
151
+ regions a step ignores; `kaloko walk --stability 3` and `kaloko stability`
152
+ find the clocks, ads and carousels worth ignoring.
153
+
84
154
  ## Design (Kaloko Design)
85
155
 
86
156
  Design the screens of a flow as live HTML with your own agent and skills, in the same scenario that development
@@ -98,6 +168,17 @@ npx kaloko share --name "Checkout v2" # a version of the whole flow
98
168
  npx kaloko reference --design --version 2 # after the team accepted it
99
169
  ```
100
170
 
171
+ Developers and their agents build from what was approved. *Dev mode* (`D` in the step detail) shows each
172
+ element's box, sizes and distances, its colors, type and spacing with the token every value comes from (or a
173
+ warning when it is not a token), contrast against WCAG AA, the `:hover`, `:focus`, `:active` and disabled states,
174
+ and its images. The same data is the step spec an agent reads, and `kaloko drift` checks the implementation
175
+ against it:
176
+
177
+ ```bash
178
+ npx kaloko spec payment # elements, styles → tokens, states (also MCP get_step_spec)
179
+ npx kaloko walk && npx kaloko drift # expected token vs actual value per element; exit 1 while it differs
180
+ ```
181
+
101
182
  On the canvas, *Prototype* (or `P`) presents the steps full screen as live pages: click through links between
102
183
  steps, switch the device, walk with the arrow keys, and share a link that opens straight into a step. The rules
103
184
  for pages that work everywhere are in `skills/kaloko/reference/portable-html.md`.
@@ -141,13 +222,19 @@ captured HTML, page text and console output.
141
222
  - **Service and evaluator**: `KALOKO_TOKEN` for `share`, `feedback`, `review`, `inbox` and clean-up (`runs`, `trash`, `prune`, `keep`, `archive`)
142
223
  (`npx kaloko login` approves one in the browser and saves it to `~/.config/kaloko/.env`, or create it in your
143
224
  organization's settings); `TYPESAFE_API_KEY` for the
144
- semantic evaluator (optional). Both names can be changed in the config.
225
+ semantic evaluator and `ANTHROPIC_API_KEY` for the vision evaluator (both
226
+ optional; without them Kaloko evaluates on your plan's allowance). The names
227
+ can be changed in the config.
145
228
 
146
229
  `npx kaloko doctor` checks that every variable the config names is set and that
147
230
  each environment lets you in.
148
231
 
149
232
  Set `browser.channel: chrome` to use an installed Chrome, or run
150
- `npx playwright install chromium`.
233
+ `npx kaloko browser install` for the Chromium of kaloko's Playwright
234
+ (`KALOKO_BROWSER_CHANNEL=chromium` then uses it, which is how CI runs).
235
+ In CI, `KALOKO_PREVIEW_URL`, `KALOKO_PR` and `KALOKO_BASE` come from
236
+ `npx kaloko ci preview-url`; `KALOKO_GITLAB_TOKEN` lets `share --pr` comment on
237
+ GitLab merge requests.
151
238
 
152
239
  ## License
153
240
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "kaloko",
3
- "version": "0.4.8",
3
+ "version": "0.4.10",
4
4
  "description": "See what your AI agents built: every screen, language and device walked, assessed and approved on one canvas.",
5
5
  "keywords": [
6
6
  "qa",
@@ -45,9 +45,42 @@
45
45
  "signals": { "type": "boolean", "default": true, "description": "Quick signals on every screen (purpose met, expected language, looks broken); never change verdicts." },
46
46
  "concurrency": { "type": "integer", "minimum": 1, "maximum": 16, "default": 6, "description": "Evaluator requests in flight at once." }
47
47
  }
48
+ },
49
+ "vision": {
50
+ "description": "Second, vision-capable evaluator (#17): sees the screenshot (a crop for scoped criteria) and answers with a probability and a one-sentence reason. Asked only for criteria with vision: true and for text answers in the uncertain band; false turns it off.",
51
+ "oneOf": [
52
+ { "const": false },
53
+ {
54
+ "type": "object",
55
+ "additionalProperties": false,
56
+ "properties": {
57
+ "provider": { "enum": ["anthropic"], "default": "anthropic" },
58
+ "model": { "type": "string", "default": "claude-opus-5-5", "description": "Model id of the provider (Anthropic Messages API with image input)." },
59
+ "api_key_env": { "type": "string", "default": "ANTHROPIC_API_KEY" },
60
+ "base_url": { "type": "string", "default": "https://api.anthropic.com" },
61
+ "when": { "enum": ["uncertain", "flagged", "always", "off"], "default": "uncertain", "description": "flagged: only criteria with vision: true; uncertain: those plus text answers in the uncertain band; always: every semantic criterion (most expensive)." },
62
+ "uncertain": { "type": "array", "items": { "type": "number", "minimum": 0, "maximum": 1 }, "minItems": 2, "maxItems": 2, "default": [0.15, 0.85], "description": "Text scores strictly between these go to the vision evaluator." },
63
+ "max_calls": { "type": "integer", "minimum": 0, "default": 40, "description": "Vision requests per evaluation (uncached); the rest stay with the text answer." },
64
+ "max_width": { "type": "integer", "minimum": 320, "maximum": 1568, "default": 1000, "description": "Images are scaled down to this width (px)." },
65
+ "tile_height": { "type": "integer", "minimum": 400, "maximum": 1568, "default": 1400, "description": "A full page is cut into tiles of this height." },
66
+ "max_tiles": { "type": "integer", "minimum": 1, "maximum": 4, "default": 2, "description": "Tiles of a full page sent at most (from the top); scoped criteria get a crop instead." },
67
+ "effort": { "enum": ["low", "medium", "high"], "default": "low" },
68
+ "reason_language": { "type": "string", "default": "en", "description": "Language of the one-sentence reasons (en, cs, …)." },
69
+ "hosted": { "type": "boolean", "default": true, "description": "Without a local key, ask through the organization's service (counts against the monthly allowance; one vision question counts as 5)." },
70
+ "concurrency": { "type": "integer", "minimum": 1, "maximum": 8, "default": 3 }
71
+ }
72
+ }
73
+ ]
48
74
  }
49
75
  }
50
76
  },
77
+ "walk": {
78
+ "type": "object",
79
+ "additionalProperties": false,
80
+ "properties": {
81
+ "concurrency": { "type": "integer", "minimum": 1, "maximum": 16, "description": "Locales walked at the same time, each in its own browser context (kaloko walk --concurrency). Default: by CPU in an own browser, one at a time on the shared browser." }
82
+ }
83
+ },
51
84
  "share": {
52
85
  "type": "object",
53
86
  "additionalProperties": false,
@@ -243,6 +243,13 @@
243
243
  "type": "string",
244
244
  "description": "Key from `sessions`; which person performs this step."
245
245
  },
246
+ "affects": {
247
+ "oneOf": [
248
+ { "const": "always" },
249
+ { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }
250
+ ],
251
+ "description": "For `kaloko walk --affected`: globs of repository files (relative to the config) that shape this step, next to the routes and imports Kaloko finds itself; `always` walks the step on every affected walk (sign-in, consent)."
252
+ },
246
253
  "packs": {
247
254
  "type": "array",
248
255
  "items": {
@@ -262,6 +269,11 @@
262
269
  "capture": {
263
270
  "$ref": "#/$defs/capture"
264
271
  },
272
+ "ignore": {
273
+ "type": "array",
274
+ "description": "Regions left out of pixel comparisons (clocks, ads, carousels): a selector, or a rectangle in page pixels of one viewport. kaloko stability suggests them.",
275
+ "items": { "$ref": "#/$defs/ignoreRegion" }
276
+ },
265
277
  "items": {
266
278
  "type": "object",
267
279
  "additionalProperties": false,
@@ -500,6 +512,13 @@
500
512
  "a11y_contrast",
501
513
  "a11y_main",
502
514
  "a11y_unique_ids",
515
+ "a11y_focus_order",
516
+ "a11y_focus_visible",
517
+ "a11y_keyboard_trap",
518
+ "a11y_skip_link",
519
+ "a11y_ax_names",
520
+ "a11y_landmarks",
521
+ "a11y_aria",
503
522
  "perf_lcp",
504
523
  "perf_cls",
505
524
  "perf_ttfb",
@@ -588,6 +607,10 @@
588
607
  "console"
589
608
  ],
590
609
  "description": "console: the evaluator reads the captured console errors with the step purpose (triage) instead of the page"
610
+ },
611
+ "vision": {
612
+ "type": "boolean",
613
+ "description": "Also show the evaluator the screenshot (a crop of scope when set): layout, imagery, brand. Needs a vision evaluator (evaluators.vision)."
591
614
  }
592
615
  }
593
616
  },
@@ -632,6 +655,22 @@
632
655
  }
633
656
  }
634
657
  },
658
+ "ignoreRegion": {
659
+ "type": "object",
660
+ "additionalProperties": false,
661
+ "oneOf": [
662
+ { "required": ["selector"] },
663
+ { "required": ["rect", "viewport"] }
664
+ ],
665
+ "properties": {
666
+ "selector": { "type": "string", "minLength": 1, "description": "CSS selector; every visible match is ignored." },
667
+ "rect": { "type": "array", "items": { "type": "number", "minimum": 0 }, "minItems": 4, "maxItems": 4, "description": "[x, y, width, height] in CSS pixels of the page." },
668
+ "viewport": { "type": "string", "description": "The viewport a rect belongs to." },
669
+ "viewports": { "type": "array", "items": { "type": "string" }, "description": "Only in these viewports (selector)." },
670
+ "locales": { "type": "array", "items": { "type": "string" }, "description": "Only in these locales." },
671
+ "note": { "type": "string" }
672
+ }
673
+ },
635
674
  "capture": {
636
675
  "type": "object",
637
676
  "additionalProperties": false,
@@ -0,0 +1,60 @@
1
+ {
2
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3
+ "$id": "https://qawalk.com/schemas/spec.v1.json",
4
+ "title": "qawalk.spec.v1",
5
+ "description": "The step spec (#42 Dev mode): what to build for one step as it was approved. Written by the in-page engine src/lib/devspec.mjs at capture (steps/<step>/<locale>/<viewport>.spec.json, and the capture record of a design revision); the canvas Dev mode runs the same engine on the live page. Served by GET …/scenarios/<id>/steps/<step>/spec, MCP get_step_spec and kaloko spec. Elements are listed in page order; a style lists only what differs from what the element inherits (page, then the parent element).",
6
+ "type": "object",
7
+ "required": ["schema", "elements"],
8
+ "$defs": {
9
+ "value": {
10
+ "type": "object",
11
+ "description": "A computed value (colors as #rrggbb[aa], lengths in px) and the token it maps to. token: the DTCG path (several, space-separated, for a shorthand such as padding; 0 for a zero side); null when the set has tokens of that kind but none fits and none is near. nearest: the closest token when the value is not one. mode: the mode whose value it is (light, dark, …). hard: the stylesheet typed the raw value although the token exists. var: the custom property the author used when it names no token.",
12
+ "required": ["value"],
13
+ "properties": {
14
+ "value": { "type": "string" },
15
+ "token": { "type": ["string", "null"] },
16
+ "nearest": { "type": "string" },
17
+ "mode": { "type": "string" },
18
+ "hard": { "type": "boolean" },
19
+ "var": { "type": "string" }
20
+ },
21
+ "additionalProperties": false
22
+ },
23
+ "styles": {
24
+ "type": "object",
25
+ "description": "CSS property → value: color, background-color, border-color, font-family, font-size, font-weight, line-height, letter-spacing, padding, gap, row-gap, column-gap, border-width, border-radius, box-shadow, opacity, display and flex or grid properties.",
26
+ "additionalProperties": { "$ref": "#/$defs/value" }
27
+ }
28
+ },
29
+ "properties": {
30
+ "schema": { "const": "qawalk.spec.v1" },
31
+ "title": { "type": "string" },
32
+ "lang": { "type": ["string", "null"] },
33
+ "viewport": { "type": "array", "items": { "type": "number" }, "minItems": 2, "maxItems": 2 },
34
+ "size": { "type": "array", "items": { "type": "number" }, "minItems": 2, "maxItems": 2, "description": "Document width and height." },
35
+ "tokens": { "type": ["object", "null"], "description": "The token set the values were mapped with (the step's tokens.json); null without one.", "properties": { "count": { "type": "integer" }, "modes": { "type": "array", "items": { "type": "string" } } } },
36
+ "page": { "$ref": "#/$defs/styles", "description": "Typography and text color of the page (body), inherited by every element that does not say otherwise." },
37
+ "elements": {
38
+ "type": "array",
39
+ "items": {
40
+ "type": "object",
41
+ "required": ["i", "parent", "tag", "sel", "styles"],
42
+ "properties": {
43
+ "i": { "type": "integer" },
44
+ "parent": { "type": "integer", "description": "Index of the nearest listed ancestor, -1 at the top." },
45
+ "tag": { "type": "string" },
46
+ "role": { "type": ["string", "null"], "description": "ARIA role, explicit or implicit (button, link, heading, main, textbox, …)." },
47
+ "name": { "type": ["string", "null"], "description": "Accessible name or own text (at most 80 characters)." },
48
+ "sel": { "type": "string", "description": "A CSS selector of the element in the design page (ids, data-testid and classes first)." },
49
+ "box": { "type": "array", "items": { "type": "number" }, "minItems": 4, "maxItems": 4, "description": "x, y, width, height in CSS px of the page." },
50
+ "styles": { "$ref": "#/$defs/styles" },
51
+ "contrast": { "type": "object", "properties": { "ratio": { "type": "number" }, "aa": { "type": "boolean" } } },
52
+ "states": { "type": "object", "description": "hover, focus, active, disabled → the styles that change in that state (the page's rules forced on the element).", "additionalProperties": { "$ref": "#/$defs/styles" } },
53
+ "asset": { "type": "object", "properties": { "kind": { "enum": ["img", "svg"] }, "src": { "type": "string" }, "size": { "type": "array" }, "alt": { "type": ["string", "null"] } } }
54
+ }
55
+ }
56
+ },
57
+ "used": { "type": "object", "description": "Token path → number of values that use it.", "additionalProperties": { "type": "integer" } },
58
+ "off": { "type": "array", "description": "Values outside the token set, most frequent first.", "items": { "type": "object", "properties": { "prop": { "type": "string" }, "value": { "type": "string" }, "count": { "type": "integer" }, "nearest": { "type": ["string", "null"] }, "sel": { "type": "string" } } } }
59
+ }
60
+ }
@@ -11,7 +11,9 @@ description: >
11
11
  reviewer rejected on a canvas, what is waiting in the Kaloko inbox, or how
12
12
  to report a run in an issue, and when designing screens of a flow in a
13
13
  design environment (live HTML steps, drafts, versions, the design
14
- reference, checking implementation against it).
14
+ reference, checking implementation against it), and when implementing a
15
+ screen that has an approved design (read the step spec, build with its
16
+ tokens, check drift).
15
17
  ---
16
18
 
17
19
  # Kaloko
@@ -32,6 +34,10 @@ the canvas, storage and collaboration are the service.
32
34
  for them (`kaloko share` prints it; `kaloko review` repeats it), and you
33
35
  read `kaloko feedback` and `kaloko inbox` before continuing on that task.
34
36
  - Changes to intent are written back to `ACCEPTANCE.md` (change log row).
37
+ - When you implemented a step from an approved design: you read its spec
38
+ (`kaloko spec <step>`), walked the implementation with `packs: [design]`,
39
+ and `kaloko drift` is clean or the PR names each difference left for a
40
+ person to decide.
35
41
  - On a design environment: the folder follows the bundle rules and
36
42
  `kaloko design lint` shows no error, `kaloko walk` and `kaloko sync` ran,
37
43
  `kaloko status` shows no conflict, and you read the draft's feedback; a
@@ -42,7 +48,9 @@ the canvas, storage and collaboration are the service.
42
48
  Start the shared browser, walk a scenario on a non-production environment,
43
49
  create test data there (users, roles, carts), capture screens and e-mails,
44
50
  evaluate, build the preview, and fix scenario mechanics (selectors, scripts)
45
- that do not change what is being accepted. On a design environment: change
51
+ that do not change what is being accepted, including applying a selector
52
+ proposal with `kaloko walk --heal` when it names the same element. Walk only
53
+ the steps a change reaches (`--affected`) and import screenshots into a run. On a design environment: change
46
54
  the design folder, walk, and sync the draft (including taking others' steps
47
55
  with `--theirs`). Move to the trash runs you uploaded in this session that
48
56
  were superseded before anyone looked at them (debug rounds: no comments, no
@@ -50,7 +58,8 @@ verdict), and run `kaloko prune` without `--yes` to see what a clean-up would do
50
58
 
51
59
  ## Ask first
52
60
 
53
- Changing acceptance criteria or scope, any production run, uploading with
61
+ Changing acceptance criteria or scope, any production run, writing or
62
+ replacing a CI workflow (`kaloko ci github|gitlab`), uploading with
54
63
  `kaloko share` (on a design environment: publishing a version), setting a
55
64
  design reference or baseline, merging a design branch, keeping your version of
56
65
  a step in a sync conflict (`--mine`), turning on `html_view`, a real clean-up
@@ -77,6 +86,15 @@ they can do; never write secrets into the config, a scenario or a script.
77
86
  [reference/stacks.md](reference/stacks.md)
78
87
  - Choosing playwright, agent or manual per step, and driving the shared
79
88
  browser with your capabilities: [reference/execution.md](reference/execution.md)
89
+ - Evaluation, thresholds, the vision evaluator and its reasons, reading
90
+ results: [reference/evaluation.md](reference/evaluation.md)
91
+ browser with your capabilities, walking only the steps a change reaches
92
+ (`--affected`), several locales at once, and selectors that moved
93
+ (`--heal`): [reference/execution.md](reference/execution.md)
94
+ - Screenshots, walkthroughs, Playwright reports and traces that an agent or a
95
+ tool already produced, laid out as a run: [reference/import.md](reference/import.md)
96
+ - CI: a workflow that walks each pull request's preview deployment and
97
+ comments on it (`kaloko ci github|gitlab`): [reference/ci.md](reference/ci.md)
80
98
  - Evaluation, JEV thresholds and reading results: [reference/evaluation.md](reference/evaluation.md)
81
99
  - Environments, access (gates, accounts, sign-in), e-mails, readonly and
82
100
  production rules, and what is yours to arrange for the project's stack:
@@ -94,6 +112,9 @@ they can do; never write secrets into the config, a scenario or a script.
94
112
  references, handoff, tokens, the design pack, canvas edits, branches, HTML
95
113
  view, presenting the prototype, exports, bringing your own design skill):
96
114
  [reference/design.md](reference/design.md)
115
+ - Implementing a screen from the approved design: read the step spec, build
116
+ with its tokens, walk, check drift, fix:
117
+ [reference/design.md#implement-from-the-approved-design](reference/design.md#implement-from-the-approved-design)
97
118
  - Writing design pages that work on the live step, from disk and in other
98
119
  tools (relative URLs, no CDNs, state in the URL, deterministic content,
99
120
  links between steps; what `kaloko design lint` checks):
@@ -20,6 +20,16 @@ unclear, ask one focused question instead of guessing.
20
20
  When intent changes, update `ACCEPTANCE.md` first, add a change-log row with
21
21
  date and commit, bump `version` in the scenario, then adjust the scenario.
22
22
 
23
+ In a project without scenarios, `kaloko init` already drafted
24
+ `qa/flows/main-pages.yml` from the route files: one screen per main route in
25
+ the locales of the i18n config, with the dev server's URL as `local`. It walks
26
+ as it is; sharpen the purposes and criteria and add the flows that matter.
27
+
28
+ Screenshots an agent or a test tool already took (a Cursor or Antigravity
29
+ walkthrough, a Playwright report or trace) can start a scenario too:
30
+ `kaloko import <folder> --adhoc` writes one manual step per screen; see
31
+ [import.md](import.md).
32
+
23
33
  For public pages of a site (landings, marketing, documentation) start from a
24
34
  draft: `npx kaloko draft https://example.com/ https://example.com/pricing`
25
35
  fetches the pages and writes a read-only production scenario and its plan
@@ -40,8 +50,21 @@ for public pages and landings; task flows keep their own criteria.
40
50
 
41
51
  - `a11y` (`AC3xx`): images with alt, links and buttons with a name, labelled
42
52
  fields, heading levels not skipped, WCAG AA text contrast, a `main`
43
- landmark, unique ids. A subset of WCAG 2.2 AA checked in the page; it does
44
- not replace an audit with a screen reader.
53
+ landmark, unique ids (`AC301`–`AC307`, read from the page). After the
54
+ screenshot the walk also presses Tab through the page: `AC308` tab order
55
+ (hidden focus, big jumps back, positive `tabindex`, controls left out or
56
+ mouse-only), `AC309` visible focus with at least 3 : 1 contrast (each stop
57
+ screenshotted focused and unfocused), `AC310` no keyboard trap (a modal
58
+ dialog may keep focus), `AC311` a skip link or main content within three
59
+ stops, `AC312` names from Chrome's accessibility tree (what a screen
60
+ reader announces). `AC313` landmarks (one `main`, labelled repeats) and
61
+ `AC314` ARIA (unknown roles and attributes, references to missing ids,
62
+ `aria-hidden` over focusable content) are read from the page. The first
63
+ five focus stops are kept as screenshots and the canvas can draw the tab
64
+ order on the capture. The walk only moves focus: nothing is clicked or
65
+ typed, and the page is left unfocused at the top. Stack steps walk the
66
+ keyboard on their first five pages. Still a subset of WCAG 2.2 AA; it
67
+ does not replace an audit with a screen reader.
45
68
  - `perf` (`AC4xx`): LCP ≤ 2.5 s, CLS ≤ 0.1, TTFB ≤ 800 ms, page weight ≤ 3 MB,
46
69
  measured in this walk's browser since the page load (lab values: the
47
70
  machine and network of the walk matter; compare runs, not absolutes).
@@ -120,6 +143,12 @@ criteria whose verdict differs between runs on the same commit as flaky;
120
143
  treat a flaky criterion as a selector or timing problem to fix in the
121
144
  scenario, not as a verdict.
122
145
 
146
+ When a step is shaped by files no route reaches (a payment library, a shared
147
+ price component), name them on the step with `affects: [lib/payments/**]`, so
148
+ `kaloko walk --affected` walks it when they change; `affects: always` marks
149
+ steps every walk needs (sign-in, consent). See
150
+ [execution.md](execution.md#only-what-changed).
151
+
123
152
  ## Naming and linkage
124
153
 
125
154
  When the work has a repository and an issue number, make them visible:
@@ -227,3 +256,22 @@ skips a session's nodes there. The canvas shows a chip with the person on
227
256
  every step, groups the step list by person and draws hand-offs between
228
257
  people as dashed arrows. Sessions run one after another, never in parallel,
229
258
  and never share cookies.
259
+
260
+ ## Ignore regions
261
+
262
+ Clocks, ads, carousels and "3 minutes ago" change on every capture and make
263
+ pixel diffs noisy. List them on the step and both the canvas diff and the
264
+ design pixel check (`AC801`) leave them out:
265
+
266
+ ```yaml
267
+ - id: home
268
+ ignore:
269
+ - { selector: "#clock" } # every visible match
270
+ - { selector: ".ad-slot", viewports: [desktop] }
271
+ - { rect: [0, 620, 1280, 240], viewport: desktop } # page pixels of one viewport
272
+ ```
273
+
274
+ Do not guess them: `kaloko walk --stability 3` notes what moves while the
275
+ page sits still, and `kaloko stability` compares the step with earlier runs
276
+ and writes suggestions (see [execution.md](execution.md)). Ignore only noise;
277
+ a region that matters to a criterion stays in.
@@ -0,0 +1,70 @@
1
+ # CI and pull request previews
2
+
3
+ Kaloko runs in CI with the same commands as on a laptop. `kaloko ci` writes the
4
+ workflow; you only add a token. Writing or replacing a CI workflow changes the
5
+ repository's pipeline: ask the person first.
6
+
7
+ ```bash
8
+ kaloko ci github [--scenario qa/flows/checkout.yml,qa/flows/signup.yml] # .github/workflows/kaloko.yml
9
+ kaloko ci gitlab # kaloko.gitlab-ci.yml (include it)
10
+ kaloko ci github --print # show it, write nothing
11
+ ```
12
+
13
+ What the generated workflow does on every preview deployment of a pull
14
+ request (Vercel, Netlify and Cloudflare Pages report their previews as GitHub
15
+ deployments; `--trigger pull_request` runs on every push instead and waits for
16
+ the deployment):
17
+
18
+ 1. installs the project (`npm ci`, or pnpm, yarn, bun by the lockfile) and the
19
+ Chromium of kaloko's Playwright (`kaloko browser install --with-deps`);
20
+ `KALOKO_BROWSER_CHANNEL=chromium` makes the run use it, so no separate CI
21
+ config is needed;
22
+ 2. `kaloko ci preview-url --wait 600 --github-env` finds the preview URL, the
23
+ pull request and its base branch and puts them into `KALOKO_PREVIEW_URL`,
24
+ `KALOKO_PR`, `KALOKO_BASE`;
25
+ 3. for each scenario: `kaloko affected --check` skips it when the change does
26
+ not reach any of its steps, otherwise `start --env preview`,
27
+ `walk --ephemeral --affected` and `share --pr` (one comment per scenario,
28
+ updated on later pushes).
29
+
30
+ `kaloko ci` also adds the `preview` environment to `kaloko.config.yml` with
31
+ `base_url: ${KALOKO_PREVIEW_URL}`. Scenarios that list `environments:` must
32
+ include `preview`, or `start` refuses them; the command says which.
33
+
34
+ ## Secrets
35
+
36
+ - `KALOKO_TOKEN`: a token with the tester role and the upload scope (Settings →
37
+ API tokens; on Enterprise a service account's token, which no person owns).
38
+ - `TYPESAFE_API_KEY` only when semantic criteria should use your own evaluator
39
+ key instead of the organization's allowance.
40
+ - Vercel deployment protection: a `VERCEL_AUTOMATION_BYPASS_SECRET` secret and
41
+ the commented `access.headers` block of the preview environment.
42
+ - GitLab: `KALOKO_GITLAB_TOKEN` (project access token, `api` scope) for the
43
+ merge request note; set `KALOKO_PREVIEW_URL` in the job when the review app
44
+ URL is known (`https://$CI_COMMIT_REF_SLUG.review.example.com`), or rely on
45
+ `CI_ENVIRONMENT_URL`.
46
+
47
+ ## Where the preview URL comes from
48
+
49
+ `kaloko ci preview-url` takes the first of: `KALOKO_PREVIEW_URL`; the
50
+ `deployment_status` event; `VERCEL_BRANCH_URL`/`VERCEL_URL`,
51
+ `DEPLOY_PRIME_URL`/`DEPLOY_URL` (Netlify), `CF_PAGES_URL`,
52
+ `CI_ENVIRONMENT_URL` (GitLab), `RENDER_EXTERNAL_URL`; then the commit's GitHub
53
+ deployments, skipping production, polled every 10 s up to `--wait` seconds.
54
+ Only the URL goes to stdout, so `export KALOKO_PREVIEW_URL="$(kaloko ci preview-url)"`
55
+ works in any CI.
56
+
57
+ ## Affected steps in CI
58
+
59
+ `walk --affected` needs the history: check out with `fetch-depth: 0` (GitHub)
60
+ or `GIT_DEPTH: "0"` (GitLab). The base is `--base`, else `KALOKO_BASE`,
61
+ `GITHUB_BASE_REF`, the merge request target, then `origin/main`. URLs captured
62
+ in the newest shared run of the scenario count too, so a fresh CI checkout
63
+ maps routes as well as a laptop with earlier runs. See
64
+ [execution.md](execution.md#only-what-changed).
65
+
66
+ ## Done means
67
+
68
+ The workflow file and the config change are committed after the person agreed,
69
+ the secret is named for the person to add, and you report the first run's PR
70
+ comment or why it did not appear (no preview deployment, missing token).
@@ -12,7 +12,9 @@ loop teams use:
12
12
  request and `gh` is signed in, `kaloko share --pr` posts one comment per
13
13
  scenario on it (summary line, canvas link, criteria that need attention
14
14
  with evidence) and updates that same comment on every later run, so the
15
- PR never fills up with Kaloko noise. Otherwise write the comment yourself:
15
+ PR never fills up with Kaloko noise. In CI `KALOKO_PR` names the pull
16
+ request of a detached checkout, and on GitLab merge requests the comment is a
17
+ note on the MR (`KALOKO_GITLAB_TOKEN`). Otherwise write the comment yourself:
16
18
  what was tested (scenario, environment, locales), the summary line
17
19
  (`n/m steps passed`), every failing or `needs_review` criterion with
18
20
  evidence, and the canvas URL. Reviewers accept, return or comment on the
@@ -255,6 +257,12 @@ Everything above is available over the API with an organization token:
255
257
  - `GET /api/v1/orgs/<org>/runs/<id>/export.zip` (`kaloko export`): the whole
256
258
  run as a zip before it expires; keep it with the issue when the evidence
257
259
  must outlive the retention period.
260
+ - Uploads by content hash: `POST …/blobs/missing` with `{"hashes": [sha256…]}`
261
+ answers what the organization does not store yet, `PUT …/blobs/<sha256>`
262
+ (header `X-QAwalk-Type: png|webp|html|json`) stores one, and the run's file
263
+ plan `POST …/runs/<id>/files` takes `{ path, bytes, blob }` entries that
264
+ point at stored blobs. `kaloko share` does this: unchanged captures are not
265
+ sent again and a share that broke off resumes when run again.
258
266
  - MCP server for agents: see [mcp.md](mcp.md).
259
267
 
260
268
  Read-only tokens may list but not change anything; keep CI tokens read-only
@@ -39,8 +39,10 @@ design/
39
39
  the bundle.
40
40
  - Links between steps are plain relative links to the page: `<a href="../payment/index.html">`. The canvas
41
41
  follows them; `data-kaloko-go="payment"` (or `data-qawalk-go`) moves to that step whatever the `href` says.
42
- - Reference token values as CSS custom properties (`var(--color-accent)`), so the inspector and the token
43
- check see where values come from.
42
+ - Reference token values as CSS custom properties named after the token path (`color.accent` →
43
+ `var(--color-accent)`), so Dev mode, the step spec and the token check see which token a value comes from.
44
+ Define interaction states in CSS (`:hover`, `:focus-visible`, `:active`, `:disabled`): Dev mode and the spec
45
+ read them from the stylesheet.
44
46
  - Write pages by the portable HTML rules ([portable-html.md](portable-html.md)): relative URLs only, bundled
45
47
  fonts, no CDNs, state in the URL instead of storage, deterministic content, viewport meta and `lang`.
46
48
  `kaloko design lint` checks the folder; `kaloko sync` runs it first and stops on errors (things that break on
@@ -94,8 +96,39 @@ Development builds from the design reference:
94
96
  - `kaloko export --live [--version n] --out design/` writes the live files into a folder.
95
97
  - MCP `get_step` (scenario, step, `ref: design`) returns the file list, a signed entry URL, the token set, the
96
98
  page structure and open comments; `list_step_changes` names the steps that changed since the last reference.
99
+ - MCP `get_step_spec` (or `kaloko spec <step>`) returns what to build, element by element, with tokens; see
100
+ [Implement from the approved design](#implement-from-the-approved-design).
97
101
  - Translate to the project's stack with the project's own skills; keep the step ids and the token names.
98
102
 
103
+ ## Implement from the approved design
104
+
105
+ Use this when you build or change a screen that has an approved design step (a design reference, or a version
106
+ the person names). The step spec says what was approved in the terms you write code in: elements in page order
107
+ with role, accessible name and selector, and their colors, typography, spacing, radius, shadow and layout, each
108
+ value with the DTCG token it comes from. People see the same data in the canvas: *Dev mode* (`D`) in the step
109
+ detail.
110
+
111
+ 1. **Read the spec** of each step you build: `kaloko spec <step>` (default `--ref design`; `--ref v4`, `draft` or a
112
+ run id), or MCP `get_step_spec`. `--json` gives `qawalk.spec.v1` (`schemas/spec.v1.json`).
113
+ 2. **Implement with the tokens.** A line such as `button "Pay" 120×44 bg #2e6bff=color.brand.primary · pad 12px
114
+ 20px=space.3 space.5` means `background: var(--color-brand-primary); padding: var(--space-3) var(--space-5)`
115
+ (the custom property is the token path with dots as dashes) or the project's own token mechanism. `not a token;
116
+ nearest x` means the design itself used a raw value: ask the designer before you copy it, and say so in the PR.
117
+ `:hover`, `:focus`, `:active` and `:disabled` lines are states to build; `(light)` names the mode of a value.
118
+ Keep the roles and names: they are how the implementation is matched with the design.
119
+ 3. **Walk it** with `packs: [design]` on the implementation steps; the capture then stores the implementation's
120
+ spec next to the screenshot.
121
+ 4. **Check drift**: `kaloko drift [step]` compares the walked run with the design reference of the linked design
122
+ branch and exits 1 while something differs. A shared run: MCP `get_step_drift` (`run`: id or `latest`).
123
+ Findings name the element, the expected token and value, and what the page has; `typed raw` means the value
124
+ is right but should come from the token; `not found` means the design's element (by role and name) is missing.
125
+ 5. **Fix and repeat** until drift is clean or what is left is a decision for a person (a missing element on
126
+ purpose, a raw value the designer accepted); report those in the PR instead of hiding them.
127
+
128
+ Specs exist for captures made with the current CLI: walk the design and sync again if `kaloko spec` says there is
129
+ none. The pixel, structure and token checks of the design pack (`AC801`–`AC803`) still run in `kaloko evaluate`;
130
+ drift is the per-element view an agent can act on.
131
+
99
132
  ## Your data is yours
100
133
 
101
134
  Everything exports in open formats, readable without Kaloko:
@@ -13,6 +13,11 @@ them; do not work around a block.
13
13
  Test data preparation for review and staging belongs to the project's CI
14
14
  prepare step; assume it exists and create only what the scenario itself needs.
15
15
 
16
+ Pull request previews (Vercel, Netlify, Cloudflare Pages, review apps) are an
17
+ environment whose URL comes from CI: `preview: { base_url: ${KALOKO_PREVIEW_URL} }`,
18
+ which `kaloko ci github|gitlab` adds and `kaloko ci preview-url` fills; see
19
+ [ci.md](ci.md).
20
+
16
21
  ## Who does what
17
22
 
18
23
  Kaloko needs evidence in its format: captures of screens and e-mails in a run.