kaloko 0.4.11 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +45 -4
- package/package.json +1 -1
- package/schemas/config.v1.json +42 -2
- package/schemas/run.v1.json +33 -2
- package/schemas/scenario.v1.json +107 -16
- package/skills/kaloko/SKILL.md +12 -4
- package/skills/kaloko/reference/acceptance.md +74 -3
- package/skills/kaloko/reference/apps.md +103 -8
- package/skills/kaloko/reference/ci.md +3 -1
- package/skills/kaloko/reference/collaboration.md +35 -6
- package/skills/kaloko/reference/design.md +39 -0
- package/skills/kaloko/reference/devices.md +109 -0
- package/skills/kaloko/reference/environments.md +6 -0
- package/skills/kaloko/reference/execution.md +15 -0
- package/skills/kaloko/reference/mcp.md +5 -2
- package/src/cli.mjs +366 -192
- package/src/lib/access.mjs +5 -2
- package/src/lib/apps/devices.mjs +187 -0
- package/src/lib/apps/drivers.mjs +11 -3
- package/src/lib/apps/ios-device.mjs +78 -0
- package/src/lib/apps/tree.mjs +9 -1
- package/src/lib/apps/webview.mjs +54 -0
- package/src/lib/apps/windows.mjs +108 -0
- package/src/lib/apps/windows.ps1 +78 -0
- package/src/lib/assertions.mjs +300 -0
- package/src/lib/build.mjs +4 -1
- package/src/lib/capture.mjs +146 -119
- package/src/lib/ci.mjs +8 -4
- package/src/lib/design-cli.mjs +35 -6
- package/src/lib/design.mjs +3 -1
- package/src/lib/devices.mjs +85 -0
- package/src/lib/devspec.mjs +3 -2
- package/src/lib/dimensions.mjs +326 -0
- package/src/lib/doctor.mjs +31 -6
- package/src/lib/drift.mjs +8 -4
- package/src/lib/evaluate.mjs +25 -7
- package/src/lib/import.mjs +2 -1
- package/src/lib/jev.mjs +3 -1
- package/src/lib/manifest-core.mjs +12 -3
- package/src/lib/probe.mjs +60 -0
- package/src/lib/recheck.mjs +6 -1
- package/src/lib/scenario.mjs +3 -0
- package/src/lib/signoff-core.mjs +138 -0
- package/src/lib/signoff.mjs +83 -0
- package/src/lib/stability-core.mjs +4 -2
- package/src/lib/stability.mjs +82 -0
- package/src/lib/walk-dimensions.mjs +83 -0
- package/viewer/index.html +739 -48
package/README.md
CHANGED
|
@@ -11,10 +11,11 @@ there, and the newest captures double as documentation.
|
|
|
11
11
|
- Product and guides: <https://kaloko.app>
|
|
12
12
|
- Service (organizations, shared canvases, reviews): <https://kaloko.app>
|
|
13
13
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
14
|
+
Pages are walked in Chromium, WebKit and Firefox, on phone, tablet and
|
|
15
|
+
desktop presets, in light and dark mode. Native apps too: Android, iOS,
|
|
16
|
+
Electron, macOS and Windows. Screens taken by any other tool (Maestro, Appium,
|
|
17
|
+
XCUITest) are imported with their UI tree. See
|
|
18
|
+
[What Kaloko can walk and design](#what-kaloko-can-walk-and-design).
|
|
18
19
|
|
|
19
20
|
## Install
|
|
20
21
|
|
|
@@ -71,8 +72,48 @@ once with `--concurrency 3`. When a step's selector no longer exists, the walk
|
|
|
71
72
|
proposes one from the page and prints the patch; `--heal` applies it once the
|
|
72
73
|
step passes.
|
|
73
74
|
|
|
75
|
+
Automatic criteria read the page itself: text, counts and attributes, the URL
|
|
76
|
+
after a step (`url_matches`), computed styles, also against a design token
|
|
77
|
+
(`style_equals` with `token: color.brand.primary`), and read-only values such
|
|
78
|
+
as `localStorage` or `dataLayer` (`js_equals`). A step script never has to
|
|
79
|
+
compute a result and leave a marker for a criterion to find. The acceptance
|
|
80
|
+
reference of the `kaloko` skill lists every check.
|
|
81
|
+
|
|
74
82
|
`npx kaloko help` lists every command.
|
|
75
83
|
|
|
84
|
+
## What Kaloko can walk and design
|
|
85
|
+
|
|
86
|
+
*Full*: Kaloko drives it, captures it and runs the checks. *Capture only*: Kaloko
|
|
87
|
+
takes the screens, your tool or a person moves between them. *Via import*: another
|
|
88
|
+
tool takes the screens and `kaloko capture --image` or `kaloko import` lays them out.
|
|
89
|
+
|
|
90
|
+
| Target | How | Checks | Design | Support |
|
|
91
|
+
|---|---|---|---|---|
|
|
92
|
+
| Web in Chromium | Playwright, the shared browser or an own one | all | live HTML steps | full |
|
|
93
|
+
| Web in WebKit (Safari's engine) and Firefox | Playwright, one pass per engine | all except the accessibility tree, focus rings and layout shifts (and the Tab walk on WebKit) | rendered in Chromium | full |
|
|
94
|
+
| Phones, foldables, tablets, laptop to 2560 px, TV | device presets: size, pixel ratio, touch, user agent | all | whole screen, device frame, safe areas | full |
|
|
95
|
+
| Light and dark, reduced motion, high contrast | emulated per capture | all; contrast per scheme | dark pages, token modes per scheme | full |
|
|
96
|
+
| Installed web app (PWA) | `display-mode: standalone` rules and `matchMedia` | all | yes | full for CSS; scripts that read the mode at load need a real install |
|
|
97
|
+
| Android: emulator, USB or Wi-Fi phone | adb: deep links, taps, typing, dark mode | UI tree, app pack | HTML mockups | full |
|
|
98
|
+
| Android WebView | DevTools over adb (debuggable builds) | web checks on the page | | full |
|
|
99
|
+
| iOS Simulator | simctl: deep links, dark mode; taps with idb | UI tree with idb, text by OCR without | HTML mockups | full with idb, capture only without |
|
|
100
|
+
| iPhone and iPad | devicectl, idevicescreenshot or pymobiledevice3 | text by OCR | HTML mockups | capture only |
|
|
101
|
+
| Electron | Playwright | all web checks | | full |
|
|
102
|
+
| macOS apps | window screenshot, accessibility tree | UI tree | | capture only |
|
|
103
|
+
| Windows apps | PowerShell and UI Automation: taps, typing | UI tree | | full on Windows, via import elsewhere |
|
|
104
|
+
| React Native, Flutter, iOS WKWebView, Maestro, Appium, XCUITest | their own runner | UI tree from a dump, or text | | via import |
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
npx kaloko devices # the presets
|
|
108
|
+
npx kaloko browser install --browsers webkit,firefox
|
|
109
|
+
npx kaloko start --scenario qa/flows/checkout.yml --env staging \
|
|
110
|
+
--browsers chromium,webkit --color-schemes light,dark --viewports iphone-se,pixel-8,desktop-hd
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The canvas switches between browsers and schemes next to the locale, draws the device frame and shows the same
|
|
114
|
+
step in another browser beside it. `npx kaloko doctor` says which engines and app tools are missing on this
|
|
115
|
+
machine and how to install them. Details: `skills/kaloko/reference/devices.md` and `apps.md`.
|
|
116
|
+
|
|
76
117
|
## Pull request previews in CI
|
|
77
118
|
|
|
78
119
|
```bash
|
package/package.json
CHANGED
package/schemas/config.v1.json
CHANGED
|
@@ -78,7 +78,12 @@
|
|
|
78
78
|
"type": "object",
|
|
79
79
|
"additionalProperties": false,
|
|
80
80
|
"properties": {
|
|
81
|
-
"concurrency": { "type": "integer", "minimum": 1, "maximum": 16, "description": "Locales walked at the same time, each in its own browser context (kaloko walk --concurrency). Default: by CPU in an own browser, one at a time on the shared browser." }
|
|
81
|
+
"concurrency": { "type": "integer", "minimum": 1, "maximum": 16, "description": "Locales walked at the same time, each in its own browser context (kaloko walk --concurrency). Default: by CPU in an own browser, one at a time on the shared browser." },
|
|
82
|
+
"browsers": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "enum": ["chromium", "webkit", "firefox"] }, "description": "Default browser engines for web runs when the scenario and kaloko start --browsers say nothing." },
|
|
83
|
+
"color_schemes": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "enum": ["light", "dark"] }, "description": "Default colour schemes (prefers-color-scheme) for every run." },
|
|
84
|
+
"reduced_motion": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "enum": ["no-preference", "reduce"] } },
|
|
85
|
+
"forced_colors": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "enum": ["none", "active"] } },
|
|
86
|
+
"display_modes": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "enum": ["browser", "standalone"] } }
|
|
82
87
|
}
|
|
83
88
|
},
|
|
84
89
|
"capture": {
|
|
@@ -120,6 +125,18 @@
|
|
|
120
125
|
}
|
|
121
126
|
]
|
|
122
127
|
},
|
|
128
|
+
"appDevices": {
|
|
129
|
+
"type": "object",
|
|
130
|
+
"description": "Devices a scenario's devices: list names, each an emulator, simulator or real device. Without an entry the name is matched against adb serials and AVDs (pixel-8-api-34 → Pixel_8_API_34) or simulators (iphone-15-ios-18 → iPhone 15 on iOS 18).",
|
|
131
|
+
"additionalProperties": {
|
|
132
|
+
"type": "object", "additionalProperties": false,
|
|
133
|
+
"properties": {
|
|
134
|
+
"serial": { "type": "string" }, "avd": { "type": "string" }, "connect": { "type": "string" },
|
|
135
|
+
"udid": { "type": "string" }, "device": { "type": "string", "description": "Simulator name (iOS) or the canvas label (Android)." }, "runtime": { "type": "string" },
|
|
136
|
+
"real": { "type": "boolean" }, "title": { "type": "string", "description": "Label on the canvas." }
|
|
137
|
+
}
|
|
138
|
+
}
|
|
139
|
+
},
|
|
123
140
|
"environment": {
|
|
124
141
|
"type": "object",
|
|
125
142
|
"additionalProperties": false,
|
|
@@ -143,7 +160,7 @@
|
|
|
143
160
|
"app": {
|
|
144
161
|
"type": "object",
|
|
145
162
|
"additionalProperties": false,
|
|
146
|
-
"description": "The app under test in this environment, per platform (scenario platform: android, ios, electron, desktop). Paths of app scenarios are deep links.",
|
|
163
|
+
"description": "The app under test in this environment, per platform (scenario platform: android, ios, electron, desktop, windows). Paths of app scenarios are deep links.",
|
|
147
164
|
"properties": {
|
|
148
165
|
"android": {
|
|
149
166
|
"type": "object", "additionalProperties": false,
|
|
@@ -155,6 +172,12 @@
|
|
|
155
172
|
"serial": { "type": "string", "description": "adb device serial; default ANDROID_SERIAL or the only device." },
|
|
156
173
|
"device": { "type": "string", "description": "Label for the canvas; default the device model." },
|
|
157
174
|
"reset": { "type": "boolean", "description": "Clear app data at the start of every locale (a new user)." },
|
|
175
|
+
"connect": { "type": "string", "description": "A real device over Wi-Fi: host:port for adb connect before the walk (pair it once with adb pair)." },
|
|
176
|
+
"avd": { "type": "string", "description": "Emulator (Android Virtual Device) to start when it is not running, e.g. Pixel_8_API_34." },
|
|
177
|
+
"emulator": { "type": "string", "description": "Path to the emulator binary (default: emulator on PATH or in ANDROID_HOME)." },
|
|
178
|
+
"wake": { "type": "boolean", "description": "Wake and unlock the screen of a real device before the walk (no PIN; default true)." },
|
|
179
|
+
"webview": { "type": "boolean", "description": "When the screen is a WebView with debugging on, capture its page (DOM, web checks) too; default false." },
|
|
180
|
+
"devices": { "$ref": "#/$defs/appDevices" },
|
|
158
181
|
"adb": { "type": "string" }, "launch_wait": { "type": "integer" }, "settle": { "type": "integer" }
|
|
159
182
|
}
|
|
160
183
|
},
|
|
@@ -170,6 +193,9 @@
|
|
|
170
193
|
"idb": { "type": "boolean", "description": "Use idb for the UI tree and taps when installed (default true)." },
|
|
171
194
|
"ocr": { "type": "boolean", "description": "Without idb, read the screen text by OCR (macOS Vision; default true)." },
|
|
172
195
|
"status_bar": { "type": "boolean", "description": "Clean status bar (9:41, full battery) for captures; default true." },
|
|
196
|
+
"runtime": { "type": "string", "description": "Simulator runtime, e.g. iOS 18 (the newest matching one)." },
|
|
197
|
+
"real": { "type": "boolean", "description": "A real iPhone or iPad (udid): installs and launches with xcrun devicectl, screenshots with idevicescreenshot (libimobiledevice); no taps or UI tree, OCR on macOS." },
|
|
198
|
+
"devices": { "$ref": "#/$defs/appDevices" },
|
|
173
199
|
"scale": { "type": "integer" }, "launch_wait": { "type": "integer" }, "settle": { "type": "integer" }
|
|
174
200
|
}
|
|
175
201
|
},
|
|
@@ -192,6 +218,20 @@
|
|
|
192
218
|
"launch": { "type": "boolean", "description": "Open the app at the start (default true)." },
|
|
193
219
|
"reset": { "type": "boolean" }, "ocr": { "type": "boolean" }, "launch_wait": { "type": "integer" }, "settle": { "type": "integer" }
|
|
194
220
|
}
|
|
221
|
+
},
|
|
222
|
+
"windows": {
|
|
223
|
+
"type": "object", "additionalProperties": false,
|
|
224
|
+
"description": "A Windows desktop app, driven through UI Automation with Windows PowerShell (no extra install).",
|
|
225
|
+
"properties": {
|
|
226
|
+
"name": { "type": "string", "description": "Process name or window title, e.g. notepad or Contoso Shop." },
|
|
227
|
+
"path": { "type": "string", "description": "Executable to start (relative to the repository or absolute)." },
|
|
228
|
+
"aumid": { "type": "string", "description": "AppUserModelID of a Store/MSIX app to start instead of path." },
|
|
229
|
+
"args": { "type": "array", "items": { "type": "string" } },
|
|
230
|
+
"launch": { "type": "boolean", "description": "Start the app at the start (default true)." },
|
|
231
|
+
"reset": { "type": "boolean", "description": "Close a running instance first." },
|
|
232
|
+
"powershell": { "type": "string", "description": "PowerShell binary (default powershell.exe; pwsh works on Windows too)." },
|
|
233
|
+
"launch_wait": { "type": "integer" }, "settle": { "type": "integer" }
|
|
234
|
+
}
|
|
195
235
|
}
|
|
196
236
|
}
|
|
197
237
|
},
|
package/schemas/run.v1.json
CHANGED
|
@@ -147,7 +147,32 @@
|
|
|
147
147
|
}
|
|
148
148
|
},
|
|
149
149
|
"viewports": {
|
|
150
|
-
"type": "object"
|
|
150
|
+
"type": "object",
|
|
151
|
+
"description": "Capture key -> [width, height]. A key is the viewport name plus the dimension values that differ from the default, joined by dots (mobile, mobile.webkit, iphone-15.webkit.dark.reduce)."
|
|
152
|
+
},
|
|
153
|
+
"variants": {
|
|
154
|
+
"type": "object",
|
|
155
|
+
"description": "Capture key -> { viewport, browser, color_scheme?, reduced_motion?, forced_colors?, display_mode?, device? }. Absent in runs without dimensions: every key is then a plain viewport in Chromium.",
|
|
156
|
+
"additionalProperties": {
|
|
157
|
+
"type": "object",
|
|
158
|
+
"properties": {
|
|
159
|
+
"viewport": { "type": "string" },
|
|
160
|
+
"browser": { "enum": ["chromium", "webkit", "firefox"] },
|
|
161
|
+
"color_scheme": { "enum": ["light", "dark"] },
|
|
162
|
+
"reduced_motion": { "enum": ["no-preference", "reduce"] },
|
|
163
|
+
"forced_colors": { "enum": ["none", "active"] },
|
|
164
|
+
"display_mode": { "enum": ["browser", "standalone"] },
|
|
165
|
+
"device": { "type": "string" }
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
},
|
|
169
|
+
"dimensions": {
|
|
170
|
+
"type": "object",
|
|
171
|
+
"description": "Values of each dimension the run walked (browsers, color_schemes, reduced_motion, forced_colors, display_modes) for the canvas switchers."
|
|
172
|
+
},
|
|
173
|
+
"devices": {
|
|
174
|
+
"type": "object",
|
|
175
|
+
"description": "Viewport name -> device preset (title, kind, screen, size, dpr, safe areas, frame) for device frames on the canvas."
|
|
151
176
|
},
|
|
152
177
|
"note": {
|
|
153
178
|
"type": "string"
|
|
@@ -405,7 +430,13 @@
|
|
|
405
430
|
"url": {
|
|
406
431
|
"type": ["string", "null"]
|
|
407
432
|
},
|
|
408
|
-
"platform": { "type": "string", "description": "App captures: android, ios, electron or
|
|
433
|
+
"platform": { "type": "string", "description": "App captures: android, ios, electron, desktop or windows." },
|
|
434
|
+
"browser": { "type": "string", "description": "Engine of a web capture when the run has dimensions; absent = chromium." },
|
|
435
|
+
"color_scheme": { "type": "string" },
|
|
436
|
+
"reduced_motion": { "type": "string" },
|
|
437
|
+
"forced_colors": { "type": "string" },
|
|
438
|
+
"display_mode": { "type": "string" },
|
|
439
|
+
"device_preset": { "type": "string", "description": "Viewport name of the device preset the capture emulated." },
|
|
409
440
|
"device": { "type": ["string", "null"] },
|
|
410
441
|
"tree": { "type": ["string", "null"], "description": "UI tree of an app capture (qawalk.tree.v1)." },
|
|
411
442
|
"http_status": {
|
package/schemas/scenario.v1.json
CHANGED
|
@@ -50,17 +50,25 @@
|
|
|
50
50
|
"minItems": 1
|
|
51
51
|
},
|
|
52
52
|
"viewports": {
|
|
53
|
-
"
|
|
54
|
-
"
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
"
|
|
58
|
-
"minimum": 200
|
|
53
|
+
"description": "Viewports by name: a size [width, height], a device preset (iphone-15, ipad-landscape, fold-open, laptop, tv-1080p, <preset>-landscape …) or { device, width?, height?, full_screen? }. A list of preset names works too: [iphone-se, pixel-8, desktop-hd].",
|
|
54
|
+
"oneOf": [
|
|
55
|
+
{
|
|
56
|
+
"type": "object",
|
|
57
|
+
"additionalProperties": { "$ref": "#/$defs/viewport" }
|
|
59
58
|
},
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
59
|
+
{
|
|
60
|
+
"type": "array",
|
|
61
|
+
"minItems": 1,
|
|
62
|
+
"items": { "oneOf": [{ "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$" }, { "type": "object", "minProperties": 1, "maxProperties": 1, "additionalProperties": { "$ref": "#/$defs/viewport" } }] }
|
|
63
|
+
}
|
|
64
|
+
]
|
|
63
65
|
},
|
|
66
|
+
"browsers": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "enum": ["chromium", "webkit", "firefox"] }, "description": "Browser engines to walk the web flow in (Playwright). Default: chromium. Captures of other engines are keyed <viewport>.<engine>." },
|
|
67
|
+
"color_schemes": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "enum": ["light", "dark"] }, "description": "prefers-color-scheme values to capture every step in (keys <viewport>.dark). Not set: the browser default, nothing emulated." },
|
|
68
|
+
"reduced_motion": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "enum": ["no-preference", "reduce"] }, "description": "prefers-reduced-motion values (keys <viewport>.reduce)." },
|
|
69
|
+
"forced_colors": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "enum": ["none", "active"] }, "description": "forced-colors (Windows high contrast) values (keys <viewport>.forced)." },
|
|
70
|
+
"display_modes": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "enum": ["browser", "standalone"] }, "description": "PWA display modes: standalone captures the page as an installed app (display-mode media rules and matchMedia; the whole screen of a device preset)." },
|
|
71
|
+
"devices": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$" }, "description": "App scenarios: devices to walk on, e.g. pixel-8-api-34, iphone-15-ios-18 (emulators, simulators or real devices named in the environment's app.<platform>.devices). One capture key per device." },
|
|
64
72
|
"environments": {
|
|
65
73
|
"type": "array",
|
|
66
74
|
"items": {
|
|
@@ -72,9 +80,9 @@
|
|
|
72
80
|
"default": false
|
|
73
81
|
},
|
|
74
82
|
"platform": {
|
|
75
|
-
"enum": ["web", "android", "ios", "electron", "desktop"],
|
|
83
|
+
"enum": ["web", "android", "ios", "electron", "desktop", "windows"],
|
|
76
84
|
"default": "web",
|
|
77
|
-
"description": "What the flow runs on. web: pages in
|
|
85
|
+
"description": "What the flow runs on. web: pages in a browser (an installed PWA too: display_modes). android/ios/electron/desktop/windows: an app, driven by the environment's app block (adb, iOS Simulator or device, Playwright Electron, macOS window, Windows UI Automation) or captured with kaloko capture --image. Node paths are deep links for apps."
|
|
78
86
|
},
|
|
79
87
|
"tags": {
|
|
80
88
|
"type": "array",
|
|
@@ -182,6 +190,23 @@
|
|
|
182
190
|
}
|
|
183
191
|
]
|
|
184
192
|
},
|
|
193
|
+
"viewport": {
|
|
194
|
+
"oneOf": [
|
|
195
|
+
{ "type": "array", "items": { "type": "integer", "minimum": 200 }, "minItems": 2, "maxItems": 2 },
|
|
196
|
+
{ "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$", "description": "A device preset name." },
|
|
197
|
+
{
|
|
198
|
+
"type": "object",
|
|
199
|
+
"additionalProperties": false,
|
|
200
|
+
"required": ["device"],
|
|
201
|
+
"properties": {
|
|
202
|
+
"device": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$" },
|
|
203
|
+
"width": { "type": "integer", "minimum": 200 },
|
|
204
|
+
"height": { "type": "integer", "minimum": 200 },
|
|
205
|
+
"full_screen": { "type": "boolean", "description": "The whole screen without browser bars (app mockups, kiosk). Design environments use it by default." }
|
|
206
|
+
}
|
|
207
|
+
}
|
|
208
|
+
]
|
|
209
|
+
},
|
|
185
210
|
"slug": {
|
|
186
211
|
"type": "string",
|
|
187
212
|
"pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$"
|
|
@@ -511,12 +536,19 @@
|
|
|
511
536
|
"count",
|
|
512
537
|
"text_matches",
|
|
513
538
|
"text_absent",
|
|
539
|
+
"text_equals",
|
|
514
540
|
"attr_equals",
|
|
541
|
+
"attr_matches",
|
|
515
542
|
"no_overflow",
|
|
516
543
|
"no_console_errors",
|
|
517
544
|
"title_matches",
|
|
518
545
|
"meta_matches",
|
|
519
546
|
"link_matches",
|
|
547
|
+
"url_matches",
|
|
548
|
+
"url_equals",
|
|
549
|
+
"style_equals",
|
|
550
|
+
"style_matches",
|
|
551
|
+
"js_equals",
|
|
520
552
|
"http_status",
|
|
521
553
|
"link_home",
|
|
522
554
|
"links_ok",
|
|
@@ -550,22 +582,73 @@
|
|
|
550
582
|
},
|
|
551
583
|
"selector": {
|
|
552
584
|
"type": "string",
|
|
553
|
-
"description": "CSS selector; links_ok: the links to check (default a[href])."
|
|
585
|
+
"description": "CSS selector; links_ok: the links to check (default a[href]); style_*: the elements whose computed style is read."
|
|
554
586
|
},
|
|
555
587
|
"pattern": {
|
|
556
588
|
"type": "string",
|
|
557
|
-
"description": "Regex; links_ok: link paths to skip."
|
|
589
|
+
"description": "Regex; links_ok: link paths to skip; url_matches: tested on the URL part; style_matches: on the computed value; attr_matches: on the attribute; js_equals: on the result."
|
|
558
590
|
},
|
|
559
591
|
"attr": {
|
|
560
592
|
"type": "string"
|
|
561
593
|
},
|
|
594
|
+
"property": {
|
|
595
|
+
"type": "string",
|
|
596
|
+
"minLength": 1,
|
|
597
|
+
"description": "style_equals, style_matches: the CSS property read with getComputedStyle (color, background-color, display, font-size, …). Use longhands, not shorthands like background."
|
|
598
|
+
},
|
|
599
|
+
"token": {
|
|
600
|
+
"type": "string",
|
|
601
|
+
"pattern": "^[^{}]+$",
|
|
602
|
+
"description": "style_equals: a DTCG token name (color.brand.primary) instead of value; resolved from the project's token set, or from the page's --color-brand-primary custom property."
|
|
603
|
+
},
|
|
604
|
+
"tolerance": {
|
|
605
|
+
"type": "number",
|
|
606
|
+
"minimum": 0,
|
|
607
|
+
"description": "style_equals: allowed difference for lengths, in px."
|
|
608
|
+
},
|
|
609
|
+
"op": {
|
|
610
|
+
"enum": [
|
|
611
|
+
"<",
|
|
612
|
+
"<=",
|
|
613
|
+
">",
|
|
614
|
+
">=",
|
|
615
|
+
"=="
|
|
616
|
+
],
|
|
617
|
+
"description": "style_equals, js_equals: compare numerically with value instead of equality (font-size >= 16px)."
|
|
618
|
+
},
|
|
619
|
+
"all": {
|
|
620
|
+
"type": "boolean",
|
|
621
|
+
"description": "style_*, text_equals, attr_matches: every match must pass (default: at least one)."
|
|
622
|
+
},
|
|
623
|
+
"part": {
|
|
624
|
+
"enum": [
|
|
625
|
+
"href",
|
|
626
|
+
"path",
|
|
627
|
+
"query",
|
|
628
|
+
"hash",
|
|
629
|
+
"host",
|
|
630
|
+
"origin"
|
|
631
|
+
],
|
|
632
|
+
"description": "url_matches: which part of the URL the pattern tests (default href, the whole URL)."
|
|
633
|
+
},
|
|
634
|
+
"ignore_query": {
|
|
635
|
+
"type": "boolean",
|
|
636
|
+
"description": "url_equals: compare without the query string and hash."
|
|
637
|
+
},
|
|
638
|
+
"expression": {
|
|
639
|
+
"type": "string",
|
|
640
|
+
"minLength": 1,
|
|
641
|
+
"maxLength": 500,
|
|
642
|
+
"description": "js_equals: one read-only JavaScript expression evaluated in the page (document.querySelectorAll('.item').length, localStorage.getItem('cart'), window.dataLayer?.length). No assignments, no ++/--, only reading calls; 2 s timeout."
|
|
643
|
+
},
|
|
562
644
|
"value": {
|
|
563
645
|
"type": [
|
|
564
646
|
"string",
|
|
565
647
|
"number",
|
|
566
|
-
"boolean"
|
|
648
|
+
"boolean",
|
|
649
|
+
"null"
|
|
567
650
|
],
|
|
568
|
-
"description": "attr_equals: the attribute value; http_status: the exact status (e.g. 404). Without value, http_status accepts min..max (default 200..299)."
|
|
651
|
+
"description": "attr_equals: the attribute value; text_equals: the whole text (whitespace collapsed); url_equals: a path (/thanks, query included unless ignore_query) or a whole URL; style_equals: the expected CSS value (colours in any CSS form, lengths in px or rem); js_equals: the expected result, or the number to compare with op; http_status: the exact status (e.g. 404). Without value, http_status accepts min..max (default 200..299)."
|
|
569
652
|
},
|
|
570
653
|
"min": {
|
|
571
654
|
"type": "integer",
|
|
@@ -591,7 +674,15 @@
|
|
|
591
674
|
"pattern": "^[imsu]{0,4}$",
|
|
592
675
|
"description": "Regex flags for pattern: i (ignore case), m, s. Page text is innerText, so CSS text-transform (uppercase buttons) applies."
|
|
593
676
|
}
|
|
594
|
-
}
|
|
677
|
+
},
|
|
678
|
+
"allOf": [
|
|
679
|
+
{"if": {"properties": {"assert": {"enum": ["style_equals", "style_matches"]}}, "required": ["assert"]}, "then": {"required": ["selector", "property"]}},
|
|
680
|
+
{"if": {"properties": {"assert": {"const": "style_equals"}}, "required": ["assert"]}, "then": {"anyOf": [{"required": ["value"]}, {"required": ["token"]}]}},
|
|
681
|
+
{"if": {"properties": {"assert": {"enum": ["style_matches", "url_matches", "attr_matches"]}}, "required": ["assert"]}, "then": {"required": ["pattern"]}},
|
|
682
|
+
{"if": {"properties": {"assert": {"const": "attr_matches"}}, "required": ["assert"]}, "then": {"required": ["selector", "attr"]}},
|
|
683
|
+
{"if": {"properties": {"assert": {"enum": ["url_equals", "text_equals"]}}, "required": ["assert"]}, "then": {"required": ["value"]}},
|
|
684
|
+
{"if": {"properties": {"assert": {"const": "js_equals"}}, "required": ["assert"]}, "then": {"required": ["expression"]}}
|
|
685
|
+
]
|
|
595
686
|
},
|
|
596
687
|
"semantic": {
|
|
597
688
|
"type": "object",
|
package/skills/kaloko/SKILL.md
CHANGED
|
@@ -5,7 +5,9 @@ description: >
|
|
|
5
5
|
evidence: writing or updating ACCEPTANCE.md or qa/scenario.yml, running a
|
|
6
6
|
scenario on local, review, staging or production (readonly), capturing
|
|
7
7
|
screens and e-mails, evaluating criteria, or building and sharing the
|
|
8
|
-
Kaloko canvas, for web pages
|
|
8
|
+
Kaloko canvas, for web pages in Chromium, WebKit and Firefox, on phone,
|
|
9
|
+
tablet and desktop presets, in light and dark mode, as well as Android, iOS,
|
|
10
|
+
macOS and Windows apps,
|
|
9
11
|
including checking many pages of one template at once (magazine articles,
|
|
10
12
|
product pages, an SEO page catalogue). Also when someone asks what a
|
|
11
13
|
reviewer rejected on a canvas, what is waiting in the Kaloko inbox, or how
|
|
@@ -50,7 +52,8 @@ Start the shared browser, walk a scenario on a non-production environment,
|
|
|
50
52
|
create test data there (users, roles, carts), capture screens and e-mails,
|
|
51
53
|
evaluate, build the preview, and fix scenario mechanics (selectors, scripts)
|
|
52
54
|
that do not change what is being accepted, including applying a selector
|
|
53
|
-
proposal with `kaloko walk --heal` when it names the same element
|
|
55
|
+
proposal with `kaloko walk --heal` when it names the same element, and pulling
|
|
56
|
+
the regions people drew to ignore into the scenario (`kaloko stability --pull`). Walk only
|
|
54
57
|
the steps a change reaches (`--affected`) and import screenshots into a run. Record steps locally
|
|
55
58
|
(`walk --record`) when a screenshot cannot show what a person has to judge, read traces, and
|
|
56
59
|
export a walkthrough video into the run folder. On a design environment: change
|
|
@@ -71,7 +74,8 @@ a whole project (`capture.record` in the config: storage), a real clean-up
|
|
|
71
74
|
anything the environment profile blocks. Never trash an accepted run, a
|
|
72
75
|
baseline, a run with review comments, or a run linked from an open PR or issue
|
|
73
76
|
comment. Never approve a
|
|
74
|
-
step or accept a run on a person's behalf
|
|
77
|
+
step or accept a run on a person's behalf, and never hand out a signed
|
|
78
|
+
acceptance record (`kaloko signoff`) for a run a person has not accepted; report what waits for them
|
|
75
79
|
(`kaloko review`, MCP `get_review_queue`) instead. Ask the person
|
|
76
80
|
for secret values (passwords, tokens, gate credentials) and for sign-ins only
|
|
77
81
|
they can do; never write secrets into the config, a scenario or a script.
|
|
@@ -82,9 +86,13 @@ they can do; never write secrets into the config, a scenario or a script.
|
|
|
82
86
|
the sample run, the first flow from a URL or a task, share, point the person
|
|
83
87
|
to the canvas): [reference/first-run.md](reference/first-run.md)
|
|
84
88
|
- Writing or updating the plan and scenario: [reference/acceptance.md](reference/acceptance.md)
|
|
85
|
-
- Android, iOS (Simulator), Electron and
|
|
89
|
+
- Android, iOS (Simulator or a real iPhone), Electron, macOS and Windows apps,
|
|
90
|
+
several devices of one app, WebViews inside apps, or screens captured by
|
|
86
91
|
another tool (Maestro, Appium, XCUITest) with their UI tree:
|
|
87
92
|
[reference/apps.md](reference/apps.md)
|
|
93
|
+
- Walking in WebKit and Firefox, on device presets (phones, foldables,
|
|
94
|
+
tablets, TV), in dark mode, with reduced motion or as an installed PWA:
|
|
95
|
+
[reference/devices.md](reference/devices.md)
|
|
88
96
|
- Many pages of one template in one step (articles, products, an SEO
|
|
89
97
|
catalogue from a sitemap or a listing), with an index on the canvas:
|
|
90
98
|
[reference/stacks.md](reference/stacks.md)
|
|
@@ -182,7 +182,8 @@ returning visitor). The plan is the source of truth; this is only the default.
|
|
|
182
182
|
Each criterion gets exactly one check:
|
|
183
183
|
|
|
184
184
|
- `deterministic` for anything countable or exact: presence, counts, text,
|
|
185
|
-
prices, attributes,
|
|
185
|
+
prices, attributes, the URL, computed styles, values in the page,
|
|
186
|
+
overflow, console errors, title/meta/canonical.
|
|
186
187
|
- `semantic` for judgement: clarity, separation of consents, whether copy
|
|
187
188
|
fits the purpose. Write the question in English, yes = met, one condition
|
|
188
189
|
per question. Add `true`/`false` for edge cases and `scope` (CSS selector)
|
|
@@ -216,6 +217,74 @@ button styled uppercase reads `VYTVOŘIT`. Add `flags: i` to ignore case
|
|
|
216
217
|
white-label check for `vyvolej` also matches the Czech verb "Vyvolej si";
|
|
217
218
|
anchor it (`vyvolej\.to`).
|
|
218
219
|
|
|
220
|
+
`text_equals` compares the whole text of an element (whitespace collapsed,
|
|
221
|
+
`flags: i` ignores case) with `value`: a total, a status, a label.
|
|
222
|
+
`attr_matches` runs `pattern` on an attribute (`selector`, `attr`). Both pass
|
|
223
|
+
when one match passes; `all: true` asks every match.
|
|
224
|
+
|
|
225
|
+
### URL, computed styles and values in the page
|
|
226
|
+
|
|
227
|
+
Never make a step script compute a check and leave a `data-qa-*` marker for
|
|
228
|
+
a criterion to find. The marker proves only that the script ran. Check the
|
|
229
|
+
URL, the style or the value directly:
|
|
230
|
+
|
|
231
|
+
```yaml
|
|
232
|
+
# the URL after the step (part: href (default), path, query, hash, host, origin)
|
|
233
|
+
- { id: AC1, text: Lands on the thank-you page, check: { type: deterministic, assert: url_matches, part: path, pattern: '^/(cs|en)/thanks$' } }
|
|
234
|
+
# exact path; ignore_query drops ?… and #…; a value with https:// compares the whole URL
|
|
235
|
+
- { id: AC2, text: Stays on the cart, check: { type: deterministic, assert: url_equals, value: /cart, ignore_query: true } }
|
|
236
|
+
# computed CSS (getComputedStyle) of the matched elements
|
|
237
|
+
- { id: AC3, text: Primary button in brand colour, check: { type: deterministic, assert: style_equals, selector: button.primary, property: background-color, token: color.brand.primary, all: true } }
|
|
238
|
+
- { id: AC4, text: Body text at least 16 px, check: { type: deterministic, assert: style_equals, selector: main p, property: font-size, op: '>=', value: 16px, all: true } }
|
|
239
|
+
- { id: AC5, text: Error text red, check: { type: deterministic, assert: style_equals, selector: .field-error, property: color, value: '#d92d20' } }
|
|
240
|
+
- { id: AC6, text: Promo banner hidden, check: { type: deterministic, assert: style_matches, selector: .promo, property: display, pattern: '^none$' } }
|
|
241
|
+
# a read-only expression in the page
|
|
242
|
+
- { id: AC7, text: Three items in the cart, check: { type: deterministic, assert: js_equals, expression: "document.querySelectorAll('.cart-item').length", value: 3 } }
|
|
243
|
+
- { id: AC8, text: Cart kept for the next visit, check: { type: deterministic, assert: js_equals, expression: "JSON.parse(localStorage.getItem('cart') || '[]').length", op: '>=', value: 1 } }
|
|
244
|
+
- { id: AC9, text: Purchase sent to analytics, check: { type: deterministic, assert: js_equals, expression: "window.dataLayer?.some((e) => e.event === 'purchase')", value: true } }
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
- `url_matches` and `url_equals` read the URL the browser shows after the
|
|
248
|
+
step, redirects included. The evidence names the URL.
|
|
249
|
+
- `style_equals` compares `property` (a longhand: `background-color`, not
|
|
250
|
+
`background`) with `value` or `token`. Colours compare in one form, so
|
|
251
|
+
`red`, `#ff0000` and `rgb(255, 0, 0)` are equal; transparency counts.
|
|
252
|
+
Lengths compare in px (`rem` uses the page's root size), with
|
|
253
|
+
`tolerance: 1` for rounding; `op: '<' | '<=' | '>' | '>=' | '=='` compares
|
|
254
|
+
numbers. Other values (`display`, `font-weight`, `font-family`) compare as
|
|
255
|
+
text without quotes or case. Visible matches count; when none is visible
|
|
256
|
+
(a check for `display: none`), every match does. `style_matches` runs
|
|
257
|
+
`pattern` on the computed value.
|
|
258
|
+
- `token: color.brand.primary` takes the expected value from the project's
|
|
259
|
+
DTCG token set (`kaloko tokens push`, or `tokens.json` of the design
|
|
260
|
+
folder) when `kaloko evaluate` runs, or from the page's
|
|
261
|
+
`--color-brand-primary` custom property during the walk. The evidence
|
|
262
|
+
reads `color is rgb(46, 107, 255) = color.brand.primary (#2e6bff)`.
|
|
263
|
+
- `:hover` and `:focus` states are not supported: the check reads the page
|
|
264
|
+
as the step left it. Hover or focus the element in the step script when
|
|
265
|
+
that state is what you accept, and say so in the criterion text.
|
|
266
|
+
- `js_equals` evaluates one `expression` in the page (2 s limit; promises are
|
|
267
|
+
awaited) and compares the result with `value` (a number value accepts
|
|
268
|
+
`"3"`; `null` matches a missing `localStorage` key), with `op` and a
|
|
269
|
+
number, or with `pattern` (objects as JSON). Without `value` and
|
|
270
|
+
`pattern` the result must be truthy. Only reading is allowed: no
|
|
271
|
+
assignment, no `++`/`--`, no `new` except `new URL(…)`, and calls only
|
|
272
|
+
from a list of reading methods (`querySelector(All)`, `getAttribute`,
|
|
273
|
+
`getItem`, `getComputedStyle`, `filter`, `some`, `JSON.parse`, …).
|
|
274
|
+
`kaloko validate` refuses anything else (`setItem`, `click`, `fetch`).
|
|
275
|
+
The guard is a convention, not a sandbox: a getter in the page can still
|
|
276
|
+
run code, so read data the page exposes (storage, `dataLayer`, the DOM)
|
|
277
|
+
and nothing that acts. Prefer `style_*`, `text_*` and `count` when they
|
|
278
|
+
can say the same.
|
|
279
|
+
|
|
280
|
+
On `kaloko evaluate --recheck` the URL checks run again from the stored URL.
|
|
281
|
+
The computed styles and expression results are stored with the capture, so a
|
|
282
|
+
recheck decides again from them when the criterion still reads the same
|
|
283
|
+
selector and property (or the same expression), for example after you fix
|
|
284
|
+
the expected value. A new selector, property or expression needs the live
|
|
285
|
+
page: the criterion is "not evaluated" with that reason until the step is
|
|
286
|
+
captured again.
|
|
287
|
+
|
|
219
288
|
Two checks need the network and run next to the capture, once per page:
|
|
220
289
|
|
|
221
290
|
- `links_ok`: every same-origin link (`selector`, default `a[href]`) answers
|
|
@@ -273,5 +342,7 @@ design pixel check (`AC801`) leave them out:
|
|
|
273
342
|
|
|
274
343
|
Do not guess them: `kaloko walk --stability 3` notes what moves while the
|
|
275
344
|
page sits still, and `kaloko stability` compares the step with earlier runs
|
|
276
|
-
and writes suggestions (see [execution.md](execution.md)).
|
|
277
|
-
a region
|
|
345
|
+
and writes suggestions (see [execution.md](execution.md)). Reviewers can also
|
|
346
|
+
draw a region on a screenshot on kaloko.app; `kaloko stability --pull` writes
|
|
347
|
+
those into `ignore:`. Ignore only noise; a region that matters to a criterion
|
|
348
|
+
stays in.
|