kaloko 0.12.1 → 0.13.0

package/README.md

@@ -1,6 +1,7 @@
 # Kaloko
 
-See what your agent built, on one canvas, and sign it off.
+Your AI agents build; Kaloko walks every screen, puts it on one canvas and
+records who approved what.
 
 Your coding agent (Claude Code, Codex or any LLM with a shell) walks your
 acceptance plan with the `kaloko` skill. Kaloko captures every screen and
@@ -169,6 +170,9 @@ messages. Click a row and the video jumps there. Web recordings need nothing
 but Chrome (WebM with VP9); with ffmpeg installed you also get MP4 for Safari,
 app recordings (Android `screenrecord`, iOS Simulator, macOS) and the
 walkthrough export. `npx kaloko video tools` says what this machine has.
+Videos play on: when one ends, the next step's video starts, and replay moves
+to the next step by itself; a step that needs a decision waits for you, and a
+step without video shows its screenshot for three seconds.
 Recordings stay in the local preview on Free and Team; Business and Enterprise
 share them with the run.
 
@@ -219,10 +223,12 @@ Every AI verdict comes with one sentence saying why. Criteria marked
 `vision: true`, and answers the text evaluator is unsure about, get a second
 look at the screenshot from a vision model (`ANTHROPIC_API_KEY`, or through
 Kaloko); `kaloko build` lists what to act on with the score and the reason.
-As an experiment you can switch on in Settings, Clef on Cloudflare also names
-the kind of problem each screen and its recording show (overlap, cut-off text,
-a raw translation key, an endless spinner); reviewers turn the right findings
-into comments with one click, and findings never change a verdict.
+As an experiment an admin switches on under Settings → Security → Experiments,
+Clef on Cloudflare also names the kind of problem each screen and its recording
+show (overlap, cut-off text, a raw translation key, an endless spinner);
+reviewers turn the right findings into comments with one click, and findings
+never change a verdict. Triage counts against the hosted-evaluation allowance,
+so it needs the Team plan or higher.
 The `a11y` pack walks the page with Tab (order, visible focus, traps, skip
 link) and reads names from the accessibility tree. Pixel diffs skip the
 regions a step ignores; `kaloko walk --stability 3` and `kaloko stability`

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "kaloko",
-  "version": "0.12.1",
-  "description": "See what your AI agents built: every screen, language and device walked, assessed and approved on one canvas.",
+  "version": "0.13.0",
+  "description": "Your AI agents build; Kaloko walks every screen, puts it on one canvas and records who approved what.",
   "keywords": [
     "qa",
     "acceptance-testing",

package/skills/kaloko/SKILL.md

@@ -20,7 +20,10 @@ description: >
   trace read (actions, requests, console), and when a user-visible change
   should land in the visual changelog (CHANGELOG.md with step references,
   kaloko changelog draft, kaloko release) or a guide should show a step or
-  be written, translated or checked (kaloko docs check).
+  be written, translated or checked (kaloko docs check), when screenshots,
+  HTML or recordings must hide personal data (capture masks, pii: redact or
+  pseudonymize, kaloko share stopping on unmasked data), and when a run has
+  Clef triage findings (experimental) to read, mention or answer.
 ---
 
 # Kaloko
@@ -128,6 +131,12 @@ only when a person says that value may be uploaded.
   (`--heal`): [reference/execution.md](reference/execution.md)
 - Evaluation, JEV thresholds, the vision evaluator and its reasons, reading
   results: [reference/evaluation.md](reference/evaluation.md)
+- Clef triage findings (experimental): what they are, how to report them apart
+  from the verdicts, `get_findings` and `answer_finding`:
+  [reference/evaluation.md#triage-findings-experimental](reference/evaluation.md#triage-findings-experimental)
+- Hiding personal data in captures (`capture.mask`, `pii: redact`,
+  `pseudonymize`, `pii: required` on an environment, the check before sharing):
+  [reference/acceptance.md#capture-quality](reference/acceptance.md#capture-quality)
 - Screenshots, walkthroughs, Playwright reports and traces that an agent or a
   tool already produced, laid out as a run: [reference/import.md](reference/import.md)
 - Video of a step or a whole flow, traces (actions, requests, console) and the

package/skills/kaloko/reference/ci.md

@@ -65,6 +65,14 @@ in the newest shared run of the scenario count too, so a fresh CI checkout
 maps routes as well as a laptop with earlier runs. See
 [execution.md](execution.md#only-what-changed).
 
+## Exit codes
+
+Every command exits with `0` when it is done, `1` when checks found problems
+or the command failed, and `2` on a usage error such as an unknown command or
+option or a missing argument. A CI step can tell a broken workflow line (2)
+from a failing check (1). `kaloko help <command>` or `kaloko <command> --help`
+shows the options of one command.
+
 ## Done means
 
 The workflow file and the config change are committed after the person agreed,

package/skills/kaloko/reference/documentation.md

@@ -47,8 +47,10 @@ scenarios.
 Settings → Integrations → Webhooks (admin): an https URL and the events
 `run.accepted` and/or `run.shared`. Each call is a JSON POST
 `{ id, type, created_at, org, data: { run, steps, steps_url, changes_url } }`
-with `QAwalk-Signature: t=<unix time>,v1=<hex HMAC-SHA256 of "<t>.<body>">`
-signed with the webhook's secret (copy it from Settings). Verify the
+with `Kaloko-Signature: t=<unix time>,v1=<hex HMAC-SHA256 of "<t>.<body>">`
+signed with the webhook's secret (copy it from Settings). The old
+`QAwalk-Signature` header is still sent with the same value, so existing
+receivers keep working; read `Kaloko-Signature` in new code. Verify the
 signature and the time (reject older than 5 minutes), answer 2xx within 5
 seconds, and do the work asynchronously. Also over the API:
 `GET|POST /api/v1/orgs/<org>/webhooks`, `DELETE …/webhooks/<id>`.

package/skills/kaloko/reference/mcp.md

@@ -2,8 +2,9 @@
 
 `https://kaloko.app/mcp` speaks MCP over Streamable HTTP (JSON-RPC 2.0,
 POST only, one message per request). Authenticate with an organization API
-token as `Authorization: Bearer qwk_…`; a read-only token is enough, every
-tool reads. Nothing binary crosses MCP: summaries carry the URLs of
+token as `Authorization: Bearer qwk_…`. The server is part of the Team plan
+and up. A read-only token is enough to read; tools that write need a token
+with that scope, as noted below. Nothing binary crosses MCP: summaries carry the URLs of
 screenshots and files, which the same token can fetch over HTTPS.
 
 ## Configure
@@ -28,6 +29,7 @@ file.
 | `list_notifications` | the token owner's inbox (`unread_only`, `limit`) |
 | `list_projects` | projects of the organization with counts; the slug is the `project` filter of `list_runs` and the value for `project:` in a scenario |
 | `get_criteria_verdicts` | reviewer verdicts on manual or overridden criteria of a run |
+| `get_stack_items` | the pages of a stack step (`run_id`, `step`; filters `status`, `criterion`, text): path, URL, title, status and failing criteria with evidence, screenshot URLs |
 | `get_step_recording` | a step's recording and trace (`run_id`, `step`, `locale`, `viewport`, `trace: true` for the full trace): video URLs, duration, its chapter of a flow recording, failed attempts, the trace summary (failed requests, console errors, slowest request); see [recording.md](recording.md). `get_run_summary` lists trace summaries per step too |
 | `get_review_queue` | what a person still has to decide in a run (`run_id`, or `scenario` / nothing for the newest run): steps in review order with reasons, decisions, approvals carried over unchanged steps with their origin, counts and `review_url` to give the person |
 | `decide_step` | records your (the agent's) decision on a step: `approve`, or `return` with a `note`; marked as an agent's and not counted as a person's. Never use it to approve for a person |
@@ -37,6 +39,8 @@ file.
 | `get_changelog` | the visual changelog of a product (`product`, `version`, `resolve: true` for the pictures, `limit`): releases newest first with the entry as written and its groups; see [docs.md](docs.md) |
 | `draft_changelog_entry` | what accepted runs since the last release (or `since`) show differently, as a Keep a Changelog draft with before/after step references (`product`, `since`) |
 | `resolve_step_image` | one step reference (`ref` = `kaloko:<scenario>/<step>?…`, `product`, `version`, `locale`, `viewport`) as a signed image URL with its run, how the version was found and canvas links |
+| `get_docs` | a product's published docs (`product`; without `path` the versions, languages and sidebar; with `path` one page with its Markdown, sections, stale sections and, with `resolve: true`, its pictures) |
+| `docs_check` | what needs attention in the published docs of a version: broken step references, references without an accepted picture, stale sections, missing or outdated translations, release items without a picture |
 | `list_step_changes` | documentation layer: steps whose capture or status changed since the previous accepted run or `since` (`scenario`, `since`, `accepted`); see [documentation.md](documentation.md) |
 | `get_started` | where the organization stands (`empty`, `sample`, `active`), its sample run, the next steps and a prompt; call it in a new project, see [first-run.md](first-run.md) |
 | `add_sample_run` | adds the sample run (or returns the one there is); needs an upload or full token and the Developer / Tester role |
@@ -51,6 +55,8 @@ file.
 | `add_comment` | comment on a step (`run_id` + `step`), the whole run (`run_id`) or the whole flow (`scenario`); `parent_id` replies; optional `pin` `{ x, y, viewport }` or `shape` (box or arrow in fractions, with `viewport` and `size`) |
 | `list_ignore_regions` | regions people drew to leave out of pixel diffs, per step, with the `ignore:` entry each becomes (`kaloko stability --pull` writes them) |
 | `add_ignore_region` | suggest such a region (`run_id`, `step`, `viewport`, `rect` in CSS px); Developer / Tester, scope upload or full |
+| `get_findings` | experimental Clef triage findings of a run (`run_id`): the kind of problem on each screen or in recording frames, with probability, severity and box, and the answers testers gave. Findings never change a verdict |
+| `answer_finding` | answers a finding (`run_id`, `node`, `locale`, `viewport`, `finding`, `kind`, `answer`: right, wrong or comment with the `comment_id` from `add_comment`); scope comment |
 | `get_signoff_record` | the signed acceptance record of an accepted run (Business): approvals with provenance, criteria, capture hashes, signature |
 | `resolve_comment` | close a thread with a `note` that says what changed (it becomes the last reply) |
 | `reopen_comment`, `edit_comment`, `delete_comment` | reopen; edit or delete your own comment |
@@ -58,14 +64,16 @@ file.
 | `list_issues` | issues created from steps with key, URL and state (`run_id`, `scenario`, `state`) |
 | `list_schedules`, `run_schedule` | hosted runs: schedules of read-only scenarios on Kaloko's browsers with the month's page budget; start one now (`id`) |
 
-Writes need a token with scope full. Follow *Processing feedback* in
+Writes here need a token with the `comment` scope (or full); `add_ignore_region`
+and `run_schedule` need upload or full. Follow *Processing feedback* in
 [collaboration.md](collaboration.md).
 
 Typical loop for an agent that owns a task: `list_runs` with the scenario →
 `get_feedback` for the latest run → fix what was returned → new run with the
 CLI (`kaloko start … share`) → post the canvas URL in the issue. Capturing,
-evaluating and uploading stay in the CLI; MCP reads, and for design drafts also
-publishes versions, sets references and step status, and comments (below).
+evaluating and uploading stay in the CLI; MCP reads, comments, records
+decisions and answers to findings, creates issues, and for design drafts also
+publishes versions and sets references and step status (below).
 
 Errors come back as tool results with `isError: true` (unknown run, expired
 run, missing argument); an invalid token is HTTP 401.

package/skills/kaloko/reference/recording.md

@@ -72,8 +72,10 @@ missing with the install command for the OS.
 The step detail gets Video next to Screenshot: the poster first, nothing loads
 until played, markers for the start, the capture, actions and failed requests.
 Technical shows the trace; a click on a row moves the video there. Cards play
-the preview on hover. Replay plays the flow recording with its chapters. The
-walkthrough video is in the top bar.
+the preview on hover. Videos play on: after one ends, the next step's video
+starts by itself, and replay with video moves to the next step on its own; a
+step that needs a decision waits, a step without video shows its screenshot for
+three seconds. The walkthrough video is in the top bar.
 
 ## Plans and sharing