kaloko 0.3.0

package/skills/kaloko/reference/execution.md

@@ -0,0 +1,75 @@
+# Execution
+
+```bash
+kaloko start --scenario docs/tasks/TASK-1/qa/scenario.yml --env local [--locales cs,en]
+kaloko browser start            # shared Chromium with CDP for the run's environment (its policy, own profile and port)
+kaloko walk                     # runs steps with strategy playwright + script
+kaloko open --step <id> --locale cs
+kaloko capture --step <id> --locale cs          # all viewports of the run
+kaloko capture --step <id> --crop ".selector" --title "What this shows"
+kaloko mail --step <id> --locale cs             # e-mail step from the environment's mailbox
+kaloko mail --step <id> --file message.eml      # e-mail you obtained yourself (.eml, .html, .json, .txt)
+kaloko auth save --env staging --account admin  # a person signs in once (SSO, MFA); walk reuses it
+kaloko session reset            # new visitor: clear cookies and storage in the shared browser
+kaloko status
+```
+
+Commands act on the current run (`CURRENT` in the output directory). When
+other agents or people may start runs in the same repository, pin yours:
+`--run <id>` on every command or `KALOKO_RUN=<id>` in the environment.
+
+`walk` resets the session before the first step of a run; agent-driven steps
+call `kaloko session reset` themselves when the plan starts as a new visitor.
+
+`start` prints a unique test e-mail address for the run; use it in forms so
+`mail` can find the message.
+
+## Choosing a strategy per step
+
+- `playwright` — stable, repeatable steps. The script exports
+  `default async ({ page, url, locale, capture, crop, acceptDialogs, env, run, session, node })`
+  (`node` is `{ id, title, kind }` of the step being walked, so one script can serve several steps);
+  `walk` captures automatically after the script unless it called
+  `capture()`. With `sessions` in the scenario, `page` and `context` belong
+  to the node's person (`session.key`, `title`, `mailbox`, `account`, and
+  `credentials` for that account in the run's environment; see
+  [environments.md](environments.md#accounts-and-sign-in)).
+- `agent` — steps that need judgement or change often. Follow
+  `execution.instructions`, then `kaloko capture --step <id>`.
+- `manual` — a person does it; you capture afterwards.
+
+Prefer turning an agent step into a playwright script once it has worked
+twice the same way.
+
+## Driving the shared browser with your capabilities
+
+- Browser tool over MCP (e.g. Playwright MCP): connect to the CDP endpoint
+  printed by `kaloko browser start`, act, then capture.
+- Computer use: run `kaloko browser start --headed` and operate the window.
+- Shell only: write a short Playwright script (`chromium.connectOverCDP`) or
+  a scenario step script and use `walk`.
+
+There is one shared browser per environment (`browser.port` + the
+environment's position in the config): a production run never shares a tab
+or a cookie jar with local work, and the CLI refuses a readonly run in a
+browser without the readonly policy. `walk` works in its own tab; agents and
+people keep theirs. Several agents on the same environment share cookies, so
+run one scenario at a time per environment: `walk` takes a lock on the shared
+browser and refuses a second run while one is walking. To walk several runs
+in parallel, give each its own browser with `kaloko walk --ephemeral` (own
+cookies and tabs; sign-in through saved account sessions still works).
+Commands leave the shared browser running: they only disconnect from it.
+
+Native dialogs (`confirm()` from Rails `data-confirm` / `data-turbo-confirm`,
+`alert`, `prompt`) are unreliable on the shared browser when more than one
+Playwright client is connected over CDP: a client without a handler can
+dismiss the dialog before yours accepts it, and the action silently does not
+happen. In step scripts call `await acceptDialogs()` before the click (it
+replaces the functions in the page, also after navigations); in your own
+scripts do the same with
+`page.evaluate(() => { window.confirm = () => true })`.
+
+The environment policy and gate apply to every context the CLI creates (the
+shared browser, `walk` sessions, one-off captures): blocked paths and, when
+readonly, non-GET requests outside `allowed_post_paths` fail and are listed as
+blocked requests in the capture; the gate's credentials go to its origins only.

package/skills/kaloko/reference/mcp.md

@@ -0,0 +1,69 @@
+# MCP server
+
+`https://kaloko.app/mcp` speaks MCP over Streamable HTTP (JSON-RPC 2.0,
+POST only, one message per request). Authenticate with an organization API
+token as `Authorization: Bearer qwk_…`; a read-only token is enough, every
+tool reads. Nothing binary crosses MCP: summaries carry the URLs of
+screenshots and files, which the same token can fetch over HTTPS.
+
+## Configure
+
+Claude Code:
+
+```bash
+claude mcp add --transport http kaloko https://kaloko.app/mcp --header "Authorization: Bearer $KALOKO_TOKEN"
+```
+
+Codex / other clients: an HTTP MCP server with URL `https://kaloko.app/mcp`
+and the same header. Keep the token in the environment, not in a committed
+file.
+
+## Tools
+
+| Tool | Use it for |
+|---|---|
+| `list_runs` | published runs, newest first; filters `scenario`, `project`, `verdict`, `author`, `environment`, `branch`, `limit` |
+| `get_run_summary` | one run: statuses per step and locale, criteria that need attention with evidence, screenshot URLs, verdict, baseline, canvas URL, open feedback |
+| `get_feedback` | run verdict with note, approved steps, unresolved rejections and comments, failed manual criteria (`run_id`, or `scenario` for its latest run) |
+| `list_notifications` | the token owner's inbox (`unread_only`, `limit`) |
+| `list_projects` | projects of the organization with counts; the slug is the `project` filter of `list_runs` and the value for `project:` in a scenario |
+| `get_criteria_verdicts` | reviewer verdicts on manual or overridden criteria of a run |
+| `get_step_assets` | documentation layer (Business): newest capture of every step of a scenario with stable image, HTML and metadata URLs (`scenario`, `step`, `locale`, `viewport`, `accepted`) |
+| `list_step_changes` | documentation layer: steps whose capture or status changed since the previous accepted run or `since` (`scenario`, `since`, `accepted`); see [documentation.md](documentation.md) |
+
+## Comment tools
+
+| Tool | Use |
+|---|---|
+| `list_comments` | open threads (or `state: all`) across the organization, filtered by `scenario` or `run_id`, with replies, pins, anchors and who wrote them |
+| `add_comment` | comment on a step (`run_id` + `step`), the whole run (`run_id`) or the whole flow (`scenario`); `parent_id` replies; optional `pin` `{ x, y, viewport }` |
+| `resolve_comment` | close a thread with a `note` that says what changed (it becomes the last reply) |
+| `reopen_comment`, `edit_comment`, `delete_comment` | reopen; edit or delete your own comment |
+
+Writes need a token with scope full. Follow *Processing feedback* in
+[collaboration.md](collaboration.md).
+
+Typical loop for an agent that owns a task: `list_runs` with the scenario →
+`get_feedback` for the latest run → fix what was returned → new run with the
+CLI (`kaloko start … share`) → post the canvas URL in the issue. Capturing,
+evaluating and uploading stay in the CLI; MCP reads, and for design drafts also
+publishes versions, sets references and step status, and comments (below).
+
+Errors come back as tool results with `isError: true` (unknown run, expired
+run, missing argument); an invalid token is HTTP 401.
+
+## Design tools (#42)
+
+The same data and permissions as the REST API; each tool is its REST route.
+
+| Tool | Use |
+|---|---|
+| `get_draft` | which revision each step shows, step status, who changed it (person or agent) |
+| `get_step` | the handoff: live files, signed entry URL, token set, page structure, open comments (`ref`: draft, design, vN, run id) |
+| `get_step_history` | versions, and revisions of a step with authors |
+| `list_step_changes` | steps that differ between two references (default design reference → draft) |
+| `get_references`, `get_tokens` | design reference and baseline; the project's token set |
+| `get_draft_feedback` | draft comments, pinned ones with their element |
+| `publish_version`, `set_reference`, `set_step_status`, `add_draft_comment` | writes: need a token with scope full; ask first where the skill says so |
+
+Files are not uploaded over MCP: the CLI (`kaloko sync`) or the REST blob endpoints carry them.

package/skills/kaloko/reference/stacks.md

@@ -0,0 +1,122 @@
+# Stack steps: many pages of one template
+
+A stack step (`kind: stack`) stands for many pages that share a template: the
+articles of a magazine, product or category pages, an SEO catalogue of
+landing pages, pages after a migration. The walk captures every page with
+the step's criteria and a few index fields; the canvas shows one node drawn
+as a stack. Clicking it opens an index of the pages with search, filters,
+sorting, thumbnails and CSV export, and each page opens in the lightbox.
+
+## When to use it
+
+- One template, many instances with different content, and the question is
+  whether *all of them* hold up. Examples: "every article has one H1, a
+  title of 10–70 chars and a cover image", "no product page returns 404",
+  "every city landing has a description and a canonical".
+- An SEO audit or design of a page catalogue: list the pages, see the gaps
+  per page, and export the failing ones for the content team.
+- A redesign or migration: walk the old URL list against the new site.
+
+Not for a flow of different screens: those stay separate steps. A stack
+step does not click through anything. It opens each page by URL. Use a
+step script only to prepare each page, for example to dismiss a dialog
+(see below).
+
+## Declaring it
+
+```yaml
+- id: articles
+  kind: stack
+  title: Magazine articles
+  purpose: Every article reads well and is findable.
+  packs: [seo]                              # the pack criteria apply to every page
+  items:
+    sitemap: /sitemap.xml                   # or urls / file / crawl / script (exactly one)
+    match: '^/clanky/'                      # regex on the path with its query; exclude: '…' drops pages
+    limit: 800                              # default 500, at most 10000
+    semantic_sample: 20                     # semantic criteria on the first 20 pages only
+    # viewports: [desktop]                  # default: the first viewport of the scenario
+  fields:                                   # index columns (max 12); default title, h1, words, http
+    - { id: title, label: Title, from: title }
+    - { id: words, label: Words, from: words }
+    - { id: http, label: HTTP, from: http }
+    - { id: author, label: Author, from: .author }
+    - { id: cover, label: Cover, from: "article img@src" }
+    - { id: links, label: Links, from: "count:main a" }
+  # capture: { full_page: true }            # default for stacks is the first screen only
+  criteria:
+    - { id: AC1, text: Has a cover image, check: { type: deterministic, assert: exists, selector: "article img" } }
+    - { id: AC2, text: Reads as an article, check: { type: semantic, question: Does the page read as a complete magazine article? } }
+```
+
+Item sources:
+
+- `urls: [/a, /b]` is a short fixed list.
+- `file: qa/data/products.csv` is a list next to the scenario. Accepted
+  formats are text with one path per line, CSV (first column), or JSON:
+  `["/a"]` or `[{ path, title, key }]`, where `key` gives a stable id.
+- `sitemap: /sitemap.xml` also follows a sitemap index (up to 50 sitemaps).
+- `crawl: { from: /clanky, links: "a.card", next: "a.next", pages: 20 }`
+  collects links from a listing and follows its pagination.
+- `script: qa/steps/items.mjs` uses the default export
+  `({ page, env, locale, url }) => [paths | { path, title, key }]` for
+  anything else, such as an API or a search.
+
+Pages are deduplicated. External URLs are allowed (e.g. an old domain).
+`{locale}` in any source string is replaced by the walked locale. This is
+how one sitemap with every language is split:
+`match: '^/guides/.*lang={locale}'`, or `sitemap: /{locale}/sitemap.xml`.
+
+Field sources: `title`, `h1`, `description`, `canonical`, `lang`, `url`,
+`path`, `words` (words in main/article/body), `http` (the navigation
+status), `count:<selector>`, `<selector>` (text) and `<selector>@<attr>`.
+
+A step `execution.script` still works. It is called with `item` in the
+context, opens `item.url` itself and may call `capture()`. Without a script,
+the walk opens each page and captures it.
+
+## Cost and size: choose deliberately
+
+- Deterministic criteria and packs run on every page. They are cheap.
+- Semantic criteria go to the evaluator. On a stack they run only on the
+  first `semantic_sample` pages (default 20). The other pages show the
+  criterion as not evaluated. Raise the sample only when the question
+  really differs between pages, and tell the person what it costs
+  (one call per page and viewport).
+- Stacks capture the first screen of each page (fast, small). Set
+  `capture: { full_page: true }` when the criteria are about the whole page.
+- Walking takes about 1–2 s per page. For a first look, use
+  `kaloko walk --limit 50`. Re-walk chosen pages with
+  `--items clanek-7,clanek-12` (matches the id or the path).
+- A shared run holds at most 20000 files and 300 MB. Each page stores about
+  four files, so a stack of a few thousand first-screen pages fits. Split
+  larger catalogues by section (`match`), one stack per section.
+
+## Reading the results
+
+- `kaloko build` writes `steps/<step>/items.json` (`qawalk.stack.v1`)
+  with every page's fields, status, failing criteria and evidence. The
+  preview inlines it, so the local `index.html` works offline.
+- The node shows the page count, a pass/fail bar and a few thumbnails, with
+  failing pages first. Each criterion carries counts (for example "3/31
+  pages" in the report).
+- In the index:
+  - status chips, a criterion filter, search over path, title and fields,
+    sort by any field, list or grid view;
+  - keyboard: `/` searches, arrows move, Enter opens, Esc closes;
+  - "Download CSV" exports the filtered rows;
+  - a page link (`#step=<stack>~<page>`) opens that page directly.
+- Over MCP: `get_run_summary` includes the stack summary.
+  `get_stack_items {run_id, step, status?, criterion?, query?, offset, limit}`
+  lists pages with fields, failing criteria, evidence and screenshot URLs.
+  This is the way to hand an agent "fix these 37 articles".
+
+## Report
+
+Say how many pages were walked, how many fail and on which criteria (with
+two or three example paths), whether the limit cut the list, and that
+semantic criteria covered only the sample. Put the full list in the CSV or
+the MCP result rather than in the issue.
+
+Not yet supported for stacks: comparison with a baseline run per page and
+the documentation layer (latest captures per step).