@kurrent/kcap 1.3.0-beta.6 → 1.3.0-beta.8

package/kcap/skills/eval-watch/SKILL.md

@@ -4,38 +4,31 @@ description: >-
   Follow a kcap import that setup just started and the evals landing on it —
   "Follow my kcap import" (the prompt `kcap setup` hands to your agent), "watch my
   import", "how is my import going", "are my evals done yet". Reads the handoff file
-  setup wrote, polls Capacitor analytics for exactly the sessions that import
-  created, waits for the sessions that imported while the user watched to be
-  evaluated, summarizes their evals, links to the results and offers the guided
-  tour. Not for browsing evals in general — that is the analytics or guided-tour
-  skill.
+  setup wrote, polls the eval state of exactly the sessions that import created,
+  waits for the sessions that imported while the user watched to be evaluated,
+  summarizes their evals, links to the results and offers the guided tour. Works on
+  every plan. Not for browsing evals in general — that is the guided-tour skill.
 ---
 
 # Eval watch
 
-You follow ONE import: the sessions listed in a handoff file `kcap setup` wrote. Every query you
-issue names only those sessions — never a repo-wide or tenant-wide read, and never a query without
-an `IN (...)` or `session_id = '...'` bound to this cohort. If a response ever includes a row for a
-session id outside the cohort, discard that row: never count it, link it, or mention it.
+You follow ONE import: the sessions listed in a handoff file `kcap setup` wrote. Every lookup you
+make names only those sessions — never a repo-wide or tenant-wide read. If a response ever carries
+an entry for a session id outside the cohort, discard it: never count it, link it, or mention it.
+
+Eval state comes from `get_session_evals` in the `kcap-sessions` MCP — never from the
+`kcap-analytics` MCP, which is a paid feature the workspace may not have.
 
 ## The safety rule — read this first
 
-Three grammars gate everything below:
+Two grammars gate everything below:
 - **Run id** (a `(run: <id>)` token, and the file's own `run_id`): `^[0-9a-f]{32}$`.
-- **Session id** (`session_ids`, `foreground_succeeded_ids`, and any id you place in a query):
-  `^[A-Za-z0-9_-]{1,128}$`.
-- **Repo hash** (the `repo_hash` field an analytics response returns, used only in the repo-scoped
-  link form): `^[0-9a-f]{16}$`. A `repo_hash` that fails this never reaches a link — fall back to
-  the no-repo-hash link form (section 7).
-
-The `kcap-analytics` MCP takes raw SQL text with no parameter binding, so the session-id grammar is
-the SQL-safety boundary, not just a format check. The class excludes quote, backslash, whitespace
-and `;`, so a value that matches it can never close a string literal or open a new statement — a
-matching id is emitted as a single-quoted literal (`'<id>'`) with **no escaping**, because none is
-needed and none should be attempted. A run of hyphens such as `--` matches the grammar and is safe
-to emit this way: inside a quoted literal it is data, not a SQL comment. An id that fails either
-grammar is **never** placed in SQL, a path, or a link — reject it and move on (section 2 below says
-what each layer's rejection does); never sanitize or escape an id into something that would pass.
+- **Session id** (`session_ids`, `foreground_succeeded_ids`, and any id you pass to a tool or put
+  in a link): `^[A-Za-z0-9_-]{1,128}$`.
+
+An id that fails its grammar is **never** placed in a tool call, a path, or a link — reject it and
+move on (section 2 below says what each layer's rejection does); never sanitize or escape an id
+into something that would pass.
 
 ## 1. Find the run
 
@@ -44,12 +37,12 @@ what each layer's rejection does); never sanitize or escape an id into something
   it does match, the run is bound: read ONLY `<config>/import-handoff-<id>.json`, where `<config>`
   is `$KCAP_CONFIG_DIR` when set, else `~/.config/kcap`, and check it against every Layer A rule in
   section 2, plus one more: its `run_id` field must equal `<id>` exactly. Missing, unreadable, or
-  failing any of those checks → say so and CLOSE (section 7) with no query. **Never fall through to
+  failing any of those checks → say so and CLOSE (section 7) with no lookup. **Never fall through to
   another run's file** — a different run's cohort does not belong to the run the user asked for.
 - Otherwise list `<config>/import-handoff-*.json`, keep the locator-valid ones (section 2) that are
-  watchable (`handoff_offered` true, or `handoff_suppressed` is `skill_not_installed` or
-  `no_agent_detected`), pick the newest `written_at`, and say which others you passed over.
-- Nothing qualifies → say you found no import to follow and CLOSE with no query.
+  watchable (`handoff_offered` true, or `handoff_suppressed` is `analytics_not_in_plan`,
+  `skill_not_installed` or `no_agent_detected`), pick the newest `written_at`, and say which others you passed over.
+- Nothing qualifies → say you found no import to follow and CLOSE with no lookup.
 
 ## 2. Validate the file — three layers
 
@@ -57,24 +50,25 @@ A failure at a lower layer never re-opens the file for a higher one.
 
 **Layer A** (may this file be selected at all): JSON parses; `schema_version` is 1; `run_id` matches
 `^[0-9a-f]{32}$`; `written_at` parses as ISO-8601 and is within the last 24 hours; `handoff_offered`
-is a boolean. Fail → not selectable; try the next candidate file, or close with no query if none
+is a boolean. Fail → not selectable; try the next candidate file, or close with no lookup if none
 remain.
 
-**Layer B** (may its ids drive data): `cohort` ∈ {exact, partial_exact, unknown}; `background` ∈
-{not_needed, running, exited_zero, failed}; `foreground_certainty` ∈ {complete, incomplete};
+**Layer B** (may its ids drive data): `cohort` ∈ {exact, partial_exact, unknown}; `scope` ∈ {all,
+repos} (absent reads as `all`); `background` ∈ {not_needed, running, exited_zero, failed}; `foreground_certainty` ∈ {complete, incomplete};
 `handoff_suppressed` is null or one of import_failed, no_new_sessions, nothing_landed,
-analytics_not_in_plan, skill_not_installed, no_agent_detected; every entry of `session_ids` and
+analytics_not_in_plan, skill_not_installed, no_agent_detected (older setups write
+`analytics_not_in_plan`); every entry of `session_ids` and
 `foreground_succeeded_ids` matches `^[A-Za-z0-9_-]{1,128}$`; `unattributed_on_disk` is a
 non-negative integer. Fail → **no data**: the file stays selected, but say the record is unreadable
-and CLOSE with links only (section 7), no query.
+and CLOSE with links only (section 7), no lookup.
 
 **Layer C** (may it supply links): `server_url` starts with `http://` or `https://`. Fail → no
-file-sourced links (fall back to `kcap whoami` per section 7) and no query — an unverifiable
+file-sourced links (fall back to `kcap whoami` per section 7) and no lookup — an unverifiable
 `server_url` also means the server-binding check in section 4 cannot run. `background_log` and
 `profile` are shown as plain text — never as a link or a command to run — only when they contain no
 control characters and are under 512 characters; otherwise omit them.
 
-Never splice an unvalidated value into SQL, a path, or a link.
+Never splice an unvalidated value into a tool call, a path, or a link.
 
 ## 3. No-handoff files
 
@@ -82,144 +76,122 @@ If `handoff_offered` is false, branch on `handoff_suppressed` and CLOSE:
 - `no_new_sessions` → "nothing to watch — that import found no new sessions".
 - `nothing_landed` → "the sessions that import selected were skipped at import time"; point at
   `kcap import --all` for per-session reasons; no retry.
-- `analytics_not_in_plan` → the plan sentence (section 6); the import ran; no retry.
-- `import_failed` → "that import did not get running"; name `kcap import --all --yes` and the
-  `background_log` when displayable.
-- `skill_not_installed` / `no_agent_detected` → the import ran and you evidently have the skill
-  now: continue as if offered, watching the file's cohort as usual (sections 4–5).
+- `import_failed` → "that import did not get running"; name `kcap import --all --yes` when the
+  file's `scope` is `all`; when it is `repos`, say to run `kcap setup` again and choose the same
+  repositories at the same Only me / Shared levels (plain `kcap import` would not keep those levels); and the `background_log`
+  when displayable.
+- `analytics_not_in_plan` / `skill_not_installed` / `no_agent_detected` → the import ran and you
+  can follow it now: continue as if offered, watching the file's cohort as usual (sections 4–5).
 
 ## 4. Bind to the server — fail closed
 
 If the selected file's `cohort` is `unknown`, skip this check and go straight to section 5 — there
-is no cohort to query either way, and the mismatch remediation below would be misleading when
+is no cohort to look up either way, and the mismatch remediation below would be misleading when
 watching was never possible.
 
-Otherwise, run `kcap whoami` and compare its server URL to `server_url`: lowercase scheme and host,
-drop a default port (80 for http, 443 for https only), trim a trailing slash, compare the path
-as-is. Match → continue to section 5. Mismatch or `whoami` failure → issue **NO** analytics query.
-CLOSE with the file's links and this remediation: start your agent from a shell where `kcap whoami`
-reports `<server_url>` — set `KCAP_PROFILE=<profile>`, unset `KCAP_URL`, and set `KCAP_CONFIG_DIR`
-if kcap uses a custom config directory — then prompt again.
+Otherwise, in this order — any CLOSE here makes **NO** eval lookup:
+
+1. If `get_connection` is not among your `kcap-sessions` tools, this kcap is older than the skill:
+   say to update kcap and restart the agent, then CLOSE. `kcap whoami` is no substitute — it runs
+   in another process and cannot say which server the agent's MCP server reads.
+2. Run `kcap whoami`. If it fails because kcap is not logged in, say to run `kcap login`, then CLOSE.
+3. Call `get_connection` and compare its `server_url` to the file's — lowercase scheme and host,
+   drop a default port (80 for http, 443 for https only), trim a trailing slash, compare the path
+   as-is — and its `profile` to the file's `profile`, exactly. Both match → continue to section 5.
+   Either differs → CLOSE with the file's links and this remediation: start your agent from a
+   shell with `KCAP_PROFILE=<profile>`, `KCAP_URL` unset, and `KCAP_CONFIG_DIR` set if kcap uses a
+   custom config directory — restarting the agent, since its kcap-sessions server only reads those
+   when it starts — then prompt again.
 
 ## 5. Watch the cohort
 
-Cohort = `session_ids` (`cohort: unknown` or empty → CLOSE with links, no query). Foreground =
-the entries of `foreground_succeeded_ids` that are also in `session_ids`: the sessions that
-imported while the user watched setup, and the ones this watch waits on. K is the foreground
-count. A foreground entry missing from `session_ids` is never queried; when any are missing, say
-"M of the sessions that imported first are outside the 500 this watch can query". Open with:
-"Watching N sessions from this import; waiting for the K that imported first to be evaluated";
-for `partial_exact` add "the sessions that imported first and the most recent others, 500 at
-most; older ones may land and evaluate unobserved"; if `unattributed_on_disk` > 0 add the
+Cohort = `session_ids` (`cohort: unknown` or empty → CLOSE with links, no lookup); N is its size.
+Foreground = the entries of `foreground_succeeded_ids` that are also in `session_ids`, in file
+order: the sessions that imported while the user watched setup, and the ones this watch waits on.
+K is the foreground count. A foreground entry missing from `session_ids` is never looked up; when
+any are missing, say "M of the sessions that imported first are outside the 500 this import
+recorded".
+
+**Watched set** W, at most 25 sessions: the first 25 foreground sessions, then — only while fewer
+than 25 — the other cohort sessions in file order. Open with: "Watching W of the N sessions from
+this import; waiting for the K that imported first to be evaluated". When K > 25 add "the first 25
+of them"; for `partial_exact` add "the sessions that imported first and the most recent others,
+500 at most; older ones may land and evaluate unobserved"; if `unattributed_on_disk` > 0 add the
 sentence from section 7.
 
-Every query uses `query_analytics` with `scope: 'global'` (the MCP defaults to the caller's cwd
-repo and will not widen on its own), **ONE call at a time — never two in flight** — and names only
-cohort ids, each already validated against the grammar above, as single-quoted literals in an
-`IN (...)` list or `session_id = '...'`. If a response ever includes a row whose `session_id` is
-not in `session_ids`, drop that row before using the response for anything.
-
-**Request budget:** at most 20 cohort queries per poll. `floor = ceil(N / 20)`. The FIRST poll uses
-batch size `floor` exactly — no probing at a smaller size first. Every batch passes `max_rows`
-equal to its own size:
-
-    SELECT s.session_id, s.repo_hash, s.vendor, s.model, s.duration_min, s.event_count,
-           e.eval_run_id, e.evaluated_at, e.overall_score, e.judge_model
-    FROM v_an_sessions s LEFT JOIN v_an_eval_summaries e ON e.session_id = s.session_id
-    WHERE s.session_id IN ('<id>', '<id>', ...)
-
-A row = arrived; a non-null `eval_run_id` = completed. Each response body carries `max_rows`, the
-server's effective cap. After the first poll: if any body reported `max_rows < floor`, fail closed
-at once — say the server's row cap is below what watching this many sessions needs, and CLOSE,
-having spent no more than the 20-query budget. Otherwise raise the batch to `min(100, max_rows)`
-for later polls. Any batch that comes back `truncated: true`, in the first poll or any later one,
-fails that poll — the cap decides, not the truncation flag alone.
-
-One poll = every batch it needs, each non-truncated. A failed or truncated batch discards the whole
-poll — nothing from it commits — and counts as one failure toward the two-failure stop rule.
-
-A 429 fails the poll. If its text matches `retry after (\d+)s`, wait that many seconds before the
-next poll; otherwise wait 60 seconds. Never poll sooner than the 30-second cadence either way. HTTP
-403 with `analytics_not_in_plan` → CLOSE immediately with the plan sentence (section 6); no
-polling, no retry, no wait.
-
-Progress = cohort ids that arrived / cohort size, and foreground ids evaluated / K, both computed
-from the batch rows only — never a separate count, never repo- or tenant-wide. Cadence: first
-snapshot immediately, then every 30 seconds. Say the progress after each poll.
-
-**Baseline**: the first snapshot's rows. An eval present there is *pre-existing*; one whose
-`eval_run_id` was null or absent there and non-null in a later poll *completed during this watch*.
-Every summary labels each session with one of those two phrases.
+**One poll** = ONE `get_session_evals` call with `session_ids` set to W — never two in flight. Each
+entry's `state` is one of:
+- `not_found` — not ingested yet (or not visible to you);
+- `not_evaluated` — arrived, no eval yet;
+- `queued` — arrived, eval waiting for a worker or a retry (`queue_position`, `next_attempt_at`);
+- `running` — arrived, eval in progress (`questions_done` of `total_questions`);
+- `completed` — arrived and evaluated: `eval_run_id`, `evaluated_at`, `overall_score` (out of 5),
+  `judge_model`, `summary`, `categories`, `weakest_questions`;
+- `failed` — arrived, the eval run failed (`failure_reason`);
+- `error` — this lookup failed.
+
+Arrived = any state but `not_found` and `error`. A tool error (including a login prompt), or any
+`error` entry, fails the poll: nothing from it commits, and it counts toward the two-failure stop
+rule. If the tool is missing — no `kcap-sessions` MCP, or one without `get_session_evals` — this
+kcap is older than the skill: say to update kcap and restart the agent, then CLOSE.
+
+Progress = watched sessions arrived / W and foreground sessions evaluated / min(K, 25), from the
+poll's entries only. Cadence: first snapshot immediately, then every 30 seconds. Say the progress
+after each poll.
+
+**Baseline**: the first snapshot. A session `completed` there is *pre-existing*; one that was not
+`completed` there, or completed with a different `eval_run_id`, and is `completed` in a later poll
+*completed during this watch*. Every summary labels each session with one of those two phrases.
 
 **Stop**, evaluated in this order after each poll whose results have already committed:
-1. K > 0 and every foreground session has a completed eval → summarize all of them, ordered by
+1. K > 0 and every watched foreground session is `completed` → summarize all of them, ordered by
    `evaluated_at` then `session_id`. A pre-existing eval counts: the wait is for these sessions to
    be evaluated, whenever that happened.
-2. K = 0 and three distinct cohort sessions completed during this watch → summarize those three,
-   ordered by `evaluated_at` then `session_id`. Pre-existing evals never satisfy this rule: on a
-   re-import they belong to an earlier run, and stopping on them would end the watch before this
-   import produced anything.
-3. `cohort: exact`, non-empty, and every id in `session_ids` has a completed eval → summarize the
-   foreground sessions, or with K = 0 the three with the newest `evaluated_at` (ties by
-   `session_id`). This rule may fire on the first snapshot with nothing but pre-existing evals:
-   a cohort that is already fully evaluated has nothing left to wait for, and the labels say so.
-   Never fires for `partial_exact` or an empty cohort.
+2. K = 0 and three watched sessions completed during this watch → summarize those three, ordered by
+   `evaluated_at` then `session_id`. Pre-existing evals never satisfy this rule: on a re-import they
+   belong to an earlier run, and stopping on them would end the watch before this import produced
+   anything.
+3. `cohort: exact`, N ≤ 25, and every cohort session is `completed` → summarize the foreground
+   sessions, or with K = 0 the three with the newest `evaluated_at` (ties by `session_id`). This
+   rule may fire on the first snapshot with nothing but pre-existing evals: a cohort that is
+   already fully evaluated has nothing left to wait for, and the labels say so.
 4. Two consecutive failed polls → stop, reporting the failures.
 5. 10 minutes since the first snapshot → stop. No new poll starts after the deadline, but a poll
    already dispatched before it finishes normally, and its outcome is still checked against rules
    1–4 first. On a deadline stop, summarize the foreground sessions that did complete and name the
-   ones still pending, with whether each has arrived; if none of them completed, summarize instead
-   up to three cohort sessions with the newest `evaluated_at` (ties by `session_id`).
+   ones still pending, with their state; if none of them completed, summarize instead up to three
+   watched sessions with the newest `evaluated_at` (ties by `session_id`).
 
 No idle early-stop — quiet polls are normal and are not a reason to stop early. Evals take minutes
 per session, and the foreground sessions are the newest, so they are usually the last to complete.
 
-**Per-session detail** (one session per query, 2 queries per detailed session, at most 10 across
-the whole run). A foreground pass takes whole session chains, so K can exceed five: the first five
-sessions in summary order get detail; every later one gets a one-line entry — link, label and
-`overall_score` — and no detail query. Both queries pass `max_rows` explicitly — a query left at
-the agent's guess truncates on the category list and throws the detail away:
-
-    -- max_rows: 25
-    SELECT category, ROUND(AVG(score), 2) AS mean FROM v_an_eval_scores
-    WHERE session_id = '<id>' AND outcome = 'assessed' GROUP BY category
-    -- max_rows: 2
-    SELECT question_id, score FROM v_an_eval_scores
-    WHERE session_id = '<id>' AND outcome = 'assessed' ORDER BY score ASC, question_id ASC LIMIT 2
-
-Strongest = highest mean, weakest = lowest, ties alphabetical by category. A truncated or failed
-detail query, or a session with no assessed rows, → summarize that session by `overall_score` alone
-and say so; this never affects poll success or the stop rules above.
-
-**What a detailed session summary says**, one short paragraph each: the link (section 7), the
-baseline label, then vendor and model, length (`duration_min`, rounded, and `event_count`),
-`overall_score` out of 5 and the judge model, strongest and weakest category with their means, and
-the two weakest questions with their scores. When the `kcap-sessions` MCP is available, call `get_session_summary` once per
-summarized session and lead with its title or first sentence when it returns one; an empty summary
-is left out silently, never called "no summary".
-
-**Cohort facts**, one line after the summaries, from the last committed poll's rows only: sessions
-arrived out of N, sessions evaluated, and the mean `overall_score` across the evaluated ones.
+**What a session summary says.** The first five sessions in summary order get one short paragraph
+each: the link (section 7), the baseline label, `overall_score` out of 5 and the judge model, the
+strongest and weakest of `categories` by `score` (ties alphabetical by name), the two
+`weakest_questions` with their scores, and the gist of the judge's `summary`. When
+`get_session_summary` returns a title or a first sentence for the session, lead with it; an empty
+summary is left out silently, never called "no summary". Every later session gets a one-line entry
+— link, label and `overall_score`. A completed entry with no categories is summarized by
+`overall_score` alone; say so.
+
+**Cohort facts**, one line after the summaries, from the last committed poll only: watched
+sessions arrived out of W, sessions evaluated, and the mean `overall_score` across the evaluated
+ones.
 
 ## 6. When little or nothing completes
 
 Explain the real gates plainly: auto-eval may be off for a repository; the server needs an eval
 agent configured; sessions under the minimum event count are skipped; already-evaluated sessions
-are not re-run; a recovery sweep runs hourly. If the analytics MCP is absent entirely, close
-immediately with the block below.
-
-Plan sentence (used here and from sections 3 and 5): "Insights isn't in this workspace's plan, so
-evals can't be followed from here; your import continues and evals appear in the Capacitor UI."
+are not re-run; a recovery sweep runs hourly. A `failed` session names its `failure_reason`.
 
 ## 7. CLOSE — always
 
 1. How to keep watching: prompt `Follow my kcap import` again (with the same `(run: <id>)` line
    when you were bound to one).
-2. Links: with a `repo_hash` matching `^[0-9a-f]{16}$`, `<server_url>/repo/<repo_hash>/sessions/<session_id>?tab=evaluation`;
-   otherwise (no `repo_hash`, or one that fails the grammar) `<server_url>/sessions/<session_id>?tab=evaluation`;
-   all results `<server_url>/sessions`. No valid file → `server_url` from `kcap whoami`; if that
-   fails too, say the results live in the Capacitor server UI and that `kcap whoami` prints the URL
-   once logged in.
+2. Links: `<server_url>/sessions/<session_id>?tab=evaluation` per session; all results
+   `<server_url>/sessions`. No valid file → `server_url` from `kcap whoami`; if that fails too, say
+   the results live in the Capacitor server UI and that `kcap whoami` prints the URL once logged
+   in.
 3. When `unattributed_on_disk` > 0: "N sessions on disk had no repository match; any of them that
    imported show without one — `kcap remap` places them."
 4. Offer the guided tour with the exact prompt `Start kcap guided tour`.