@iceinvein/agent-skills 0.18.3 → 0.20.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@iceinvein/agent-skills",
-  "version": "0.18.3",
+  "version": "0.20.0",
   "description": "Install agent skills into AI coding tools",
   "author": "iceinvein",
   "license": "MIT",

package/skills/index.json

@@ -283,7 +283,7 @@
     "name": "sluice",
     "description": "Routes work by change shape into four channels (bypass, fast, main, deep) and applies only the rules each channel needs, so a one-line fix does not pay the cost of a multi-subsystem build. Carries seven rules as one-liners in the router and the full treatment in references read only on friction. Checks the finished plan with plan.sh validate rather than trusting it to memory, seeds the run state from it, keeps a deep run's task breakdown in .sluice/run.json so a statusline segment, one status command and a SessionStart hook can answer where the run is (the hook prints a live run at every session start, compaction included), and closes each run with a ledger read out of the session transcript: elapsed, tools, tokens, and what each dispatched agent cost where the transcript recorded it. Claude Code only; stands down where the superpowers pipeline governs the repo.",
     "type": "prompt",
-    "version": "0.19.3"
+    "version": "0.21.0"
   },
   {
     "name": "temporal-coupling-detector",

package/skills/sluice/SKILL.md

@@ -36,6 +36,12 @@ them and an announcement worded otherwise reports as `not announced`. `bypass`
 says nothing at all, because a question that gets announced stops being a
 question.
 
+A session that changes the tree without ever announcing is stopped once and
+asked to route, because nothing else catches it: the run state that everything
+else reads is written by the channels, so a session that skipped the router
+leaves nothing behind to notice it skipped. The stop names no channel for you.
+Route what you have already done and say which one it was.
+
 **`root-cause`, `finish`, `meter` and `show-or-say` are not channel-assigned.**
 The code misbehaving triggers the first: a bug report, a red test, behaviour you
 cannot account for. An integration event, merging, pushing, or opening a PR,
@@ -96,7 +102,10 @@ your partner states a preference. Get the design signed off before code.
 Take the design stop through the harness's plan mode where there is one. Its
 gate is enforced rather than requested and it holds edits shut while it is open,
 so nothing gets built against a design nobody signed. It carries the first stop
-only; pre-flight still wants answers, and an approval is not one.
+only; pre-flight still wants answers, and an approval is not one. Where there is
+no plan mode, nothing is holding the draft: write the design to its file before
+you end the turn on it, because a design that lives only in the message you just
+sent is gone at the next compaction.
 
 The run's state goes in `.sluice/run.json`, written a command at a time by
 `scripts/status.sh`. That is what makes the breakdown readable from outside the

package/skills/sluice/evals/README.md

@@ -0,0 +1,100 @@
+# sluice evals
+
+Eight cases for `claude plugin eval`. Six pin the routing decision: which
+channel the announcement names, and whether the behaviour that channel owes
+actually happened. Two pin what a deep run does after pre-flight, where the
+question is no longer which channel but whether the run keeps going.
+
+| Case | Signal under test | What it pins |
+|---|---|---|
+| `bypass-question-stays-silent` | A question, no code change | Answers it, announces nothing, writes nothing |
+| `fast-flag-on-existing-command` | A new flag on an existing command | Fast channel; test edited and run before the source |
+| `main-new-interface` | Adds a port the repo does not have | Main channel; shape stated with a recommendation before building |
+| `deep-plan-across-subsystems` | A plan asked for, three subsystems | Deep channel; design written to `docs/specs/`; stops before code |
+| `explicit-instruction-collapses-to-fast` | Main-shaped work plus "just do it" | Collapses to fast; no design, no proposal |
+| `superpowers-conflict-stands-down` | Repo mandates the superpowers sequence | Stands down once, names no channel |
+
+Execution, where the run is already past both stops:
+
+| Case | Signal under test | What it pins |
+|---|---|---|
+| `deep-run-finishes-every-task` | Signed-off plan, three tasks left | All three reach done in one turn; no checking in between tasks |
+| `deep-run-blocks-on-a-real-decision` | Task 4 collides with a published contract | Tasks 2 and 3 land, Task 4 is marked `blocked`, the turn ends on one question |
+
+## Running
+
+Seven cases scaffold a small Node repo and then change it, so they need the
+scaffold flag and a tool grant. From the repo root:
+
+```bash
+claude plugin eval skills/sluice --scaffold --allow-tools Bash Write Edit
+```
+
+`--case` takes one glob and is not repeatable, so the one case that needs
+less runs on its own:
+
+```bash
+claude plugin eval skills/sluice --case 'bypass-*'
+```
+
+Useful while iterating on graders: `--ablation none` drops the no-plugin arm
+and halves the cost, `--runs 1` drops the repeats, and `--judge-model sonnet`
+settles an `llm` grader that keeps flipping.
+
+To gate CI, pick a floor and let a miss fail the job:
+
+```bash
+claude plugin eval skills/sluice --scaffold --allow-tools Bash Write Edit \
+  --trust-plugin --threshold 0.8
+```
+
+## Scoring notes
+
+`tool_used: Skill` graders are excluded from the score in a two-arm run and
+reported as pass/fail indicators, because they can never pass without the
+plugin. They are there to tell you whether a score came from sluice or from
+the model's own habits.
+
+The two stops a deep run is allowed are both before Task 1: design sign-off and
+plan-plus-pre-flight. `deep-plan-across-subsystems` pins the first. Everything
+after pre-flight is covered by the two execution cases, which is where a run
+that checks in per task would show up. Their `.sluice/run.json` fixtures were
+produced by `status.sh` and `plan.sh import` rather than typed by hand, so the
+tiers and the graph columns match what the plan actually says; the plans pass
+`plan.sh validate` with no errors and the scaffolded suites start green.
+
+The fixtures are deliberately small. `fixture.sh` is duplicated across the
+cases that use it rather than shared, because `context.scaffold_script` reads
+only from the case's own directory.
+
+## Verification status
+
+Every case has been run end to end at least once and scored 1.00. The two that
+needed the least (`bypass-question-stays-silent`, `superpowers-conflict-stands-down`)
+were run at `--runs 1 --ablation none`; the other six were run the same way, and
+`deep-plan-across-subsystems` and `main-new-interface` twice each after the
+fixes below.
+
+Three defects the first full pass turned up, all in the suite rather than in
+sluice:
+
+- `announces-<channel>-channel` matched `<channel> channel` anywhere in the
+  trace, and the trace carries SKILL.md's routing table, which names all four.
+  Those graders passed whenever the skill loaded. They now anchor on the
+  announcement opening an assistant message, the same anchor `run-stats.sh`
+  meters by. `fast-flag-on-existing-command` and
+  `explicit-instruction-collapses-to-fast` still carry the old pattern.
+- `deep-plan-across-subsystems` shipped no `fixture.sh`, so the run landed in an
+  empty tree and the case flipped between designing against the prompt alone and
+  stopping to ask where the repo was. It has a fixture now: three callers
+  through one upstream client. Its `no-implementation-yet` grader went with it,
+  because `file_exists: 'src/**', exists: false` reported absent against a tree
+  holding four source files.
+- `shape-agreed-before-building` was an `llm` grader over the trace, and the
+  judge is given a head-and-tail window of it. In a run this long the shape
+  statement lands in the dropped middle, so the judge voted FAIL six times out
+  of six on runs that had stated the shape plainly. `focus` accepts only
+  `last_message`, `trace` or a file, and `main` agrees in a message rather than
+  a file, so there was no slice to point it at. It is a regex over the
+  chronological trace now, which pins the order but not whether a
+  recommendation came with the shape.

package/skills/sluice/evals/bypass-question-stays-silent/graders/answers-the-question.md

@@ -0,0 +1,9 @@
+---
+type: llm
+weight: 2
+---
+
+The reply should answer a conceptual question about two identifiers used in payment APIs.
+
+PASS if the reply explains that an idempotency key is supplied by the caller to make a retried write safe (the server returns the original result instead of performing the operation twice), and that a request ID identifies one call for tracing, logging, or support, without affecting what the server does.
+FAIL if the reply conflates the two, describes only one of them, asks a clarifying question instead of answering, or answers with a plan of work rather than an explanation.

package/skills/sluice/evals/bypass-question-stays-silent/graders/no-channel-announcement.md

@@ -0,0 +1,7 @@
+---
+type: regex
+pattern: '(bypass|fast|main|deep)\s+channel'
+flags: i
+match: not_contains
+target: last_message
+---

package/skills/sluice/evals/bypass-question-stays-silent/graders/writes-nothing.md

@@ -0,0 +1,5 @@
+---
+type: file_exists
+path: '**/*'
+exists: false
+---

package/skills/sluice/evals/bypass-question-stays-silent/prompt.md

@@ -0,0 +1,10 @@
+---
+name: bypass-question-stays-silent
+description: A question that changes no code must be answered without a channel announcement.
+tags: [routing, bypass, readonly]
+max_turns: 6
+allowed_tools: [Skill]
+expected_outcome: A direct answer about idempotency keys, with no "<channel> channel" line anywhere in the reply.
+---
+
+What's the difference between an idempotency key and a request ID? I keep seeing both in payment APIs and I'm not sure when each one earns its place.

package/skills/sluice/evals/deep-plan-across-subsystems/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: deep-plan-across-subsystems
+context:
+  scaffold_script: fixture.sh

package/skills/sluice/evals/deep-plan-across-subsystems/fixture.sh

@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Smallest repo that makes the rate limiter a real three-subsystem change: a
+# CLI, a webhook handler and a worker, each calling the same upstream client,
+# and nothing between them and the API.
+set -euo pipefail
+
+mkdir -p src/cli src/webhook src/worker src/upstream tests
+
+cat > package.json <<'JSON'
+{
+  "name": "relay",
+  "version": "0.4.0",
+  "type": "module",
+  "scripts": {
+    "test": "node --test \"tests/*.test.js\""
+  }
+}
+JSON
+
+cat > src/upstream/client.js <<'JS'
+const BASE = "https://api.upstream.example";
+
+export async function call(path, body, fetchImpl = fetch) {
+  const response = await fetchImpl(`${BASE}${path}`, {
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body: JSON.stringify(body),
+  });
+
+  if (!response.ok) {
+    throw new Error(`upstream ${response.status} on ${path}`);
+  }
+
+  return response.json();
+}
+JS
+
+cat > src/cli/push.js <<'JS'
+import { call } from "../upstream/client.js";
+
+export async function push(records, log = console.log) {
+  for (const record of records) {
+    await call("/v1/records", record);
+    log(`pushed ${record.id}`);
+  }
+}
+JS
+
+cat > src/webhook/handler.js <<'JS'
+import { call } from "../upstream/client.js";
+
+export async function handle(event) {
+  const result = await call("/v1/events", { type: event.type, payload: event.payload });
+  return { status: 202, id: result.id };
+}
+JS
+
+cat > src/worker/backfill.js <<'JS'
+import { call } from "../upstream/client.js";
+
+export async function backfill(queue) {
+  let sent = 0;
+
+  while (queue.length > 0) {
+    await call("/v1/records", queue.shift());
+    sent += 1;
+  }
+
+  return sent;
+}
+JS
+
+cat > tests/upstream.test.js <<'JS'
+import assert from "node:assert/strict";
+import { test } from "node:test";
+import { call } from "../src/upstream/client.js";
+
+function fakeFetch(ok, body) {
+  return async () => ({ ok, status: ok ? 200 : 429, json: async () => body });
+}
+
+test("a successful call returns the parsed body", async () => {
+  const result = await call("/v1/records", { id: "a" }, fakeFetch(true, { id: "a" }));
+  assert.deepEqual(result, { id: "a" });
+});
+
+test("a rejected call reports the status and the path", async () => {
+  await assert.rejects(
+    () => call("/v1/records", { id: "a" }, fakeFetch(false)),
+    /upstream 429 on \/v1\/records/,
+  );
+});
+JS
+
+cat > README.md <<'MD'
+# relay
+
+Three callers, one upstream API: `src/cli/push.js`, `src/webhook/handler.js`
+and `src/worker/backfill.js` all go through `src/upstream/client.js`.
+`npm test` runs the suite.
+MD
+
+git init --quiet
+git add -A
+git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "relay 0.4.0"

package/skills/sluice/evals/deep-plan-across-subsystems/graders/announces-deep-channel.md

@@ -0,0 +1,11 @@
+---
+type: regex
+# The trace carries SKILL.md's own routing table, which names every channel, so
+# `contains` passes whenever the skill loads. Anchor on how the agent says it
+# instead, the same anchor run-stats.sh meters: the words open an assistant
+# message, or follow a label such as "Sluice:" on the same line.
+pattern: '"text":"(?:[^.!?\n"]{0,100}[:=]\s*)?[\s*_#>\\]*deep channel'
+flags: i
+target: trace
+weight: 2
+---

package/skills/sluice/evals/deep-plan-across-subsystems/graders/design-written-to-docs.md

@@ -0,0 +1,5 @@
+---
+type: file_exists
+path: 'docs/specs/*.md'
+weight: 2
+---

package/skills/sluice/evals/deep-plan-across-subsystems/graders/no-implementation-yet.md

@@ -0,0 +1,10 @@
+---
+type: llm
+focus: { source: file, path: src/upstream/client.js }
+weight: 2
+---
+
+This is the upstream client after a turn that was asked to plan a shared rate limiter. Every caller reaches the upstream API through it, so it is the file a limiter has to land in.
+
+PASS if it is still the fixture's client and nothing else: a `call()` that posts the body, throws on a non-ok response, and returns the parsed JSON.
+FAIL if any limiting has been built into it: a token bucket, a counter, a budget check, a store or Redis client, a sleep, a queue, or a wrapper that decides whether the call may proceed.

package/skills/sluice/evals/deep-plan-across-subsystems/graders/sluice-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:[\w-]+:)?sluice"'
+---

package/skills/sluice/evals/deep-plan-across-subsystems/graders/stopped-for-signoff.md

@@ -0,0 +1,8 @@
+---
+type: llm
+---
+
+The agent was asked to plan a rate limiter shared by three subsystems.
+
+PASS if the final message hands a written design back for sign-off: it says where the design was written and asks for approval, or otherwise makes clear it is waiting before any code is written.
+FAIL if the final message reports implementation work as done, moves straight into building, or presents the design as settled with no sign-off asked for.

package/skills/sluice/evals/deep-plan-across-subsystems/prompt.md

@@ -0,0 +1,11 @@
+---
+name: deep-plan-across-subsystems
+description: A plan asked for across three subsystems is the deep channel. Pins the announcement, the written design, and stopping before code.
+tags: [routing, deep, write]
+max_turns: 20
+timeout_seconds: 900
+allowed_tools: [Read, Glob, Grep, Skill, Write]
+expected_outcome: Announces the deep channel, writes a design under docs/specs/, stops for sign-off, and writes no implementation code.
+---
+
+Plan out a shared rate limiter for us. The CLI, the webhook handler and the background worker all hammer the same upstream API and all three need to sit behind one budget, so whatever we build has to be reachable from each of them and hold its counters somewhere they can all see.

package/skills/sluice/evals/deep-run-blocks-on-a-real-decision/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: deep-run-blocks-on-a-real-decision
+context:
+  scaffold_script: fixture.sh