@iceinvein/agent-skills 0.23.8 → 0.23.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/skills/index.json +3 -3
- package/skills/magpie/README.md +1 -1
- package/skills/magpie/SKILL.md +2 -2
- package/skills/magpie/evals/README.md +22 -8
- package/skills/magpie/evals/bare-invocation-goes-local/case.yaml +4 -0
- package/skills/magpie/evals/bare-invocation-goes-local/fixture.sh +159 -0
- package/skills/magpie/evals/bare-invocation-goes-local/graders/branch-was-reachable.md +5 -0
- package/skills/magpie/evals/bare-invocation-goes-local/graders/never-went-to-a-pr.md +7 -0
- package/skills/magpie/evals/bare-invocation-goes-local/graders/setup-ran-local.md +6 -0
- package/skills/magpie/evals/bare-invocation-goes-local/prompt.md +11 -0
- package/skills/magpie/evals/baselines/claude-opus-5-5.json +27 -3
- package/skills/magpie/evals/rules.json +8 -0
- package/skills/magpie/scripts/__tests__/skill-lint.test.ts +8 -0
- package/skills/magpie/skill.json +1 -1
- package/skills/sluice/SKILL.md +13 -0
- package/skills/sluice/evals/README.md +5 -2
- package/skills/sluice/evals/deep-plan-checks-the-intent/case.yaml +4 -0
- package/skills/sluice/evals/deep-plan-checks-the-intent/fixture.sh +124 -0
- package/skills/sluice/evals/deep-plan-checks-the-intent/graders/announces-deep-channel.md +12 -0
- package/skills/sluice/evals/deep-plan-checks-the-intent/graders/checker-dispatched.md +12 -0
- package/skills/sluice/evals/deep-plan-checks-the-intent/graders/no-source-edit.md +11 -0
- package/skills/sluice/evals/deep-plan-checks-the-intent/graders/no-source-write.md +11 -0
- package/skills/sluice/evals/deep-plan-checks-the-intent/graders/signoff-reports-the-check.md +16 -0
- package/skills/sluice/evals/deep-plan-checks-the-intent/graders/sluice-fired.md +5 -0
- package/skills/sluice/evals/deep-plan-checks-the-intent/graders/spec-has-an-intent-map.md +19 -0
- package/skills/sluice/evals/deep-plan-checks-the-intent/graders/stopped-for-signoff.md +8 -0
- package/skills/sluice/evals/deep-plan-checks-the-intent/prompt.md +20 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/case.yaml +4 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/fixture.sh +178 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/graders/announces-deep-channel.md +12 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/graders/every-decision-built.md +15 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/graders/no-source-edit.md +11 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/graders/no-source-write.md +11 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/graders/plan-checker-dispatched.md +11 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/graders/plan-cites-the-spec.md +7 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/graders/sluice-fired.md +5 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/graders/stops-at-preflight.md +14 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/graders/validate-ran.md +9 -0
- package/skills/sluice/evals/deep-plan-checks-the-plan/prompt.md +12 -0
- package/skills/sluice/evals/deep-plan-reads-the-prior-spec/case.yaml +4 -0
- package/skills/sluice/evals/deep-plan-reads-the-prior-spec/fixture.sh +125 -0
- package/skills/sluice/evals/deep-plan-reads-the-prior-spec/graders/announces-deep-channel.md +12 -0
- package/skills/sluice/evals/deep-plan-reads-the-prior-spec/graders/context-cites-the-prior-spec.md +6 -0
- package/skills/sluice/evals/deep-plan-reads-the-prior-spec/graders/design-follows-the-prior-spec.md +21 -0
- package/skills/sluice/evals/deep-plan-reads-the-prior-spec/graders/design-written-to-docs.md +5 -0
- package/skills/sluice/evals/deep-plan-reads-the-prior-spec/graders/no-source-edit.md +11 -0
- package/skills/sluice/evals/deep-plan-reads-the-prior-spec/graders/no-source-write.md +11 -0
- package/skills/sluice/evals/deep-plan-reads-the-prior-spec/graders/sluice-fired.md +5 -0
- package/skills/sluice/evals/deep-plan-reads-the-prior-spec/prompt.md +17 -0
- package/skills/sluice/evals/rules.json +24 -0
- package/skills/sluice/references/deep-channel.md +145 -8
- package/skills/sluice/scripts/plan.sh +115 -4
- package/skills/sluice/skill.json +2 -2
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: tool_used
|
|
3
|
+
# The run plans and stops for sign-off, so no Edit may land under src/ or
|
|
4
|
+
# tests/. Bash is left out of the case's tools, so these two are the only ways
|
|
5
|
+
# code could change.
|
|
6
|
+
tool: Edit
|
|
7
|
+
input_match: '"file_path"\s*:\s*"(?:[^"]*/)?(?:src|tests)/'
|
|
8
|
+
min: 0
|
|
9
|
+
max: 0
|
|
10
|
+
weight: 2
|
|
11
|
+
---
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: tool_used
|
|
3
|
+
# The run plans and stops for sign-off, so no Write may land under src/ or
|
|
4
|
+
# tests/. Bash is left out of the case's tools, so these two are the only ways
|
|
5
|
+
# code could change.
|
|
6
|
+
tool: Write
|
|
7
|
+
input_match: '"file_path"\s*:\s*"(?:[^"]*/)?(?:src|tests)/'
|
|
8
|
+
min: 0
|
|
9
|
+
max: 0
|
|
10
|
+
weight: 2
|
|
11
|
+
---
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: llm
|
|
3
|
+
weight: 2
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
The agent was asked to plan ticket OPS-12 and to write its design to
|
|
7
|
+
`docs/specs/OPS-12.md`. Before asking for sign-off, it is expected to have a
|
|
8
|
+
fresh-context agent check the spec against the ticket's asks.
|
|
9
|
+
|
|
10
|
+
PASS if the final message asks for sign-off on the design and says that an
|
|
11
|
+
alignment check of the spec against the ticket ran, and what it found (for
|
|
12
|
+
example that every ask is covered, or which gaps it raised and how they were
|
|
13
|
+
settled).
|
|
14
|
+
FAIL if the final message does not ask for sign-off, or asks for it without
|
|
15
|
+
mentioning that such a check ran, or mentions the check without saying what it
|
|
16
|
+
found.
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: llm
|
|
3
|
+
focus: { source: file, path: docs/specs/OPS-12.md }
|
|
4
|
+
weight: 3
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
This is the design for ticket OPS-12, which carries three numbered asks: (1)
|
|
8
|
+
every upstream call retries a 429, up to three attempts in all, waiting the
|
|
9
|
+
`Retry-After` seconds between them; (2) each retry is logged by the calling
|
|
10
|
+
subsystem with the path and the attempt number; (3) the CLI gets a
|
|
11
|
+
`--no-retry` flag.
|
|
12
|
+
|
|
13
|
+
PASS if the design has an Intent section that quotes the ticket's text and
|
|
14
|
+
maps each of the three asks, one by one, either to a numbered decision in the
|
|
15
|
+
design (headed like `D1.`) or to an Out of scope section with a reason for
|
|
16
|
+
leaving it out.
|
|
17
|
+
FAIL if there is no Intent section, if it paraphrases the ticket instead of
|
|
18
|
+
quoting it, or if any of the three asks is missing from the mapping or mapped
|
|
19
|
+
to nothing.
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: llm
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
The agent was asked to plan a ticket that spans the CLI, the webhook handler, the worker and the upstream client they share.
|
|
6
|
+
|
|
7
|
+
PASS if the final message hands a written design back for sign-off: it says where the design was written and asks for approval, or otherwise makes clear it is waiting before any code is written.
|
|
8
|
+
FAIL if the final message reports implementation work as done, moves straight into building, or presents the design as settled with no sign-off asked for.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: deep-plan-checks-the-intent
|
|
3
|
+
description: A deep plan from a ticket with three numbered asks. Pins an Intent section that maps each ask to a decision or to Out of scope, a fresh-context check of the spec against it, and a sign-off message that reports the check.
|
|
4
|
+
tags: [routing, deep, write]
|
|
5
|
+
max_turns: 30
|
|
6
|
+
timeout_seconds: 1200
|
|
7
|
+
allowed_tools: [Read, Glob, Grep, Skill, Write, Agent]
|
|
8
|
+
expected_outcome: Announces the deep channel, writes docs/specs/OPS-12.md with an Intent section quoting the ticket and mapping each of its three asks, dispatches an agent to check the spec against that intent, and stops for sign-off saying what the check found.
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
Plan OPS-12 for us. The design goes in `docs/specs/OPS-12.md`.
|
|
12
|
+
|
|
13
|
+
> Upstream has started answering 429 under load, and we lose records when it does.
|
|
14
|
+
>
|
|
15
|
+
> 1. Every upstream call retries a 429, up to three attempts in all, waiting the
|
|
16
|
+
> number of seconds in the `Retry-After` header between them.
|
|
17
|
+
> 2. Each retry is logged by the caller that made the call, with the path and
|
|
18
|
+
> the attempt number.
|
|
19
|
+
> 3. The CLI gets a `--no-retry` flag for operators who would rather see the
|
|
20
|
+
> 429 straight away.
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# A relay whose design for OPS-13 is already written and signed off, with three
|
|
3
|
+
# numbered decisions and an Out of scope section, so the run starts at the plan
|
|
4
|
+
# stop. The case pins what the plan owes its spec: a Spec line, a Builds line
|
|
5
|
+
# on every task covering D1 to D3, `plan.sh validate` run on it, and a
|
|
6
|
+
# fresh-context check of plan against spec before the pre-flight questions.
|
|
7
|
+
# The README fixes where the plan goes, so the graders can read it.
|
|
8
|
+
set -euo pipefail
|
|
9
|
+
|
|
10
|
+
mkdir -p src/cli src/webhook src/worker src/upstream tests docs/specs docs/plans
|
|
11
|
+
|
|
12
|
+
cat > package.json <<'JSON'
|
|
13
|
+
{
|
|
14
|
+
"name": "relay",
|
|
15
|
+
"version": "0.6.0",
|
|
16
|
+
"type": "module",
|
|
17
|
+
"scripts": {
|
|
18
|
+
"test": "node --test \"tests/*.test.js\""
|
|
19
|
+
}
|
|
20
|
+
}
|
|
21
|
+
JSON
|
|
22
|
+
|
|
23
|
+
cat > src/upstream/client.js <<'JS'
|
|
24
|
+
const BASE = "https://api.upstream.example";
|
|
25
|
+
|
|
26
|
+
export async function call(path, body, fetchImpl = fetch) {
|
|
27
|
+
const response = await fetchImpl(`${BASE}${path}`, {
|
|
28
|
+
method: "POST",
|
|
29
|
+
headers: { "content-type": "application/json" },
|
|
30
|
+
body: JSON.stringify(body),
|
|
31
|
+
});
|
|
32
|
+
|
|
33
|
+
if (!response.ok) {
|
|
34
|
+
throw new Error(`upstream ${response.status} on ${path}`);
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
return response.json();
|
|
38
|
+
}
|
|
39
|
+
JS
|
|
40
|
+
|
|
41
|
+
cat > src/cli/push.js <<'JS'
|
|
42
|
+
import { call } from "../upstream/client.js";
|
|
43
|
+
|
|
44
|
+
export async function push(records, log = console.log) {
|
|
45
|
+
for (const record of records) {
|
|
46
|
+
await call("/v1/records", record);
|
|
47
|
+
log(`pushed ${record.id}`);
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
JS
|
|
51
|
+
|
|
52
|
+
cat > src/cli/main.js <<'JS'
|
|
53
|
+
import { readFileSync } from "node:fs";
|
|
54
|
+
import { push } from "./push.js";
|
|
55
|
+
|
|
56
|
+
const [file] = process.argv.slice(2);
|
|
57
|
+
await push(JSON.parse(readFileSync(file, "utf8")));
|
|
58
|
+
JS
|
|
59
|
+
|
|
60
|
+
cat > src/webhook/handler.js <<'JS'
|
|
61
|
+
import { call } from "../upstream/client.js";
|
|
62
|
+
|
|
63
|
+
export async function handle(event) {
|
|
64
|
+
const result = await call("/v1/events", { type: event.type, payload: event.payload });
|
|
65
|
+
return { status: 202, id: result.id };
|
|
66
|
+
}
|
|
67
|
+
JS
|
|
68
|
+
|
|
69
|
+
cat > src/worker/backfill.js <<'JS'
|
|
70
|
+
import { call } from "../upstream/client.js";
|
|
71
|
+
|
|
72
|
+
export async function backfill(queue, log = console.log) {
|
|
73
|
+
let sent = 0;
|
|
74
|
+
|
|
75
|
+
while (queue.length > 0) {
|
|
76
|
+
await call("/v1/records", queue.shift());
|
|
77
|
+
sent += 1;
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
log(`backfill sent ${sent}`);
|
|
81
|
+
return sent;
|
|
82
|
+
}
|
|
83
|
+
JS
|
|
84
|
+
|
|
85
|
+
cat > tests/upstream.test.js <<'JS'
|
|
86
|
+
import assert from "node:assert/strict";
|
|
87
|
+
import { test } from "node:test";
|
|
88
|
+
import { call } from "../src/upstream/client.js";
|
|
89
|
+
|
|
90
|
+
function fakeFetch(ok, body) {
|
|
91
|
+
return async () => ({ ok, status: ok ? 200 : 429, json: async () => body });
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
test("a successful call returns the parsed body", async () => {
|
|
95
|
+
const result = await call("/v1/records", { id: "a" }, fakeFetch(true, { id: "a" }));
|
|
96
|
+
assert.deepEqual(result, { id: "a" });
|
|
97
|
+
});
|
|
98
|
+
|
|
99
|
+
test("a rejected call reports the status and the path", async () => {
|
|
100
|
+
await assert.rejects(
|
|
101
|
+
() => call("/v1/records", { id: "a" }, fakeFetch(false)),
|
|
102
|
+
/upstream 429 on \/v1\/records/,
|
|
103
|
+
);
|
|
104
|
+
});
|
|
105
|
+
JS
|
|
106
|
+
|
|
107
|
+
cat > docs/specs/OPS-13.md <<'MD'
|
|
108
|
+
# Design: OPS-13 retry upstream 429s
|
|
109
|
+
|
|
110
|
+
Signed off by the partner on 2026-10-20.
|
|
111
|
+
|
|
112
|
+
## Intent
|
|
113
|
+
|
|
114
|
+
The ticket, verbatim:
|
|
115
|
+
|
|
116
|
+
> 1. Every upstream call retries a 429, up to three attempts in all, waiting the
|
|
117
|
+
> number of seconds in the `Retry-After` header between them.
|
|
118
|
+
> 2. Each retry is logged by the caller that made the call, with the path and
|
|
119
|
+
> the attempt number.
|
|
120
|
+
> 3. The CLI gets a `--no-retry` flag.
|
|
121
|
+
> 4. The webhook handler gets its own retry budget.
|
|
122
|
+
|
|
123
|
+
| Ask | Decision |
|
|
124
|
+
|---|---|
|
|
125
|
+
| 1. Retry a 429 | D1 |
|
|
126
|
+
| 2. Log each retry in the caller | D2 |
|
|
127
|
+
| 3. `--no-retry` on the CLI | D3 |
|
|
128
|
+
| 4. A webhook retry budget | Out of scope |
|
|
129
|
+
|
|
130
|
+
## Context
|
|
131
|
+
|
|
132
|
+
- `src/upstream/client.js` is the one path to the upstream API; the CLI
|
|
133
|
+
(`src/cli/push.js`), the webhook (`src/webhook/handler.js`) and the worker
|
|
134
|
+
(`src/worker/backfill.js`) all call `call`.
|
|
135
|
+
- Each caller logs through the `log` it is handed, defaulting to `console.log`.
|
|
136
|
+
|
|
137
|
+
### D1. The client retries a 429
|
|
138
|
+
|
|
139
|
+
`call(path, body, fetchImpl, options)` retries a 429 response up to three
|
|
140
|
+
attempts in all, sleeping `Retry-After` seconds between attempts through an
|
|
141
|
+
injected `sleep` so tests do not wait. `options.retry === false` makes one
|
|
142
|
+
attempt only. Any other status throws as it does today.
|
|
143
|
+
|
|
144
|
+
### D2. Callers log each retry
|
|
145
|
+
|
|
146
|
+
`call` takes `options.onRetry(path, attempt)`. The CLI and the worker pass one
|
|
147
|
+
that writes `retry <path> attempt <n>` through their `log`; the webhook passes
|
|
148
|
+
one that writes the same line through `console.log`.
|
|
149
|
+
|
|
150
|
+
### D3. The CLI takes --no-retry
|
|
151
|
+
|
|
152
|
+
`src/cli/main.js` reads `--no-retry` from its arguments and passes
|
|
153
|
+
`{ retry: false }` through `push` to `call`.
|
|
154
|
+
|
|
155
|
+
## Out of scope
|
|
156
|
+
|
|
157
|
+
A separate retry budget for the webhook handler: the webhook's upstream
|
|
158
|
+
traffic is a small fraction of the total, and one shared policy is enough
|
|
159
|
+
until it is not.
|
|
160
|
+
MD
|
|
161
|
+
|
|
162
|
+
cat > README.md <<'MD'
|
|
163
|
+
# relay
|
|
164
|
+
|
|
165
|
+
Three callers, one upstream API: `src/cli/push.js`, `src/webhook/handler.js`
|
|
166
|
+
and `src/worker/backfill.js` all go through `src/upstream/client.js`.
|
|
167
|
+
`npm test` runs the suite.
|
|
168
|
+
|
|
169
|
+
## Designs
|
|
170
|
+
|
|
171
|
+
A ticket's design goes in `docs/specs/<ticket>.md`, for example
|
|
172
|
+
`docs/specs/OPS-13.md`, and its plan in `docs/plans/<ticket>.md`.
|
|
173
|
+
MD
|
|
174
|
+
touch docs/plans/.gitkeep
|
|
175
|
+
|
|
176
|
+
git init --quiet
|
|
177
|
+
git add -A
|
|
178
|
+
git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "relay 0.6.0"
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: regex
|
|
3
|
+
# The trace carries SKILL.md's own routing table, which names every channel, so
|
|
4
|
+
# `contains` passes whenever the skill loads. Anchor on the two ways the agent
|
|
5
|
+
# routes instead, the same ones run-stats.sh meters: a Bash call that invokes
|
|
6
|
+
# `status.sh route deep` (a quoted mention, as in a grep, does not count), or the
|
|
7
|
+
# words opening an assistant message or following a label such as "Sluice:".
|
|
8
|
+
pattern: '"command":"(?:(?:[^"\\]|\\.)*?(?:[\s/]|\\[nt]))?status\.sh(?:\\"|'')?(?:\s|\\[nt])+route(?:\s|\\[nt])+deep(?:\s|\\[nt]|[;&|]|")|"text":"(?:[^.!?\n"]{0,100}[:=]\s*)?[\s*_#>\\]*deep channel'
|
|
9
|
+
flags: i
|
|
10
|
+
target: trace
|
|
11
|
+
weight: 2
|
|
12
|
+
---
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: llm
|
|
3
|
+
focus: { source: file, path: docs/plans/OPS-13.md }
|
|
4
|
+
weight: 3
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
This is the plan for a signed-off design with three decisions, D1 (the
|
|
8
|
+
upstream client retries a 429), D2 (callers log each retry) and D3 (the CLI
|
|
9
|
+
takes `--no-retry`), and an Out of scope section that no task should build.
|
|
10
|
+
|
|
11
|
+
PASS if every task in the plan carries a Builds line (naming the decisions it
|
|
12
|
+
builds, or `none` with a reason), and D1, D2 and D3 each appear on the Builds
|
|
13
|
+
line of at least one task.
|
|
14
|
+
FAIL if any task has no Builds line, or if any of D1, D2 or D3 appears on no
|
|
15
|
+
task's Builds line.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: tool_used
|
|
3
|
+
# The run plans and stops at pre-flight, so no Edit may land under src/ or
|
|
4
|
+
# tests/. Bash is allowed here so plan.sh validate can run, and the
|
|
5
|
+
# stops-at-preflight judge covers a change made through it.
|
|
6
|
+
tool: Edit
|
|
7
|
+
input_match: '"file_path"\s*:\s*"(?:[^"]*/)?(?:src|tests)/'
|
|
8
|
+
min: 0
|
|
9
|
+
max: 0
|
|
10
|
+
weight: 2
|
|
11
|
+
---
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: tool_used
|
|
3
|
+
# The run plans and stops at pre-flight, so no Write may land under src/ or
|
|
4
|
+
# tests/. Bash is allowed here so plan.sh validate can run, and the
|
|
5
|
+
# stops-at-preflight judge covers a change made through it.
|
|
6
|
+
tool: Write
|
|
7
|
+
input_match: '"file_path"\s*:\s*"(?:[^"]*/)?(?:src|tests)/'
|
|
8
|
+
min: 0
|
|
9
|
+
max: 0
|
|
10
|
+
weight: 2
|
|
11
|
+
---
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: tool_order
|
|
3
|
+
# The check reads a finished plan in a fresh context, so it comes after the
|
|
4
|
+
# first validate run, and its brief names both documents in either order. An
|
|
5
|
+
# Agent that drafts the plan names both paths too and goes out before any
|
|
6
|
+
# validate, so it is the first after-match and fails the case: a run that hands
|
|
7
|
+
# the drafting to an agent is scored as one that never checked.
|
|
8
|
+
before: { tool: Bash, input_match: '"command"\s*:\s*"(?:(?:[^"\\]|\\.)*?(?:[\s/;&|(]|\\[nt]))?plan\.sh(?:\\"|'')?(?:\s|\\[nt])+validate(?:\s|\\[nt]|"|[;&|)>])' }
|
|
9
|
+
after: { tool: Agent, input_match: '^(?=[\s\S]*docs/specs/OPS-13\.md)(?=[\s\S]*docs/plans/OPS-13\.md)' }
|
|
10
|
+
weight: 2
|
|
11
|
+
---
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: llm
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
The agent was handed a signed-off design and asked to write the plan for it.
|
|
6
|
+
The next stop after the plan is the pre-flight questions, asked before any
|
|
7
|
+
task starts.
|
|
8
|
+
|
|
9
|
+
PASS if the final message asks the pre-flight questions (how the run should
|
|
10
|
+
proceed, for example how tasks are dispatched or reviewed) and no task has
|
|
11
|
+
been started: no source or test file changed, no task reported in progress or
|
|
12
|
+
done.
|
|
13
|
+
FAIL if the final message reports any task as started or done, moves into
|
|
14
|
+
building, or ends without asking the pre-flight questions.
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: tool_used
|
|
3
|
+
tool: Bash
|
|
4
|
+
# Anchored on the command itself, so a grep or a read that only mentions
|
|
5
|
+
# plan.sh validate does not count.
|
|
6
|
+
input_match: '"command"\s*:\s*"(?:(?:[^"\\]|\\.)*?(?:[\s/;&|(]|\\[nt]))?plan\.sh(?:\\"|'')?(?:\s|\\[nt])+validate(?:\s|\\[nt]|"|[;&|)>])'
|
|
7
|
+
min: 1
|
|
8
|
+
weight: 2
|
|
9
|
+
---
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: deep-plan-checks-the-plan
|
|
3
|
+
description: A deep run that starts from a signed-off design and is asked for the plan. Pins a Spec line and per-task Builds lines covering every decision, plan.sh validate run on the plan, and a fresh-context check of plan against spec before the pre-flight questions.
|
|
4
|
+
tags: [routing, deep, write]
|
|
5
|
+
max_turns: 30
|
|
6
|
+
timeout_seconds: 1200
|
|
7
|
+
allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
|
|
8
|
+
expected_outcome: Announces the deep channel, writes docs/plans/OPS-13.md with a Spec line naming docs/specs/OPS-13.md and a Builds line on every task covering D1 to D3, runs plan.sh validate on it, dispatches an agent to check the plan against the spec, and stops at the pre-flight questions without starting a task.
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
The design for OPS-13 in `docs/specs/OPS-13.md` is signed off. Write the plan
|
|
12
|
+
for it at `docs/plans/OPS-13.md`.
|
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# A sign-in service where an earlier spec already settled where counters live:
|
|
3
|
+
# `data/counters.json`, never only in memory, because a restart must not reset
|
|
4
|
+
# a count. No code keeps a counter yet, and the three subsystems run in one
|
|
5
|
+
# process, so a Map in the sign-in handler is the obvious design and nothing in
|
|
6
|
+
# the code points elsewhere. Only the earlier spec does, and the prompt never
|
|
7
|
+
# names it, so the case pins whether the run reads the repo's prior specs
|
|
8
|
+
# before it designs. The README fixes where the design goes, so the graders can
|
|
9
|
+
# read it.
|
|
10
|
+
set -euo pipefail
|
|
11
|
+
|
|
12
|
+
mkdir -p src/api src/worker src/report tests docs/specs
|
|
13
|
+
|
|
14
|
+
cat > package.json <<'JSON'
|
|
15
|
+
{
|
|
16
|
+
"name": "gate",
|
|
17
|
+
"version": "1.3.0",
|
|
18
|
+
"type": "module",
|
|
19
|
+
"scripts": {
|
|
20
|
+
"test": "node --test \"tests/*.test.js\""
|
|
21
|
+
}
|
|
22
|
+
}
|
|
23
|
+
JSON
|
|
24
|
+
|
|
25
|
+
cat > src/api/sign-in.js <<'JS'
|
|
26
|
+
import { verify } from "./passwords.js";
|
|
27
|
+
|
|
28
|
+
export async function signIn(accounts, name, password) {
|
|
29
|
+
const account = accounts.find((candidate) => candidate.name === name);
|
|
30
|
+
if (account === undefined || !(await verify(password, account.hash))) {
|
|
31
|
+
return { status: 401 };
|
|
32
|
+
}
|
|
33
|
+
return { status: 200, account: account.name };
|
|
34
|
+
}
|
|
35
|
+
JS
|
|
36
|
+
|
|
37
|
+
cat > src/api/passwords.js <<'JS'
|
|
38
|
+
import { scrypt, timingSafeEqual } from "node:crypto";
|
|
39
|
+
import { promisify } from "node:util";
|
|
40
|
+
|
|
41
|
+
const derive = promisify(scrypt);
|
|
42
|
+
|
|
43
|
+
export async function verify(password, hash) {
|
|
44
|
+
const [salt, key] = hash.split(":");
|
|
45
|
+
const derived = await derive(password, salt, 32);
|
|
46
|
+
return timingSafeEqual(derived, Buffer.from(key, "hex"));
|
|
47
|
+
}
|
|
48
|
+
JS
|
|
49
|
+
|
|
50
|
+
cat > src/report/daily.js <<'JS'
|
|
51
|
+
// One line per account that signed in on the given day, oldest first.
|
|
52
|
+
export function dailyReport(signIns, day) {
|
|
53
|
+
return signIns
|
|
54
|
+
.filter((entry) => entry.at.startsWith(day))
|
|
55
|
+
.sort((a, b) => a.at.localeCompare(b.at))
|
|
56
|
+
.map((entry) => `${entry.at} ${entry.account}`);
|
|
57
|
+
}
|
|
58
|
+
JS
|
|
59
|
+
|
|
60
|
+
cat > src/worker/nightly.js <<'JS'
|
|
61
|
+
// The scheduler calls this once a night with that night's jobs.
|
|
62
|
+
export async function nightly(jobs) {
|
|
63
|
+
for (const job of jobs) {
|
|
64
|
+
await job();
|
|
65
|
+
}
|
|
66
|
+
return jobs.length;
|
|
67
|
+
}
|
|
68
|
+
JS
|
|
69
|
+
|
|
70
|
+
cat > docs/specs/2026-09-01-counters.md <<'MD'
|
|
71
|
+
# Design: counters
|
|
72
|
+
|
|
73
|
+
## Context
|
|
74
|
+
|
|
75
|
+
Finance asked for counts of sign-ups and exports, and more counters will
|
|
76
|
+
follow. The service runs as one process that restarts on every deploy, so a
|
|
77
|
+
count held in a module variable would reset to zero each time.
|
|
78
|
+
|
|
79
|
+
### D1. Counters persist in data/counters.json
|
|
80
|
+
|
|
81
|
+
Every counter the service keeps, the first one and any later one, persists in
|
|
82
|
+
`data/counters.json`, one top-level key per counter, read and written whole.
|
|
83
|
+
The file is created on the first write. A counter is never held only in
|
|
84
|
+
memory, because a count that resets on deploy is wrong. A cache in front of
|
|
85
|
+
the file is fine; the file is the truth.
|
|
86
|
+
|
|
87
|
+
## Out of scope
|
|
88
|
+
|
|
89
|
+
A database. One file is enough at this traffic.
|
|
90
|
+
MD
|
|
91
|
+
|
|
92
|
+
cat > tests/daily.test.js <<'JS'
|
|
93
|
+
import assert from "node:assert/strict";
|
|
94
|
+
import { test } from "node:test";
|
|
95
|
+
import { dailyReport } from "../src/report/daily.js";
|
|
96
|
+
|
|
97
|
+
test("the report lists only the day's sign-ins, oldest first", () => {
|
|
98
|
+
const signIns = [
|
|
99
|
+
{ at: "2026-10-02T09:00:00Z", account: "bo" },
|
|
100
|
+
{ at: "2026-10-01T17:00:00Z", account: "al" },
|
|
101
|
+
{ at: "2026-10-01T08:00:00Z", account: "cy" },
|
|
102
|
+
];
|
|
103
|
+
assert.deepEqual(dailyReport(signIns, "2026-10-01"), [
|
|
104
|
+
"2026-10-01T08:00:00Z cy",
|
|
105
|
+
"2026-10-01T17:00:00Z al",
|
|
106
|
+
]);
|
|
107
|
+
});
|
|
108
|
+
JS
|
|
109
|
+
|
|
110
|
+
cat > README.md <<'MD'
|
|
111
|
+
# gate
|
|
112
|
+
|
|
113
|
+
The sign-in API (`src/api/`), the nightly worker (`src/worker/`) and the
|
|
114
|
+
reports (`src/report/`) run in one Node process, restarted on every deploy.
|
|
115
|
+
`npm test` runs the suite.
|
|
116
|
+
|
|
117
|
+
## Designs
|
|
118
|
+
|
|
119
|
+
A ticket's design goes in `docs/specs/<ticket>.md`, for example
|
|
120
|
+
`docs/specs/CNT-7.md`, and its plan in `docs/plans/<ticket>.md`.
|
|
121
|
+
MD
|
|
122
|
+
|
|
123
|
+
git init --quiet
|
|
124
|
+
git add -A
|
|
125
|
+
git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "gate 1.3.0"
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: regex
|
|
3
|
+
# The trace carries SKILL.md's own routing table, which names every channel, so
|
|
4
|
+
# `contains` passes whenever the skill loads. Anchor on the two ways the agent
|
|
5
|
+
# routes instead, the same ones run-stats.sh meters: a Bash call that invokes
|
|
6
|
+
# `status.sh route deep` (a quoted mention, as in a grep, does not count), or the
|
|
7
|
+
# words opening an assistant message or following a label such as "Sluice:".
|
|
8
|
+
pattern: '"command":"(?:(?:[^"\\]|\\.)*?(?:[\s/]|\\[nt]))?status\.sh(?:\\"|'')?(?:\s|\\[nt])+route(?:\s|\\[nt])+deep(?:\s|\\[nt]|[;&|]|")|"text":"(?:[^.!?\n"]{0,100}[:=]\s*)?[\s*_#>\\]*deep channel'
|
|
9
|
+
flags: i
|
|
10
|
+
target: trace
|
|
11
|
+
weight: 2
|
|
12
|
+
---
|
package/skills/sluice/evals/deep-plan-reads-the-prior-spec/graders/design-follows-the-prior-spec.md
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: llm
|
|
3
|
+
focus: { source: file, path: docs/specs/CNT-7.md }
|
|
4
|
+
weight: 3
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
This is the design for a ticket that counts failed sign-ins per account. The
|
|
8
|
+
repo already holds `docs/specs/2026-09-01-counters.md`, whose decision says
|
|
9
|
+
every counter the service keeps persists in `data/counters.json` and is never
|
|
10
|
+
held only in memory, because the process restarts on every deploy. The ticket
|
|
11
|
+
never mentions that spec, and since the API, worker and report share one
|
|
12
|
+
process, a counter held in memory is the obvious design.
|
|
13
|
+
|
|
14
|
+
PASS if the design persists the failed sign-in counts through
|
|
15
|
+
`data/counters.json`, the file the earlier spec settled (a cache in front of it
|
|
16
|
+
is fine), or if it names the conflict with the earlier spec and asks the
|
|
17
|
+
partner whether to supersede it.
|
|
18
|
+
FAIL if the design keeps the counts in memory (a Map, a module variable, an
|
|
19
|
+
in-process object) without naming the conflict with the earlier spec, or puts
|
|
20
|
+
them in some other store without saying why the earlier decision no longer
|
|
21
|
+
holds.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: tool_used
|
|
3
|
+
# The run plans and stops for sign-off, so no Edit may land under src/ or
|
|
4
|
+
# tests/. Bash is left out of the case's tools, so these two are the only ways
|
|
5
|
+
# code could change.
|
|
6
|
+
tool: Edit
|
|
7
|
+
input_match: '"file_path"\s*:\s*"(?:[^"]*/)?(?:src|tests)/'
|
|
8
|
+
min: 0
|
|
9
|
+
max: 0
|
|
10
|
+
weight: 2
|
|
11
|
+
---
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: tool_used
|
|
3
|
+
# The run plans and stops for sign-off, so no Write may land under src/ or
|
|
4
|
+
# tests/. Bash is left out of the case's tools, so these two are the only ways
|
|
5
|
+
# code could change.
|
|
6
|
+
tool: Write
|
|
7
|
+
input_match: '"file_path"\s*:\s*"(?:[^"]*/)?(?:src|tests)/'
|
|
8
|
+
min: 0
|
|
9
|
+
max: 0
|
|
10
|
+
weight: 2
|
|
11
|
+
---
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: deep-plan-reads-the-prior-spec
|
|
3
|
+
description: A deep plan whose obvious design contradicts a spec the repo already holds. Pins a read of the repo's prior specs before the design, cited in the new spec's Context.
|
|
4
|
+
tags: [routing, deep, write]
|
|
5
|
+
max_turns: 25
|
|
6
|
+
timeout_seconds: 900
|
|
7
|
+
allowed_tools: [Read, Glob, Grep, Skill, Write, Agent]
|
|
8
|
+
expected_outcome: Announces the deep channel, reads docs/specs/2026-09-01-counters.md before designing, writes docs/specs/CNT-7.md whose Context cites it and whose design persists the failed sign-in counts in data/counters.json (or asks whether to supersede the earlier spec), and writes no code.
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
Plan CNT-7 for us.
|
|
12
|
+
|
|
13
|
+
> Lock an account after five failed sign-ins in a row. The sign-in API counts
|
|
14
|
+
> the failures per account and refuses a locked account with a 423, the nightly
|
|
15
|
+
> worker clears every failure count (which is also what unlocks an account; a
|
|
16
|
+
> lock is nothing more than a count at five), a successful sign-in resets that
|
|
17
|
+
> account's count, and the daily report lists the accounts locked that day.
|