@slowdini/slow-powers-opencode 0.4.3 → 0.4.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md
CHANGED
|
@@ -26,7 +26,7 @@ Slow-powers guides agents through an evidence-backed, no-guess debugging approac
|
|
|
26
26
|
|
|
27
27
|
Skills for writing skills! Slow-powers skills are all written and evaluated following the same guidelines and processes it ships. Back up your own skills with real stats, and understand their cost in time and tokens.
|
|
28
28
|
|
|
29
|
-
Skill evaluations are powered by [
|
|
29
|
+
Skill evaluations are powered by [eval-magic](https://github.com/slowdini/eval-magic)
|
|
30
30
|
|
|
31
31
|
## Installation
|
|
32
32
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@slowdini/slow-powers-opencode",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.5",
|
|
4
4
|
"description": "Slow-powers — structured development workflows for coding agents (TDD, debugging, verification, git hygiene)",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./opencode/plugins/slow-powers.js",
|
|
@@ -36,19 +36,19 @@
|
|
|
36
36
|
},
|
|
37
37
|
"scripts": {
|
|
38
38
|
"test": "bun test --path-ignore-patterns='skills-workspace/**'",
|
|
39
|
-
"evals": "
|
|
40
|
-
"evals:snapshot": "
|
|
41
|
-
"evals:validate": "
|
|
42
|
-
"evals:ingest": "
|
|
43
|
-
"evals:finalize": "
|
|
44
|
-
"evals:record-runs": "
|
|
45
|
-
"evals:fill-transcripts": "
|
|
46
|
-
"evals:detect-stray-writes": "
|
|
47
|
-
"evals:teardown-guard": "
|
|
48
|
-
"evals:teardown": "
|
|
49
|
-
"evals:grade": "
|
|
50
|
-
"evals:aggregate": "
|
|
51
|
-
"evals:promote-baseline": "
|
|
39
|
+
"evals": "eval-magic run --skill-dir ./skills --bootstrap ./bootstrap.md",
|
|
40
|
+
"evals:snapshot": "eval-magic snapshot --skill-dir ./skills",
|
|
41
|
+
"evals:validate": "eval-magic validate --skill-dir ./skills",
|
|
42
|
+
"evals:ingest": "eval-magic ingest --skill-dir ./skills",
|
|
43
|
+
"evals:finalize": "eval-magic finalize --skill-dir ./skills",
|
|
44
|
+
"evals:record-runs": "eval-magic record-runs --skill-dir ./skills",
|
|
45
|
+
"evals:fill-transcripts": "eval-magic fill-transcripts --skill-dir ./skills",
|
|
46
|
+
"evals:detect-stray-writes": "eval-magic detect-stray-writes --skill-dir ./skills",
|
|
47
|
+
"evals:teardown-guard": "eval-magic teardown-guard --skill-dir ./skills",
|
|
48
|
+
"evals:teardown": "eval-magic teardown --skill-dir ./skills",
|
|
49
|
+
"evals:grade": "eval-magic grade --skill-dir ./skills",
|
|
50
|
+
"evals:aggregate": "eval-magic aggregate --skill-dir ./skills",
|
|
51
|
+
"evals:promote-baseline": "eval-magic promote-baseline --skill-dir ./skills",
|
|
52
52
|
"check": "biome check --write .",
|
|
53
53
|
"check:ci": "biome check --error-on-warnings .",
|
|
54
54
|
"typecheck": "tsc --noEmit",
|
|
@@ -5,7 +5,7 @@ description: Use when testing whether a new skill improves agent behavior, or wh
|
|
|
5
5
|
|
|
6
6
|
# Evaluating Skills
|
|
7
7
|
|
|
8
|
-
Skill development has two phases: **drafting** (`slow-powers:writing-skills`) and **evaluation** (this skill). This skill owns the *craft* of evaluation — deciding whether a change needs measuring, designing test cases, devising pressure-testing scenarios, writing assertions, and reading results. The *mechanics* of actually running an eval — building the workspace, staging skills, dispatching subagents, grading, aggregating — are owned by a dedicated tool, **[
|
|
8
|
+
Skill development has two phases: **drafting** (`slow-powers:writing-skills`) and **evaluation** (this skill). This skill owns the *craft* of evaluation — deciding whether a change needs measuring, designing test cases, devising pressure-testing scenarios, writing assertions, and reading results. The *mechanics* of actually running an eval — building the workspace, staging skills, dispatching subagents, grading, aggregating — are owned by a dedicated tool, **[eval-magic](https://github.com/slowdini/eval-magic)**, which ships as a dependency-less prebuilt binary you invoke as `eval-magic`. See [Running the eval](#running-the-eval) for the hand-off.
|
|
9
9
|
|
|
10
10
|
## Overview
|
|
11
11
|
|
|
@@ -95,7 +95,7 @@ A test case has these parts:
|
|
|
95
95
|
- **files** (optional): fixture files the prompt references
|
|
96
96
|
- **skill_should_trigger** (optional, default `true`): set `false` for a *negative* eval where correct behavior is the skill **not** firing (e.g. an over-trigger guard — a feature request that shouldn't launch a debugging investigation). Negative evals are excluded from the skill-invocation rate, so a correct non-invocation isn't mistaken for the skill failing to fire.
|
|
97
97
|
|
|
98
|
-
Cases live in `<skill>/evals/evals.json`. For the file shape, see the
|
|
98
|
+
Cases live in `<skill>/evals/evals.json`. For the file shape, see the author-template example in the eval-magic README and validate against the bundled schema with `eval-magic validate`; for worked, maintained examples, read the live suites in this repo — e.g. `skills/verifying-development-work/evals/evals.json` and `skills/hardening-plans/evals/evals.json`.
|
|
99
99
|
|
|
100
100
|
Tips for writing good prompts:
|
|
101
101
|
|
|
@@ -142,7 +142,7 @@ Keep the seeded turns short and concrete; the point is to establish momentum, no
|
|
|
142
142
|
|
|
143
143
|
**The ceiling — state it plainly.** A seed is *text the subagent reads*, not a state it operates under. It cannot place the agent in a harness-injected mode — a real plan mode, an enforced multi-phase workflow, genuine context-window pressure — it can only *describe* one. So when the wild failure you're chasing was *caused* by such a mode (the documented case: an agent in plan mode that invoked **zero** skills because the mode's own procedure made loading them feel redundant), a text seed cannot fully reproduce it — the causal layer is exactly the one a prompt string can't inject. A seeded **pass is therefore necessary but not sufficient** — it under-estimates real-session difficulty — and a seed that *fails* to reproduce a known wild failure is usually hitting this ceiling, not testing a bad seed. Treat seeded results as a stronger-than-cold signal, not as ground truth, and don't let downstream work over-trust them.
|
|
144
144
|
|
|
145
|
-
**Narrowing the gap — `--plan-mode`.** For the documented plan-mode case, the runner offers the highest-fidelity in-runner approximation: its `--plan-mode` flag injects the harness's *verbatim* plan-mode procedure into every dispatch as an operating-context layer the subagent is told it is operating under, rather than a paraphrase the agent merely reads in the seed prose. This narrows the gap (verbatim procedure > paraphrase) but does **not** close it: it is still text the agent reads, not an injected mode, so the necessary-not-sufficient ceiling above stands unchanged. Use it as the strongest in-runner signal and pair it with a paraphrase-seed arm. See
|
|
145
|
+
**Narrowing the gap — `--plan-mode`.** For the documented plan-mode case, the runner offers the highest-fidelity in-runner approximation: its `--plan-mode` flag injects the harness's *verbatim* plan-mode procedure into every dispatch as an operating-context layer the subagent is told it is operating under, rather than a paraphrase the agent merely reads in the seed prose. This narrows the gap (verbatim procedure > paraphrase) but does **not** close it: it is still text the agent reads, not an injected mode, so the necessary-not-sufficient ceiling above stands unchanged. Use it as the strongest in-runner signal and pair it with a paraphrase-seed arm. See `eval-magic run --help` for the flag and the per-harness profiles it depends on.
|
|
146
146
|
|
|
147
147
|
## Writing assertions
|
|
148
148
|
|
|
@@ -186,19 +186,19 @@ Once a run is graded and aggregated, the headline is the **delta**: what the ski
|
|
|
186
186
|
|
|
187
187
|
## Running the eval
|
|
188
188
|
|
|
189
|
-
The mechanics of executing a run live in **[
|
|
189
|
+
The mechanics of executing a run live in **[eval-magic](https://github.com/slowdini/eval-magic)** — the `eval-magic` binary. eval-magic's README is the complete operating guide, and every flag is documented in the tool's own help.
|
|
190
190
|
|
|
191
191
|
| Need | Where |
|
|
192
192
|
|------|-------|
|
|
193
|
-
| Quickstart, install, the two modes end-to-end | the
|
|
194
|
-
| Every subcommand and flag; the `--skill-dir` model; workspace layout | `
|
|
195
|
-
| Full run mechanics: dispatch loop, transcript access, grading, aggregating, baselines |
|
|
196
|
-
| Claude Code
|
|
193
|
+
| Quickstart, install, the two modes end-to-end | the eval-magic README |
|
|
194
|
+
| Every subcommand and flag; the `--skill-dir` model; workspace layout | `eval-magic --help` and `eval-magic <subcommand> --help` |
|
|
195
|
+
| Full run mechanics: dispatch loop, transcript access, grading, aggregating, baselines | the eval-magic README |
|
|
196
|
+
| Claude Code & Codex harness specifics — isolating from installed plugins, the guard, judging | the README's Harnesses section |
|
|
197
197
|
| What a harness needs to reach Claude-Code-tier support | `docs/harness-parity.md` |
|
|
198
198
|
|
|
199
199
|
## See also
|
|
200
200
|
|
|
201
201
|
- `slow-powers:writing-skills` — drafting a skill (Phase 1)
|
|
202
202
|
- `pressure-scenarios.md` — pressure-scenario taxonomy for authoring prompts that stress discipline-enforcing skills
|
|
203
|
-
-
|
|
203
|
+
- eval-magic (the `eval-magic` tool) — runs the evals this skill teaches you to author
|
|
204
204
|
- agentskills.io/skill-creation/evaluating-skills — the methodology this skill is derived from
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Baseline — evaluating-skills
|
|
2
2
|
|
|
3
3
|
Committed reference output from a canonical eval run. Regenerate with
|
|
4
|
-
`
|
|
4
|
+
`eval-magic promote-baseline --skill evaluating-skills --iteration <N>` after aggregating. The ephemeral workspace (run records, timing,
|
|
5
5
|
dispatch files, produced outputs) stays gitignored under `skills-workspace/`
|
|
6
|
-
and is reclaimable by `
|
|
6
|
+
and is reclaimable by `eval-magic teardown` once promoted (this commit's marker).
|
|
7
7
|
|
|
8
8
|
| Field | Value |
|
|
9
9
|
|-------|-------|
|
|
@@ -20,4 +20,3 @@ and is reclaimable by `skill-eval teardown` once promoted (this commit's marker)
|
|
|
20
20
|
Files:
|
|
21
21
|
- `benchmark.json` — aggregate pass-rate / duration / token deltas.
|
|
22
22
|
- `grading/<eval-id>__<condition>.json` — per-run assertion results and judge rationales.
|
|
23
|
-
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Baseline — hardening-plans
|
|
2
2
|
|
|
3
3
|
Committed reference output from a canonical eval run. Regenerate with
|
|
4
|
-
`
|
|
4
|
+
`eval-magic promote-baseline --skill hardening-plans --iteration <N>` after aggregating. The ephemeral workspace (run records, timing,
|
|
5
5
|
dispatch files, produced outputs) stays gitignored under `skills-workspace/`
|
|
6
|
-
and is reclaimable by `
|
|
6
|
+
and is reclaimable by `eval-magic teardown` once promoted (this commit's marker).
|
|
7
7
|
|
|
8
8
|
| Field | Value |
|
|
9
9
|
|-------|-------|
|
|
@@ -24,4 +24,3 @@ the named-hand-off requirement). `new_skill` = the working tree at promotion
|
|
|
24
24
|
Files:
|
|
25
25
|
- `benchmark.json` — aggregate pass-rate / duration / token deltas.
|
|
26
26
|
- `grading/<eval-id>__<condition>.json` — per-run assertion results and judge rationales.
|
|
27
|
-
|