cyber-aced 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +18 -0
- package/.plugin/pins.json +1 -0
- package/.plugin/plugin.json +18 -0
- package/agents/aced-case-judge.md +165 -0
- package/agents/aced-impl-judge.md +55 -0
- package/agents/aced-scenario-writer.md +76 -0
- package/agents/aced-spec-validator.md +66 -0
- package/governances/README.md +52 -0
- package/governances/agent-tool-output.md +86 -0
- package/governances/cli-resolution.md +102 -0
- package/governances/skill-design.md +258 -0
- package/governances/skill-repo-structure.md +173 -0
- package/package.json +44 -0
- package/readme.md +124 -0
- package/skills/aced-architect-impl/README.md +9 -0
- package/skills/aced-architect-impl/SKILL.md +73 -0
- package/skills/aced-builder-impl/README.md +14 -0
- package/skills/aced-builder-impl/SKILL.md +46 -0
- package/skills/aced-builder-spec/README.md +15 -0
- package/skills/aced-builder-spec/SKILL.md +130 -0
- package/skills/aced-fit/README.md +21 -0
- package/skills/aced-fit/SKILL.md +105 -0
- package/skills/aced-impl-producer/SKILL.md +75 -0
- package/skills/add-scenario/README.md +25 -0
- package/skills/add-scenario/SKILL.md +75 -0
- package/skills/bench/README.md +64 -0
- package/skills/bench/SKILL.md +199 -0
- package/skills/bench/scripts/bench-claude-code.mts +187 -0
- package/skills/bench/scripts/bench-compare.mts +369 -0
- package/skills/bench/scripts/bench-record.mts +133 -0
- package/skills/bench/scripts/bench.mts +930 -0
- package/skills/check-freshness/README.md +57 -0
- package/skills/check-freshness/SKILL.md +76 -0
- package/skills/check-freshness/scripts/check-freshness.mts +399 -0
- package/skills/compare/README.md +25 -0
- package/skills/compare/SKILL.md +87 -0
- package/skills/contribute-skill/README.md +25 -0
- package/skills/contribute-skill/SKILL.md +241 -0
- package/skills/contribute-skill/scripts/source.mts +188 -0
- package/skills/define-agent/README.md +40 -0
- package/skills/define-agent/SKILL.md +279 -0
- package/skills/define-command/README.md +36 -0
- package/skills/define-command/SKILL.md +174 -0
- package/skills/define-governance/SKILL.md +151 -0
- package/skills/define-skill/README.md +37 -0
- package/skills/define-skill/SKILL.md +200 -0
- package/skills/extract-situation/README.md +32 -0
- package/skills/extract-situation/SKILL.md +54 -0
- package/skills/extract-situation/scripts/extract-situation.mts +287 -0
- package/skills/improve/README.md +25 -0
- package/skills/improve/SKILL.md +100 -0
- package/skills/improve-skill/README.md +38 -0
- package/skills/improve-skill/SKILL.md +169 -0
- package/skills/improve-skill/references/check-definitions.md +215 -0
- package/skills/improve-skill/scripts/validate.mts +1033 -0
- package/skills/init-aced/README.md +5 -0
- package/skills/init-aced/SKILL.md +76 -0
- package/skills/init-aced/scripts/ensure-results-ignored.mts +117 -0
- package/skills/list-skills/README.md +19 -0
- package/skills/list-skills/SKILL.md +55 -0
- package/skills/list-skills/scripts/list-skills.mts +210 -0
- package/skills/manage/README.md +35 -0
- package/skills/manage/SKILL.md +65 -0
- package/skills/manage-model-runners/README.md +25 -0
- package/skills/manage-model-runners/SKILL.md +113 -0
- package/skills/manage-skill-dirs/README.md +17 -0
- package/skills/manage-skill-dirs/SKILL.md +54 -0
- package/skills/manage-skill-dirs/scripts/manage-skill-dirs.mts +328 -0
- package/skills/repair-private-skills/README.md +17 -0
- package/skills/repair-private-skills/SKILL.md +59 -0
- package/skills/repair-private-skills/scripts/repair-private-skills.mts +279 -0
- package/skills/report/README.md +25 -0
- package/skills/report/SKILL.md +74 -0
- package/skills/run/README.md +25 -0
- package/skills/run/SKILL.md +220 -0
- package/skills/skillify/README.md +25 -0
- package/skills/skillify/SKILL.md +141 -0
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# define-skill
|
|
2
|
+
|
|
3
|
+
Create or improve a workflow skill — a process, tool-based, or standard SKILL.md — then hand it to the ACED eval loop to spec and score.
|
|
4
|
+
|
|
5
|
+
## When to use
|
|
6
|
+
|
|
7
|
+
Use this skill when you want to author a new workflow skill or formalize an existing ad-hoc workflow into one.
|
|
8
|
+
|
|
9
|
+
Good triggers include:
|
|
10
|
+
|
|
11
|
+
- "Create a skill that does X" / "I want a skill for X"
|
|
12
|
+
- Turning a documented multi-step process into a reusable skill
|
|
13
|
+
- Filling out an incomplete or weakly-triggering existing SKILL.md
|
|
14
|
+
- Scaffolding the SKILL.md + README + runtime symlinks in one pass
|
|
15
|
+
|
|
16
|
+
Defer to a sibling when the request is really about an agent or persona (`define-agent`), a reference-only rule set (`define-governance`), extracting the current session (`skillify`), or scoring / diagnosing an existing config (`run` / `add-scenario` / `improve`).
|
|
17
|
+
|
|
18
|
+
## What it does
|
|
19
|
+
|
|
20
|
+
The skill walks you through the shape before writing anything:
|
|
21
|
+
|
|
22
|
+
1. **Route** — confirm this is a workflow skill, not an agent, governance, or session extraction
|
|
23
|
+
2. **Settle scope** — the five design questions (scope, trigger, output contract, quality bar, out-of-scope)
|
|
24
|
+
3. **Pattern** — process, tool-based, or standard (personas go to `define-agent`)
|
|
25
|
+
4. **Placement** — user-global, project-private, or project-public, plus which runtime symlinks to create
|
|
26
|
+
|
|
27
|
+
It then drafts the SKILL.md (kebab-case name, a 150–400 char trigger-bearing description, a step body under the size bar), adds a README for a public skill, runs the structural audit and fixes CRITICAL/HIGH findings, and points you at the ACED eval loop (`start-mission` / `add-scenario` / `run`) to spec and score it.
|
|
28
|
+
|
|
29
|
+
## Enhanced from create-skill
|
|
30
|
+
|
|
31
|
+
`define-skill` is the ACED-native successor to the legacy `create-skill` skill: same scaffold → audit → link lifecycle, but the test step is the ACED eval loop (a frozen `.feature` with inline `@rubric` scenarios scored over N runs) instead of a one-shot trigger-query file, and fit is classified up front so authoring effort matches the subject.
|
|
32
|
+
|
|
33
|
+
## Install
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
npx skills add cyberuni/cyberplace --skill aced/define-skill
|
|
37
|
+
```
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: define-skill
|
|
3
|
+
description: 'Use this skill when the user wants to create, scaffold, or formalize a workflow skill — a process, tool-based, or standard SKILL.md — from scratch or from an existing ad-hoc workflow. Trigger on "create a skill", "write a skill for X", "I want a skill that…", "turn this workflow into a skill", or "help me build this skill", even if they do not say "skill" explicitly. Not for agent definitions or personas (define-agent), reference-only rule sets (define-governance), or extracting the current session (skillify).'
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Define Skill
|
|
7
|
+
|
|
8
|
+
Create or improve a **workflow skill** — a process, tool-based, or standard SKILL.md — then hand it
|
|
9
|
+
to the ACED eval loop to spec and score.
|
|
10
|
+
|
|
11
|
+
When the conductor dispatches this skill as a generic builder (`produced-by sdd:automaton`) for the
|
|
12
|
+
ACED **impl-producer** role (implement mode, against a frozen `.feature`), it builds the **SKILL.md**
|
|
13
|
+
to pass the frozen suite. The **verification is the frozen `.feature` itself** — its inline `@rubric`
|
|
14
|
+
scenarios and `@trigger` `Examples`, authored by `aced-scenario-writer` at explore — so no separate
|
|
15
|
+
eval suite is authored here; `eval.md` carries only the `subject` binding and run policy. As
|
|
16
|
+
impl-producer it self-aligns to `sdd:ownership-governance` plus the resolved **builder-impl +
|
|
17
|
+
architect-impl** bars (the ACED builder-impl is `aced:aced-builder-impl`). If the impl-judge reports
|
|
18
|
+
scenario failures, load `aced-impl-producer` to run the diagnose-and-refine loop rather than
|
|
19
|
+
re-deriving it here.
|
|
20
|
+
|
|
21
|
+
There are two other entry points, both with **no frozen `.feature`** and **only the SKILL.md**
|
|
22
|
+
produced:
|
|
23
|
+
|
|
24
|
+
- **Standalone** — the user invokes this skill directly, outside any CR. Scaffold, then offer the
|
|
25
|
+
ACED eval loop (see "Report and hand off" below) rather than assuming it.
|
|
26
|
+
- **Escaped** — the gateway or `start-mission` invokes this skill directly after resolving the
|
|
27
|
+
request `non-durable` (the escape hatch, before any CR opens). There is no mission to hand off
|
|
28
|
+
within: scaffold, audit, report, and **stop** — do not mention the ACED eval loop.
|
|
29
|
+
|
|
30
|
+
## Route the request first
|
|
31
|
+
|
|
32
|
+
`define-skill` owns **workflow skills**. Defer when the intent belongs to a sibling that carries the
|
|
33
|
+
same config vocabulary:
|
|
34
|
+
|
|
35
|
+
| The request is really about… | Defer to |
|
|
36
|
+
|---|---|
|
|
37
|
+
| a named reusable **agent** / delegated worker / an in-context **persona** | `define-agent` |
|
|
38
|
+
| a **reference-only rule set / governance** other skills load but never execute as steps | `define-governance` |
|
|
39
|
+
| generalizing **the current session's** work into a skill | `skillify` |
|
|
40
|
+
| **scoring** an existing config, or **adding** an eval case | `run` / `add-scenario` |
|
|
41
|
+
| **diagnosing** why an existing skill's evals fail | `improve` |
|
|
42
|
+
|
|
43
|
+
A persona request that surfaces mid-authoring is handed to `define-agent` — do not scaffold a persona
|
|
44
|
+
skill here.
|
|
45
|
+
|
|
46
|
+
## Load the governance
|
|
47
|
+
|
|
48
|
+
Before writing content, load the **skill-design** governance and read stdout as the authoritative
|
|
49
|
+
rules for principles, progressive disclosure, description structure, and when to extract deterministic
|
|
50
|
+
logic to a script:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
npx cyberplace@0.2.4 governance show skill-design # resolve <version> via: npm view cyberplace version — never @latest
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
If the skill will ship a `scripts/` directory or document CLI commands agents run, also load
|
|
57
|
+
**agent-tool-output** for stdout / JSON / non-interactive / stderr rules.
|
|
58
|
+
|
|
59
|
+
## Settle the shape — five questions
|
|
60
|
+
|
|
61
|
+
Answer all five before writing a line of SKILL.md. If any cannot be inferred from what the user has
|
|
62
|
+
said, **ask the user to resolve it** — never invent it:
|
|
63
|
+
|
|
64
|
+
| Question | What to decide |
|
|
65
|
+
|---|---|
|
|
66
|
+
| Scope | What exactly does this skill do? One workflow only. |
|
|
67
|
+
| Trigger condition | When should it fire? List both explicit and implicit phrasings. |
|
|
68
|
+
| Output contract | What artifact does it produce — a file path, a report format, a decision? |
|
|
69
|
+
| Quality bar | How will you know a run succeeded? A concrete pass condition. |
|
|
70
|
+
| Out of scope | What must this skill explicitly NOT do? |
|
|
71
|
+
|
|
72
|
+
## Choose the pattern
|
|
73
|
+
|
|
74
|
+
Capture the workflow shape (separate from placement). A persona is **not** a pattern here — route it
|
|
75
|
+
to `define-agent`.
|
|
76
|
+
|
|
77
|
+
| Pattern | When to use it | Body shape |
|
|
78
|
+
|---|---|---|
|
|
79
|
+
| Process | Ordered multi-step workflow with decision points | Numbered steps |
|
|
80
|
+
| Tool-based | Consistent use of tools, systems, or connectors | Tool usage + guardrails |
|
|
81
|
+
| Standard | Tone, format, structure, or quality enforcement | Rules + pass conditions |
|
|
82
|
+
|
|
83
|
+
## Resolve placement and runtimes
|
|
84
|
+
|
|
85
|
+
Pick placement before scaffolding; if unclear from context, ask:
|
|
86
|
+
|
|
87
|
+
| Placement | Location | Notes |
|
|
88
|
+
|---|---|---|
|
|
89
|
+
| User-global | `~/.agents/skills/<name>/` | Personal, across all projects; no README |
|
|
90
|
+
| Project-private | `.agents/skills/<name>/` | Contributor tooling; add `metadata: internal: true` |
|
|
91
|
+
| Project-public | `skills/<name>/` (or `plugins/<plugin>/skills/<name>/`) | Shipped with a package; **requires a README** |
|
|
92
|
+
|
|
93
|
+
Check whether `npx skills` is available, then scaffold at the derived path:
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
npx skills --version 2>/dev/null && npx skills init <name> --dir <placement-dir> # else: mkdir -p <placement-dir>/<name>
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Ask which runtimes to target; the SKILL.md is the canonical file, each runtime location is a symlink
|
|
100
|
+
to it. Create and **verify** one symlink per selected runtime:
|
|
101
|
+
|
|
102
|
+
| Runtime | Symlink location |
|
|
103
|
+
|---|---|
|
|
104
|
+
| Claude Code | `~/.claude/skills/<name>` (or `.claude/skills/<name>`) |
|
|
105
|
+
| Cursor | `~/.cursor/skills/<name>` |
|
|
106
|
+
| Codex | `~/.codex/skills/<name>` |
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
ln -sf <relative-path-to-canonical> <runtime-location> && ls <runtime-location> # verify it resolves
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
## Draft the SKILL.md
|
|
113
|
+
|
|
114
|
+
Write the file at the canonical path:
|
|
115
|
+
|
|
116
|
+
```markdown
|
|
117
|
+
---
|
|
118
|
+
name: <name> # kebab-case, must match the directory name exactly
|
|
119
|
+
description: <capability> + "Use when <trigger>" + an implicit-phrasing example ("even if they say …")
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
# <Title>
|
|
123
|
+
|
|
124
|
+
## When to use
|
|
125
|
+
<the trigger condition, restated for the body>
|
|
126
|
+
|
|
127
|
+
## Instructions
|
|
128
|
+
1. <step>
|
|
129
|
+
2. <step>
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
- **Description** — the only field loaded at startup; it carries the whole triggering burden. Target
|
|
133
|
+
150–400 characters (≤1024 hard limit): capability + "Use when…" + an implicit phrasing. For a
|
|
134
|
+
**partial skill** (a reusable part other skills call by name, not user-triggered), set
|
|
135
|
+
`user-invocable: false` and lead the description with the `"Partial Skill:"` prefix (recommended
|
|
136
|
+
form `"Partial Skill: invoke by name only — <identity>. <caller>."`), kept minimal and
|
|
137
|
+
non-trigger-shaped so it does not self-activate.
|
|
138
|
+
- **Body** — step-by-step, under 500 lines. Keep the *when* here; move any **deterministic,
|
|
139
|
+
fixed-output** step to a `scripts/` file or an existing CLI and have the body only say when to run
|
|
140
|
+
it.
|
|
141
|
+
- **Project-public** skills get a `README.md` beside the SKILL.md (title, when-to-use, what-it-does,
|
|
142
|
+
install line). User-global skills get none.
|
|
143
|
+
- **Binding rules get a `## Validate` section.** If the body states rules the agent must, must not,
|
|
144
|
+
always, or never do — most commonly the **Standard** pattern, but any pattern can carry one —
|
|
145
|
+
add a `## Validate` section: one assertion per rule, phrased as an observable, falsifiable claim
|
|
146
|
+
about the resulting artifact (a diff, a file's presence, a structural shape), not a restatement of
|
|
147
|
+
the rule or a claim about the agent's intent. Write it so it works run twice, from two vantage
|
|
148
|
+
points: as the agent's own pre-handoff self-check before declaring the work done, and as an
|
|
149
|
+
independent re-check by a cold reader later, against the same artifact rather than against
|
|
150
|
+
anything the first agent said about itself. Prefer a mechanical check (script, grep, diff) wherever
|
|
151
|
+
the rule permits one. A skill whose body is only ordered steps with no binding rules on the
|
|
152
|
+
*outcome* does not need one — this is for rules a result can violate, not for the steps themselves.
|
|
153
|
+
|
|
154
|
+
### Improving an existing skill
|
|
155
|
+
|
|
156
|
+
If the target SKILL.md already exists, **read it first** and change **only** the gaps or issues found
|
|
157
|
+
— leave every sound part intact. This fills gaps in a skill's *definition*; diagnosing failing evals
|
|
158
|
+
is `improve`.
|
|
159
|
+
|
|
160
|
+
## Name gate/case scorers by role
|
|
161
|
+
|
|
162
|
+
For a subagent this skill scaffolds (realized as a partial skill, e.g. loaded via the ACED
|
|
163
|
+
impl-producer/impl-judge pattern): if its role is to **score or verify a specific gate or case**,
|
|
164
|
+
name it by that gate/scope, not a bare action verb — `<domain>-<gate>-judge` for a gate scorer (e.g.
|
|
165
|
+
`aced-impl-judge`), `<domain>-case-judge` for a case scorer (e.g. `aces-case-judge`). Reject
|
|
166
|
+
`implementer`, `judge`, `validator`, `reviewer`, `checker` alone. A producer subagent (e.g.
|
|
167
|
+
`scenario-writer`, `doc-writer`) keeps its action-oriented name — this check does not fire for it.
|
|
168
|
+
|
|
169
|
+
Evaluate this **in-skill quality check** — separate from the mechanical `audit validate` below — before
|
|
170
|
+
handing the skill back: a drafted gate/case scorer named with a bare action verb is a **HIGH** severity
|
|
171
|
+
finding; fix it (rename to the gate-and-scope form) before presenting the skill.
|
|
172
|
+
|
|
173
|
+
## Audit before handing back
|
|
174
|
+
|
|
175
|
+
Run the structural audit and fix any CRITICAL or HIGH finding **before** presenting the skill:
|
|
176
|
+
|
|
177
|
+
```bash
|
|
178
|
+
npx cyberplace@0.2.4 audit validate --path <placement-dir>/<name>
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
For a fuller pass, invoke the `improve-skill` skill. Do not present a skill with an
|
|
182
|
+
open CRITICAL finding.
|
|
183
|
+
|
|
184
|
+
## Report and hand off to the ACED eval loop
|
|
185
|
+
|
|
186
|
+
Summarize:
|
|
187
|
+
|
|
188
|
+
- Canonical SKILL.md path
|
|
189
|
+
- README path (project-public only)
|
|
190
|
+
- Runtime symlinks created and verified
|
|
191
|
+
- Audit outcome
|
|
192
|
+
|
|
193
|
+
**Escaped** entry: stop here — the artifact resolved `non-durable`, so there is no mission to hand
|
|
194
|
+
off to.
|
|
195
|
+
|
|
196
|
+
**Standalone or impl-producer** entry: point the user at the **ACED eval loop** to spec and score
|
|
197
|
+
the skill — run `sdd:start-mission` (the conductor resolves the ACED roles for the `skill`
|
|
198
|
+
artifact-type) to author its frozen `.feature` (with inline `@rubric`), or `add-scenario` / `run` to
|
|
199
|
+
grow and score it. Do **not** embed a legacy trigger-query eval file as the test step — scoring is the
|
|
200
|
+
ACED loop's job.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# extract-situation
|
|
2
|
+
|
|
3
|
+
Internal, non-invokable ACED engine — loaded by `aced-case-judge`, not triggered directly. It reads
|
|
4
|
+
one scenario out of a frozen `.feature` and emits only the situation a simulating agent may see.
|
|
5
|
+
|
|
6
|
+
## What it does
|
|
7
|
+
|
|
8
|
+
Parses the `.feature`, finds the named scenario, and emits its `Given` and `When` steps verbatim, in
|
|
9
|
+
file order. It withholds the scenario name, every `Then`, the inline `@rubric` docstring, the tags,
|
|
10
|
+
the `Feature:` description, and every sibling scenario.
|
|
11
|
+
|
|
12
|
+
An `And` inherits the keyword above it, so `Then A / And B` withholds `And B` while
|
|
13
|
+
`Given X / And Y` emits `And Y`. For a `Scenario Outline`, `<placeholder>` tokens are preserved and
|
|
14
|
+
an `Examples` column survives only when an emitted step references it.
|
|
15
|
+
|
|
16
|
+
Every failure exits non-zero and emits nothing: an unreadable or unparseable file, a scenario name
|
|
17
|
+
that is absent or matches twice, or a missing argument.
|
|
18
|
+
|
|
19
|
+
## Why it exists
|
|
20
|
+
|
|
21
|
+
ACED's per-case scorer used to be handed a scenario's name, its `Then`, and its rubric — and then
|
|
22
|
+
asked to simulate an agent and score that simulation in one context. The simulator was told the
|
|
23
|
+
answer before it answered, so a passing score could not distinguish an agent that reasons from one
|
|
24
|
+
handed the key. This engine composes the simulator's brief mechanically, so the redaction is done by
|
|
25
|
+
tested code rather than by a judge deciding what to withhold from itself.
|
|
26
|
+
|
|
27
|
+
## Usage
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
node "<skill>/scripts/extract-situation.mts" --feature <path-to-.feature> --scenario "<exact name>"
|
|
31
|
+
node "<skill>/scripts/extract-situation.mts" --feature <path> --scenario "<name>" --format json
|
|
32
|
+
```
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: extract-situation
|
|
3
|
+
description: "Partial Skill: invoke by name only — the ACED blind-brief extractor — loaded by aced-case-judge to compose a simulating context's brief, not user-triggered."
|
|
4
|
+
user-invocable: false
|
|
5
|
+
metadata:
|
|
6
|
+
internal: true
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# extract-situation
|
|
10
|
+
|
|
11
|
+
The **blind-brief extractor** — the deterministic engine that reads one scenario out of a frozen
|
|
12
|
+
`.feature` and emits **only the situation**: its `Given` and `When` steps. The scenario **name**,
|
|
13
|
+
every `Then`, the inline `@rubric`, and the tags are **never** emitted. `aced-case-judge` composes a
|
|
14
|
+
simulating context's brief **only** through this engine; it never decides by its own judgment what to
|
|
15
|
+
withhold. It is **read-only** — it never writes a `.feature`, a brief, or any other file.
|
|
16
|
+
|
|
17
|
+
## Invocation
|
|
18
|
+
|
|
19
|
+
```
|
|
20
|
+
node "<skill>/scripts/extract-situation.mts" --feature <path-to-.feature> --scenario "<exact name>"
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
`--format json` emits `{ steps, given, when, placeholders, examples }` instead of the markdown brief —
|
|
24
|
+
`steps` is the emitted brief in file order (`given`/`when` regroup the same steps by keyword), and
|
|
25
|
+
`examples` carries a `Scenario Outline`'s surviving columns.
|
|
26
|
+
|
|
27
|
+
## What it emits, and what it must never emit
|
|
28
|
+
|
|
29
|
+
| Element | Emitted | Why |
|
|
30
|
+
|---|---|---|
|
|
31
|
+
| `Given` steps (and an `And`/`But` inheriting `Given`) | **yes** | the situation the agent is in |
|
|
32
|
+
| `When` steps (and an `And`/`But` inheriting `When`) | **yes** | the act under evaluation |
|
|
33
|
+
| Scenario **name** | never | names state the verdict (`red tests block the commit`) |
|
|
34
|
+
| `Then` steps, and any `And`/`But` inheriting `Then` | never | the expected outcome is the answer key |
|
|
35
|
+
| Inline `@rubric` docstring | never | the scoring ladders are worked answers |
|
|
36
|
+
| Tags, `Feature:` description, sibling scenarios | never | not the situation |
|
|
37
|
+
|
|
38
|
+
**Keyword inheritance is the load-bearing rule.** An `And` takes the keyword of the step above it:
|
|
39
|
+
`Given X / And Y` makes `And Y` a `Given` (emit); `Then A / And B` makes `And B` a `Then` (withhold).
|
|
40
|
+
Tracking the current keyword is what keeps the answer key out of the brief.
|
|
41
|
+
|
|
42
|
+
For a `Scenario Outline`, `<placeholder>` tokens stay intact and an `Examples` column is emitted
|
|
43
|
+
**only when an emitted step references its placeholder** — a column only a `Then` reads is withheld.
|
|
44
|
+
|
|
45
|
+
## Fail closed
|
|
46
|
+
|
|
47
|
+
Every failure exits non-zero and emits **no brief** — an empty brief must never be mistakable for a
|
|
48
|
+
valid one. Unreadable or unparseable `.feature`; scenario name absent; the name matching **two**
|
|
49
|
+
scenarios; `--feature` or `--scenario` missing.
|
|
50
|
+
|
|
51
|
+
## References
|
|
52
|
+
|
|
53
|
+
The contract this engine serves: `aced-case-judge`'s blind two-pass protocol, spec'd at
|
|
54
|
+
`.agents/specs/aced/sdd-roles/judge/`.
|
|
@@ -0,0 +1,287 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// extract-situation — mechanically redacts a named .feature scenario down to its SITUATION: the
|
|
3
|
+
// Given/When steps only. This is the fix for the "eval cannot fail" defect in
|
|
4
|
+
// plugins/aced/agents/aced-case-judge.md — that judge is handed the scenario's Then/And/But
|
|
5
|
+
// assertions AND its inline @rubric in the SAME context it simulates the subject in, so the
|
|
6
|
+
// simulator sees the answer key before it acts. A simulating context must see ONLY this engine's
|
|
7
|
+
// output: never the scenario name, never a Then/And/But, never a docstring, never a tag, never a
|
|
8
|
+
// sibling scenario.
|
|
9
|
+
//
|
|
10
|
+
// The keyword-inheritance rule is the load-bearing logic: `And`/`But` inherit the keyword of the
|
|
11
|
+
// concrete step above them (`Given X` / `And Y` → Y is a Given; `Then A` / `And B` → B is a
|
|
12
|
+
// Then). Each step's EFFECTIVE keyword — never its literal `And`/`But` text — decides inclusion.
|
|
13
|
+
//
|
|
14
|
+
// Pure functions are exported for node:test; running the file directly drives the CLI. No
|
|
15
|
+
// dependencies — plain node strips the types.
|
|
16
|
+
|
|
17
|
+
import { readFileSync, realpathSync } from 'node:fs'
|
|
18
|
+
import { pathToFileURL } from 'node:url'
|
|
19
|
+
|
|
20
|
+
// ─── types ────────────────────────────────────────────────────────────────────
|
|
21
|
+
|
|
22
|
+
export type ConcreteKeyword = 'Given' | 'When' | 'Then'
|
|
23
|
+
|
|
24
|
+
// An `And`/`But` with no concrete step above it has no keyword to inherit. It resolves to
|
|
25
|
+
// `unresolved` and is withheld: a step whose keyword cannot be established is never emitted.
|
|
26
|
+
export type EffectiveKeyword = ConcreteKeyword | 'unresolved'
|
|
27
|
+
|
|
28
|
+
export interface Step {
|
|
29
|
+
text: string
|
|
30
|
+
effectiveKeyword: EffectiveKeyword
|
|
31
|
+
// A docstring belongs to the step above it and inherits that step's fate. Under a `Given`/`When`
|
|
32
|
+
// it IS the situation (the prompt under test); under a `Then` it is the answer key (the rubric).
|
|
33
|
+
docstring: { fence: string; lines: string[] } | null
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
export interface ParsedScenario {
|
|
37
|
+
name: string
|
|
38
|
+
steps: Step[]
|
|
39
|
+
isOutline: boolean
|
|
40
|
+
examples: { header: string[]; rows: string[][] } | null
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
export interface Situation {
|
|
44
|
+
// Every emitted step, in the order the file lists them. `given`/`when` regroup the same steps
|
|
45
|
+
// by keyword; the brief is rendered from `steps`, so an interleaved scenario reads as written.
|
|
46
|
+
steps: string[]
|
|
47
|
+
given: string[]
|
|
48
|
+
when: string[]
|
|
49
|
+
placeholders: string[]
|
|
50
|
+
examples: { header: string[]; rows: string[][] } | null
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
// ─── parse ────────────────────────────────────────────────────────────────────
|
|
54
|
+
|
|
55
|
+
// Finds every scenario in the file (name, steps with resolved keywords, outline/Examples). Steps
|
|
56
|
+
// carry the ORIGINAL line text (minus indentation) so the brief is verbatim; only the keyword used
|
|
57
|
+
// for inclusion is resolved.
|
|
58
|
+
//
|
|
59
|
+
// Docstring lines are skipped wholesale rather than left to fall through the step regex. A rubric
|
|
60
|
+
// ladder routinely opens a line with a step keyword ("When the agent stages only related files,
|
|
61
|
+
// award 3"); read as a step, that line both leaks the answer key into the brief AND overwrites
|
|
62
|
+
// `lastKeyword`, so the collapsing `And ... at least the threshold` below it inherits `When` and
|
|
63
|
+
// leaks too.
|
|
64
|
+
export function parseScenarios(text: string): ParsedScenario[] {
|
|
65
|
+
const lines = text.split('\n')
|
|
66
|
+
const scenarios: ParsedScenario[] = []
|
|
67
|
+
let current: ParsedScenario | null = null
|
|
68
|
+
let lastKeyword: ConcreteKeyword | null = null
|
|
69
|
+
let docstringFence: string | null = null
|
|
70
|
+
let docstringOwner: Step | null = null
|
|
71
|
+
|
|
72
|
+
for (const raw of lines) {
|
|
73
|
+
const line = raw.trimStart()
|
|
74
|
+
|
|
75
|
+
// Docstring lines are captured onto their owning step, never re-read as steps. Letting them
|
|
76
|
+
// fall through the step regex is what leaks a rubric ladder that opens with a step keyword
|
|
77
|
+
// ("When the agent stages only related files, award 3") into the brief, and overwrites
|
|
78
|
+
// `lastKeyword` so the collapsing `And ... at least the threshold` below it leaks too.
|
|
79
|
+
const fence = /^("""|```)/.exec(line)?.[1]
|
|
80
|
+
if (docstringFence) {
|
|
81
|
+
if (fence === docstringFence) {
|
|
82
|
+
docstringFence = null
|
|
83
|
+
docstringOwner = null
|
|
84
|
+
} else if (docstringOwner?.docstring) {
|
|
85
|
+
docstringOwner.docstring.lines.push(raw)
|
|
86
|
+
}
|
|
87
|
+
continue
|
|
88
|
+
}
|
|
89
|
+
if (fence) {
|
|
90
|
+
docstringFence = fence
|
|
91
|
+
const owner = current?.steps.at(-1) ?? null
|
|
92
|
+
if (owner) {
|
|
93
|
+
owner.docstring = { fence, lines: [] }
|
|
94
|
+
docstringOwner = owner
|
|
95
|
+
}
|
|
96
|
+
continue
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
if (/^Scenario:/i.test(line) || /^Scenario Outline:/i.test(line)) {
|
|
100
|
+
if (current) scenarios.push(current)
|
|
101
|
+
const isOutline = /^Scenario Outline:/i.test(line)
|
|
102
|
+
const name = line.replace(/^Scenario(?: Outline)?:/i, '').trim()
|
|
103
|
+
current = { name, steps: [], isOutline, examples: null }
|
|
104
|
+
lastKeyword = null
|
|
105
|
+
continue
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
if (current && /^Examples:/i.test(line)) {
|
|
109
|
+
current.examples = { header: [], rows: [] }
|
|
110
|
+
continue
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
if (current?.examples && line.startsWith('|')) {
|
|
114
|
+
const cells = line
|
|
115
|
+
.split('|')
|
|
116
|
+
.slice(1, -1)
|
|
117
|
+
.map((c) => c.trim())
|
|
118
|
+
if (current.examples.header.length === 0) current.examples.header = cells
|
|
119
|
+
else current.examples.rows.push(cells)
|
|
120
|
+
continue
|
|
121
|
+
}
|
|
122
|
+
|
|
123
|
+
const stepMatch = /^(Given|When|Then|And|But)\b/i.exec(line)
|
|
124
|
+
if (current && stepMatch) {
|
|
125
|
+
const literal = stepMatch[1]
|
|
126
|
+
const resolved: EffectiveKeyword = /^(Given|When|Then)$/i.test(literal)
|
|
127
|
+
? ((literal[0].toUpperCase() + literal.slice(1).toLowerCase()) as ConcreteKeyword)
|
|
128
|
+
: (lastKeyword ?? 'unresolved')
|
|
129
|
+
current.steps.push({ text: line, effectiveKeyword: resolved, docstring: null })
|
|
130
|
+
if (resolved !== 'unresolved') lastKeyword = resolved
|
|
131
|
+
}
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
if (current) scenarios.push(current)
|
|
135
|
+
return scenarios
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
// ─── extraction ───────────────────────────────────────────────────────────────
|
|
139
|
+
|
|
140
|
+
export class ScenarioNotFoundError extends Error {}
|
|
141
|
+
export class AmbiguousScenarioError extends Error {}
|
|
142
|
+
export class UnparseableFeatureError extends Error {}
|
|
143
|
+
// A scenario the file holds but which carries no Given/When has no situation to simulate from. It
|
|
144
|
+
// fails closed rather than emitting an empty brief: a simulator handed nothing simulates nothing,
|
|
145
|
+
// the case scores low, and the eval reads as a defect in the SUBJECT rather than in the extraction.
|
|
146
|
+
export class EmptySituationError extends Error {}
|
|
147
|
+
// A `Scenario Outline` row index that the Examples table does not hold.
|
|
148
|
+
export class RowOutOfRangeError extends Error {}
|
|
149
|
+
|
|
150
|
+
// Extracts the redacted situation for the named scenario: its Given/When steps only, by EFFECTIVE
|
|
151
|
+
// keyword (never literal text) — every Then/And/But-after-Then step, every tag, the scenario name
|
|
152
|
+
// itself, and every sibling scenario are excluded by construction (never read into the result).
|
|
153
|
+
// Fails closed: an unparseable file, a missing name, or a duplicate name throws rather than
|
|
154
|
+
// returning an empty-but-plausible brief a caller could mistake for a valid one.
|
|
155
|
+
export function extractSituation(text: string, scenarioName: string, fileLabel: string, row?: number): Situation {
|
|
156
|
+
if (!/^\s*Feature:/im.test(text)) {
|
|
157
|
+
throw new UnparseableFeatureError(`${fileLabel}: not a valid .feature file (no Feature: line)`)
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
const scenarios = parseScenarios(text)
|
|
161
|
+
const matches = scenarios.filter((s) => s.name === scenarioName)
|
|
162
|
+
|
|
163
|
+
if (matches.length === 0) {
|
|
164
|
+
throw new ScenarioNotFoundError(`${fileLabel}: scenario not found: "${scenarioName}"`)
|
|
165
|
+
}
|
|
166
|
+
if (matches.length > 1) {
|
|
167
|
+
throw new AmbiguousScenarioError(
|
|
168
|
+
`${fileLabel}: scenario name is ambiguous (${matches.length} matches): "${scenarioName}"`,
|
|
169
|
+
)
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
const scenario = matches[0]
|
|
173
|
+
const emitted = scenario.steps.filter((s) => s.effectiveKeyword === 'Given' || s.effectiveKeyword === 'When')
|
|
174
|
+
const steps = emitted.flatMap((s) =>
|
|
175
|
+
s.docstring ? [s.text, s.docstring.fence, ...s.docstring.lines, s.docstring.fence] : [s.text],
|
|
176
|
+
)
|
|
177
|
+
const withDoc = (s: Step) =>
|
|
178
|
+
s.docstring ? [s.text, s.docstring.fence, ...s.docstring.lines, s.docstring.fence] : [s.text]
|
|
179
|
+
const given = emitted.filter((s) => s.effectiveKeyword === 'Given').flatMap(withDoc)
|
|
180
|
+
const when = emitted.filter((s) => s.effectiveKeyword === 'When').flatMap(withDoc)
|
|
181
|
+
|
|
182
|
+
if (steps.length === 0) {
|
|
183
|
+
throw new EmptySituationError(`${fileLabel}: scenario has no Given or When steps: "${scenarioName}"`)
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
const placeholders = [...new Set(steps.flatMap((s) => [...s.matchAll(/<([^>]+)>/g)].map((m) => m[1])))]
|
|
187
|
+
|
|
188
|
+
let examples: Situation['examples'] = null
|
|
189
|
+
if (scenario.isOutline && scenario.examples && placeholders.length > 0) {
|
|
190
|
+
const keepIdx = scenario.examples.header
|
|
191
|
+
.map((col, i) => (placeholders.includes(col) ? i : -1))
|
|
192
|
+
.filter((i) => i !== -1)
|
|
193
|
+
if (keepIdx.length > 0) {
|
|
194
|
+
// One row is one case: judge scores a single invoke decision per invocation, so an outline
|
|
195
|
+
// is extracted a row at a time rather than handed over whole.
|
|
196
|
+
const allRows = scenario.examples.rows
|
|
197
|
+
if (row !== undefined && (row < 0 || row >= allRows.length)) {
|
|
198
|
+
throw new RowOutOfRangeError(
|
|
199
|
+
`${fileLabel}: row ${row} is outside the Examples table (${allRows.length} rows): "${scenarioName}"`,
|
|
200
|
+
)
|
|
201
|
+
}
|
|
202
|
+
const picked = row === undefined ? allRows : [allRows[row]]
|
|
203
|
+
examples = {
|
|
204
|
+
header: keepIdx.map((i) => scenario.examples!.header[i]),
|
|
205
|
+
rows: picked.map((r) => keepIdx.map((i) => r[i])),
|
|
206
|
+
}
|
|
207
|
+
}
|
|
208
|
+
}
|
|
209
|
+
|
|
210
|
+
return { steps, given, when, placeholders, examples }
|
|
211
|
+
}
|
|
212
|
+
|
|
213
|
+
// ─── format ───────────────────────────────────────────────────────────────────
|
|
214
|
+
|
|
215
|
+
function formatTable(table: { header: string[]; rows: string[][] }): string {
|
|
216
|
+
const line = (cells: string[]) => `| ${cells.join(' | ')} |`
|
|
217
|
+
return [line(table.header), ...table.rows.map(line)].join('\n')
|
|
218
|
+
}
|
|
219
|
+
|
|
220
|
+
export function formatMarkdown(situation: Situation): string {
|
|
221
|
+
const lines: string[] = ['## Situation', '']
|
|
222
|
+
for (const step of situation.steps) lines.push(step)
|
|
223
|
+
if (situation.examples) {
|
|
224
|
+
lines.push('', 'Examples:', formatTable(situation.examples))
|
|
225
|
+
}
|
|
226
|
+
return `${lines.join('\n')}\n`
|
|
227
|
+
}
|
|
228
|
+
|
|
229
|
+
export function formatJson(situation: Situation): string {
|
|
230
|
+
return `${JSON.stringify(situation, null, 2)}\n`
|
|
231
|
+
}
|
|
232
|
+
|
|
233
|
+
// ─── CLI ──────────────────────────────────────────────────────────────────────
|
|
234
|
+
|
|
235
|
+
function flag(argv: string[], name: string): string | undefined {
|
|
236
|
+
const i = argv.indexOf(name)
|
|
237
|
+
return i === -1 ? undefined : argv[i + 1]
|
|
238
|
+
}
|
|
239
|
+
|
|
240
|
+
export function main(argv: string[]): number {
|
|
241
|
+
const featurePath = flag(argv, '--feature')
|
|
242
|
+
const scenarioName = flag(argv, '--scenario')
|
|
243
|
+
const rowRaw = flag(argv, '--row')
|
|
244
|
+
const format = flag(argv, '--format') ?? (argv.includes('--json') ? 'json' : 'text')
|
|
245
|
+
|
|
246
|
+
if (!featurePath || !scenarioName) {
|
|
247
|
+
console.error('✗ usage: extract-situation.mts --feature <path> --scenario "<name>" [--row <n>] [--format json]')
|
|
248
|
+
return 1
|
|
249
|
+
}
|
|
250
|
+
|
|
251
|
+
let row: number | undefined
|
|
252
|
+
if (rowRaw !== undefined) {
|
|
253
|
+
row = Number(rowRaw)
|
|
254
|
+
if (!Number.isInteger(row) || row < 0) {
|
|
255
|
+
console.error(`✗ --row must be a non-negative integer, got "${rowRaw}"`)
|
|
256
|
+
return 1
|
|
257
|
+
}
|
|
258
|
+
}
|
|
259
|
+
|
|
260
|
+
let text: string
|
|
261
|
+
try {
|
|
262
|
+
text = readFileSync(featurePath, 'utf8')
|
|
263
|
+
} catch {
|
|
264
|
+
console.error(`✗ ${featurePath}: cannot read file`)
|
|
265
|
+
return 1
|
|
266
|
+
}
|
|
267
|
+
|
|
268
|
+
try {
|
|
269
|
+
const situation = extractSituation(text, scenarioName, featurePath, row)
|
|
270
|
+
process.stdout.write(format === 'json' ? formatJson(situation) : formatMarkdown(situation))
|
|
271
|
+
return 0
|
|
272
|
+
} catch (err) {
|
|
273
|
+
console.error(`✗ ${(err as Error).message}`)
|
|
274
|
+
return 1
|
|
275
|
+
}
|
|
276
|
+
}
|
|
277
|
+
|
|
278
|
+
// `import.meta.main` is Node >=24.2, but this repo's engines floor is >=22, where it is `undefined`
|
|
279
|
+
// — the CLI would never run, printing nothing and exiting 0. A caller keying its BLOCKER on a
|
|
280
|
+
// non-zero exit would read that empty brief as a success and simulate from nothing.
|
|
281
|
+
//
|
|
282
|
+
// `pathToFileURL`, not `file://${process.argv[1]}`: `import.meta.url` percent-encodes, so the naive
|
|
283
|
+
// concat mismatches on any path holding a space (or #, ?, %) and silently reproduces that same
|
|
284
|
+
// never-fires bug — the trigger merely moves from a Node version to an install path.
|
|
285
|
+
if (process.argv[1] && import.meta.url === pathToFileURL(realpathSync(process.argv[1])).href) {
|
|
286
|
+
process.exit(main(process.argv.slice(2)))
|
|
287
|
+
}
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# improve
|
|
2
|
+
|
|
3
|
+
Improve an existing agent configuration of any kind — a skill, subagent, command, or AGENTS.md section — whether by diagnosing failing ACED evals or by general review against fit and quality bars.
|
|
4
|
+
|
|
5
|
+
## When to use
|
|
6
|
+
|
|
7
|
+
Use this skill when a config isn't working as well as it should and you want concrete proposed edits, not just a diagnosis.
|
|
8
|
+
|
|
9
|
+
Good triggers include:
|
|
10
|
+
|
|
11
|
+
- "Improve this skill" / "Make this agent better"
|
|
12
|
+
- "Why does this config keep failing?"
|
|
13
|
+
- Reviewing a config that has no eval suite yet, before deciding whether it's worth building one
|
|
14
|
+
|
|
15
|
+
Defer to a sibling when the request is really about scaffolding something new (`define-skill` / `define-agent` / `define-governance`), just scoring (`run`), just adding a case (`add-scenario`), just diffing two versions (`compare`), or auditing a `SKILL.md`'s structure specifically (`improve-skill`).
|
|
16
|
+
|
|
17
|
+
## What it does
|
|
18
|
+
|
|
19
|
+
If the target is ACED-tracked (has an `eval.md`), it loads the internal impl-producer loop to classify failing cases by pattern and propose before/after edits, then verifies with `compare`. If not yet tracked, it runs a general review — fit classification plus the relevant builder bar — and proposes edits for any gap found, offering to hand off to `sdd:start-mission` to formalize an eval suite.
|
|
20
|
+
|
|
21
|
+
## Install
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
npx skills add cyberuni/cyberplace --skill aced/improve
|
|
25
|
+
```
|