@nurix/etna 0.19.33 → 0.19.35
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/data/harness-index.json +5 -1
- package/data/harnesses/code/CLAUDE.md +21 -11
- package/data/harnesses/code/harness.json +2 -2
- package/data/harnesses/code/rules/general.md +1 -0
- package/data/harnesses/code/rules/testing.md +1 -1
- package/data/harnesses/code/rules/verification.md +51 -0
- package/data/harnesses/code/skills/test-plan/SKILL.md +3 -3
- package/data/harnesses/delivery/.prettierignore +29 -0
- package/data/harnesses/delivery/.prettierrc.json +3 -0
- package/data/harnesses/delivery/.vscode/settings.json +19 -0
- package/data/harnesses/delivery/CLAUDE.md +123 -0
- package/data/harnesses/delivery/agents/brainstorm.md +144 -0
- package/data/harnesses/delivery/agents/data-generator.md +23 -0
- package/data/harnesses/delivery/agents/eval-runner.md +23 -0
- package/data/harnesses/delivery/agents/judge.md +23 -0
- package/data/harnesses/delivery/agents/lld-mode.md +56 -0
- package/data/harnesses/delivery/agents/maintenance.md +229 -0
- package/data/harnesses/delivery/agents/prompt-iterator.md +23 -0
- package/data/harnesses/delivery/agents/red-team.md +23 -0
- package/data/harnesses/delivery/agents/regression-runner.md +28 -0
- package/data/harnesses/delivery/agents/retro.md +46 -0
- package/data/harnesses/delivery/codex/agents/brainstorm.toml +142 -0
- package/data/harnesses/delivery/codex/agents/data-generator.toml +22 -0
- package/data/harnesses/delivery/codex/agents/eval-runner.toml +22 -0
- package/data/harnesses/delivery/codex/agents/judge.toml +22 -0
- package/data/harnesses/delivery/codex/agents/lld-mode.toml +52 -0
- package/data/harnesses/delivery/codex/agents/maintenance.toml +228 -0
- package/data/harnesses/delivery/codex/agents/prompt-iterator.toml +22 -0
- package/data/harnesses/delivery/codex/agents/red-team.toml +22 -0
- package/data/harnesses/delivery/codex/agents/regression-runner.toml +27 -0
- package/data/harnesses/delivery/codex/agents/retro.toml +45 -0
- package/data/harnesses/delivery/harness.json +11 -0
- package/data/harnesses/delivery/hooks/generate-ownership-index.mjs +148 -0
- package/data/harnesses/delivery/hooks/hooks.json +16 -0
- package/data/harnesses/delivery/rules/changelog.md +19 -0
- package/data/harnesses/delivery/rules/evals.md +23 -0
- package/data/harnesses/delivery/rules/model-tiers.md +18 -0
- package/data/harnesses/delivery/rules/nomenclature.md +34 -0
- package/data/harnesses/delivery/rules/prompting.md +23 -0
- package/data/harnesses/delivery/skills/brainstorm/README.md +56 -0
- package/data/harnesses/delivery/skills/brainstorm/SKILL.md +173 -0
- package/data/harnesses/delivery/skills/brainstorm/references/doctrine.md +98 -0
- package/data/harnesses/delivery/skills/brainstorm/references/harvest.md +86 -0
- package/data/harnesses/delivery/skills/brainstorm/references/session-schema.md +186 -0
- package/data/harnesses/delivery/skills/brainstorm/scripts/detector.mjs +460 -0
- package/data/harnesses/delivery/skills/cost-report/SKILL.md +24 -0
- package/data/harnesses/delivery/skills/documentation/SKILL.md +129 -0
- package/data/harnesses/delivery/skills/documentation/references/examples.md +279 -0
- package/data/harnesses/delivery/skills/documentation/references/frontmatter.md +44 -0
- package/data/harnesses/delivery/skills/documentation/references/style-guide.md +340 -0
- package/data/harnesses/delivery/skills/documentation/references/untemplated.md +38 -0
- package/data/harnesses/delivery/skills/documentation/scripts/doc-lint.mjs +140 -0
- package/data/harnesses/delivery/skills/documentation/templates/Readme.md +31 -0
- package/data/harnesses/delivery/skills/documentation/templates/api-reference.md +20 -0
- package/data/harnesses/delivery/skills/documentation/templates/architecture-decision.md +121 -0
- package/data/harnesses/delivery/skills/documentation/templates/backlog.md +33 -0
- package/data/harnesses/delivery/skills/documentation/templates/changelog-entry.md +57 -0
- package/data/harnesses/delivery/skills/documentation/templates/changelog-index.md +41 -0
- package/data/harnesses/delivery/skills/documentation/templates/docs-readme.md +72 -0
- package/data/harnesses/delivery/skills/documentation/templates/domain-model.md +30 -0
- package/data/harnesses/delivery/skills/documentation/templates/environment-registry.md +15 -0
- package/data/harnesses/delivery/skills/documentation/templates/feature.md +149 -0
- package/data/harnesses/delivery/skills/documentation/templates/glossary.md +11 -0
- package/data/harnesses/delivery/skills/documentation/templates/memory-main.md +25 -0
- package/data/harnesses/delivery/skills/documentation/templates/migration-runbook.md +34 -0
- package/data/harnesses/delivery/skills/documentation/templates/non-goals.md +11 -0
- package/data/harnesses/delivery/skills/documentation/templates/operations-procedure.md +36 -0
- package/data/harnesses/delivery/skills/documentation/templates/plan.md +64 -0
- package/data/harnesses/delivery/skills/documentation/templates/protected-files.md +37 -0
- package/data/harnesses/delivery/skills/documentation/templates/root-readme.md +104 -0
- package/data/harnesses/delivery/skills/documentation/templates/security-overview.md +27 -0
- package/data/harnesses/delivery/skills/documentation/templates/structure.md +65 -0
- package/data/harnesses/delivery/skills/documentation/templates/system-overview.md +32 -0
- package/data/harnesses/delivery/skills/documentation/templates/testing-strategy.md +29 -0
- package/data/harnesses/delivery/skills/escalate/SKILL.md +25 -0
- package/data/harnesses/delivery/skills/prompt-eval/SKILL.md +22 -0
- package/data/harnesses/delivery/skills/prompt-library/SKILL.md +36 -0
- package/data/harnesses/delivery/skills/route/SKILL.md +21 -0
- package/data/harnesses/etna/CLAUDE.md +1 -1
- package/data/harnesses/etna/harness.json +2 -2
- package/data/harnesses/etna/rules/general.md +1 -0
- package/data/harnesses/etna/rules/testing.md +1 -1
- package/data/harnesses/hr/harness.json +1 -1
- package/package.json +1 -1
package/data/harness-index.json
CHANGED
|
@@ -1,10 +1,14 @@
|
|
|
1
1
|
{
|
|
2
|
-
"fingerprint": "
|
|
2
|
+
"fingerprint": "22a721cdaeb5",
|
|
3
3
|
"harnesses": {
|
|
4
4
|
"code": {
|
|
5
5
|
"plugin": "code",
|
|
6
6
|
"description": "the foundation every Nurix project starts with"
|
|
7
7
|
},
|
|
8
|
+
"delivery": {
|
|
9
|
+
"plugin": "delivery",
|
|
10
|
+
"description": "cost-governed prompt engineering"
|
|
11
|
+
},
|
|
8
12
|
"discover": {
|
|
9
13
|
"plugin": "discover",
|
|
10
14
|
"description": "Product Discovery harness"
|
|
@@ -21,7 +21,7 @@ Invoke the `maintenance` agent whenever the docs look like they need it — not
|
|
|
21
21
|
|
|
22
22
|
## Workflows
|
|
23
23
|
|
|
24
|
-
Three peer modes plus
|
|
24
|
+
Three peer modes plus Verification and Completion — enter wherever the user's thinking already is. Each mode is implemented by an agent + skill; the behavioural detail lives in those files, not here.
|
|
25
25
|
|
|
26
26
|
| Mode | Enter when | Implemented by | Exits to |
|
|
27
27
|
| --- | --- | --- | --- |
|
|
@@ -33,18 +33,19 @@ Three peer modes plus a Completion step — enter wherever the user's thinking a
|
|
|
33
33
|
|
|
34
34
|
**On Plan Completion:**
|
|
35
35
|
|
|
36
|
-
1.
|
|
37
|
-
2.
|
|
38
|
-
3.
|
|
39
|
-
4.
|
|
36
|
+
1. **The verification gate passes first** — discharge every row of the plan's Test Obligations, driving the real surface before pinning tests (see Verification). A plan with outstanding obligations does not reach `Status: Complete`, whatever the code does.
|
|
37
|
+
2. Freeze the plan — `Status: Complete`, fill the Completed date. The file is now frozen.
|
|
38
|
+
3. Flip the feature LLD to `status: active`; extend it in place with the how-it-works sections; confirm its `owns:` globs against the actual code.
|
|
39
|
+
4. The `boundary` and `test-plan` skills must already have gated this plan before its first implementation step — see Ownership, Boundaries, Verification & Documentation.
|
|
40
|
+
5. Derive/update the behavioral checklist at `docs/tests/{feature}.md` via the `documentation` skill (a truth document — present-tense, referencing the LLD's promises, restating no intent), and record the gate's results in the changelog entry.
|
|
40
41
|
|
|
41
42
|
**On Brainstorm Close** (when a session ends, not when a plan completes): close + harvest is the skill's `stop` procedure — converge, a user-approved harvest manifest, apply, then a uselessness test; conclusions disperse by scope (architecture docs, decision records, backlog), the record moves from `docs/research/{topic}-brainstorm/` to `docs/history/brainstorms/{topic}/`, and the user decides whether the moved folder is deleted or committed.
|
|
42
43
|
|
|
43
44
|
---
|
|
44
45
|
|
|
45
|
-
## Ownership, Boundaries & Documentation
|
|
46
|
+
## Ownership, Boundaries, Verification & Documentation
|
|
46
47
|
|
|
47
|
-
These
|
|
48
|
+
These four mechanisms work in tandem to keep the repo self-updating and to prevent half-finished changes: **ownership** routes an edit to the canonical doc before the code is touched, the **boundary** check reads both sides of every contract a change participates in so one side never ships alone, **verification** proves the change works in the product before it is called done, and **documentation** updates close the loop so the next session inherits the truth. Skipping any leg breaks the others.
|
|
48
49
|
|
|
49
50
|
### Ownership — route before editing
|
|
50
51
|
|
|
@@ -65,7 +66,15 @@ Boundaries are **diagnosed, not documented** — the `boundary` skill derives th
|
|
|
65
66
|
|
|
66
67
|
### Test obligations — specify before implementing
|
|
67
68
|
|
|
68
|
-
**After `boundary` passes and its amendments are folded, invoke the `test-plan` skill** — same plan gate, ordered second. It derives the tests the plan owes from its Proposed Changes and the owning LLD, and writes a **Test Obligations** section into the plan (one row per promised behavior: layer · target file · oracle · fixtures). It writes no test files — the tests land
|
|
69
|
+
**After `boundary` passes and its amendments are folded, invoke the `test-plan` skill** — same plan gate, ordered second. It derives the tests the plan owes from its Proposed Changes and the owning LLD, and writes a **Test Obligations** section into the plan (one row per promised behavior: layer · target file · oracle · fixtures). It writes no test files — the tests land at the verification gate, and the manifest is the ledger that gate discharges. This makes obligations survive dispatch to an executor that sees only the plan file, and lets a later doc-health pass prove the tests match the promises. On a boundary **BLOCK** it does not run — re-plan first. The three-moment split (gate → verification → Completion checklist), the layer rules, and the e2e altitude gate live in the skill.
|
|
70
|
+
|
|
71
|
+
### Verification — prove before completing
|
|
72
|
+
|
|
73
|
+
**Deferring tests is what keeps the build loop fast, and the ledger is what makes deferring safe.** A plan-driven build writes no test files while the code is taking shape; the obligations accumulate in the plan and are discharged in one pass at the **verification gate** — triggered when the user says the work is ready to verify, and mandatory before a plan reaches `Status: Complete`. The gate, the two regimes, the mechanism ladder, and what earns a durable test are `.claude/rules/verification.md`.
|
|
74
|
+
|
|
75
|
+
- **Only a plan may defer.** A bug fix and any ad-hoc change ship their test immediately — deferral needs a ledger and a gate, and non-plan work has neither.
|
|
76
|
+
- **Drive the real surface before pinning tests.** Browser automation for web UI, the real CLI, HTTP against the running worker; the suite pins what must not regress afterward.
|
|
77
|
+
- **The gate runs in the main session.** Verification is never delegated — see the Browser Automation guardrail and Runtime Notes.
|
|
69
78
|
|
|
70
79
|
### Documentation — close the loop
|
|
71
80
|
|
|
@@ -120,9 +129,10 @@ When a plan's implementation surfaces work that is important but outside the pla
|
|
|
120
129
|
|
|
121
130
|
Defaults — ❌ prohibitions hold unless the user explicitly asks otherwise; ✅ and ⚠️ items are standing policy:
|
|
122
131
|
|
|
123
|
-
- ✅ **
|
|
124
|
-
- **
|
|
125
|
-
- **
|
|
132
|
+
- ✅ **Verification — batched at a gate, not continuous.** A plan-driven build loop ships code without accompanying test files; the tests and the driven checks land together at the **verification gate**, which the user triggers and which is mandatory before a plan reaches `Status: Complete`. The gate, the mechanism ladder, and what earns a durable test are `.claude/rules/verification.md`; framework, test-tree placement, and shared fixtures stay `.claude/rules/testing.md` — the repo's existing runner and layout always win.
|
|
133
|
+
- **Only a plan defers**: a bug fix and any ad-hoc change ship their test immediately — a regression test that fails before the change and passes after. Deferral needs the Test Obligations ledger and the gate that discharges it, and non-plan work has neither.
|
|
134
|
+
- **Drive before pinning**: prove the change in the real product first — browser automation for web UI, the real CLI, HTTP against the running worker — then pin as suite tests only the behaviors that must not regress.
|
|
135
|
+
- **Discipline**: run the affected part of the suite at the gate and report results honestly. When an LLD or plan intentionally changes behavior, updating the affected tests to the new contract is part of the change — the ban is on weakening, skipping, or deleting a test to mask an unexplained failure.
|
|
126
136
|
- **Throwaway checks are not tests**: temporary verification scripts go to scratch space and are deleted, never committed into the test tree.
|
|
127
137
|
- ❌ **Dependencies** – Never upgrade frameworks, libraries, or dependencies.
|
|
128
138
|
- ❌ **Compatibility** – Never maintain backward compatibility unless specified.
|
|
@@ -22,6 +22,7 @@ Runtime versions, the canonical package list, and GitHub Actions pins — the wo
|
|
|
22
22
|
| `@reduxjs/toolkit` | ^2.11.2 | State management | `zustand`, `jotai` |
|
|
23
23
|
| `react-redux` | ^9.2.0 | Redux React bindings | — |
|
|
24
24
|
| `axios` | ^1.13.2 | HTTP requests | `fetch` |
|
|
25
|
+
| `@microsoft/fetch-event-source` | ^2.0.1 | SSE over POST — streaming agent output | hand-rolled `getReader()` loops, `EventSource` |
|
|
25
26
|
| `date-fns` | ^4.1.0 | Date formatting | `moment`, `dayjs` |
|
|
26
27
|
| `@dnd-kit/core` | ^6.3.1 | Drag and drop | `react-beautiful-dnd` |
|
|
27
28
|
| `@xyflow/react` | ^12.11.0 | Node-based flow charts | `reactflow`, `react-flow-renderer` |
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Testing
|
|
2
2
|
|
|
3
|
-
Test technology and placement — framework per stack, suite layout, mocking policy;
|
|
3
|
+
Test technology and placement — framework per stack, suite layout, mocking policy; **when** a test is written and how work is proven is [`verification.md`](verification.md), not here. Framework choice is **never** ad hoc and never the user's problem mid-task: detect the stack, use its row; if the repo already has a runner, that runner wins — **never** introduce a second framework beside it.
|
|
4
4
|
|
|
5
5
|
## Frameworks by stack
|
|
6
6
|
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Verification
|
|
2
|
+
|
|
3
|
+
**When** work is proven and **by what mechanism** — the gate, the two regimes, and the ladder of ways to drive a change. Framework choice, suite layout, and mocking policy are [`testing.md`](testing.md); the plan-gate obligation manifest is the `test-plan` skill. This rule owns the moment, not the technology.
|
|
4
|
+
|
|
5
|
+
## Two regimes
|
|
6
|
+
|
|
7
|
+
**Deferral is a property of plans, not of code.** A plan carries a Test Obligations ledger and a gate that discharges it; nothing else does, so nothing else may defer.
|
|
8
|
+
|
|
9
|
+
| Regime | During the work | Proven |
|
|
10
|
+
| --- | --- | --- |
|
|
11
|
+
| **Plan-driven build** | No test files. Code only — the loop stays fast | At the verification gate, before `Status: Complete` |
|
|
12
|
+
| **Bug fix** | The regression test ships with the fix | Immediately — it fails before the change, passes after |
|
|
13
|
+
| **Ad-hoc / non-plan change** | The test ships with the change | Immediately, same discipline as a bug fix |
|
|
14
|
+
|
|
15
|
+
- **A bug fix cannot defer, because its central property is only observable at fix time** — "fails before, passes after" is unprovable once the fix is already in the tree.
|
|
16
|
+
- **Ad-hoc work cannot defer, because there is no ledger to defer into.** An obligation recorded nowhere is an obligation forgotten; the plan file is the only durable home, so work without a plan pays as it goes.
|
|
17
|
+
|
|
18
|
+
## The gate
|
|
19
|
+
|
|
20
|
+
**The gate is a discrete step, not a background habit** — it runs when the user says the work is ready to verify, or when a plan's implementation is complete. It is **mandatory before a plan reaches `Status: Complete`**: a plan with outstanding obligations is not complete, whatever the code does.
|
|
21
|
+
|
|
22
|
+
- **The gate runs in the main session, never delegated.** A sub-agent cannot hold a browser connection (`CLAUDE.md` § Runtime Notes), so a delegated gate silently degrades from *driving the product* to *describing how someone else could*. Fan-out may prepare fixtures or draft test bodies; the driving and the verdict stay inline.
|
|
23
|
+
- **The plan's Test Obligations table is the checklist.** Every row is discharged — satisfied, or restated with the reason it owes nothing after all. A row left unexamined blocks the gate.
|
|
24
|
+
- **Report what happened, including failures.** A gate that reports green while a suite is red is worse than no gate; state the failing test and its output.
|
|
25
|
+
|
|
26
|
+
## The mechanism ladder
|
|
27
|
+
|
|
28
|
+
**Drive the real thing first, then pin what must not regress.** Ordering matters: driving finds the defects that matter to a user, and shows which behaviors are worth the permanent cost of a test.
|
|
29
|
+
|
|
30
|
+
| Rung | Mechanism | Proves |
|
|
31
|
+
| --- | --- | --- |
|
|
32
|
+
| 1 | **Drive the real surface** — browser automation for web UI, the actual CLI binary, HTTP against the running worker | The change works in the product, against real wiring |
|
|
33
|
+
| 2 | **Pin the durable behaviors** as suite tests per [`testing.md`](testing.md) | The behavior survives the next change |
|
|
34
|
+
| 3 | **Run the affected suite + the project's e2e smoke gate** | Nothing else broke |
|
|
35
|
+
|
|
36
|
+
- **Browser automation is the default for a web surface, and needs no permission to use** — a rendered page driven through real interaction is the only oracle that catches wiring, styling, and state defects together. Where the runtime offers none, the gate emits a browser-verification checklist (URLs, actions, expected outcomes) for the user to run, and says so rather than claiming verification.
|
|
37
|
+
- **A driven check is evidence, not an artifact.** It proves the change works today; it leaves nothing behind. That is why rung 2 exists.
|
|
38
|
+
|
|
39
|
+
## What earns a durable test
|
|
40
|
+
|
|
41
|
+
**A test is a permanent maintenance cost, so it is bought deliberately** — the ledger names behaviors, and the gate decides which of them are worth pinning.
|
|
42
|
+
|
|
43
|
+
| Earns a suite test | Verified by driving alone |
|
|
44
|
+
| --- | --- |
|
|
45
|
+
| A promise an LLD makes about the feature's contract | Layout, spacing, copy — restated by a test, not protected by one |
|
|
46
|
+
| A bug that occurred once — its regression test | A one-off migration or backfill's output |
|
|
47
|
+
| An error path, race, or idempotency guarantee | Behavior a type or the compiler already makes unshippable |
|
|
48
|
+
|
|
49
|
+
- **A behavior that is compiler-guarded owes no test** — if it cannot ship green-but-broken, a test adds cost and no safety.
|
|
50
|
+
- **Never weaken, skip, or delete a test to mask an unexplained failure.** When a plan or LLD intentionally changes behavior, updating the affected tests to the new contract is part of that change.
|
|
51
|
+
- **Throwaway checks are not tests.** Temporary scripts written to drive a change live in scratch space and are deleted — never committed into the test tree.
|
|
@@ -16,10 +16,10 @@ Testing spans three moments in the plan lifecycle. This skill owns **only the fi
|
|
|
16
16
|
| Moment | Artifact | Owner |
|
|
17
17
|
| --- | --- | --- |
|
|
18
18
|
| **Plan gate** (here, after boundary) | Test Obligations section written into the plan | this skill |
|
|
19
|
-
| **
|
|
20
|
-
| **Completion** | behavioral checklist at `docs/tests/{feature}.md`;
|
|
19
|
+
| **Verification gate** (before `Status: Complete`) | the obligated tests written and green, plus the driven checks against the real surface | the implementer at the gate (`.claude/rules/verification.md`) |
|
|
20
|
+
| **Completion** | behavioral checklist at `docs/tests/{feature}.md`; record the gate's results in the changelog | the `documentation` skill + Completion steps |
|
|
21
21
|
|
|
22
|
-
Writing tests *before* the code they cover either fabricates implementation coupling (labels/routes/copy the plan hasn't fixed) or leaves the suite red on the shared branch for the plan's whole duration.
|
|
22
|
+
Writing tests *before* the code they cover either fabricates implementation coupling (labels/routes/copy the plan hasn't fixed) or leaves the suite red on the shared branch for the plan's whole duration. Writing them *during* the build loop taxes every iteration for behavior still in flux. So the plan gate emits obligations, the build loop stays free of test files, and the manifest is discharged in one pass at the verification gate.
|
|
23
23
|
|
|
24
24
|
## Contract sources
|
|
25
25
|
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Dependencies & build output
|
|
2
|
+
node_modules
|
|
3
|
+
dist
|
|
4
|
+
build
|
|
5
|
+
|
|
6
|
+
# Lockfiles
|
|
7
|
+
pnpm-lock.yaml
|
|
8
|
+
package-lock.json
|
|
9
|
+
yarn.lock
|
|
10
|
+
|
|
11
|
+
# Harness-managed — the installer overwrites all of these, so formatting them is churn
|
|
12
|
+
# that comes straight back on the next `npx @nurix/etna`.
|
|
13
|
+
.claude
|
|
14
|
+
.codex
|
|
15
|
+
.agents
|
|
16
|
+
CLAUDE.md
|
|
17
|
+
AGENTS.md
|
|
18
|
+
|
|
19
|
+
# Frozen records — a plan, changelog entry, or migration runbook is a point-in-time
|
|
20
|
+
# artifact that is never edited after it freezes. Reformatting one is an edit, and the
|
|
21
|
+
# `maintenance` agent is forbidden from making it — so a format gate over this tree is
|
|
22
|
+
# a demand nothing in the repo is allowed to satisfy.
|
|
23
|
+
docs/history
|
|
24
|
+
|
|
25
|
+
# Source catalogs — authored markdown; prettier collapses template/example blocks
|
|
26
|
+
skills
|
|
27
|
+
rules
|
|
28
|
+
agents
|
|
29
|
+
harness
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
{
|
|
2
|
+
// Markdown: Prettier unwraps prose to one logical line per paragraph
|
|
3
|
+
// (see .prettierrc → proseWrap:never) on save; the editor soft-wraps for
|
|
4
|
+
// reading, so no hard line breaks get baked into the source.
|
|
5
|
+
"[markdown]": {
|
|
6
|
+
"editor.defaultFormatter": "esbenp.prettier-vscode",
|
|
7
|
+
"editor.formatOnSave": true,
|
|
8
|
+
"editor.wordWrap": "on"
|
|
9
|
+
},
|
|
10
|
+
// Hide harness-authoring scaffolding from the file explorer
|
|
11
|
+
"files.exclude": {
|
|
12
|
+
".claude": true,
|
|
13
|
+
".vscode": true,
|
|
14
|
+
".agents": true,
|
|
15
|
+
".codex": true,
|
|
16
|
+
"**/node_modules": true,
|
|
17
|
+
"**/apollo.service.json": true
|
|
18
|
+
}
|
|
19
|
+
}
|
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
# Agent Instructions
|
|
2
|
+
|
|
3
|
+
This repo is a **prompt-engineering project**: its artifacts are prompts and the eval suites that certify them, not application source. "Code" below means prompt artifacts — a feature LLD owns `prompts/{family}/**` and `evals/{family}/**`, not `src/`. This policy is deliberately forked from the `code` harness, reframed for prompt work — a capability added to `code` never silently lands here. Rules marked **[override]** deliberately replace a runtime default; the rule here wins.
|
|
4
|
+
|
|
5
|
+
**Policy style: self-improving, scoped to prompt-engineering teams.** The routing table below is living policy: escalation records feed `retro`, `retro` proposes routing amendments, and the table moves. Nothing in this file names a model — tier labels (economy / standard / frontier) are the vocabulary here; the tier → model mapping lives only in `.claude/rules/model-tiers.md`, and enforcement lives in each owned agent's `model:` pin.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## The Routing Policy
|
|
10
|
+
|
|
11
|
+
The model decision belongs to this table, not to the human and not to session judgment. Classify the task (the `route` skill), read the row, dispatch the pinned agent.
|
|
12
|
+
|
|
13
|
+
| Task type | Tier | Executing agent |
|
|
14
|
+
| --- | --- | --- |
|
|
15
|
+
| Eval execution, regression runs, scoring against golden sets | economy | `eval-runner`, `regression-runner` |
|
|
16
|
+
| Synthetic test data, edge-case corpora | economy | `data-generator` |
|
|
17
|
+
| Task classification (`route` itself) | economy | inline, cheapest capable tier |
|
|
18
|
+
| Prompt drafting and iteration loops | standard | `prompt-iterator` |
|
|
19
|
+
| Ship gate (judge), user-requested second opinion | frontier | `judge` |
|
|
20
|
+
| Adversarial probing | frontier | `red-team` |
|
|
21
|
+
|
|
22
|
+
**The escalation ladder.** Every task starts at its row's tier. Promotion requires a **recorded failing eval at the current tier** — run the `escalate` skill, which writes the evidence as a changelog entry (`Class: escalation`). Never promote on judgment alone. Symmetrically, demotion experiments run continuously: `regression-runner` re-runs passing prompts one tier down, and a pass is a `retro`-visible signal to move the routing row.
|
|
23
|
+
|
|
24
|
+
**Routing guardrails:**
|
|
25
|
+
|
|
26
|
+
- ❌ **Model names** — never name a model outside `.claude/rules/model-tiers.md`. Tier labels are free everywhere; a model name anywhere else in this repo's instruction surface is a defect.
|
|
27
|
+
- ❌ **Frontier fan-out** — never dispatch the frontier tier for mechanical, parallel, or repetitive work; frontier is for judgment (the ship gate, adversarial probing, an explicitly requested second opinion).
|
|
28
|
+
- ❌ **Unevidenced promotion** — never run a task above its routed tier without the `escalate` record.
|
|
29
|
+
- ✅ **Escalations are changelog entries**, so `retro` can propose routing-table amendments from escalation counts — the cost policy tunes itself from history.
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Maintenance
|
|
34
|
+
|
|
35
|
+
Invoke the `maintenance` agent whenever the docs look like they need it: a missing `docs/structure.md`, a `harness-version` marker behind the installed version, docs contradicting the prompt families they describe, unowned prompt paths, or drift noticed while working. Never reconcile by hand — the agent auto-selects its mode and hands back. **This harness's own parity contract is part of the audit**: the routing table above, the owned agents' `model:` pins, and `.claude/rules/model-tiers.md` must agree three ways — a row with no pinned agent, a pin with no row, or a tier the rule doesn't map is drift to flag.
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## Getting Started
|
|
40
|
+
|
|
41
|
+
- Read **[`docs/structure.md`](docs/structure.md)** for the documentation map and folder authority.
|
|
42
|
+
- Read **[`docs/memory/`](docs/memory/)** for agent memory files.
|
|
43
|
+
- Prefer `docs/` over `README.md` — `docs/` is canonical for agents.
|
|
44
|
+
- Prompt families live under `prompts/{family}/`; their golden sets under `evals/{family}/`. The feature doc that owns a family is the first read before touching it.
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
## Workflows
|
|
49
|
+
|
|
50
|
+
Three peer modes plus a Completion step — enter wherever the user's thinking already is. Each mode is implemented by an agent + skill; the behavioural detail lives in those files, not here.
|
|
51
|
+
|
|
52
|
+
| Mode | Enter when | Implemented by | Exits to |
|
|
53
|
+
| --- | --- | --- | --- |
|
|
54
|
+
| **Brainstorming** | explicit ask (`/brainstorm start`); or offer a session — never auto-enter — when the user brings a large piece of work or an utterance contradicts a recorded pillar | `brainstorm` agent + skill | LLD, Plan, or close + harvest |
|
|
55
|
+
| **LLD** (low-level design) | a new prompt family, task type, or eval suite | `lld-mode` agent + `documentation` skill | Plan |
|
|
56
|
+
| **Plan** | an LLD is approved | `documentation` skill → `docs/history/plans/YYYY-MM-DD-{slug}.md` | Completion |
|
|
57
|
+
|
|
58
|
+
**On Plan Completion:** freeze the plan (`Status: Complete`, Completed date); flip the feature LLD to `status: active` and extend it in place; confirm its `owns:` globs against the actual prompt files; write the changelog entry as a draft immediately.
|
|
59
|
+
|
|
60
|
+
**On Brainstorm Close:** the skill's `stop` procedure — converge, a user-approved harvest manifest, apply, uselessness test; the record moves to `docs/history/brainstorms/{topic}/`.
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## Ownership & Documentation
|
|
65
|
+
|
|
66
|
+
- **Ownership routes before editing.** A feature doc (flat in `docs/features/`) declares the prompt paths it owns in its `owns:` frontmatter (`prompts/{family}/**`, `evals/{family}/**`); the SessionStart hook injects the union as the ownership index. Match the path or topic, read the owning doc first. No index in context ⇒ the repo isn't reconciled — see Maintenance.
|
|
67
|
+
- **Every markdown document goes through the `documentation` skill** — templates, the two-natures rule (truth documents present-tense; history only in records), and doc-lint.
|
|
68
|
+
- **[override]** Keep `docs/` updated when a change alters documented behaviour — creating and updating docs is part of the work.
|
|
69
|
+
- When moving or renaming a prompt family: update the owning doc's `owns:` globs in the same change.
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## Prompt Artifacts — eval-first
|
|
74
|
+
|
|
75
|
+
**[override]** The gated-verification posture of the `code` harness is reframed: here the suite discharged at the gate is the eval set.
|
|
76
|
+
|
|
77
|
+
- **A prompt ships with its golden set.** A new prompt family lands with the evals that certify it; a prompt change re-runs its family's evals before it ships. A failing eval is reported honestly, never bypassed by weakening the rubric.
|
|
78
|
+
- **Certification is per tier.** A prompt-library entry records the tier it is certified for, its eval score, and the certification date (the `prompt-library` skill owns the entry format). A model-generation change invalidates certifications — `regression-runner` re-certifies.
|
|
79
|
+
- **Throwaway prompt experiments are not library entries** — scratch iterations stay out of `prompts/**` until they pass their evals.
|
|
80
|
+
- Conventions — golden-set shape, rubric design, pass thresholds: `.claude/rules/evals.md`. House prompt style: `.claude/rules/prompting.md`.
|
|
81
|
+
|
|
82
|
+
---
|
|
83
|
+
|
|
84
|
+
## Changelog
|
|
85
|
+
|
|
86
|
+
The working session is the last holder of the refinement context that produced a prompt: the variants rejected, the eval deltas, the escalations and why. The `changelog` rule carries the trigger and lifecycle; the policy here matches the `code` harness — every non-mechanical commit is covered by an entry (`docs/history/changelog/YYYY-MM-DD-{three-words}.md` + a `docs/history/CHANGELOG.md` row), drafts are mutable and commits freeze, and the `retro` agent distills on demand. Two genres are load-bearing for this harness:
|
|
87
|
+
|
|
88
|
+
- **Escalation entries** (`Class: escalation`, written by the `escalate` skill) — the raw data for routing-table tuning.
|
|
89
|
+
- **Prompt refinement entries** — rejected variants and eval deltas quoted with the same discipline the template's failure fields demand.
|
|
90
|
+
|
|
91
|
+
---
|
|
92
|
+
|
|
93
|
+
## Project Memory
|
|
94
|
+
|
|
95
|
+
Project learnings live in `docs/memory/` (git-tracked): `main.md` for project-wide patterns, `{agent-name}.md` for each owned agent's strategy memory (what rubrics drift, which adversarial classes keep working). User facts go to the runtime's user-level store. When recording a learning: _true of this project, or true of this user?_
|
|
96
|
+
|
|
97
|
+
---
|
|
98
|
+
|
|
99
|
+
## Out-of-Scope Discovery
|
|
100
|
+
|
|
101
|
+
Work surfaced outside the current plan's scope: append to `docs/history/backlog.md` under the appropriate priority (P0–P3), format `- **[P1]** Description — discovered by [plan or agent]`. Do NOT ask permission; append and mention it in the reply.
|
|
102
|
+
|
|
103
|
+
---
|
|
104
|
+
|
|
105
|
+
## Guardrails & Defaults
|
|
106
|
+
|
|
107
|
+
❌ prohibitions hold unless the user explicitly asks otherwise; ✅ and ⚠️ items are standing policy:
|
|
108
|
+
|
|
109
|
+
- ❌ **Dependencies** — never upgrade frameworks, libraries, or dependencies.
|
|
110
|
+
- ❌ **Branches & Worktrees [override]** — never self-create git branches or worktrees; commit straight to the checked-out branch. Exception: a user-requested pull request creates the branch it requires.
|
|
111
|
+
- ❌ **Protected Files** — if `docs/operations/protected-files.md` exists, read it before editing and never modify a listed path; never modify `AGENTS.md` or that file itself.
|
|
112
|
+
- ✅ **Fan-out is a tool, not a default** — but where the `code` harness leaves tier choice to session judgment, here dispatch runs through the routing table. Delegation questions are routing questions: consult the table first.
|
|
113
|
+
- ⚠️ **Plans** — created in `docs/history/plans/`; frozen at `Complete`/`Abandoned`, never edited after.
|
|
114
|
+
- ⚠️ **Tool Issues** — report unexpected CLI/MCP tool behavior to the user with the exact tool, command, and error.
|
|
115
|
+
|
|
116
|
+
---
|
|
117
|
+
|
|
118
|
+
## Runtime Notes
|
|
119
|
+
|
|
120
|
+
Everything above is runtime-neutral; this section is the only place runtime-specific facts may live.
|
|
121
|
+
|
|
122
|
+
- **Claude Code**: sub-agent dispatch is the Agent tool; the owned agents' frontmatter `model:` pins enforce the routing table's tiers mechanically. Skills are slash commands (`/route`, `/escalate`). The ownership index injects at SessionStart.
|
|
123
|
+
- **Runtimes without pin support** (Codex and other AGENTS.md readers): agent tier pins are dropped from the projected twins, so the routing table above and each agent's stated tier are the policy — advisory rather than enforced. The table is still binding on the humans and sessions that dispatch.
|
|
@@ -0,0 +1,144 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: brainstorm
|
|
3
|
+
description: The brainstorm twin — the paper-owning half of the brainstorm capability. Dispatched at session gates (start, on demand, pause, stop) by the `brainstorm` skill, never self-triggered and never conversing with the user. Scribe (distills the ops capture ledger into the working surface), editor (recomposes the thesis whole, diffs surfaced), harvester (drives the stop pipeline — manifest, apply, uselessness test, move), blindspot analyst (cold-reads the surface for unknown unknowns behind a four-test quality gate — grounded, material, new, answerable). Consumes the detector script for mechanical integrity and audits the five semantic sources the detector cannot judge. Writes every working-surface file; the facilitator writes only ops/capture.md and sync-log dispatch lines.
|
|
4
|
+
tools: Read, Glob, Grep, Write, Edit, Bash, Task, Skill
|
|
5
|
+
model: opus
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Brainstorm Twin
|
|
9
|
+
|
|
10
|
+
You own the **paper** of a brainstorm session; the facilitator (the main loop running the
|
|
11
|
+
`brainstorm` skill) owns the conversation. You are dispatched at gates with a session-folder path,
|
|
12
|
+
a gate name, and a scope. You never talk to the user — anything needing their judgment goes back
|
|
13
|
+
in your final report for the facilitator to relay.
|
|
14
|
+
|
|
15
|
+
The format contract is the skill's `references/session-schema.md` (resolve it beside this harness:
|
|
16
|
+
`.claude/skills/brainstorm/references/` in a consumer repo, `skills/brainstorm/references/` in
|
|
17
|
+
etna; also read `references/doctrine.md` for the epistemics). The mechanical half is the detector:
|
|
18
|
+
|
|
19
|
+
```
|
|
20
|
+
node <skill>/scripts/detector.mjs <session-folder> [--gate=manifest|move] [--json]
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
Run it at the start of every dispatch and again before you return. If node or the script is
|
|
24
|
+
unavailable, say so loudly in your report and run the checks by hand from the schema — a missing
|
|
25
|
+
detector is a flagged fact, never a silent skip.
|
|
26
|
+
|
|
27
|
+
## The write protocol you enforce
|
|
28
|
+
|
|
29
|
+
One writer per file. You write `state.md`, `thesis.md`, `forks.md`, `ledger.md`, `fragments/`,
|
|
30
|
+
manifest, and your own sync-log **result** lines. You never write `ops/capture.md` (the
|
|
31
|
+
facilitator's) or sync-log dispatch lines. The user may hand-edit anything: before rewriting any
|
|
32
|
+
file, read its current state in full and treat content you did not write as **first-class user
|
|
33
|
+
signal — fold it forward, never clobber it**. A hand-edit that contradicts the ledger is a design
|
|
34
|
+
question: file a fork, don't "fix" the file.
|
|
35
|
+
|
|
36
|
+
## Every dispatch, in order
|
|
37
|
+
|
|
38
|
+
1. **Detector first** — `--gate=session`. Fix what is yours to fix (scribe-calls, below); report
|
|
39
|
+
what isn't.
|
|
40
|
+
2. **Read the whole surface** plus every `ops/capture.md` entry since the last reconciled sync.
|
|
41
|
+
3. **Scribe pass** — distill new capture into the surface, per the doctrine's epistemic rules
|
|
42
|
+
(distilled never verbatim; examples as examples; no naked pointers; both sides captured).
|
|
43
|
+
Residue that doesn't warrant filing stays in ops — that is what ops is for.
|
|
44
|
+
4. **Semantic audit** — the five sources the detector cannot judge:
|
|
45
|
+
- **Cross-pillar contradiction** — across any two session files, `state.md` included (a state
|
|
46
|
+
file describing a session the pillars contradict is the historically strongest signal).
|
|
47
|
+
- **Aspect adequacy** — the detector checks that every mapped aspect *links* to content; you judge
|
|
48
|
+
whether the content actually covers it. Covered-in-name-only → a fork.
|
|
49
|
+
- **Basis drift** — decided divergence from `basis/` is fine; accidental divergence → a fork.
|
|
50
|
+
- **Tech conflicts** — anything violating the tech fragment (or the repo's `.claude/rules/`)
|
|
51
|
+
without a recorded deviation → a fork.
|
|
52
|
+
- **Self-containment** — read as the stranger you are (you were dispatched context-free; use
|
|
53
|
+
that): undefined terms, naked pointers, enumerations cited but never printed.
|
|
54
|
+
5. **The scribe-call / design-call line** — an edit whose resolution doesn't change the design
|
|
55
|
+
(wording, bookkeeping, printing an enumeration, glossing a term) you apply directly and list.
|
|
56
|
+
An edit whose resolution changes the idea becomes a fork (`Raised by: twin (S#)` — cite your
|
|
57
|
+
own sync id). Never route a design question through a direct edit.
|
|
58
|
+
6. **Editor pass** — recompose `thesis.md` whole (claim → argument → what changed), stamp
|
|
59
|
+
`Current as of:`, mirror the stamp into state.md `Last compose`, update the posture line.
|
|
60
|
+
Filing an insight must never push the session away from coherent: orphan evidence is a
|
|
61
|
+
scribe-note in your report, never a fork; free-floating insight may feed a thesis aspect.
|
|
62
|
+
7. **Close the log** — append your result line to `ops/sync-log.md`:
|
|
63
|
+
`S# → reconciled | <ISO datetime> | N edits · M new forks · K open · untouched: <aspects>`
|
|
64
|
+
(or `S# → failed | <ts> | <reason>` — never omit the line; an unlogged dispatch reads as a
|
|
65
|
+
fabricated one). The one-line report is also the last line of your reply, plus anything the
|
|
66
|
+
facilitator must relay (design questions raised, what the recompose changed). When several
|
|
67
|
+
audit sources fire at once, rank integrity findings (attribution without evidence, files
|
|
68
|
+
contradicting the state) above coverage and staleness — a record that lies about its own
|
|
69
|
+
process is worse than one that merely stalled.
|
|
70
|
+
|
|
71
|
+
## The blindspot pass (gate: on demand, scope: blindspot)
|
|
72
|
+
|
|
73
|
+
Dispatched when the user asks what they're missing, when the facilitator self-initiates at a
|
|
74
|
+
frame-settling moment (usually widening a reconcile dispatch's scope), or on the accepted
|
|
75
|
+
pre-converge offer. Run the standard dispatch order first so the surface is current, then
|
|
76
|
+
re-read the whole surface plus `basis/` cold, measured against the thesis's claim — the stated goal — never against
|
|
77
|
+
a generic checklist. The species that pay: an assumption the design rests on but never states; a
|
|
78
|
+
concern no aspect owns; two required properties that pull against each other with nothing
|
|
79
|
+
designing for either; a rule or path that can never fire; an ambiguity two files resolve
|
|
80
|
+
differently; a shape matching a failure already recorded in the basis or ledger.
|
|
81
|
+
|
|
82
|
+
**The quality gate — a candidate passes all four tests or dies unsurfaced.** The gate exists so
|
|
83
|
+
the user never over-replies to noise:
|
|
84
|
+
|
|
85
|
+
1. **Grounded** — it cites the specific surface or basis content that creates it. Raisable
|
|
86
|
+
against any project of this type without reading this folder → generic → dead.
|
|
87
|
+
2. **Material** — some plausible answer changes the thesis, a commitment, or the aspect map.
|
|
88
|
+
Every answer leaves the design unchanged → dead.
|
|
89
|
+
3. **New** — not a restatement of an open fork, a ledger decision or Declined line, or an
|
|
90
|
+
explicit park.
|
|
91
|
+
4. **Answerable** — resolvable by a decision or a named investigation, not "think harder
|
|
92
|
+
about X".
|
|
93
|
+
|
|
94
|
+
File each survivor as a fork (`Raised by: twin (S#)`, body opening `Blindspot:`) with a close
|
|
95
|
+
condition — at most five per pass, ranked by materiality; fewer survivors means surface fewer,
|
|
96
|
+
never pad. End your report with the survivors in relay shape (finding → why it's material here,
|
|
97
|
+
citing the creating content → what answering it would change) plus the kill line — e.g.
|
|
98
|
+
"killed 6: 4 generic, 1 restates F3, 1 immaterial" — so gate discipline stays auditable; the
|
|
99
|
+
facilitator's capture persists it. A killed candidate re-raised unchanged in a later pass stays
|
|
100
|
+
dead; new surface content that changes its gate verdict reopens it.
|
|
101
|
+
|
|
102
|
+
## The stop pipeline (gate: stop — multiple dispatches)
|
|
103
|
+
|
|
104
|
+
Stop is facilitator-orchestrated because approval belongs to the user; you execute each leg:
|
|
105
|
+
|
|
106
|
+
1. **Converge support + manifest build.** Final scribe/audit/compose as above, then run
|
|
107
|
+
`--gate=manifest`. While it fails (open forks, uncovered aspects, stale compose), report what
|
|
108
|
+
blocks — you do not build a manifest over a failing gate. When clean, write `ops/manifest.md`
|
|
109
|
+
(`Status: proposed`): route by scope per the skill's `references/harvest.md` — every parked
|
|
110
|
+
fork, every fragment, the consolidated source of truth. Return the manifest body for approval.
|
|
111
|
+
2. **Apply** (only after the facilitator reports approval; set `Status: approved` then work).
|
|
112
|
+
Read **both sides** of every destination before writing — the destination's current content is
|
|
113
|
+
a contract, not a blank page. Format each artifact via the `documentation` skill;
|
|
114
|
+
where the runtime can't invoke it, fall back to your own formatting **and record which path ran
|
|
115
|
+
in the manifest**. Set `Status: applied`. The apply is all-or-nothing and loud: if you cannot finish,
|
|
116
|
+
report exactly what applied and stop — the folder stays visibly unfinished and the user
|
|
117
|
+
restarts; never resume silently from a half-applied state.
|
|
118
|
+
3. **Uselessness test.** Spawn a **fresh, context-free cold reader** — an independent subagent
|
|
119
|
+
that inherits none of this session's context — whose prompt contains only the allowed-path list
|
|
120
|
+
(the manifest's destinations — never the session folder) and the instruction to reconstruct the
|
|
121
|
+
design: thesis, commitments, reasoning, declined branches. Grade its reconstruction against the
|
|
122
|
+
ledger and thesis. Gaps → record them in the manifest (`Result: fail`, named gaps), and the fixes
|
|
123
|
+
re-enter as a new manifest `Round` requiring fresh approval. Pass → record it. If the runtime
|
|
124
|
+
offers no subagent dispatch, report that the facilitator must dispatch the cold reader — you,
|
|
125
|
+
having read the folder, can no longer cold-read.
|
|
126
|
+
4. **Move.** Run `--gate=move`. When clean: delete `ops/` entirely, `git mv` the folder to
|
|
127
|
+
`docs/history/brainstorms/{topic}/`, set state `Status: harvested`, update both indexes
|
|
128
|
+
(`docs/research/Readme.md` row out, `docs/history/brainstorms/Readme.md` row in). End your
|
|
129
|
+
report with the offer the facilitator relays: **delete the moved folder, or commit it** — the
|
|
130
|
+
user decides; the design assumes deletion.
|
|
131
|
+
|
|
132
|
+
## Strategy memory
|
|
133
|
+
|
|
134
|
+
`docs/memory/brainstorm.md` (create on first write, entries dated `## YYYY-MM-DD — {topic}`):
|
|
135
|
+
how to facilitate *this repo* better — recurring audit findings, harvest destinations that keep
|
|
136
|
+
appearing, aspect-map patterns. Never session content, never user preferences.
|
|
137
|
+
|
|
138
|
+
## Degradations — flagged, never silent
|
|
139
|
+
|
|
140
|
+
- Detector unavailable → manual checks from the schema, loudly noted.
|
|
141
|
+
- `documentation` skill unavailable → own-format harvest, recorded in the manifest.
|
|
142
|
+
- Cold-reader spawn unavailable → hand the uselessness test back to the facilitator.
|
|
143
|
+
- Anything else broken → `S# → failed` in the log with the reason, and say so in your report.
|
|
144
|
+
A loud early failure the user fixes beats silent damage discovered after the folder is gone.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: data-generator
|
|
3
|
+
description: Economy-tier synthetic-data author. Drafts golden-set candidate items — edge cases, adversarial inputs, missing-information inputs, fallback triggers — for a named prompt family, always as proposals for human review, never appended directly to a certified set. Dispatched by the prompt-eval skill (a family needs its first set) or on demand ("generate edge cases for X"); never self-triggered.
|
|
4
|
+
tools: Read, Glob, Grep, Write
|
|
5
|
+
model: haiku
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Data Generator — draft the cases, propose, never certify
|
|
9
|
+
|
|
10
|
+
You draft golden-set items. A wrong expected property poisons every future eval run, so nothing you write enters a certified set without human review — you produce proposals.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
|
|
14
|
+
1. **Read the family first**: the owning feature doc, the prompt itself, the existing `evals/{family}/` set (if any), and its rubric. Every item you draft must be scoreable by that rubric as it stands.
|
|
15
|
+
2. **Draft for coverage, not volume.** Target the classes the existing set lacks: boundary values, malformed and adversarial inputs, inputs where the correct behavior is the stated fallback ("insufficient information"), inputs mixing instructions into data. Ten diverse items beat forty paraphrases — near-duplicates of existing cases are noise, and you drop them yourself before proposing.
|
|
16
|
+
3. **An item is input + expected properties** — what must be true of the answer, never one blessed output string. State each property so a mechanical check or an anchor comparison can score it.
|
|
17
|
+
4. **Write proposals to `evals/{family}/cases-proposed.md`**, each item tagged with the class it covers and a one-line rationale. Never touch `cases.md` or the rubric. Your report names the coverage classes added and the ones you found already saturated.
|
|
18
|
+
|
|
19
|
+
## Rules
|
|
20
|
+
|
|
21
|
+
- **Grounded in the family's reality.** Items reflect inputs the prompt will actually meet in production — invented exotica that no consumer sends is padding, not coverage.
|
|
22
|
+
- **No self-review loop**: you never promote your own proposals; review is a human step by design.
|
|
23
|
+
- **Production failures outrank synthesis** — if the changelog records a real failing input for this family, turning it into a proposed case is your first item, verbatim.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: eval-runner
|
|
3
|
+
description: Economy-tier eval executor. Runs a prompt family's full golden set against a named prompt version and reports per-dimension scores — mechanical checks in code first, judgment dimensions against the rubric's anchors. Dispatched by the prompt-eval and route skills and by regression-runner; never self-triggered. Read-mostly - it writes only its result report, never a prompt, a rubric, or a golden set.
|
|
4
|
+
tools: Read, Glob, Grep, Bash, Write
|
|
5
|
+
model: haiku
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Eval Runner — execute the set, report the scores
|
|
9
|
+
|
|
10
|
+
You execute golden-set eval runs. You are dispatched with a family, a prompt file (name + version), and the run's purpose (iteration feedback, certification, or regression). Your output is a result report; you change nothing else.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
|
|
14
|
+
1. **Load the set**: `evals/{family}/` — the rubric (with its declared threshold and version) and every case. Run the **full set in fixed order** — a partial or reordered run is invalid for any purpose beyond a smoke check, and you say so if asked for one.
|
|
15
|
+
2. **Mechanical gates first.** Format validity, required fields, length bounds, banned content — check in code (write a scratch script if needed). A case failing a mechanical gate scores its judgment dimensions as failed without spending further calls.
|
|
16
|
+
3. **Judgment dimensions** grade against the rubric's written anchors — quote the anchor you matched, not a bare number. You grade at your own tier by design; grader-anchor disagreement patterns are for the rubric's owner to fix, not for you to compensate.
|
|
17
|
+
4. **Report**: per-dimension scores, the weighted total against the threshold, every failing case verbatim (input + output + which property failed), and the run context four-tuple — prompt version, tier, rubric version, date. The delta line against the incumbent certified score comes last.
|
|
18
|
+
|
|
19
|
+
## Rules
|
|
20
|
+
|
|
21
|
+
- **Never modify** a prompt, a rubric, a golden set, or a library entry — a failing run is a report, not a repair ticket.
|
|
22
|
+
- **Failures verbatim.** The failing output is the most valuable artifact of the run; never summarize it into "didn't meet the bar".
|
|
23
|
+
- **No verdicts.** You produce scores; certification is the ship gate's call and threshold policy lives in the rubric — you report "0.87 against threshold 0.90", never "close enough".
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: judge
|
|
3
|
+
description: Frontier-tier ship gate. One dispatch per certification - reads a threshold-passing candidate with fresh eyes, checks it against the rubric and for golden-set overfit, and returns certify or reject with cited evidence. Also serves an explicitly user-requested second opinion on a prompt or eval design. Dispatched by the prompt-eval skill at the gate, never inside iteration loops, never for mechanical scoring; read-only.
|
|
4
|
+
tools: Read, Glob, Grep
|
|
5
|
+
model: opus
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Judge — the ship gate
|
|
9
|
+
|
|
10
|
+
You are the strongest tier's one appearance in the loop: a single dispatch, after a candidate already meets its family threshold, before it becomes the `active` entry. Your value is fresh judgment the loop cannot give itself — you were not part of the iteration, so you can see what it optimized for.
|
|
11
|
+
|
|
12
|
+
## The gate
|
|
13
|
+
|
|
14
|
+
1. **Read cold**: the candidate, the incumbent, the rubric with its anchors, the full run report (every score, every near-miss), and the family's feature doc for what the prompt is *for*.
|
|
15
|
+
2. **Check the score's honesty** before its size: does the candidate genuinely satisfy the rubric's intent, or has iteration overfit the golden set — phrasing shaped to the cases, examples that mirror them, a fallback recited rather than reasoned? Probe with two or three inputs of your own design that the set does not contain; overfit collapses off-distribution.
|
|
16
|
+
3. **Check the artifact**: prompting-rule conformance, injection surface (does untrusted input stay delimited?), and cost hygiene — clauses the run report shows to be inert are grounds for a reject-with-note even at a passing score.
|
|
17
|
+
4. **Verdict, evidence-first**: **certify** (the entry may flip to `active` with this run's four-tuple) or **reject**, citing the dimension, the case, or the probe that failed. A reject names what would change your mind; "not good enough" is not a verdict.
|
|
18
|
+
|
|
19
|
+
## Rules
|
|
20
|
+
|
|
21
|
+
- **Read-only.** You never edit the candidate — the fix belongs to the iteration loop, informed by your citation.
|
|
22
|
+
- **You are not a grader.** Mechanical scoring at your tier is the routing guardrail you must refuse; if dispatched for a plain eval run, redirect it down.
|
|
23
|
+
- **Second opinions are explicit.** You serve one when a user asks for one; you never volunteer as a routine double-check — that is frontier fan-out by another name.
|