task-pipeline-skill 1.79.1 → 1.81.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/CHANGELOG.md +224 -0
  2. package/CONTRIBUTING.md +15 -0
  3. package/README.md +4 -3
  4. package/SKILL-CARD.md +1 -1
  5. package/cursor/rules/task-pipeline.mdc +3 -1
  6. package/evals/RESULTS.md +212 -9
  7. package/evals/evidence-docs.evals.json +109 -0
  8. package/evals/project-audit.evals.json +108 -0
  9. package/evals/run.py +47 -19
  10. package/package.json +4 -2
  11. package/plugins/task-pipeline/.claude-plugin/plugin.json +3 -2
  12. package/plugins/task-pipeline/commands/task-pipeline.md +2 -1
  13. package/plugins/task-pipeline/hooks/gate-observer.sh +19 -2
  14. package/plugins/task-pipeline/skills/evidence-docs/SKILL.md +1 -0
  15. package/plugins/task-pipeline/skills/project-audit/SKILL.md +7 -2
  16. package/plugins/task-pipeline/skills/task-pipeline/SKILL.md +49 -56
  17. package/plugins/task-pipeline/skills/task-pipeline/pipeline.example.json +1 -1
  18. package/plugins/task-pipeline/skills/task-pipeline/references/acceptance.md +1 -1
  19. package/plugins/task-pipeline/skills/task-pipeline/references/adoption.md +15 -4
  20. package/plugins/task-pipeline/skills/task-pipeline/references/artifacts.md +4 -2
  21. package/plugins/task-pipeline/skills/task-pipeline/references/brainstorm.md +6 -1
  22. package/plugins/task-pipeline/skills/task-pipeline/references/build.md +11 -1
  23. package/plugins/task-pipeline/skills/task-pipeline/references/certification.md +7 -0
  24. package/plugins/task-pipeline/skills/task-pipeline/references/companion-skills.md +33 -2
  25. package/plugins/task-pipeline/skills/task-pipeline/references/documentation.md +2 -2
  26. package/plugins/task-pipeline/skills/task-pipeline/references/exposure.md +7 -3
  27. package/plugins/task-pipeline/skills/task-pipeline/references/gates.md +17 -185
  28. package/plugins/task-pipeline/skills/task-pipeline/references/model-tiering.md +13 -0
  29. package/plugins/task-pipeline/skills/task-pipeline/references/portability.md +1 -0
  30. package/plugins/task-pipeline/skills/task-pipeline/references/probing.md +202 -0
  31. package/plugins/task-pipeline/skills/task-pipeline/references/progress.md +8 -4
  32. package/plugins/task-pipeline/skills/task-pipeline/references/spec.md +60 -1
  33. package/plugins/task-pipeline/skills/task-pipeline/references/stages.md +32 -54
  34. package/plugins/task-pipeline/skills/task-pipeline/references/work-graph.md +1 -1
  35. package/plugins/task-pipeline/skills/task-pipeline/scripts/graph.py +28 -5
  36. package/plugins/task-pipeline/skills/task-pipeline/templates/backlog.md +6 -2
  37. package/plugins/task-pipeline/skills/task-pipeline/templates/run.md +2 -2
package/CHANGELOG.md CHANGED
@@ -1,5 +1,229 @@
1
1
  # Changelog
2
2
 
3
+ ## v1.81.1 — the anchors are derived, and a dormant plant stops reading as a pass
4
+
5
+ > **Why 1.81.1 and not 1.81.0.** A `v1.81.0` tag was cut from outside this run, against a tree that did not yet carry the run stamp, and its release round failed on the release-gap check. Nothing published; npm never left 1.80.0. The tag ruleset forbids deletion, so `v1.81.0` is declared dead in `docs/evidence/retro.md` → *Releases that carry no stamp* and this version carries the payload.
6
+
7
+ Board row **B-113**, filed 2026-08-22 and confirmed four times since: *negative-test
8
+ anchors are pinned to literals that the releases they guard move, so a release
9
+ disarms its own checks.* Every confirmation had the same cause and the same reason
10
+ nobody noticed — the plants were disarmed by the repository getting **healthier**.
11
+ The board re-derived its ages and `bd5`'s needle stopped existing; the first blind
12
+ eval runs shipped and `pf1`'s guard went dormant; a release finally carried an honest
13
+ run stamp and `gap1`'s precondition emptied. All four were found by the 35-minute
14
+ negatives suite at release time, one release after the damage, and until it ran
15
+ everything was green.
16
+
17
+ **The census, measured rather than estimated.** `test/anchors.py` reads all 422
18
+ negative self-tests out of the workflow and answers the row's own question — *is the
19
+ number the plant WRITES, or the number it LOOKS FOR?* — from the AST rather than by
20
+ grep. A needle is only what a plant reads out of a file, before it writes that path:
21
+ provenance is tracked from `open(...).read()` through derived names, read-backs are
22
+ tracked per path (a plant-wide "after the first write" rule exempts `pf1`'s second
23
+ file, which is a genuine anchor), regex shape is stripped (`[0-9a-f]{7,40}` is a
24
+ shape, `\u2192` in a raw string is an escape and not the year 2192), and a
25
+ `validate.py | grep 'message'` is assertion 3 rather than a needle. Counted over the
26
+ corpus at the commit this release branched from: **26 of 419 plants pinned a value a
27
+ release can move.** The row's own count of 27 came from a grep that both over- and
28
+ under-reported — it missed `bd5`'s single-digit `| 1 | **6** |` entirely — and this
29
+ release's own first figure of 21 was wrong for the same reason one level in: the
30
+ detector could not see a needle read through `enumerate()`, `pathlib` or a compiled
31
+ pattern. The R-005 reader found that by fixture, and it is why the number moved twice
32
+ before it was true.
33
+
34
+ **25 of the 26 are derived now**, one declared. Each reads its anchor out of the tree at run
35
+ time: the high-water mark by shape, the rule number the stage map binds, the seeded
36
+ templates' open row and worked example, the carry-over ledgers' board pointers, the
37
+ live board's rows, the criterion numbers, the caps, the container count and its date,
38
+ the recipe fence. The live-board plants were the sharpest: `bd10` pinned `| B-008 |`
39
+ and `tbl`/`tbl2` pinned `| B-015 |`, so all three would have stopped landing the day
40
+ those rows closed. One literal stays and says so: `res15` pins `^13\.` because the
41
+ criterion NUMBER is the *guard's* contract — `test/validate.py` locates the teardown
42
+ criterion as `^13\.` and its refusal names 13 — and deriving it would prove some other
43
+ criterion's guard fired instead.
44
+
45
+ **Dormant is no longer counted as passing.** `test/negatives.py` folded every plant
46
+ that printed `SKIP` into *all N guards provably reject their planted defect*: a check
47
+ that could not construct its precondition, reported inside the claim that every check
48
+ rejected one. Skips now have their own bucket, print with the reason they gave, and
49
+ the aggregate says *N of M · K DORMANT, named above — not counted as passing*. In CI
50
+ each plant is its own step and a green step carrying SKIP is invisible, so the half
51
+ that is checkable everywhere is the declaration: every plant that can decline to run
52
+ carries `# dormant-when: <the state it cannot construct, and what that costs>`, and
53
+ all four now do.
54
+
55
+ **The class is self-reporting, in the cheap gate rather than the 35-minute one.**
56
+ `test/validate.py` refuses a plant whose needle pins a moving value unless it declares
57
+ why, refuses a declaration that resolves to nothing the plant reads, refuses a skip
58
+ branch with no dormancy declaration, and refuses an empty census — because "no anchors
59
+ found" and "the parser matched nothing" are otherwise the same sentence. Four checks
60
+ ride into the suite with it, each watched red before shipping, and `test/anchors_test.py`
61
+ is **47 whole-workflow fixtures, 27 of them watched firing** — a number that itself moved
62
+ three times as the reader found what the census could not see. **Eighteen of the 45 cases assert SILENCE, and 16 of
63
+ those 18 are retractions** — the other two are positive controls (`a derived needle passes`,
64
+ `a declared anchor is accepted`) (the other two of the 47 checks are the empty-corpus
65
+ census and a source check, which assert neither) — a `{7,40}` quantifier read as a year, `\u2192` in a raw
66
+ string read as 2192, a payload built from local strings, a `validate.py | grep` read as a
67
+ needle, a plant echoing `B-008` into its own replacement, `json.dump(d, open(p, "w"))`
68
+ unrecognised as a write, a comprehension variable outliving its comprehension, and an
69
+ unquoted `grep` taking a filename for its pattern. The other 25 fire, and four of them
70
+ name blind spots the R-005 reader found: `enumerate()`, `pathlib.Path.read_text()`,
71
+ `re.compile` and a heredoc spelling.
72
+
73
+ **The R-005 reader found five blocking defects the author's own fixtures did not, and
74
+ this is the part worth keeping.** A dispatched independent reader wrote twenty
75
+ fixtures against the census and nineteen of them slipped past it. Provenance was lost
76
+ through `enumerate()`, and **two live plants — `wv1` and `wv2` — were still pinned to
77
+ `B-005` behind that blind spot** while the census called the corpus clean; the same for
78
+ `pathlib.Path(...).read_text()` (seven plants) and `re.compile` (one). `bd2` and `bd13`
79
+ had collapsed onto one row, so the dangling-id guard's well-formed branch went unprobed
80
+ and `bd13` no longer proved what its name claims. And three documents restated a census
81
+ of 21 that the detector itself put at 24. All five are closed, each with its own
82
+ fixture; the true figure moved **21 → 24 → 26** as the detector stopped being blind, and
83
+ that is the whole argument for the rule the reader exists to serve.
84
+
85
+ Consequently the census now reads provenance through builtin wrappers, `pathlib`,
86
+ `with`-handles, compiled patterns, `json.load`/`yaml.safe_load` and their structured
87
+ looks, `sed`, `awk`, an unquoted `grep` and a `subprocess` argv; it keys read-backs by
88
+ path SOURCE TEXT so a computed path matches itself; it refuses a heredoc spelling it
89
+ cannot read rather than reporting silence — while ignoring one quoted inside a payload,
90
+ because a guard reads what would RUN; and it judges a declaration on characters **and**
91
+ distinct words, because sixty dots satisfy a length floor.
92
+
93
+ **Round two of the same reader found four more, and one of them is this row's own
94
+ class for the third time.** `json.dump(d, open(p, "w"))` was not recorded as a write, so
95
+ the own-read-back of every plant that writes that way was one string literal from being
96
+ refused — 39 plants write that way and **none was flagged yet**, which is a latent false
97
+ positive and is said as one — the write side had been extended to
98
+ `Path().write_text()` and not to `json.dump`, two halves of one rule landing unmatched in
99
+ one commit. A loop or comprehension variable kept its file provenance for the rest of the
100
+ body, so an unrelated later local read as file text. Needles 891 → **832**, and most of that
101
+ drop is a second defect: a comprehension was dispatched twice, so needles inside its `iter`
102
+ and `ifs` were counted twice over. The size of that share is **not stated here** — two
103
+ methods of counting it disagreed (50 and 88) and neither has a command written beside it,
104
+ which is the same defect one level up. 891 shipped as a measurement in a ledger row.
105
+ The no-needle disclosure explained 39 plants as *a JSON key, which raises rather than
106
+ passing*, and by the classifier that now ships exactly **1 of those 39** was a JSON or
107
+ dict key: 13 were shell-only, 11 read no file, 10 compared whole-file bytes and 4 lost
108
+ provenance at a helper function — so the breakdown is computed now — **five**
109
+ reasons at this tree, none of them a gloss, and the fifth (`provenance stopped at a helper
110
+ function`) is the one the gloss was hiding. And the fixture count was **stated as 18/8 in one sentence
111
+ and 35/20 in the next**, which is B-113 itself, third occurrence inside its own fix
112
+ (21 → 24 → 26, then 18/8 → 44/25 → 47/27). Every number in this entry and in the ledger rows is
113
+ now pasted from the command that prints it.
114
+
115
+ **A pure module for the arithmetic nobody can reach.** The dormant branch cannot fire on
116
+ a healthy machine — all four skip-capable plants ran and their guards fired when the
117
+ reader drove them end to end, because the runner rebuilds `.git` in its snapshot and CI
118
+ fetches tags. So the two declarations that claimed otherwise were corrected after
119
+ measurement, and `verdict()` and `claim()` became pure functions with
120
+ `test/runner_test.py` calling them directly: 14 cases, two of them the empty-set branch
121
+ that used to print `PASS: K DORMANT` over nothing at all.
122
+
123
+ **One defect this change surfaced in a neighbour, fixed in the same commit.**
124
+ `CONTRIBUTING.md`'s invariant list cites the guard literal that enforces each entry, and
125
+ `test/validate.py` looked for those literals in its own source only — so the first
126
+ citation of an anchors-census refusal was reported as *an enforcement that does not
127
+ exist* while the check printing it sat in the same suite. The corpus is now this file
128
+ plus the sibling modules it imports, discovered rather than listed. Invariant **64**
129
+ carries the rule.
130
+
131
+ **The loop guard tripped, and the trip is the release's own best evidence.** The fixture
132
+ count returned twice with the same stale figure — inside the paragraph that narrates a
133
+ stale figure as the row's lesson — because each fix APPENDED a corrected sentence instead
134
+ of correcting the one already there. `build.md:417` says a tripped guard is not another
135
+ round: stop, name the two shapes, escalate to the layer that owns the conflict. The layer
136
+ is canon 8, and the escalation is mechanical: **`anchor-census fixtures` and
137
+ `anchor-census fixtures watched firing` are registered claim classes now**, computed from
138
+ the fixture file, with a plant watching them fail. Registered classes 13 → 15. Three
139
+ figures this entry cannot compute are **deleted rather than corrected** — the share of the
140
+ 891 that was double-counted, the number of plants narrowing by order, and one the reader
141
+ retracted itself — because a number whose definition is not written beside it is the
142
+ defect this whole release is about.
143
+
144
+ Guards: 419 → **423** — the re-pinned needle, the undeclared dormancy and the
145
+ declaration that resolves to nothing, plus a property check that the census prints its
146
+ disclosure beside the verdict.
147
+
148
+ ## v1.80.0 — the doctrine pack: one home per rule, and the boundaries said out loud
149
+
150
+ The largest doctrine reconciliation since the corpus was audited whole. The
151
+ wave-2/3 audit read all ~717KB of shipped doctrine against itself and this
152
+ release closes its deferred pack: every contradiction between a short-form
153
+ gate and its long form, every count restated where a list already was the
154
+ count, and the two closure protocols that had no boundary sentence between
155
+ them.
156
+
157
+ **Two closure protocols, one boundary.** A work-graph node closes through the
158
+ three blind tiers and `graph.py certify` (ceiling 3); a prose-plan task closes
159
+ through `build.md` §4.4's single reviewer (five-round cap). Both were true and
160
+ nothing said which applied — a run could pick whichever was cheaper at the
161
+ moment of closing. The boundary sentence now lives in `SKILL.md`'s routing
162
+ row, `certification.md` and `build.md`: the artifact the queue is decides.
163
+
164
+ **The COPY/VISUAL track doctrine moved into `spec.md` whole** — its one home —
165
+ and `stages.md`'s stage 3 keeps a routed summary; the example config's stage-3
166
+ check now demands the tracks and their convergence record. `references/
167
+ probing.md` is new: the four probe-authoring sections extracted from
168
+ `gates.md` (plant/run/restore, how a probe rots, the neighbour probe, the
169
+ landed-mutation rule), routed from the gate doctrine they prove; the guards
170
+ and plants that read those sections moved with them, and the bundle's
171
+ reference-file counts were re-derived to 38 by the registry that refuses a
172
+ stale one.
173
+
174
+ **The `judgment` gate type reaches every surface that taught `auto`/`manual`**
175
+ — SKILL.md (three places), stages.md, the command, the Cursor rule, README,
176
+ plugin.json — and a new cross-surface guard compares the schema's enum against
177
+ the sentence that teaches it, so the next type cannot sit seven releases in
178
+ the schema alone. The browser gate cells say when the look binds and what the
179
+ weaker claim is; brainstorm's GATE names the queue-artifact and loop-arming
180
+ criteria; the loop arms where `run.loop.arm` says, not unconditionally at
181
+ preflight; stage-0 seeding covers the three gate scripts stages 5–10 run;
182
+ the template board's State vocabulary admits `parked`/`waived`; project-audit
183
+ prices with the board header's declared formula and never effort, and its
184
+ disambiguation table gains the fifth audit (`/seo-aeo-audit`). Four domain
185
+ companions (sheleg-dev, agent-stack, telegram-dev, seo-aeo-audit) enter the
186
+ matrix, the preflight block and the stages they bind — recommended, never a
187
+ gate. `model-tiering.md` gains the per-generation stress-test rule and the
188
+ judgment gate the rubric-steering note, both with dated sources.
189
+
190
+ **Mechanics: the PR gate now runs what `npm test` documents.** `graph_test.py`
191
+ and `project_audit_test.py` reached CI only at tag time; a PR breaking
192
+ `graph.py` merged green. The blind rule covers every prose field of a tier
193
+ report — a cross-tier citation hiding in `scope` or a finding's `what`/`fix`
194
+ is refused, fixtured, and mutation-proven (17/17 noticed, control 22).
195
+ `gate-observer.sh` reads the documented `tool_response` payload with
196
+ `tool_output` as fallback — the exit-code branch had been dead on the
197
+ documented shape — with fixtures on both shapes, each watched red against the
198
+ reverted hook. The upstream strict validation is its own CI job, so an npm
199
+ outage and a house failure stop masking each other.
200
+
201
+ **Ledgers: the board re-ranks on real ages.** Every open row's Age and Prio
202
+ re-derived against its Source date — rows filed 08-17/08-20 still said Age 0
203
+ on 08-29 under a header claiming re-derivation. The verification ledger's
204
+ `Shipped state` header is gated against `package.json` (it had sat six
205
+ releases stale), the two 2026-08-22 run stamps stop claiming retro entries
206
+ that do not exist, the two rows frozen at `unreleased` gain superseding rows
207
+ naming v1.72.0, and v1.76.0's absence from the ledger is stated beside the
208
+ 08-20 note. Five board rows filed from the harness-engineering articles
209
+ (B-114…B-117) and from this release's own probes (B-118).
210
+
211
+ **Evals: three blind runs, one per model.** evidence-docs and project-audit
212
+ gain behavioural suites with near-miss negatives («аудит модуля» one word
213
+ from «аудит проекта»); `evals/run.py` discovers every suite instead of the
214
+ one filename it shipped pinned to; and 147 fresh-subagent probes (49 queries
215
+ × haiku/sonnet/opus) produced the first dated blind rows in both RESULTS
216
+ files — opus 42/42, sonnet 40/42, haiku 39/42, with haiku's three misses
217
+ sharing one shape that is now B-118. Method and limits stated in the ledgers,
218
+ INSTR cases recorded as not reproducible rather than guessed.
219
+
220
+ `compatibility` frontmatter on task-pipeline and project-audit; a 3-line
221
+ Degradation block in SKILL.md paid for by the model blockquote it displaced;
222
+ `$schema` in both manifests.
223
+
224
+ Guards: 417 → **419** — the gate-type vocabulary plant and the stale
225
+ shipped-state-header plant, both watched red locally before shipping.
226
+
3
227
  ## v1.79.1 — the probe that a healthy release disarmed
4
228
 
5
229
  The first round of `v1.79.0` failed in the release job on the
package/CONTRIBUTING.md CHANGED
@@ -592,6 +592,21 @@ file about itself. Checked in the live retro and in `templates/retro.md`, which
592
592
  false bound into every host project.
593
593
  *(guard: `declares the bound` and `and nothing enforces it`)*
594
594
 
595
+ **64. A negative self-test may not promise that the tree will not change, and a plant that
596
+ declines to run is not a plant that passed.** B-113, confirmed four times: a needle pinned to a
597
+ value a release moves stops landing, and reports green while it does — and three of the four
598
+ instances were caused by the repository getting *healthier* (a re-derived board, the first blind
599
+ eval runs, the first honest run stamp), so nothing about them looked like a regression. Every
600
+ needle a plant reads off disk before writing that path is extracted from the AST by
601
+ `test/anchors.py`; one carrying a value a release can move must be derived at run time or
602
+ declared in the plant's own body with what would falsify the declaration. Separately, any check
603
+ that can print `SKIP` — a negative self-test **or a property check** — declares the state it
604
+ cannot construct and what that costs, and `test/negatives.py` reports it by name instead of
605
+ counting it inside *all N guards provably reject* or *N property check(s) printed what they
606
+ assert*. Where nothing that ran proved anything the runner reports INCONCLUSIVE, because a
607
+ pass over an empty set is a refused measurement.
608
+ *(guard: `is anchored on a value a release can move` and `can print SKIP and declares no` and `the anchor census found no negative self-tests`)*
609
+
595
610
  ## Adding or changing doctrine
596
611
 
597
612
  - **Change one idea per PR.** These files are read by agents under load; a PR that
package/README.md CHANGED
@@ -587,7 +587,7 @@ must say so.
587
587
  ### Held to Anthropic's own Skill authoring guidance
588
588
 
589
589
  Audited against the four Agent Skills pages. Most of it already held — `name`
590
- 13/64 chars, `description` inside 1024, `SKILL.md` 334/500 lines, all 37 references
590
+ 13/64 chars, `description` inside 1024, `SKILL.md` 334/500 lines, all 38 references
591
591
  linked **directly** from `SKILL.md`, 436 KB against a 30 MB ceiling. What did not,
592
592
  now does:
593
593
 
@@ -787,7 +787,7 @@ Stages 0→10 above are the plugin's **example** flow. It is a machine-readable
787
787
  written against a universal contract
788
788
  ([`pipeline.schema.json`](plugins/task-pipeline/skills/task-pipeline/pipeline.schema.json)):
789
789
  copy the example to `pipeline.json` in your repo and rewrite it with your own stages
790
- (any count), your own `skills[]`, and your own `auto`/`manual` gate types. The
790
+ (any count), your own `skills[]`, and your own `auto`/`judgment`/`manual` gate types. The
791
791
  framework bakes in no fixed stage count and no opinion on which gates are manual.
792
792
 
793
793
  ```jsonc
@@ -932,7 +932,8 @@ recommendation, so you arm the whole run in one exchange. Detail:
932
932
  | [`references/artifacts.md`](plugins/task-pipeline/skills/task-pipeline/references/artifacts.md) | the canonical document layout each stage writes to |
933
933
  | [`references/conventions.md`](plugins/task-pipeline/skills/task-pipeline/references/conventions.md) | how stages 6–10 read the host project's `CLAUDE.md`, and how the documentation regime is detected |
934
934
  | [`references/documentation.md`](plugins/task-pipeline/skills/task-pipeline/references/documentation.md) | the doc system: the inventory, registers and ids, SSOT, the Doc Loop, supersede semantics, the propagation matrix, intent vs as-built |
935
- | [`references/gates.md`](plugins/task-pipeline/skills/task-pipeline/references/gates.md) | the three axes, the promotion ladder, gate anatomy, the probe recipe, ratchet floors, where a gate runs |
935
+ | [`references/gates.md`](plugins/task-pipeline/skills/task-pipeline/references/gates.md) | the three axes, the promotion ladder, gate anatomy, ratchet floors, where a gate runs |
936
+ | [`references/probing.md`](plugins/task-pipeline/skills/task-pipeline/references/probing.md) | the probe recipe — plant, run, restore; how a probe rots; the neighbour probe; the landed-mutation rule |
936
937
  | [`references/deploy-targets.md`](plugins/task-pipeline/skills/task-pipeline/references/deploy-targets.md) | stages 7–8: writing the runbook when there is none, the template, per-platform deploy and log verbs, the verification trio |
937
938
  | [`references/hooks.md`](plugins/task-pipeline/skills/task-pipeline/references/hooks.md) | the `PreToolUse` contract, the fail-open hazard, placement, and the Claude-Code-only limit |
938
939
  | [`references/knowledge-graph.md`](plugins/task-pipeline/skills/task-pipeline/references/knowledge-graph.md) | the code graph: install line, stage-0 reach queries, the stage-9 refresh, the graph↔docs divergence check |
package/SKILL-CARD.md CHANGED
@@ -12,7 +12,7 @@ harmless.
12
12
  |---|---|
13
13
  | **Purpose** | Runs a substantial task through ten gated delivery stages — intake grill, docs study, brainstorm, spec, plan, subagent build, tests, lint/deploy, post-deploy, docs+registers, acceptance — refusing to advance until each gate passes |
14
14
  | **Owner** | ssheleg ([github.com/ssheleg/task-pipeline](https://github.com/ssheleg/task-pipeline)) |
15
- | **Version** | 1.79.1 |
15
+ | **Version** | 1.81.1 |
16
16
  | **Surface** | Claude Code (filesystem skill + plugin) and the vercel `skills` CLI. **Not** uploaded to the Skills API; custom Skills do not sync across surfaces |
17
17
  | **Dependencies** | None required. Optional: `context7` (MCP), `figma` (MCP), super-ux, agent-sync, graphify, obsidian-wiki, and **one of two browser channels** — `playwright` (CLI or MCP) or `chrome-devtools` (MCP); either satisfies the browser step and neither is required. Every stage's doctrine ships in-repo; the one conditional requirement is super-ux for the stage-3 UX track on a user-facing task |
18
18
  | **Evaluation status** | Suite authored, 5 categories. One recorded run, **self-observed by the author**; **zero blind runs on zero of three models** — the split, and the numbers, live in [`evals/RESULTS.md`](evals/RESULTS.md) and are computed by `evals/run.py` |
@@ -499,7 +499,9 @@ type the run produced, then run the documentation gate and print its ratchet cou
499
499
  ## Gates: three axes, and how to build one that cannot lie
500
500
 
501
501
  **Axis A — the stage gate type.** `auto` = verify the check yourself and stop on
502
- fail; `manual` = wait for an explicit operator go. An auto gate never substitutes
502
+ fail; `judgment` = a named judge rules where no deterministic check exists, and the
503
+ ruling is recorded as judgement, never as a measurement; `manual` = wait for an
504
+ explicit operator go. Neither of the first two ever substitutes
503
505
  for a required manual approval.
504
506
 
505
507
  **Axis B — where the rule is actually enforced**, and a rule climbs this ladder
package/evals/RESULTS.md CHANGED
@@ -1,9 +1,10 @@
1
1
  # Evaluation results — task-pipeline
2
2
 
3
- **Status: the suite is authored. One run is recorded and it was self-observed by the
4
- author; no blind run has been made on any model.** Recorded this way rather than left
5
- blank, because an empty results file and an unrun suite look identical, and this
6
- repository's own doctrine calls that the failure — a skip is not a pass.
3
+ **Status: three suites authored (task-pipeline, evidence-docs, project-audit);
4
+ one self-observed run and, since 2026-08-31, three blind trigger runs one per
5
+ model (haiku, sonnet, opus).** The instruction-following cases remain unexecuted
6
+ and say so per run, because they need a full interactive session — a skip is not
7
+ a pass, and a guessed verdict is worse than a vacant row.
7
8
 
8
9
  **The numbers below are computed, not asserted.** `python3 evals/run.py` counts the
9
10
  suite and the dated run headings in this file. A value typed here that disagrees with
@@ -39,11 +40,11 @@ or it belongs in a check.
39
40
 
40
41
  | Metric | Value | Computed by | As of |
41
42
  |---|---|---|---|
42
- | Evals authored | 15 | `python3 evals/run.py` → `suite: N evals` | 2026-08-08 |
43
- | Categories covered | 5 of 5 | the suite's own `category` fields | 2026-08-08 |
44
- | Dated runs recorded | **1** | `python3 evals/run.py` → `recorded runs: N` | 2026-08-08 |
45
- | …of those, **blind** | **0** | run headings not marked `self-observed` | 2026-08-08 |
46
- | Models exercised blind | **0 of 3** | distinct models across blind runs | 2026-08-08 |
43
+ | Evals authored | 46 (three suites: task-pipeline 28, evidence-docs 9, project-audit 9) | `python3 evals/run.py` → `suite: N evals` | 2026-08-31 |
44
+ | Categories covered | 5 of 5, per suite | the suite's own `category` fields | 2026-08-31 |
45
+ | Dated runs recorded | **4** | `python3 evals/run.py` → `recorded runs: N` | 2026-08-31 |
46
+ | …of those, **blind** | **3** | run headings not marked `self-observed` | 2026-08-31 |
47
+ | Models exercised blind | **3 of 3** | distinct models across blind runs | 2026-08-31 |
47
48
 
48
49
  **The last three rows are the honest state of this skill's behavioural evidence**, and
49
50
  the split matters more than the total: a run the author watched, knowing the expected
@@ -67,6 +68,208 @@ green structural suite is never read as "the skill is known to work".
67
68
  Newest first. Each entry states the model, the task, and how the evidence was
68
69
  obtained — a self-observed run and a blind one are not the same claim.
69
70
 
71
+ **Method (wave-3 subagent protocol, and its limits).** One FRESH general-purpose
72
+ subagent per query per model, launched from a Claude Code harness with the model
73
+ pinned. Each received: the query verbatim; the family's 28 skill
74
+ names-with-descriptions (built once from the members' `SKILL.md` frontmatters,
75
+ including evidence-docs and project-audit from this plugin); the instruction to
76
+ answer with one skill name from that list, or `none`. Blind: no expected answer
77
+ was named. Limits, stated rather than implied: the harness's OWN installed-skill
78
+ inventory is also visible to a subagent's system prompt — a real coexistence
79
+ environment rather than a cleanroom, and four early sonnet probes answered from
80
+ it without opening the provided list (one chose `build-dashboard`, a skill
81
+ outside the family; recorded as the miss it is, and the remaining prompts were
82
+ tightened to "choose only from that file's list"). A trigger `pass` means the
83
+ model NAMED the skill; it does not prove a full session loaded and obeyed the
84
+ doctrine — the INSTR cases need exactly that and are recorded as **not
85
+ reproducible from this harness**, not guessed.
86
+
87
+ ## 2026-08-31 · haiku · blind trigger probes, subagent protocol
88
+
89
+ **39 of 42 scored cases passed** across the three suites' trigger-shaped
90
+ categories (should_trigger, should_not_trigger; ambiguous and coexistence
91
+ recorded unscored). Three misses, one shape: checkup/setup/prod-check phrasings («что у нас накопилось непроверенного?», «проверь документацию проекта…», «проверь, всё ли живо в проде…») routed to `project-audit` instead of task-pipeline's own checkup/setup modes — the two skills now share the "what is unverified/true" ground and the cheaper model takes the newer, narrower description.
92
+
93
+ task-pipeline suite:
94
+
95
+ | id | verdict | what happened |
96
+ |---|---|---|
97
+ | TRIG-01 | pass | named `task-pipeline` |
98
+ | TRIG-02 | pass | named `task-pipeline` |
99
+ | TRIG-03 | pass | named `task-pipeline` |
100
+ | NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
101
+ | NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
102
+ | NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
103
+ | AMB-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
104
+ | AMB-02 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
105
+ | COEX-01 | observed | answered `ux-flows` (ambiguous/coexistence — recorded, not scored) |
106
+ | TRIG-04 | pass | named `task-pipeline` |
107
+ | NOTRIG-04 | pass | answered `none` — did not route to the excluded skill |
108
+ | TRIG-05 | fail | answered `project-audit` |
109
+ | TRIG-06 | fail | answered `project-audit` |
110
+ | NOTRIG-05 | pass | answered `none` — did not route to the excluded skill |
111
+ | TRIG-07 | pass | named `task-pipeline` |
112
+ | TRIG-08 | pass | named `task-pipeline` |
113
+ | TRIG-09 | fail | answered `project-audit` |
114
+ | TRIG-10 | pass | named `task-pipeline` |
115
+ | NOTRIG-06 | pass | answered `seo-aeo-audit` — did not route to the excluded skill |
116
+ | NOTRIG-07 | pass | answered `ux-audit` — did not route to the excluded skill |
117
+ | NOTRIG-08 | pass | answered `make-skill` — did not route to the excluded skill |
118
+
119
+ evidence-docs suite:
120
+
121
+ | id | verdict | what happened |
122
+ |---|---|---|
123
+ | TRIG-01 | pass | named `evidence-docs` |
124
+ | TRIG-02 | pass | named `evidence-docs` |
125
+ | TRIG-03 | pass | named `evidence-docs` |
126
+ | NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
127
+ | NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
128
+ | NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
129
+ | AMB-01 | observed | answered `evidence-docs` (ambiguous/coexistence — recorded, not scored) |
130
+ | COEX-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
131
+
132
+ project-audit suite:
133
+
134
+ | id | verdict | what happened |
135
+ |---|---|---|
136
+ | TRIG-01 | pass | named `project-audit` |
137
+ | TRIG-02 | pass | named `project-audit` |
138
+ | TRIG-03 | pass | named `project-audit` |
139
+ | NOTRIG-01 | pass | answered `task-pipeline` — did not route to the excluded skill |
140
+ | NOTRIG-02 | pass | answered `task-pipeline` — did not route to the excluded skill |
141
+ | NOTRIG-03 | pass | answered `make-skill` — did not route to the excluded skill |
142
+ | AMB-01 | observed | answered `project-audit` (ambiguous/coexistence — recorded, not scored) |
143
+ | COEX-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
144
+
145
+ INSTR-01…07 (task-pipeline) and INSTR-01 (evidence-docs, project-audit): **not
146
+ reproducible from this harness** — each needs a full interactive pipeline run,
147
+ and a guessed verdict would be the exact substitution this file exists to refuse.
148
+
149
+ ## 2026-08-31 · sonnet · blind trigger probes, subagent protocol
150
+
151
+ **40 of 42 scored cases passed** across the three suites' trigger-shaped
152
+ categories (should_trigger, should_not_trigger; ambiguous and coexistence
153
+ recorded unscored). Two misses: TRIG-03 (`build a support-agent dashboard…`) answered `build-dashboard` — a skill OUTSIDE the offered list, from the harness's own inventory, before the choose-only-from-list tightening; TRIG-10 («посмотри PR #24 и скажи, что там не так») answered `none` — the PR-review findings class did not reach sonnet from the description alone.
154
+
155
+ task-pipeline suite:
156
+
157
+ | id | verdict | what happened |
158
+ |---|---|---|
159
+ | TRIG-01 | pass | named `task-pipeline:task-pipeline` |
160
+ | TRIG-02 | pass | named `task-pipeline` |
161
+ | TRIG-03 | fail | answered `build-dashboard` |
162
+ | NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
163
+ | NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
164
+ | NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
165
+ | AMB-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
166
+ | AMB-02 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
167
+ | COEX-01 | observed | answered `ux-flows` (ambiguous/coexistence — recorded, not scored) |
168
+ | TRIG-04 | pass | named `task-pipeline` |
169
+ | NOTRIG-04 | pass | answered `none` — did not route to the excluded skill |
170
+ | TRIG-05 | pass | named `task-pipeline` |
171
+ | TRIG-06 | pass | named `task-pipeline` |
172
+ | NOTRIG-05 | pass | answered `none` — did not route to the excluded skill |
173
+ | TRIG-07 | pass | named `task-pipeline` |
174
+ | TRIG-08 | pass | named `task-pipeline` |
175
+ | TRIG-09 | pass | named `task-pipeline` |
176
+ | TRIG-10 | fail | answered `none` |
177
+ | NOTRIG-06 | pass | answered `none` — did not route to the excluded skill |
178
+ | NOTRIG-07 | pass | answered `super-ux:ux-audit` — did not route to the excluded skill |
179
+ | NOTRIG-08 | pass | answered `make-skill:skill-audit` — did not route to the excluded skill |
180
+
181
+ evidence-docs suite:
182
+
183
+ | id | verdict | what happened |
184
+ |---|---|---|
185
+ | TRIG-01 | pass | named `evidence-docs` |
186
+ | TRIG-02 | pass | named `evidence-docs` |
187
+ | TRIG-03 | pass | named `evidence-docs` |
188
+ | NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
189
+ | NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
190
+ | NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
191
+ | AMB-01 | observed | answered `evidence-docs` (ambiguous/coexistence — recorded, not scored) |
192
+ | COEX-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
193
+
194
+ project-audit suite:
195
+
196
+ | id | verdict | what happened |
197
+ |---|---|---|
198
+ | TRIG-01 | pass | named `task-pipeline:project-audit` |
199
+ | TRIG-02 | pass | named `task-pipeline:project-audit` |
200
+ | TRIG-03 | pass | named `project-audit` |
201
+ | NOTRIG-01 | pass | answered `task-pipeline` — did not route to the excluded skill |
202
+ | NOTRIG-02 | pass | answered `task-pipeline` — did not route to the excluded skill |
203
+ | NOTRIG-03 | pass | answered `make-skill:skill-audit` — did not route to the excluded skill |
204
+ | AMB-01 | observed | answered `task-pipeline:project-audit` (ambiguous/coexistence — recorded, not scored) |
205
+ | COEX-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
206
+
207
+ INSTR-01…07 (task-pipeline) and INSTR-01 (evidence-docs, project-audit): **not
208
+ reproducible from this harness** — each needs a full interactive pipeline run,
209
+ and a guessed verdict would be the exact substitution this file exists to refuse.
210
+
211
+ ## 2026-08-31 · opus · blind trigger probes, subagent protocol
212
+
213
+ **42 of 42 scored cases passed** across the three suites' trigger-shaped
214
+ categories (should_trigger, should_not_trigger; ambiguous and coexistence
215
+ recorded unscored). No misses. All four audit near-misses routed away correctly (module audit → task-pipeline, skill standard → make-skill, landing-page traffic → seo-aeo-audit, UX conformance → ux-audit).
216
+
217
+ task-pipeline suite:
218
+
219
+ | id | verdict | what happened |
220
+ |---|---|---|
221
+ | TRIG-01 | pass | named `task-pipeline` |
222
+ | TRIG-02 | pass | named `task-pipeline` |
223
+ | TRIG-03 | pass | named `task-pipeline` |
224
+ | NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
225
+ | NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
226
+ | NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
227
+ | AMB-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
228
+ | AMB-02 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
229
+ | COEX-01 | observed | answered `ux-flows` (ambiguous/coexistence — recorded, not scored) |
230
+ | TRIG-04 | pass | named `task-pipeline` |
231
+ | NOTRIG-04 | pass | answered `none` — did not route to the excluded skill |
232
+ | TRIG-05 | pass | named `task-pipeline` |
233
+ | TRIG-06 | pass | named `task-pipeline` |
234
+ | NOTRIG-05 | pass | answered `none` — did not route to the excluded skill |
235
+ | TRIG-07 | pass | named `task-pipeline` |
236
+ | TRIG-08 | pass | named `task-pipeline` |
237
+ | TRIG-09 | pass | named `task-pipeline` |
238
+ | TRIG-10 | pass | named `task-pipeline` |
239
+ | NOTRIG-06 | pass | answered `seo-aeo-audit` — did not route to the excluded skill |
240
+ | NOTRIG-07 | pass | answered `ux-audit` — did not route to the excluded skill |
241
+ | NOTRIG-08 | pass | answered `make-skill` — did not route to the excluded skill |
242
+
243
+ evidence-docs suite:
244
+
245
+ | id | verdict | what happened |
246
+ |---|---|---|
247
+ | TRIG-01 | pass | named `evidence-docs` |
248
+ | TRIG-02 | pass | named `evidence-docs` |
249
+ | TRIG-03 | pass | named `evidence-docs` |
250
+ | NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
251
+ | NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
252
+ | NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
253
+ | AMB-01 | observed | answered `evidence-docs` (ambiguous/coexistence — recorded, not scored) |
254
+ | COEX-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
255
+
256
+ project-audit suite:
257
+
258
+ | id | verdict | what happened |
259
+ |---|---|---|
260
+ | TRIG-01 | pass | named `project-audit` |
261
+ | TRIG-02 | pass | named `project-audit` |
262
+ | TRIG-03 | pass | named `project-audit` |
263
+ | NOTRIG-01 | pass | answered `task-pipeline` — did not route to the excluded skill |
264
+ | NOTRIG-02 | pass | answered `task-pipeline` — did not route to the excluded skill |
265
+ | NOTRIG-03 | pass | answered `make-skill` — did not route to the excluded skill |
266
+ | AMB-01 | observed | answered `project-audit` (ambiguous/coexistence — recorded, not scored) |
267
+ | COEX-01 | observed | answered `project-audit` (ambiguous/coexistence — recorded, not scored) |
268
+
269
+ INSTR-01…07 (task-pipeline) and INSTR-01 (evidence-docs, project-audit): **not
270
+ reproducible from this harness** — each needs a full interactive pipeline run,
271
+ and a guessed verdict would be the exact substitution this file exists to refuse.
272
+
70
273
  ## 2026-08-03 · opus · self-observed, not a blind run
71
274
 
72
275
  **Scope of this evidence.** The author ran the skill on a real task