task-pipeline-skill 1.46.0 → 1.47.0

package/CHANGELOG.md

@@ -1,5 +1,49 @@
 # Changelog
 
+## v1.47.0 — a green suite that cannot speak for an agent
+
+The suite gate at stage 6 assumes the thing under test is deterministic: run it twice,
+get the same answer, and a pass means something. An agent breaks that assumption, and
+until now this pipeline had nothing to say about it — a grep across the plugin for
+`llm-as-judge`, `eval suite`, `regression fixture` and `trace id` returned nothing.
+
+**`tdd.md` gains *When the thing under test is an agent*.** The artifact under test is
+the execution record, not the source: the code says what the agent is allowed to do,
+only a run says what it did. Three tiers with unequal authority at the gate — step and
+turn **block**, thread **reports** — plus the two rules an ordinary suite never needs.
+**Assert the side effect, not the sentence**, because an agent that says it saved the
+preference and did not passes trajectory and response and is broken. And **a model
+judging a model is not a check until it has been calibrated** against human labels on
+cases known to be bad. The suite is grown from production failures, minimised, and kept
+permanently — a fixed defect that silently returns is the whole reason.
+
+The gate states what it does **not** cover, as canon 6 requires: an offline suite speaks
+for the cases already known and for nothing else, which is why production observation
+stays a stage-8 concern.
+
+**Three canons extended rather than an eleventh added.** Canon 3 says every fact has one
+home, so the agent case belongs inside the canons it is an instance of, not beside them:
+
+- **Canon 1** — where the subject is non-deterministic, the address is a **trace id and
+  the assertion that ran against it**; a rerun is not the same run.
+- **Canon 5** — a model used as a judge is the same object as a check: until it has been
+  seen disagreeing with a human on a known-bad case, its pass is an opinion with a
+  number attached.
+- **Canon 8** — a score produced by a model is an estimate in every report that quotes
+  it, however many decimal places it carries.
+
+**`evidence-docs` gains one routing row** and no doctrine — it is a navigator. The row
+names its target instead of linking it: this navigator already carries eleven
+out-of-directory links that break wherever a packager ships the skill alone, and a
+twelfth would widen a known defect. Stated rather than fixed, and the count is unchanged
+at eleven — measured before and after, not assumed.
+
+`Guards: 291 → 291`. This release adds doctrine, not enforcement: nothing here is
+machine-checkable yet, and the count says so rather than leaving a reader to infer it.
+The gate written into `tdd.md` is a specification a host project runs against its own
+agent — this repository has no agent to run it against, so arming it here would be a
+check with nothing to look at, which `gates.md` calls dormant and forbids passing.
+
 ## v1.46.0 — what a run leaves running, what "done" costs to say, and what a check is for
 
 Three rules this pipeline had been following by disposition rather than by doctrine,

package/SKILL-CARD.md

@@ -12,7 +12,7 @@ harmless.
 |---|---|
 | **Purpose** | Runs a substantial task through ten gated delivery stages — intake grill, docs study, brainstorm, spec, plan, subagent build, tests, lint/deploy, post-deploy, docs+registers, acceptance — refusing to advance until each gate passes |
 | **Owner** | ssheleg ([github.com/ssheleg/task-pipeline](https://github.com/ssheleg/task-pipeline)) |
-| **Version** | 1.46.0 |
+| **Version** | 1.47.0 |
 | **Surface** | Claude Code (filesystem skill + plugin) and the vercel `skills` CLI. **Not** uploaded to the Skills API; custom Skills do not sync across surfaces |
 | **Dependencies** | None required. Optional: `context7` (MCP), `figma` (MCP), super-ux, agent-sync, graphify, obsidian-wiki. Every stage's doctrine ships in-repo; the one conditional requirement is super-ux for the stage-3 UX track on a user-facing task |
 | **Evaluation status** | Suite authored, 5 categories. One recorded run, **self-observed by the author**; **zero blind runs on zero of three models** — the split, and the numbers, live in [`evals/RESULTS.md`](evals/RESULTS.md) and are computed by `evals/run.py` |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "task-pipeline-skill",
-  "version": "1.46.0",
+  "version": "1.47.0",
   "description": "Full-cycle delivery pipeline for coding agents: a mandatory built-in intake grill, then 10 gated stages (docs, brainstorm+decompose, spec, plan, build, tests, lint/deploy, post-deploy, docs/wiki, acceptance). Every stage's doctrine ships inside the skill — no companion plugin required. This package is the installer CLI.",
   "bin": {
     "task-pipeline": "bin/task-pipeline.js"

package/plugins/task-pipeline/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "task-pipeline",
   "displayName": "Task Pipeline",
   "description": "Runs a substantial task through a mandatory built-in intake grill, then 10 gated stages (docs, brainstorm+decompose, spec, plan, subagent build, tests, lint/deploy, post-deploy, docs/wiki, acceptance). Every stage's doctrine is built into the skill — no companion plugin required — with typed auto/manual gates, a frozen requirement spine that closes with evidence, a work board and a verification ledger that outlive a run, an exposure line naming what shipped unconfirmed, a progress rail computed from the project's own config, a loop guard whose review ceiling measures rather than stops, and stage-3 tracks for what a product does, how it sounds and how it looks. Two modes need no task: `checkup` (what is unverified) and `setup` (audit existing docs). Retro insights can publish upstream as issues, opt-in and redacted.",
-  "version": "1.46.0",
+  "version": "1.47.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/task-pipeline/skills/evidence-docs/SKILL.md

@@ -45,6 +45,7 @@ thing, the canon is the *why* and the rule is the *how*.
 | audit documentation a project already has | [`setup.md`](../task-pipeline/references/setup.md) | seven passes, cheapest first, output is a fix plan |
 | carry a lesson to the next run | [`retrospective.md`](../task-pipeline/references/retrospective.md) | stamp first (the cold trigger reads it), then prune to a cap of ten; every lesson names its commit |
 | seed a gate into a host project | [`templates/docgate.sh`](../task-pipeline/templates/docgate.sh) | it seeds **green**: dormant where there is no input yet |
+| claim that an **agent** behaves | `tdd.md` → *When the thing under test is an agent* — named rather than linked, because this navigator's out-of-directory links break wherever a packager ships this skill alone | the address is a trace id and the assertion that ran (canon 1); a judge nobody watched disagree is a green nobody watched turn red (canon 5) |
 | take a whole change through to acceptance | [`task-pipeline`](../task-pipeline/SKILL.md) | this skill is the standard; that one is how a change reaches the repository |
 
 ## When this applies

package/plugins/task-pipeline/skills/task-pipeline/references/documentation.md

@@ -42,8 +42,11 @@ exists to prevent.
 **1. A claim carries its address.** Every fact that lands in a document names where it
 can be checked: `file:line`, a command with its output, a test name. A lesson names the
 commit that earned it. *"We verified it" is the sentence that passes every review and
-proves nothing.* → the retro's SHA-resolution guard; the finding shape in
-[`setup.md`](setup.md).
+proves nothing.* Where the subject is non-deterministic — an agent, a model call — the
+address is **a trace id and the assertion that ran against it**, because a rerun is not
+the same run and a description of the behaviour is not the behaviour. → the retro's
+SHA-resolution guard; the finding shape in [`setup.md`](setup.md); [`tdd.md`](tdd.md) →
+*When the thing under test is an agent*.
 
 **2. Numbers are computed, never restated.** A count in prose is a number that was true
 once. Derive it at check time and compare the stated one against the computed one as the
@@ -60,8 +63,11 @@ under every link checker, because the checker resolves from the file's home. →
 
 **5. Green nobody watched turn red is not evidence.** A check must be seen rejecting a
 planted defect before its pass means anything — and the plant must be proven to have
-landed in the text the check actually parses. → [`gates.md`](gates.md) → *Probing*;
-[`learned.md`](learned.md) rules 4 and 5.
+landed in the text the check actually parses. **A model used as a judge is the same
+object**: until it has been seen disagreeing with a human label on a case known to be
+bad, its pass is an opinion with a number attached. → [`gates.md`](gates.md) →
+*Probing*; [`learned.md`](learned.md) rules 4 and 5; [`tdd.md`](tdd.md) → *When the
+thing under test is an agent*.
 
 **6. A check proves its scope and nothing beyond it.** Every gate carries what it does
 **not** cover, and quoting it wider is how "the gate is green" becomes a false statement
@@ -74,8 +80,9 @@ wrong, it is not evidence. → [`gates.md`](gates.md) → *False success*.
 
 **8. An estimate is never announced as a measurement.** A rule that fires on a judgement
 states its **evidence condition** — the observable signal that licenses it. A false
-alarm does not cost one interruption; it costs the alarm. → [`continuity.md`](continuity.md)
-→ *The context budget*.
+alarm does not cost one interruption; it costs the alarm. A score produced by a model
+is an estimate and stays one in every report that quotes it, however many decimal places
+it carries. → [`continuity.md`](continuity.md) → *The context budget*.
 
 **9. What was not checked is printed beside what was.** Absence is a finding with one
 side, so it never surfaces by comparison. Carry it as a named, counted set next to every

package/plugins/task-pipeline/skills/task-pipeline/references/tdd.md

@@ -14,6 +14,7 @@ into this skill; nothing to install.
 - The green from residue
 - Tests that stay honest
 - Stage 6 — consolidation and the suite gate
+- When the thing under test is an agent
 - When stuck
 - What a case consumes, and why a timeout is unclassified
 - Rationalizations
@@ -125,6 +126,51 @@ changed code is covered. No `skip` / `xfail` / commented-out assertion smuggles
 red suite past the gate. A partial or red run never advances to deploy; report it
 honestly instead.
 
+## When the thing under test is an agent
+
+An agent's behaviour is not in its source. The code says what it is *allowed* to do;
+only a run says what it did. So the artifact under test is the **execution record**, and
+the suite above needs one more tier before *green* means anything.
+
+Three tiers, and they do not carry equal authority at the gate:
+
+| Tier | Fixture | Asserts | At the gate |
+|---|---|---|---|
+| **Step** | one serialized model call with its prompt, tools and context | the decision — tool chosen, argument shape | **blocks** |
+| **Turn** | one whole execution | the trajectory, the final response, **and the state change** | **blocks** |
+| **Thread** | a scripted multi-turn session | what carried across turns, checked after **every** turn, failing fast | **reports** |
+
+Two rules the ordinary suite does not need:
+
+- **Assert the side effect, not the sentence.** An agent that says it saved the
+  preference and did not passes trajectory and response and is broken. The memory row,
+  the written file, the created record — inspect the thing, not the prose about it.
+- **A model judging a model is not a check until it has been calibrated.** Label the
+  same traces by hand, measure the agreement, and only then let it score unattended.
+  This is canon 5 with a different subject: a judge nobody has watched disagree is a
+  green nobody has watched turn red. Cheap deterministic checks — schema, exact match,
+  business rule, tool-call correctness — run first and take everything they can decide,
+  because they cost nothing and cannot drift.
+
+**The suite is grown, never authored.** Every production failure and every thumbs-down
+is minimised to the smallest input that still reproduces it, filed into the tier that
+isolates it, and **kept there permanently** — a fixed defect that silently returns is
+the whole reason this rule exists.
+
+**GATE (auto):** step and turn tiers green, and every fixture added by this change
+present in the suite. Thread results and any online, reference-free checks are
+**reported beside the verdict, not folded into it** — canon 9, which is why a
+non-blocking tier still has to print.
+
+**What this gate does not cover:** anything the fixtures do not contain. An offline
+suite proves you did not regress the cases already known; it cannot speak for inputs
+nobody has seen, which is canon 6 and the reason production observation is a stage-8
+concern rather than a stage-6 one.
+
+For building the suite this gate reads — the primitives, the tiers, judge design,
+annotation queues and simulated users — see the `agent-evals` skill in the
+`agent-stack` plugin.
+
 ## When stuck
 
 | Problem | What it means |