@ssheleg/agent-stack 0.11.0 → 0.12.0

package/CHANGELOG.md

@@ -1,5 +1,68 @@
 # Changelog
 
+## v0.12.0 — 2026-08-19
+
+Three places where this pack's own doctrine disagreed with the Proof of Done manifesto it
+is built on. Each was confirmed from *inside* the pack before anything was changed.
+
+### The gate before every convergence asked how sure a branch was, never what it could show
+
+The manifesto's checker contract is *arrived · matches its contract · **carries its
+evidence** · does not contradict a sibling*. This pack named five things and substituted
+**under-confident — a confidence signal below the bar** for the evidence item. In a family
+whose first value is Evidence over confidence, and whose own text says "an uncalibrated
+judge is an opinion with a number attached", the gate guarding every convergence never
+asked what a branch could show.
+
+Six mandatory items now, in run order, with `unevidenced` as item 3 — an assertion with no
+receipt, explicitly distinguished from *wrong*. **Arrival is a count**, not an accident: a
+never-returning branch is caught because it is missing, not because the host happened to
+null it. The confidence signal is kept and demoted to a hint: **low confidence flags,
+absent evidence blocks.**
+
+### A node was defined by three of its five fields, and read as whole
+
+`One input, one output, one job` — against the manifesto's five, which add **one owner** and
+**its own completion test**. Both texts then gave the same justification in nearly the same
+words, which is what let the abridged version read as complete.
+
+Worktree isolation answers **when** — two writers cannot corrupt one file. Ownership answers
+**whose** — which node's version is authoritative once the branches return. Isolation without
+ownership loses nothing during the run and moves the loss to the merge, where a silently
+one-sided merge is indistinguishable from a convergence that only ever had one answer. The
+`check` field matches `task-pipeline`'s exactly — same name, same one-string rule, same
+`parked` exemption — rather than inventing a second vocabulary for one idea.
+
+### The suite that could never be authored up front had to gate the first release
+
+`Never author the suite up front` read as an absolute because the pack had no word for the
+other tier: `grep -ci observable` and `grep -ci requirement` both returned **0**. Taken as
+written the rule made a first release ungateable — §3 names the offline suite "this is the
+gate", and a suite that may never be authored up front cannot exist before there is
+production to grow it from.
+
+Two clocks now. The **observable** is a criterion, written before the implementation. The
+**corpus** is a sample, grown from production. The original imperative survives word for
+word and gains only its subject: *"Never author the suite up front — the corpus, that is:
+the inputs."* Neither rule softens the other, because they govern different objects.
+
+Each contract is declared machine-readably with a floor and named keys, so an item cannot
+silently go missing again. Validator 10 → 13 checks; negative self-tests 15 → 19.
+
+## v0.11.1 — 2026-08-16
+
+**This pack was the one place nothing was looking.** The family umbrella's shared checker
+had an early exit for a member carrying no routed triggers, and `agent-stack` is that
+member — so when a sibling's front matter turned out to be invalid YAML (B-56: a
+colon-space inside an unquoted scalar, which every regex-based gate in the family reads
+happily and a real parser refuses), nothing here would have caught the same mistake.
+
+`test/validate.py` now asks that checker, which no longer exits early: it validates the
+shipped front matter of every skill first, and only then the routed triggers a member may
+or may not have. The table it reads is not copied here, so there is nothing to drift.
+Watched refusing a planted `Broken: now a nested mapping.` in `agent-orchestrator`'s
+description, and green after restore.
+
 ## [0.11.0] — 2026-08-16
 
 ### Changed

package/package.json

@@ -1,7 +1,10 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.11.0",
-  "description": "Production patterns for AI agent orchestrators \u2014 tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay \u2014 plus the wallet side of reselling LLM access. This package is the installer CLI.",
+  "version": "0.12.0",
+  "scripts": {
+    "test": "python3 test/validate.py && python3 test/plant_guard_test.py"
+  },
+  "description": "Production patterns for AI agent orchestrators — tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay — plus the wallet side of reselling LLM access. This package is the installer CLI.",
   "bin": {
     "agent-stack": "bin/agent-stack.js"
   },

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Two skills: agent-orchestrator \u2014 tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals \u2014 run/trace/thread evals, judges, and fixtures grown from production.",
-  "version": "0.11.0",
+  "version": "0.12.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/agent-stack/skills/agent-evals/SKILL.md

@@ -20,7 +20,8 @@ license: MIT
 
 An agent's behaviour does not exist in its source. The code says what it is *allowed* to
 do; only a run says what it *did*. So the artifact under test is the execution record,
-and the suite is grown from production rather than authored up front.
+and the suite runs on **two clocks**: the *observable* is authored up front, before the
+implementation exists, and the *corpus* it runs against is grown from production (§6).
 
 Three claims follow, and they are what makes this different from testing ordinary code:
 
@@ -28,7 +29,8 @@ Three claims follow, and they are what makes this different from testing ordinar
 - **Every natural-language input is unique**, so the edge cases cannot be enumerated
   offline. Production is not only where you catch what you missed — it is where you
   discover what to test for.
-- **Traces become test cases.** The suite grows from what actually happened.
+- **Traces become test cases.** The corpus grows from what actually happened; the
+  criterion it is measured against does not.
 
 ---
 
@@ -169,17 +171,27 @@ depends on **it** rather than on the branches — otherwise the gate has a bypas
 
 It matters here because it is the same machinery as §5, positioned differently:
 
+<!-- checker-contract: missing, empty, unevidenced, malformed, contradictory, off-topic | optional: under-confident -->
+
 | It catches | Decided by |
 |---|---|
-| an empty or null output | a code check |
-| a confidence signal below the downstream bar | a code check |
-| a shape that will break the consumer's parsing | a code check |
-| two outputs that cannot both be true | a judge |
-| an output answering a different question than the one asked | a judge |
-
-Three of five are free. Run them first — §5's *cheap checks first*, applied to a position
+| a **missing** branch — the arrival count falls short of the fan-out promised | a code check |
+| an **empty** or null result | a code check |
+| an **unevidenced** claim — nothing attached that a reader could re-check | a code check |
+| a **malformed** shape the consuming node cannot parse | a code check |
+| **contradictory** siblings, including two that paraphrased one shared assumption | a judge |
+| an **off-topic** answer to something nobody asked | a judge |
+
+Four of six are free. Run them first — §5's *cheap checks first*, applied to a position
 in the graph rather than to a suite.
 
+**An under-confident branch is a hint, not a row: its own confidence number is optional
+and deliberately not one of the six.** §5 above is the reason: a score from an
+uncalibrated source is an opinion, so it may order retries and it may not open a gate.
+The gate asks for the third row instead — a receipt — which is the same reason this pack
+ranks evidence over confidence everywhere else. The contract's home, with the argument in
+full, is `agent-orchestrator/references/graph-engineering.md` §6.
+
 **A checker is a node, so it can be wrong, and its failure mode is silent approval.** A
 model checker that has never been shown a bad input passes everything, and a graph with a
 checker that always says yes is **worse** than one with none: the missing checkpoint has
@@ -198,10 +210,34 @@ and they are eval work rather than orchestration work:
 Where the checker sits in the shape, and why the convergence needs one at all:
 `agent-orchestrator/references/graph-engineering.md`.
 
-## 6. The corpus grows from production
+## 6. Two clocks — the observable up front, the corpus from production
 
-Never author the suite up front. Every production failure and every thumbs-down becomes a
-fixture:
+Two different objects get called *the eval*, and they are written at opposite ends of the
+work. Naming them apart is what stops either rule from reading as the other's exception.
+
+<!-- eval-tiers: observable, corpus -->
+
+| Tier | Is | Written | Because |
+|---|---|---|---|
+| **Observable** | the criterion that would show one requirement was met — a pass/fail rubric (§4), a trajectory or state-change assertion (§2) | **before the implementation exists** | a requirement with no observable is unfinished: attach one afterwards and you are inventing the test having already seen the code, so the output has decided what counts as success |
+| **Corpus** | the inputs those criteria run against — fixtures, datasets, minimised production failures | **from production, never up front** | every natural-language input is unique, so the edge cases cannot be enumerated offline; inputs invented in advance test your imagination |
+
+**Neither rule softens the other, because they govern different objects.** An observable is
+a *criterion* — what would count as success. A corpus is a *sample* — which inputs you
+happen to have. The criterion costs nothing to write early and can only be written honestly
+early; the sample written early is green on inputs no user sends. Both therefore hold at
+full strength: **a requirement that ships without an observable is unfinished, and a corpus
+with no production in it is imagination.** The requirement itself gets its id and its
+definition of done from `task-pipeline`'s REQ spine — what this pack owns is the
+observable's *form*, not the register it hangs on.
+
+**The first release has no production, so its offline gate is observables only** (§3). That
+is not the corpus rule suspended for a special case: the corpus is empty because nothing has
+run yet, and it fills from the first real traces. Inventing *inputs* to fill it sooner would
+still be imagination.
+
+**Never author the suite up front** — the *corpus*, that is: the inputs. Every production
+failure and every thumbs-down becomes a fixture:
 
 1. Capture the state at the failure point.
 2. Minimise it to the smallest input that still reproduces.
@@ -257,6 +293,8 @@ None of the above runs without these, and they are the part people skip:
 - [ ] Every checker node watched refusing a planted output, its verdicts stored as scores,
       and its rejection rate on the dashboard — a checker at zero rejections is a finding
 - [ ] Domain-expert review for output a general judge cannot grade
+- [ ] Every requirement carries an **observable** written before the implementation — the
+      corpus waits for production, the criterion does not
 - [ ] Every production failure minimised into a permanent fixture
 - [ ] Annotation queue with filters, and the two reviewer roles kept separate
 - [ ] Simulated users trained on real transcripts, with adversarial personas

package/plugins/agent-stack/skills/agent-harness/SKILL.md

@@ -179,4 +179,5 @@ prompt.
 - [ ] One technique chosen per problem, with a reason — not ReAct because it was in a paper
 - [ ] Sub-agents return **distilled summaries**, not transcripts
 - [ ] The agent can be observed: which tool, which arguments, which observation, how many tokens
-- [ ] An eval exists before the prompt is tuned, or the tuning is folklore
+- [ ] An observable before the implementation, an eval before the prompt is tuned, or the
+      tuning is folklore — only the corpus waits for production

package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md

@@ -365,7 +365,8 @@ heading — the ones that were learned by getting them wrong:
       to the input is a function call wearing a costume
 - [ ] Model, window and price are resolved at one boundary from configuration or the
       provider — never from a table of vendor ids in source
-- [ ] An eval exists before the prompt is tuned, or the tuning is folklore
+- [ ] An observable before the implementation, an eval before the prompt is tuned, or the
+      tuning is folklore — only the corpus waits for production
 
 ## References
 

package/plugins/agent-stack/skills/agent-orchestrator/references/graph-engineering.md

@@ -49,10 +49,55 @@ Four things are **this pack's**, not the source's, and each is marked where it a
 
 ## 1. Node and edge
 
-**A node is one unit of work.** One input, one output, one job. Not *"research the topic,
-summarise it, and check the sources"* — that is three nodes wearing one name. The
-smaller and more defined the job, the more useful the node, because a node is also the
-unit you retry, cache, review and replace.
+**A node is one unit of work, and it has five fields.** One input, one job, one output,
+**one owner**, and **its own completion test**. Not *"research the topic, summarise it,
+and check the sources"* — that is three nodes wearing one name. The smaller and more
+defined the job, the more useful the node, because a node is also the unit you retry,
+cache, review and replace.
+
+<!-- node-contract: input, job, output, owner, check -->
+
+The first three get drawn every time. The last two are the ones a graph is usually drawn
+without — this file shipped without them until 2026-08-19 — and each has a failure that
+surfaces only at the convergence, where it is cheapest to mistake for success.
+
+**One owner — exactly one node is answerable for a given artifact.** Not one *agent*: the
+same subagent may own several nodes, and a node owned by a person is still a node. The
+unit of ownership is the artifact, and the assignment is made when the layer is built, not
+discovered when the writes come back.
+
+**Ownership is not what `isolation: "worktree"` gives you, and the two are complementary
+rather than alternatives.** §9's primitive table calls a private checkout the only safe way
+to fan out writers, and against a *race* that is true. Isolation answers **when**: two
+writers cannot corrupt one file, because neither can see the other's copy while it runs.
+Ownership answers **whose**: which node's version is authoritative once the branches
+return. Take isolation without ownership and nothing is lost during the run — the loss
+moves to the merge, which is the quieter place for it, because a convergence that silently
+takes one side is indistinguishable from a convergence that only ever had one answer. §4's
+first rule is the *detection* half of this: two independent workers writing one file are
+one node with a race in it. Ownership is the *assignment* half, and it is what makes
+fanning writers out decidable rather than merely survivable.
+
+**Its own completion test — the node names what will close it, before it runs.** One
+string: a command a verifier can run, or a named judgement standing in where no command
+can, with the judge named and the verdict recorded *as* judgement. Whatever it prints is
+that node's evidence row.
+
+The field's name across this family is **`check`** — `task-pipeline`'s node schema
+(`graph.schema.json`) carries it as one required string per node and exempts only a
+`parked` one, the single node nobody intends to close. Same name and same meaning here,
+one string and never a list, because a node needing two unrelated checks is a node doing
+two jobs and the answer is to split it. One gate built from two commands is `a && b`,
+which is still one gate.
+
+**A completion test is not a checker node, and carrying both is not redundancy.** §6's
+checker sits between a parallel layer and its convergence and judges *arriving siblings*
+from the outside — including the two things no node can establish about itself: whether it
+contradicts a neighbour, and whether it answered the question the layer was actually
+asked. The completion test is the node's own gate on its own output, run by whoever owns
+it. Drop the per-node test and the checker becomes the only gate in the graph, which is how
+a convergence ends up re-deriving what each branch should have proved about itself; drop
+the checker and every branch can pass its own test and still disagree with its neighbour.
 
 **An edge is a dependency, and it carries data.** It exists when the second node
 genuinely consumes what the first produced. Not when the second merely *happens after*
@@ -151,19 +196,44 @@ A node between the parallel layer and the convergence whose only job is to decid
 each output may proceed. It synthesises nothing and writes nothing. It answers *is this
 usable* and then passes, flags, retries or drops.
 
-Five things it must catch — the list is the contract, and a checker that cannot say which
-of the five it is asserting is not a checker:
-
-1. **Empty or null** — the node returned nothing usable.
-2. **Mutually contradictory** — two outputs that cannot both be true.
-3. **Off-topic** — an output that answers a different question than the one asked.
-4. **Under-confident** — a confidence signal below the bar for the downstream decision.
-5. **Malformed** — a shape that will break the convergence node's parsing.
-
-Three of the five are code checks (1, 4, 5) and cost nothing; only 2 and 3 need a model.
-Run them in that order — this is `agent-evals` §5's *cheap checks first*, applied to a
+Six things it must catch — the list is the contract, and a checker that cannot say which
+of the six it is asserting is not a checker:
+
+<!-- checker-contract: missing, empty, unevidenced, malformed, contradictory, off-topic | optional: under-confident -->
+
+1. **Missing** — a branch the split promised never came back at all. This item needs an
+   **arrival count**: the fan-out is fixed when the layer is built, so the checker is
+   handed the number of results to expect and compares it with the number it is holding.
+   Without that number a vanished branch is caught only when its slot happens to hold
+   nothing — item 2, by luck rather than by design, and only on a host that fills the slot
+   at all (§9's primitive table records what this one does with a thrower). A host that
+   simply drops it makes the gap invisible, and a short list looks like a complete one.
+2. **Empty or null** — the result arrived and carries nothing usable.
+3. **Unevidenced** — an assertion with no receipt: no citation, no tool result, no file
+   and line, nothing a later reader could re-check. Whether the claim is *wrong* is item 5
+   or 6's business; this item is about a claim nobody downstream can verify at all, and it
+   is the one most checkers are missing.
+4. **Malformed** — a shape that will break the convergence node's parsing.
+5. **Contradictory** — two results that cannot both be true, **including two that used
+   different words for one shared assumption**. The disagreement hides inside the
+   paraphrase, so comparing strings is not enough to find it.
+6. **Off-topic** — an output that answers a different question than the one asked.
+
+Four of the six are code checks (1–4) and cost nothing; only 5 and 6 need a model. Run
+them in that order — this is `agent-evals` §5's *cheap checks first*, applied to a
 position in the graph rather than to a test suite.
 
+**A confidence signal is optional, and it is not the evidence item.** *Under-confident*
+was item 4 of this list until 2026-08-19; it is kept, demoted to a hint. A branch's own
+number is worth carrying as exactly that — which result to retry first, which to hand to a
+person — and it is not a gate. It never stands in for item 3, and the argument is this
+pack's own, made twice already: *an uncalibrated judge is an opinion with a number
+attached* (`agent-evals` §5), and for anything consequential the control is a
+deterministic limit or a human, **never a classifier's confidence**
+(`references/governance.md`). A gate that asks a branch how sure it is and never asks
+what it can show has graded the run instead of measuring it. So: low confidence *flags*,
+absent evidence *blocks*.
+
 **What a checker costs, and how it turns into a rubber stamp — this pack's addition.** A
 checker is a node, so it has the failure mode of every node: it can be wrong. A model
 checker that has never been shown a bad input will pass everything, and a graph with a