@ssheleg/agent-stack 0.11.1 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +89 -0
- package/package.json +5 -2
- package/plugins/agent-stack/.claude-plugin/plugin.json +1 -1
- package/plugins/agent-stack/skills/agent-evals/SKILL.md +53 -16
- package/plugins/agent-stack/skills/agent-harness/SKILL.md +2 -1
- package/plugins/agent-stack/skills/agent-harness/references/audit.md +46 -11
- package/plugins/agent-stack/skills/agent-interop/SKILL.md +5 -5
- package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md +12 -12
- package/plugins/agent-stack/skills/agent-orchestrator/references/graph-engineering.md +85 -15
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,94 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## v0.13.0 — 2026-08-20 — the audit refused a score and computed one
|
|
4
|
+
|
|
5
|
+
`references/audit.md` said *"a prioritized change plan … **not** a score"* at `:20`, argued
|
|
6
|
+
at `:22-25` that *"a number compresses away the only useful information"*, and then computed
|
|
7
|
+
`P = blast × confidence / effort` at `:114` and ordered the plan by it at `:128`. The
|
|
8
|
+
manifesto backs the refusal in the same words (`manifesto.md:424` — *"these axes are not a
|
|
9
|
+
fake numerical score. They are a reason the team can inspect"*) and names **four** axes at
|
|
10
|
+
`:419-422`: impact, **irreversibility**, uncertainty, **coordination**. Two of them appeared
|
|
11
|
+
nowhere in the pack (`grep -ci irreversib` → 0, `grep -ci coordinat` → 0) and `effort` — a
|
|
12
|
+
cost, not a risk — had been substituted for both.
|
|
13
|
+
|
|
14
|
+
**Position taken: publish the axes, drop the arithmetic.** The number never did what its own
|
|
15
|
+
paragraph claimed — `3 × 1 / 3` and `1 × 1 / 1` both print 1, so two findings a reviewer
|
|
16
|
+
would rank very differently ranked identically.
|
|
17
|
+
|
|
18
|
+
Three more, each with its plant:
|
|
19
|
+
|
|
20
|
+
- **The ledger graded a tree, not an artifact.** Three sections said *unreleased* and
|
|
21
|
+
*"the version stays 0.11.1 and the CHANGELOG is untouched"* over 35 rows reading
|
|
22
|
+
`verified` — while v0.12.0 was tagged and published — against this file's own rule that a
|
|
23
|
+
row sits at `never` until its check has been watched passing **on the shipped artifact**.
|
|
24
|
+
Re-run against `git archive v0.12.0`, both commands exit 0.
|
|
25
|
+
- **Three of four skill descriptions were past the house working limit**, one with five
|
|
26
|
+
characters of headroom before the platform's hard 1024. 1019 / 986 / 983 → **964 / 963 /
|
|
27
|
+
970**, every trigger intact (19→19, 12→12, 14→14), and 970 is a gate now.
|
|
28
|
+
- **`test/plant_guard_test.py` leaked eight nameless temp trees per run and the gate said
|
|
29
|
+
nothing.** `test/residue.py` is **ported** from `make-skill`, not rewritten; the shared
|
|
30
|
+
pile went from growing to flat (2568 → 2576 measured, 0 growth from this suite).
|
|
31
|
+
|
|
32
|
+
**Three guards were wrong first, and watching them fail is the only reason that is known:**
|
|
33
|
+
the scalar check refused the paragraph that records the formula's removal; the ledger check
|
|
34
|
+
read its own citation as a claim; and its `shipped in vX` pattern was **lowercase-only**
|
|
35
|
+
while every real claim is capitalised — so it reported green over a file it had never read.
|
|
36
|
+
|
|
37
|
+
Also corrected: `checks = 9 + len(skill_dirs)` was a hand-bumped literal that five ledger
|
|
38
|
+
rows quote as evidence a guard was added. The true count at v0.12.0 was **10**, not 13.
|
|
39
|
+
|
|
40
|
+
Negative self-tests 19 → **26**.
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
## v0.12.0 — 2026-08-19
|
|
44
|
+
|
|
45
|
+
Three places where this pack's own doctrine disagreed with the Proof of Done manifesto it
|
|
46
|
+
is built on. Each was confirmed from *inside* the pack before anything was changed.
|
|
47
|
+
|
|
48
|
+
### The gate before every convergence asked how sure a branch was, never what it could show
|
|
49
|
+
|
|
50
|
+
The manifesto's checker contract is *arrived · matches its contract · **carries its
|
|
51
|
+
evidence** · does not contradict a sibling*. This pack named five things and substituted
|
|
52
|
+
**under-confident — a confidence signal below the bar** for the evidence item. In a family
|
|
53
|
+
whose first value is Evidence over confidence, and whose own text says "an uncalibrated
|
|
54
|
+
judge is an opinion with a number attached", the gate guarding every convergence never
|
|
55
|
+
asked what a branch could show.
|
|
56
|
+
|
|
57
|
+
Six mandatory items now, in run order, with `unevidenced` as item 3 — an assertion with no
|
|
58
|
+
receipt, explicitly distinguished from *wrong*. **Arrival is a count**, not an accident: a
|
|
59
|
+
never-returning branch is caught because it is missing, not because the host happened to
|
|
60
|
+
null it. The confidence signal is kept and demoted to a hint: **low confidence flags,
|
|
61
|
+
absent evidence blocks.**
|
|
62
|
+
|
|
63
|
+
### A node was defined by three of its five fields, and read as whole
|
|
64
|
+
|
|
65
|
+
`One input, one output, one job` — against the manifesto's five, which add **one owner** and
|
|
66
|
+
**its own completion test**. Both texts then gave the same justification in nearly the same
|
|
67
|
+
words, which is what let the abridged version read as complete.
|
|
68
|
+
|
|
69
|
+
Worktree isolation answers **when** — two writers cannot corrupt one file. Ownership answers
|
|
70
|
+
**whose** — which node's version is authoritative once the branches return. Isolation without
|
|
71
|
+
ownership loses nothing during the run and moves the loss to the merge, where a silently
|
|
72
|
+
one-sided merge is indistinguishable from a convergence that only ever had one answer. The
|
|
73
|
+
`check` field matches `task-pipeline`'s exactly — same name, same one-string rule, same
|
|
74
|
+
`parked` exemption — rather than inventing a second vocabulary for one idea.
|
|
75
|
+
|
|
76
|
+
### The suite that could never be authored up front had to gate the first release
|
|
77
|
+
|
|
78
|
+
`Never author the suite up front` read as an absolute because the pack had no word for the
|
|
79
|
+
other tier: `grep -ci observable` and `grep -ci requirement` both returned **0**. Taken as
|
|
80
|
+
written the rule made a first release ungateable — §3 names the offline suite "this is the
|
|
81
|
+
gate", and a suite that may never be authored up front cannot exist before there is
|
|
82
|
+
production to grow it from.
|
|
83
|
+
|
|
84
|
+
Two clocks now. The **observable** is a criterion, written before the implementation. The
|
|
85
|
+
**corpus** is a sample, grown from production. The original imperative survives word for
|
|
86
|
+
word and gains only its subject: *"Never author the suite up front — the corpus, that is:
|
|
87
|
+
the inputs."* Neither rule softens the other, because they govern different objects.
|
|
88
|
+
|
|
89
|
+
Each contract is declared machine-readably with a floor and named keys, so an item cannot
|
|
90
|
+
silently go missing again. Validator 10 → 13 checks; negative self-tests 15 → 19.
|
|
91
|
+
|
|
3
92
|
## v0.11.1 — 2026-08-16
|
|
4
93
|
|
|
5
94
|
**This pack was the one place nothing was looking.** The family umbrella's shared checker
|
package/package.json
CHANGED
|
@@ -1,7 +1,10 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ssheleg/agent-stack",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
3
|
+
"version": "0.13.0",
|
|
4
|
+
"scripts": {
|
|
5
|
+
"test": "python3 test/validate.py && python3 test/plant_guard_test.py"
|
|
6
|
+
},
|
|
7
|
+
"description": "Production patterns for AI agent orchestrators — tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay — plus the wallet side of reselling LLM access. This package is the installer CLI.",
|
|
5
8
|
"bin": {
|
|
6
9
|
"agent-stack": "bin/agent-stack.js"
|
|
7
10
|
},
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"name": "agent-stack",
|
|
3
3
|
"displayName": "Agent Stack",
|
|
4
4
|
"description": "Two skills: agent-orchestrator \u2014 tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals \u2014 run/trace/thread evals, judges, and fixtures grown from production.",
|
|
5
|
-
"version": "0.
|
|
5
|
+
"version": "0.13.0",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "ssheleg",
|
|
8
8
|
"url": "https://x.com/sshlg93"
|
|
@@ -7,12 +7,11 @@ description: >-
|
|
|
7
7
|
evals. Covers the three observability primitives (run, trace, thread) crossed with three
|
|
8
8
|
eval granularities (single-step, full-turn, multi-turn), the offline/online/ad-hoc
|
|
9
9
|
timing axis, pass-fail rubrics over scalar scores, cheap code checks before model
|
|
10
|
-
judges, the checker node as an evaluator
|
|
11
|
-
|
|
12
|
-
possible. Triggers - "agent eval", "eval suite", "LLM judge", "regression fixture",
|
|
10
|
+
judges, the checker node as an evaluator inside the graph, simulated users with adversarial
|
|
11
|
+
personas, annotation queues, and what to instrument for any of it. Triggers - "agent eval", "eval suite", "LLM judge", "regression fixture",
|
|
13
12
|
"trajectory eval", "checker node", "is the agent getting better", "эвалы агента",
|
|
14
13
|
"оценка агента", "LLM-судья", "регрессионный набор", "как проверить агента". Not for
|
|
15
|
-
unit tests of ordinary code, or
|
|
14
|
+
unit tests of ordinary code, or benchmarking a model.
|
|
16
15
|
license: MIT
|
|
17
16
|
---
|
|
18
17
|
|
|
@@ -20,7 +19,8 @@ license: MIT
|
|
|
20
19
|
|
|
21
20
|
An agent's behaviour does not exist in its source. The code says what it is *allowed* to
|
|
22
21
|
do; only a run says what it *did*. So the artifact under test is the execution record,
|
|
23
|
-
and the suite
|
|
22
|
+
and the suite runs on **two clocks**: the *observable* is authored up front, before the
|
|
23
|
+
implementation exists, and the *corpus* it runs against is grown from production (§6).
|
|
24
24
|
|
|
25
25
|
Three claims follow, and they are what makes this different from testing ordinary code:
|
|
26
26
|
|
|
@@ -28,7 +28,8 @@ Three claims follow, and they are what makes this different from testing ordinar
|
|
|
28
28
|
- **Every natural-language input is unique**, so the edge cases cannot be enumerated
|
|
29
29
|
offline. Production is not only where you catch what you missed — it is where you
|
|
30
30
|
discover what to test for.
|
|
31
|
-
- **Traces become test cases.** The
|
|
31
|
+
- **Traces become test cases.** The corpus grows from what actually happened; the
|
|
32
|
+
criterion it is measured against does not.
|
|
32
33
|
|
|
33
34
|
---
|
|
34
35
|
|
|
@@ -169,17 +170,27 @@ depends on **it** rather than on the branches — otherwise the gate has a bypas
|
|
|
169
170
|
|
|
170
171
|
It matters here because it is the same machinery as §5, positioned differently:
|
|
171
172
|
|
|
173
|
+
<!-- checker-contract: missing, empty, unevidenced, malformed, contradictory, off-topic | optional: under-confident -->
|
|
174
|
+
|
|
172
175
|
| It catches | Decided by |
|
|
173
176
|
|---|---|
|
|
174
|
-
|
|
|
175
|
-
|
|
|
176
|
-
|
|
|
177
|
-
|
|
|
178
|
-
|
|
|
179
|
-
|
|
180
|
-
|
|
177
|
+
| a **missing** branch — the arrival count falls short of the fan-out promised | a code check |
|
|
178
|
+
| an **empty** or null result | a code check |
|
|
179
|
+
| an **unevidenced** claim — nothing attached that a reader could re-check | a code check |
|
|
180
|
+
| a **malformed** shape the consuming node cannot parse | a code check |
|
|
181
|
+
| **contradictory** siblings, including two that paraphrased one shared assumption | a judge |
|
|
182
|
+
| an **off-topic** answer to something nobody asked | a judge |
|
|
183
|
+
|
|
184
|
+
Four of six are free. Run them first — §5's *cheap checks first*, applied to a position
|
|
181
185
|
in the graph rather than to a suite.
|
|
182
186
|
|
|
187
|
+
**An under-confident branch is a hint, not a row: its own confidence number is optional
|
|
188
|
+
and deliberately not one of the six.** §5 above is the reason: a score from an
|
|
189
|
+
uncalibrated source is an opinion, so it may order retries and it may not open a gate.
|
|
190
|
+
The gate asks for the third row instead — a receipt — which is the same reason this pack
|
|
191
|
+
ranks evidence over confidence everywhere else. The contract's home, with the argument in
|
|
192
|
+
full, is `agent-orchestrator/references/graph-engineering.md` §6.
|
|
193
|
+
|
|
183
194
|
**A checker is a node, so it can be wrong, and its failure mode is silent approval.** A
|
|
184
195
|
model checker that has never been shown a bad input passes everything, and a graph with a
|
|
185
196
|
checker that always says yes is **worse** than one with none: the missing checkpoint has
|
|
@@ -198,10 +209,34 @@ and they are eval work rather than orchestration work:
|
|
|
198
209
|
Where the checker sits in the shape, and why the convergence needs one at all:
|
|
199
210
|
`agent-orchestrator/references/graph-engineering.md`.
|
|
200
211
|
|
|
201
|
-
## 6.
|
|
212
|
+
## 6. Two clocks — the observable up front, the corpus from production
|
|
202
213
|
|
|
203
|
-
|
|
204
|
-
|
|
214
|
+
Two different objects get called *the eval*, and they are written at opposite ends of the
|
|
215
|
+
work. Naming them apart is what stops either rule from reading as the other's exception.
|
|
216
|
+
|
|
217
|
+
<!-- eval-tiers: observable, corpus -->
|
|
218
|
+
|
|
219
|
+
| Tier | Is | Written | Because |
|
|
220
|
+
|---|---|---|---|
|
|
221
|
+
| **Observable** | the criterion that would show one requirement was met — a pass/fail rubric (§4), a trajectory or state-change assertion (§2) | **before the implementation exists** | a requirement with no observable is unfinished: attach one afterwards and you are inventing the test having already seen the code, so the output has decided what counts as success |
|
|
222
|
+
| **Corpus** | the inputs those criteria run against — fixtures, datasets, minimised production failures | **from production, never up front** | every natural-language input is unique, so the edge cases cannot be enumerated offline; inputs invented in advance test your imagination |
|
|
223
|
+
|
|
224
|
+
**Neither rule softens the other, because they govern different objects.** An observable is
|
|
225
|
+
a *criterion* — what would count as success. A corpus is a *sample* — which inputs you
|
|
226
|
+
happen to have. The criterion costs nothing to write early and can only be written honestly
|
|
227
|
+
early; the sample written early is green on inputs no user sends. Both therefore hold at
|
|
228
|
+
full strength: **a requirement that ships without an observable is unfinished, and a corpus
|
|
229
|
+
with no production in it is imagination.** The requirement itself gets its id and its
|
|
230
|
+
definition of done from `task-pipeline`'s REQ spine — what this pack owns is the
|
|
231
|
+
observable's *form*, not the register it hangs on.
|
|
232
|
+
|
|
233
|
+
**The first release has no production, so its offline gate is observables only** (§3). That
|
|
234
|
+
is not the corpus rule suspended for a special case: the corpus is empty because nothing has
|
|
235
|
+
run yet, and it fills from the first real traces. Inventing *inputs* to fill it sooner would
|
|
236
|
+
still be imagination.
|
|
237
|
+
|
|
238
|
+
**Never author the suite up front** — the *corpus*, that is: the inputs. Every production
|
|
239
|
+
failure and every thumbs-down becomes a fixture:
|
|
205
240
|
|
|
206
241
|
1. Capture the state at the failure point.
|
|
207
242
|
2. Minimise it to the smallest input that still reproduces.
|
|
@@ -257,6 +292,8 @@ None of the above runs without these, and they are the part people skip:
|
|
|
257
292
|
- [ ] Every checker node watched refusing a planted output, its verdicts stored as scores,
|
|
258
293
|
and its rejection rate on the dashboard — a checker at zero rejections is a finding
|
|
259
294
|
- [ ] Domain-expert review for output a general judge cannot grade
|
|
295
|
+
- [ ] Every requirement carries an **observable** written before the implementation — the
|
|
296
|
+
corpus waits for production, the criterion does not
|
|
260
297
|
- [ ] Every production failure minimised into a permanent fixture
|
|
261
298
|
- [ ] Annotation queue with filters, and the two reviewer roles kept separate
|
|
262
299
|
- [ ] Simulated users trained on real transcripts, with adversarial personas
|
|
@@ -179,4 +179,5 @@ prompt.
|
|
|
179
179
|
- [ ] One technique chosen per problem, with a reason — not ReAct because it was in a paper
|
|
180
180
|
- [ ] Sub-agents return **distilled summaries**, not transcripts
|
|
181
181
|
- [ ] The agent can be observed: which tool, which arguments, which observation, how many tokens
|
|
182
|
-
- [ ] An eval
|
|
182
|
+
- [ ] An observable before the implementation, an eval before the prompt is tuned, or the
|
|
183
|
+
tuning is folklore — only the corpus waits for production
|
|
@@ -11,7 +11,7 @@ inherited codebase, or a "why is this unreliable" investigation.
|
|
|
11
11
|
- Run the scanner first
|
|
12
12
|
- Seven tracks
|
|
13
13
|
- Evidence tiers
|
|
14
|
-
- Priority,
|
|
14
|
+
- Priority — four axes, and no scalar
|
|
15
15
|
- The report
|
|
16
16
|
- Traps
|
|
17
17
|
|
|
@@ -109,23 +109,56 @@ Every finding carries one, and the tier is part of the finding:
|
|
|
109
109
|
**Never present judgement as measured.** A finding whose tier is honest survives the meeting
|
|
110
110
|
where it is challenged; one that is inflated loses the whole report.
|
|
111
111
|
|
|
112
|
-
## Priority,
|
|
112
|
+
## Priority — four axes, and no scalar
|
|
113
113
|
|
|
114
|
-
`P = blast × confidence / effort`
|
|
114
|
+
`P = blast × confidence / effort` used to sit here, and it contradicted the two sections
|
|
115
|
+
above it. "Not a score" and *pass/fail with a named failure condition beats a scalar that
|
|
116
|
+
names no fix* cannot share a file with a number the plan is then ordered by.
|
|
115
117
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
118
|
+
**And the number does not do what it claimed to do.** The section said the ranking "can be
|
|
119
|
+
argued with on its inputs", but multiplication destroys them: `3 × 1 / 3` and `1 × 1 / 1`
|
|
120
|
+
both print **1**, so *harms a user, judgement-tier, its own project* and *annoys a future
|
|
121
|
+
maintainer, judgement-tier, an hour* arrive at one priority and nobody reading the output
|
|
122
|
+
can tell which is which. A product is a one-way function on the very inputs the argument
|
|
123
|
+
needs.
|
|
119
124
|
|
|
120
|
-
|
|
121
|
-
|
|
125
|
+
<!-- priority-axes: impact, irreversibility, uncertainty, coordination -->
|
|
126
|
+
|
|
127
|
+
So the inputs are published and the arithmetic is not. The axes are the manifesto's four
|
|
128
|
+
(`~/DATA/pod-manifesto/manifesto.md:419-422`, under *these axes are not a fake numerical
|
|
129
|
+
score* at `:424`), and two of them were absent here entirely while `effort` — a **cost**,
|
|
130
|
+
not a risk axis — had been substituted into their place:
|
|
131
|
+
|
|
132
|
+
| Axis | Question | High · Medium · Low |
|
|
133
|
+
|---|---|---|
|
|
134
|
+
| **Impact** | What is harmed if the finding is right? | a user of the system · the operator · a future maintainer |
|
|
135
|
+
| **Irreversibility** | How hard is the harm to undo once it lands? | unrecoverable · recoverable with work · trivially reversible |
|
|
136
|
+
| **Uncertainty** | How much of the behaviour cannot be checked deterministically? | unmeasurable here · measurable but unmeasured · measured |
|
|
137
|
+
| **Coordination** | How many agents, repositories, services and owners meet at the fix? | many · two · one |
|
|
138
|
+
|
|
139
|
+
**Ordering rule: the first axis that separates two findings decides, in that order.**
|
|
140
|
+
Impact, then Irreversibility, then Uncertainty, then Coordination. It is inspectable in the
|
|
141
|
+
direction the old formula was not — a reader who disagrees with the order of two findings
|
|
142
|
+
can point at the axis that decided it and argue about that axis alone.
|
|
143
|
+
|
|
144
|
+
**Effort keeps its job and loses its rank.** It is recorded per finding — under an hour · a
|
|
145
|
+
session · its own project — and it never moves a finding up or down. It sizes the *first
|
|
146
|
+
three items of the plan* so they can start immediately, which is the only decision it was
|
|
147
|
+
ever good for. A cost that divides a risk is how "too expensive to fix" becomes "not
|
|
148
|
+
important".
|
|
149
|
+
|
|
150
|
+
**Uncertainty is not the old `confidence` renamed.** `confidence` graded the auditor's
|
|
151
|
+
evidence; the axis grades what the *system* cannot be made to prove. The evidence grade
|
|
152
|
+
still exists and is still required — it is the tier on every finding, one section up — and
|
|
153
|
+
a finding whose tier is `judgement` says so there rather than being quietly discounted here.
|
|
122
154
|
|
|
123
155
|
## The report
|
|
124
156
|
|
|
125
157
|
1. **One paragraph** — what the system is, which layer, and the single most important thing.
|
|
126
158
|
2. **The scanner output**, including its blind-spot list, verbatim.
|
|
127
|
-
3. **Findings by track**, each with observation, tier and
|
|
128
|
-
4. **The plan** — ordered by
|
|
159
|
+
3. **Findings by track**, each with observation, tier, the four axes and its effort.
|
|
160
|
+
4. **The plan** — ordered by the axes above, first separating axis wins, with the first
|
|
161
|
+
three items sized by effort so they can start immediately.
|
|
129
162
|
5. **What was not looked at**, and why. An audit that does not say where it stopped is read
|
|
130
163
|
as complete.
|
|
131
164
|
|
|
@@ -135,7 +168,9 @@ is fixed before lunch, and the ranking can be argued with on its inputs.
|
|
|
135
168
|
is often not in the repository at all — ask where it lives before concluding it is fine.
|
|
136
169
|
- **Filing "no permission model" against a harness that delegates by design.** Check the
|
|
137
170
|
layer first (`layers.md`).
|
|
138
|
-
- **Grading instead of planning.** A score ends the conversation the audit was meant to
|
|
171
|
+
- **Grading instead of planning.** A score ends the conversation the audit was meant to
|
|
172
|
+
start — including a score assembled from honest axes. Publish the axes; do not multiply
|
|
173
|
+
them.
|
|
139
174
|
- **Confusing "no evals" with "not measured yet."** It is the root finding; put it first,
|
|
140
175
|
because every other conclusion inherits it.
|
|
141
176
|
- **Reading a silent scanner as a clean system.** It is silent about what it can see.
|
|
@@ -3,15 +3,15 @@ name: agent-interop
|
|
|
3
3
|
description: >-
|
|
4
4
|
Use when an agent must talk to something outside its own process — building or consuming an
|
|
5
5
|
MCP server, exposing or calling another agent over A2A, publishing to the MCP Registry, or
|
|
6
|
-
putting a gateway in front of agent traffic. Carries the MCP 2026-07-28 wire surface and what
|
|
7
|
-
|
|
8
|
-
subscriptions/listen; sampling, roots, logging and dynamic client registration
|
|
6
|
+
putting a gateway in front of agent traffic. Carries the MCP 2026-07-28 wire surface and what it
|
|
7
|
+
deprecated (server/discover, stateless per-request _meta, elicitation in form and URL mode,
|
|
8
|
+
subscriptions/listen; sampling, roots, logging and dynamic client registration going),
|
|
9
9
|
A2A 1.0 agent cards, task states and three bindings, registry namespaces and server.json, tool
|
|
10
10
|
federation, and what a gateway must do that an API gateway does not. Triggers - "MCP server",
|
|
11
11
|
"MCP client", "A2A", "agent card", "agent interoperability", "MCP registry", "server.json",
|
|
12
12
|
"agentgateway", "tool federation", "MCP-сервер", "карточка агента", "интероперабельность
|
|
13
|
-
агентов", "реестр MCP", "шлюз для агентов". Not for designing one server's tool set,
|
|
14
|
-
|
|
13
|
+
агентов", "реестр MCP", "шлюз для агентов". Not for designing one server's tool set, nor for
|
|
14
|
+
a skill's own construction — that is make-skill.
|
|
15
15
|
---
|
|
16
16
|
|
|
17
17
|
# Agent interop — the protocols an agent speaks outside its own process
|
|
@@ -2,17 +2,16 @@
|
|
|
2
2
|
name: agent-orchestrator
|
|
3
3
|
description: >-
|
|
4
4
|
Use when building an agent system — an orchestrator, an LLM-powered tool, a chatbot with tool
|
|
5
|
-
use, an AI pipeline — or
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
"
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
or for prompt wording.
|
|
5
|
+
use, an AI pipeline — or metering and billing the LLM access it burns. Covers tool-calling
|
|
6
|
+
loops, pipelines with human checkpoints, provider routing with fallback/retry, four-layer
|
|
7
|
+
memory with confidence decay, context budgets, sub-agent coordination, error hierarchies; the
|
|
8
|
+
work as a graph — parallel layers, fake edges, a checker before convergence; for resale:
|
|
9
|
+
tiered wallets, one markup boundary, two-phase commit across database and provider API,
|
|
10
|
+
spend-delta polling, budget and loop guards, per-tenant keys. Triggers - "agent",
|
|
11
|
+
"orchestrator", "tool calling", "sub-agent", "LLM router", "fallback chain", "human in the
|
|
12
|
+
loop", "memory layer", "LLM billing", "token wallet", "checker node", "агент", "оркестратор",
|
|
13
|
+
"суб-агент", "роутер моделей", "человек в цикле", "слой памяти", "биллинг LLM", "граф задач".
|
|
14
|
+
Not for a single LLM call in a script, or prompt wording.
|
|
16
15
|
---
|
|
17
16
|
|
|
18
17
|
# Agent Orchestrator — Production Best Practices
|
|
@@ -365,7 +364,8 @@ heading — the ones that were learned by getting them wrong:
|
|
|
365
364
|
to the input is a function call wearing a costume
|
|
366
365
|
- [ ] Model, window and price are resolved at one boundary from configuration or the
|
|
367
366
|
provider — never from a table of vendor ids in source
|
|
368
|
-
- [ ] An eval
|
|
367
|
+
- [ ] An observable before the implementation, an eval before the prompt is tuned, or the
|
|
368
|
+
tuning is folklore — only the corpus waits for production
|
|
369
369
|
|
|
370
370
|
## References
|
|
371
371
|
|
|
@@ -49,10 +49,55 @@ Four things are **this pack's**, not the source's, and each is marked where it a
|
|
|
49
49
|
|
|
50
50
|
## 1. Node and edge
|
|
51
51
|
|
|
52
|
-
**A node is one unit of work.** One input, one
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
52
|
+
**A node is one unit of work, and it has five fields.** One input, one job, one output,
|
|
53
|
+
**one owner**, and **its own completion test**. Not *"research the topic, summarise it,
|
|
54
|
+
and check the sources"* — that is three nodes wearing one name. The smaller and more
|
|
55
|
+
defined the job, the more useful the node, because a node is also the unit you retry,
|
|
56
|
+
cache, review and replace.
|
|
57
|
+
|
|
58
|
+
<!-- node-contract: input, job, output, owner, check -->
|
|
59
|
+
|
|
60
|
+
The first three get drawn every time. The last two are the ones a graph is usually drawn
|
|
61
|
+
without — this file shipped without them until 2026-08-19 — and each has a failure that
|
|
62
|
+
surfaces only at the convergence, where it is cheapest to mistake for success.
|
|
63
|
+
|
|
64
|
+
**One owner — exactly one node is answerable for a given artifact.** Not one *agent*: the
|
|
65
|
+
same subagent may own several nodes, and a node owned by a person is still a node. The
|
|
66
|
+
unit of ownership is the artifact, and the assignment is made when the layer is built, not
|
|
67
|
+
discovered when the writes come back.
|
|
68
|
+
|
|
69
|
+
**Ownership is not what `isolation: "worktree"` gives you, and the two are complementary
|
|
70
|
+
rather than alternatives.** §9's primitive table calls a private checkout the only safe way
|
|
71
|
+
to fan out writers, and against a *race* that is true. Isolation answers **when**: two
|
|
72
|
+
writers cannot corrupt one file, because neither can see the other's copy while it runs.
|
|
73
|
+
Ownership answers **whose**: which node's version is authoritative once the branches
|
|
74
|
+
return. Take isolation without ownership and nothing is lost during the run — the loss
|
|
75
|
+
moves to the merge, which is the quieter place for it, because a convergence that silently
|
|
76
|
+
takes one side is indistinguishable from a convergence that only ever had one answer. §4's
|
|
77
|
+
first rule is the *detection* half of this: two independent workers writing one file are
|
|
78
|
+
one node with a race in it. Ownership is the *assignment* half, and it is what makes
|
|
79
|
+
fanning writers out decidable rather than merely survivable.
|
|
80
|
+
|
|
81
|
+
**Its own completion test — the node names what will close it, before it runs.** One
|
|
82
|
+
string: a command a verifier can run, or a named judgement standing in where no command
|
|
83
|
+
can, with the judge named and the verdict recorded *as* judgement. Whatever it prints is
|
|
84
|
+
that node's evidence row.
|
|
85
|
+
|
|
86
|
+
The field's name across this family is **`check`** — `task-pipeline`'s node schema
|
|
87
|
+
(`graph.schema.json`) carries it as one required string per node and exempts only a
|
|
88
|
+
`parked` one, the single node nobody intends to close. Same name and same meaning here,
|
|
89
|
+
one string and never a list, because a node needing two unrelated checks is a node doing
|
|
90
|
+
two jobs and the answer is to split it. One gate built from two commands is `a && b`,
|
|
91
|
+
which is still one gate.
|
|
92
|
+
|
|
93
|
+
**A completion test is not a checker node, and carrying both is not redundancy.** §6's
|
|
94
|
+
checker sits between a parallel layer and its convergence and judges *arriving siblings*
|
|
95
|
+
from the outside — including the two things no node can establish about itself: whether it
|
|
96
|
+
contradicts a neighbour, and whether it answered the question the layer was actually
|
|
97
|
+
asked. The completion test is the node's own gate on its own output, run by whoever owns
|
|
98
|
+
it. Drop the per-node test and the checker becomes the only gate in the graph, which is how
|
|
99
|
+
a convergence ends up re-deriving what each branch should have proved about itself; drop
|
|
100
|
+
the checker and every branch can pass its own test and still disagree with its neighbour.
|
|
56
101
|
|
|
57
102
|
**An edge is a dependency, and it carries data.** It exists when the second node
|
|
58
103
|
genuinely consumes what the first produced. Not when the second merely *happens after*
|
|
@@ -151,19 +196,44 @@ A node between the parallel layer and the convergence whose only job is to decid
|
|
|
151
196
|
each output may proceed. It synthesises nothing and writes nothing. It answers *is this
|
|
152
197
|
usable* and then passes, flags, retries or drops.
|
|
153
198
|
|
|
154
|
-
|
|
155
|
-
of the
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
199
|
+
Six things it must catch — the list is the contract, and a checker that cannot say which
|
|
200
|
+
of the six it is asserting is not a checker:
|
|
201
|
+
|
|
202
|
+
<!-- checker-contract: missing, empty, unevidenced, malformed, contradictory, off-topic | optional: under-confident -->
|
|
203
|
+
|
|
204
|
+
1. **Missing** — a branch the split promised never came back at all. This item needs an
|
|
205
|
+
**arrival count**: the fan-out is fixed when the layer is built, so the checker is
|
|
206
|
+
handed the number of results to expect and compares it with the number it is holding.
|
|
207
|
+
Without that number a vanished branch is caught only when its slot happens to hold
|
|
208
|
+
nothing — item 2, by luck rather than by design, and only on a host that fills the slot
|
|
209
|
+
at all (§9's primitive table records what this one does with a thrower). A host that
|
|
210
|
+
simply drops it makes the gap invisible, and a short list looks like a complete one.
|
|
211
|
+
2. **Empty or null** — the result arrived and carries nothing usable.
|
|
212
|
+
3. **Unevidenced** — an assertion with no receipt: no citation, no tool result, no file
|
|
213
|
+
and line, nothing a later reader could re-check. Whether the claim is *wrong* is item 5
|
|
214
|
+
or 6's business; this item is about a claim nobody downstream can verify at all, and it
|
|
215
|
+
is the one most checkers are missing.
|
|
216
|
+
4. **Malformed** — a shape that will break the convergence node's parsing.
|
|
217
|
+
5. **Contradictory** — two results that cannot both be true, **including two that used
|
|
218
|
+
different words for one shared assumption**. The disagreement hides inside the
|
|
219
|
+
paraphrase, so comparing strings is not enough to find it.
|
|
220
|
+
6. **Off-topic** — an output that answers a different question than the one asked.
|
|
221
|
+
|
|
222
|
+
Four of the six are code checks (1–4) and cost nothing; only 5 and 6 need a model. Run
|
|
223
|
+
them in that order — this is `agent-evals` §5's *cheap checks first*, applied to a
|
|
165
224
|
position in the graph rather than to a test suite.
|
|
166
225
|
|
|
226
|
+
**A confidence signal is optional, and it is not the evidence item.** *Under-confident*
|
|
227
|
+
was item 4 of this list until 2026-08-19; it is kept, demoted to a hint. A branch's own
|
|
228
|
+
number is worth carrying as exactly that — which result to retry first, which to hand to a
|
|
229
|
+
person — and it is not a gate. It never stands in for item 3, and the argument is this
|
|
230
|
+
pack's own, made twice already: *an uncalibrated judge is an opinion with a number
|
|
231
|
+
attached* (`agent-evals` §5), and for anything consequential the control is a
|
|
232
|
+
deterministic limit or a human, **never a classifier's confidence**
|
|
233
|
+
(`references/governance.md`). A gate that asks a branch how sure it is and never asks
|
|
234
|
+
what it can show has graded the run instead of measuring it. So: low confidence *flags*,
|
|
235
|
+
absent evidence *blocks*.
|
|
236
|
+
|
|
167
237
|
**What a checker costs, and how it turns into a rubber stamp — this pack's addition.** A
|
|
168
238
|
checker is a node, so it has the failure mode of every node: it can be wrong. A model
|
|
169
239
|
checker that has never been shown a bad input will pass everything, and a graph with a
|