@ssheleg/agent-stack 0.20.0 → 0.22.0

package/CHANGELOG.md

@@ -1,3 +1,83 @@
+## v0.22.0 — the failures named from outside, and the control that is absence
+
+**`graph-engineering.md` was argued entirely from what breaks, with no citation behind it.**
+*Why Do Multi-Agent LLM Systems Fail?* (2025) measured the same ground: traces from seven
+mainstream frameworks — MetaGPT, ChatDev, AG2, Magentic-One — with human annotators
+independently analysing ~**150 traces** at **Cohen's kappa = 0.88**, yielding **14 failure
+modes in three groups**.
+
+The reason to carry it is that the three groups land on this file's own five-field node
+contract without being bent to fit: *system design flaws* on the interface and the `owner`,
+*inter-agent alignment failures* on the edge payload — §3's **Carries** column — and
+*missing task verification* on `check` and the checker node. An outside measurement arriving
+at the same joints is worth more than another argument from failure.
+
+**And its load-bearing result is the negative one.** Better prompts, more explicit roles and
+retries bought ChatDev **15.6%**, and the authors conclude the modes are **architectural
+rather than bugs** — the same claim §1 makes about the two fields nobody draws. The section
+is written as a review checklist, not a taxonomy to admire: per node, is the interface
+stated, is there exactly one owner, and does the check run on something other than the
+node's own claim of success.
+
+**`governance.md` put per-tool authorisation at the moment of invocation, and there is a
+strictly stronger control one layer earlier.** *The model cannot reason about capabilities
+it does not know exist.* A tool absent from the schema cannot be invoked, argued for, or
+probed for a bypass; a tool present and refused at call time is a negotiation, and
+negotiations are won sometimes. Sub-agent isolation needs both halves — schema filtering at
+construction **and** `message_history = None` at execution, because a sub-agent handed its
+parent's transcript has been told about every capability you removed from its schema.
+
+**Two numbers that point in opposite directions on purpose.** Eagerly loading every MCP tool
+schema at startup consumed **40% of the context budget before the first user message**; a
+metadata index with schemas fetched on selection takes it **under 5%**. Yet the same system
+builds *its own* prompt and tool schemas **eagerly**, in the constructor. The rule is not
+*lazy is better*: it is **eager for what you own and always need, lazy for what is foreign
+and might not be used.**
+
+**Approval fatigue is filed as a safety failure, not a UX complaint.** An approval system
+with no persistence makes users re-approve the same operations every session, which produces
+blanket auto-approval and defeats the safety system entirely — through the user rather than
+through a bug, so nothing in the logs looks wrong. **A control that is asked too often is a
+control on its way to being switched off.**
+
+## v0.21.0 — the risk one tool cannot show you, and the money an iteration refund does not cover
+
+Two findings, both of them about a rule that is right on one axis and silently assumed to
+cover a second.
+
+**Tool annotations are a risk vocabulary, and their defaults are asymmetric on purpose.**
+`readOnlyHint` **false**, `destructiveHint` **true**, `idempotentHint` **false**,
+`openWorldHint` **true** — so a server author who omits annotations entirely has declared
+the *most dangerous* shape. That is the correct fail-closed choice and the opposite of what
+most authors think they are doing: annotation **narrows** an assumption, silence widens it.
+And every one is a hint rather than a contract — the specification says a client must treat
+them as untrusted unless the server is, so an annotation may inform a UI or a policy default
+and may never be the thing that decides whether a destructive call runs.
+
+**The lethal trifecta now has a name here.** Private data, untrusted content, and the
+ability to communicate externally: any two are safe, all three in one session are an
+exfiltration path that no prompt-level instruction reliably closes. It lands in
+`agent-harness/references/tools.md` rather than in a permission section for a structural
+reason — **it is a property of the tool set assembled in a session, so per-tool analysis
+cannot see it by construction.** Every tool can pass its own review and the combination
+still be unsafe; adding a `fetch(url)` beside a private-data reader closes the triangle and
+will not look like a security change. `agent-interop/references/gateway.md` gains the
+composer's half: a gateway's whole value is assembling many servers into one surface, which
+is exactly the operation that can create a risk none of its inputs had, so **granting a role
+one more server is a trifecta question, not only a least-privilege one.**
+
+**Refund the iteration, charge the money.** Our loop refunds an iteration on a recoverable
+provider error — a 502 should not consume one of the ten attempts. That is right, and it
+says nothing about money: the provider still billed the call. A response that arrived and
+then failed to parse was generated, metered and charged. The observed failure is precise —
+a harness charged its cost on the **success path** of `query()`, so a format error meant the
+cost was never added, and the code compensated inside the exception handler. Where a cost
+ceiling is the *primary* bound (that harness ships `cost_limit = 3.0` with the step limit
+**off**), a leak there is a leak in the only guard. Accounting belongs in a `finally`, and
+**a budget that under-counts is worse than one that over-counts**: over-counting stops a run
+early and visibly, under-counting is invisible until the invoice and biases toward the
+failing runs, which are the expensive ones.
+
 ## v0.20.0 — what a trajectory cannot carry across a vendor, and where the capability goes
 
 Three findings from the harvest, all landing on the same question: **which model does what,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.20.0",
+  "version": "0.22.0",
   "scripts": {
     "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js"
   },

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
-  "version": "0.20.0",
+  "version": "0.22.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/agent-stack/skills/agent-harness/references/tools.md

@@ -126,6 +126,48 @@ the mistake cannot be made**, rather than documenting the mistake.
 - A required `confirm: true` on a destructive action, so a partially-formed call fails
   closed.
 
+## Annotations, and the risk one tool cannot show you
+
+MCP tools carry four hints, and their defaults are asymmetric on purpose:
+
+| Hint | Default | So an unannotated tool is assumed to be |
+|---|---|---|
+| `readOnlyHint` | `false` | one that writes |
+| `destructiveHint` | `true` | destructive |
+| `idempotentHint` | `false` | unsafe to repeat |
+| `openWorldHint` | `true` | reaching outside your system |
+
+**A server author who omits annotations entirely has declared the most dangerous shape**,
+which is the correct fail-closed choice and the opposite of what most authors assume they
+are doing. Annotate to *narrow* the assumption; silence widens it.
+
+**Every one is a hint, not a contract.** The specification is explicit that a client must
+treat descriptions and annotations as untrusted unless the server itself is trusted — so an
+annotation informs a UI and a policy default, and may never be the thing that decides
+whether a destructive call runs.
+
+### The lethal trifecta — a property of the session, not of a tool
+
+Three capabilities that are individually ordinary and jointly an exfiltration path:
+
+1. access to **private data**,
+2. exposure to **untrusted content**,
+3. the ability to **communicate externally**.
+
+Any two are safe. All three in one session mean untrusted content can instruct the agent to
+read private data and send it out, and no prompt-level instruction reliably prevents it.
+
+**The reason it belongs here rather than in a permission check:** the trifecta is a property
+of *the tool set assembled in a session*, so **per-tool analysis cannot see it**. Every tool
+can pass its own review and the combination still be unsafe — which is why a review that
+walks a server's tools one at a time answers a different question than "what can this
+session do". Look at the set, and at what a gateway composes into it.
+
+**The practical consequence for a tool author:** a tool that only reads private data is
+fine; adding a `fetch(url)` beside it is what closes the triangle, and it will not look like
+a security change in review.
+
+
 ## Evaluating tools
 
 Tools deserve **thorough documentation and testing**, and testing means running the agent
@@ -150,5 +192,7 @@ Fix the interface, not the prompt, when the fault is in this table.
   can do nothing with.
 - **Treating tool output as trusted.** It is attacker-controlled input if the server is; the
   specification says descriptions and annotations are untrusted unless the server is.
+- **Reviewing tools one at a time.** The lethal trifecta is a property of the assembled set;
+  a per-tool review cannot see it by construction.
 - **Adding a tool to fix a prompt problem.** The set grows, selection degrades, and the
   original defect is still there.

package/plugins/agent-stack/skills/agent-interop/references/gateway.md

@@ -147,6 +147,25 @@ authorization** — the last being the per-tool control that a generic gateway l
 That is the strongest single argument for it in a cluster that already runs Gateway API: the
 routing objects are ones your platform team already reviews.
 
+## The gateway is where the lethal trifecta gets assembled
+
+A gateway's whole value is composing many servers into one surface, and that is exactly the
+operation that can create a risk none of its inputs had. The **lethal trifecta** — private
+data, untrusted content, and the ability to communicate externally — is a property of the
+**tool set in a session**, so every upstream can pass its own review and the composition
+still be unsafe.
+
+Two consequences for whoever runs the gateway:
+
+- **A role is a trifecta decision, not only a least-privilege one.** Granting a role one
+  more server is the moment to ask which of the three corners it just completed — not
+  whether that server is individually trustworthy.
+- **Per-tool annotations do not answer it.** They are hints about one tool (`readOnlyHint`,
+  `destructiveHint`, `idempotentHint`, `openWorldHint`, and their defaults assume the
+  dangerous shape — `agent-harness/references/tools.md`), and no combination of per-tool
+  hints computes a session-level property.
+
+
 ## Traps
 
 - **Introducing a gateway and leaving the direct paths open.** The policy is then advisory.

package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md

@@ -86,7 +86,8 @@ was gathered rather than returning nothing. The full listing is in
 - **In-loop trimming**: At ~80% capacity, collapse older assistant+tool pairs into one-liner summaries
 - **Token limit recovery**: On `LLMTokenLimitError`, compress to 60% and retry once. If still fails, return partial answer
 - **Max iterations guard**: Always have a hard limit. On exhaustion, compose best-effort answer from data gathered so far
-- **Iteration refund**: a recoverable provider error is not charged to that guard
+- **Iteration refund**: a recoverable provider error is not charged to that guard — the
+  money still is, and accounting on the success path alone under-counts the worst runs
 - **Budget awareness**: tell the model what is left, or 300 steps performs like 30
 
 ---

package/plugins/agent-stack/skills/agent-orchestrator/references/governance.md

@@ -36,6 +36,45 @@ guardrails" so often means only the first.
 The last one is the one most designs miss: a sub-agent that inherits its caller's
 authority silently widens every permission the caller had.
 
+## The cheapest control is absence
+
+The **Tool call** row above puts per-tool authorisation at the moment of invocation. There
+is a control one layer earlier and it is strictly stronger:
+
+> **The model cannot reason about capabilities it does not know exist.**
+
+A tool absent from the schema cannot be invoked, cannot be argued for, and cannot be probed
+for a bypass — there is nothing to jailbreak toward. A tool present in the schema and
+refused at call time is a negotiation, and negotiations are won sometimes.
+
+So **filter the schema at build time**, and treat runtime authorisation as the second line
+rather than the first. Sub-agent isolation comes from exactly two mechanisms used together:
+schema filtering when the agent is constructed, and no inherited conversation
+(`message_history = None`) when it runs. The second matters as much as the first — a
+sub-agent handed its parent's transcript has been told about every capability you carefully
+removed from its schema.
+
+**Two numbers, and they point in opposite directions on purpose.** Eagerly loading every
+MCP tool schema at startup consumed **40% of the context budget before the first user
+message**; a metadata index at startup with the full schema fetched on selection takes it
+**under 5%**. But the same system builds *its own* prompt and tool schemas **eagerly**, in
+the constructor. The rule underneath is not *lazy is better*: it is **eager for what you
+own and always need, lazy for what is foreign and might not be used** — the first removes
+latency and race conditions from the hot path, the second removes a cost you cannot predict.
+
+### Approval fatigue is a safety failure, not a UX complaint
+
+> Without persistence, users must re-approve the same operations every session, causing
+> approval fatigue that leads to **blanket auto-approval, defeating the safety system
+> entirely**.
+
+An approval system with no memory converts itself into no approval system, and it does so
+through the user rather than through a bug — so nothing in the logs looks wrong. The
+remedy is on the same axis as the section above: **decide once what does not need asking,
+remove it from the question, and spend the prompts on what genuinely changes.** A control
+that is asked too often is a control on its way to being switched off.
+
+
 ## Guardrails, and their honest limit
 
 The content-layer checks worth having, roughly in order of reliability:

package/plugins/agent-stack/skills/agent-orchestrator/references/graph-engineering.md

@@ -28,6 +28,7 @@ well-measured answer to the wrong question.
 - [9. What Claude Code actually executes](#9-what-claude-code-actually-executes)
 - [10. Barrier or no barrier](#10-barrier-or-no-barrier)
 - [11. Project defaults, written once](#11-project-defaults-written-once)
+- [11a. MAST — the failures, named from outside](#11a-mast--the-failures-named-from-outside)
 - [12. The source's four diagrams, and what each one is for](#12-the-sources-four-diagrams-and-what-each-one-is-for)
 - [Where this file disagrees with its source](#where-this-file-disagrees-with-its-source)
 
@@ -375,6 +376,32 @@ The last line is `context-engineering.md`'s *filesystem as context* stated as a
 rule: an edge that carries a path costs a few tokens, and an edge that carries a
 transcript costs the window.
 
+## 11a. MAST — the failures, named from outside
+
+Everything above is argued from what breaks. *Why Do Multi-Agent LLM Systems Fail?* (2025)
+measured it: execution traces from seven mainstream frameworks — MetaGPT, ChatDev, AG2 and
+Magentic-One among them — with human annotators independently analysing roughly **150
+traces** at **Cohen's kappa = 0.88**, producing **14 failure modes in three groups**.
+
+The three groups land on this file's own node contract, which is the reason to carry them:
+
+| MAST group | Modes include | The field it lands on |
+|---|---|---|
+| **System design flaws** | unclear interfaces between agents, overlapping roles, wrong tool configuration | `input` / `output` (the interface), and `owner` (overlapping roles) |
+| **Inter-agent alignment failures** | inconsistent understanding of the objective, downstream misinterpretation, logically contradictory operations | the **edge payload** — what actually crosses, §3's *Carries* column |
+| **Missing task verification** | an agent reports *completed* and the result does not meet the requirement | `check`, and §6's checker node |
+
+**The load-bearing result is the negative one.** Straightforward fixes — better prompts,
+more explicit role descriptions, retries — bought ChatDev only **15.6%**. The authors
+conclude the modes are **architectural rather than bugs**, which is the same claim §1 makes
+about the two fields nobody draws: an owner and a completion test are not documentation of
+a graph, they are the parts that make it a graph.
+
+Read the mapping as a review checklist rather than a taxonomy to admire: for each node, is
+the interface stated, is there exactly one owner, and does the *check* run on something
+other than the node's own claim of success.
+
+
 ## 12. The source's four diagrams, and what each one is for
 
 The article carries four hand-drawn figures. They are not decoration — each one is doing

package/plugins/agent-stack/skills/agent-orchestrator/references/llm-proxy-billing.md

@@ -24,6 +24,7 @@ the patterns hold for any upstream that issues per-tenant keys with limits.
 - [Discovering spend you do not control](#discovering-spend-you-do-not-control)
 - [Guardrails: budgets, loops, auto-pause](#guardrails-budgets-loops-auto-pause)
 - [Key lifecycle and healing](#key-lifecycle-and-healing)
+- [Refund the iteration, charge the money](#refund-the-iteration-charge-the-money)
 - [The refund waterfall](#the-refund-waterfall)
 - [Model routing and fallbacks](#model-routing-and-fallbacks)
 
@@ -226,6 +227,41 @@ handled three different ways.
 
 ---
 
+## Refund the iteration, charge the money
+
+These are two axes and it is easy to ship one rule for both.
+
+`agent-orchestrator`'s loop **refunds the iteration** on a recoverable provider error: a
+502 should not consume one of the ten attempts the agent has to finish its work. That is
+right, and it says nothing about money.
+
+**The provider still billed the call.** A response that arrived and then failed to parse
+was generated, metered and charged upstream; so was the one that arrived truncated, and the
+one whose tool call was malformed. The tempting symmetry — *the attempt did not count, so
+it did not cost* — is how a spend guard under-counts on exactly the runs that go worst.
+
+The failure mode is specific and worth naming, because it hides where nobody looks:
+
+> A harness charged `self.cost` on the **success path** of its `query()`. When parsing the
+> response raised a format error, `query()` never returned, so the cost was never added —
+> and the code compensated for it explicitly inside the exception handler. Where a cost
+> ceiling is the *primary* bound — that harness ships `cost_limit = 3.0` with the step
+> limit **off** — a leak in that accounting is a leak in the only guard there is.
+
+The rule, in one line each:
+
+- **Iteration:** refunded on a recoverable provider error, never on a misconfiguration.
+- **Money:** charged whenever the provider generated tokens, including on every path that
+  raises after the response arrived.
+- **Therefore:** accounting belongs in a `finally`, or in the exception handler as well as
+  the success path. If the only place your cost is added is the line after a successful
+  parse, the guard is quietly optimistic.
+
+A budget that under-counts is worse than one that over-counts: over-counting stops a run
+early and is visible immediately; under-counting is invisible until the invoice, and it
+biases toward the failing runs, which are the expensive ones.
+
+
 ## The refund waterfall
 
 A payment refund has to come out of somewhere, and the money has usually moved.