@ssheleg/agent-stack 0.19.1 → 0.21.0

package/CHANGELOG.md

@@ -1,3 +1,77 @@
+## v0.21.0 — the risk one tool cannot show you, and the money an iteration refund does not cover
+
+Two findings, both of them about a rule that is right on one axis and silently assumed to
+cover a second.
+
+**Tool annotations are a risk vocabulary, and their defaults are asymmetric on purpose.**
+`readOnlyHint` **false**, `destructiveHint` **true**, `idempotentHint` **false**,
+`openWorldHint` **true** — so a server author who omits annotations entirely has declared
+the *most dangerous* shape. That is the correct fail-closed choice and the opposite of what
+most authors think they are doing: annotation **narrows** an assumption, silence widens it.
+And every one is a hint rather than a contract — the specification says a client must treat
+them as untrusted unless the server is, so an annotation may inform a UI or a policy default
+and may never be the thing that decides whether a destructive call runs.
+
+**The lethal trifecta now has a name here.** Private data, untrusted content, and the
+ability to communicate externally: any two are safe, all three in one session are an
+exfiltration path that no prompt-level instruction reliably closes. It lands in
+`agent-harness/references/tools.md` rather than in a permission section for a structural
+reason — **it is a property of the tool set assembled in a session, so per-tool analysis
+cannot see it by construction.** Every tool can pass its own review and the combination
+still be unsafe; adding a `fetch(url)` beside a private-data reader closes the triangle and
+will not look like a security change. `agent-interop/references/gateway.md` gains the
+composer's half: a gateway's whole value is assembling many servers into one surface, which
+is exactly the operation that can create a risk none of its inputs had, so **granting a role
+one more server is a trifecta question, not only a least-privilege one.**
+
+**Refund the iteration, charge the money.** Our loop refunds an iteration on a recoverable
+provider error — a 502 should not consume one of the ten attempts. That is right, and it
+says nothing about money: the provider still billed the call. A response that arrived and
+then failed to parse was generated, metered and charged. The observed failure is precise —
+a harness charged its cost on the **success path** of `query()`, so a format error meant the
+cost was never added, and the code compensated inside the exception handler. Where a cost
+ceiling is the *primary* bound (that harness ships `cost_limit = 3.0` with the step limit
+**off**), a leak there is a leak in the only guard. Accounting belongs in a `finally`, and
+**a budget that under-counts is worse than one that over-counts**: over-counting stops a run
+early and visibly, under-counting is invisible until the invoice and biases toward the
+failing runs, which are the expensive ones.
+
+## v0.20.0 — what a trajectory cannot carry across a vendor, and where the capability goes
+
+Three findings from the harvest, all landing on the same question: **which model does what,
+and what survives when that changes mid-run.** §6 shipped three traps about routing, and all
+three quietly assume the *request* is what moves.
+
+- **A trajectory carries a vendor credential, and it may not be attached to the reasoning.**
+  Tool calls and results are portable — different structure, same meaning, re-render and
+  send. Reasoning is portable *text* plus a **non-portable credential** the vendor attaches
+  to prove the reasoning is its own, and vendors disagree on what they demand: one end
+  validates nothing, the other rejects any credential it did not issue. The credential
+  sometimes sits on the **tool call** rather than the reasoning — which is why *"just strip
+  all reasoning before failing over"*, the policy that sounds safest, is the one that
+  produces a 400. Store trajectories in a neutral internal format, keep the text, discard
+  the credential, re-render per vendor at send time, and put the failover boundary
+  **between turns**. A fallback chain never exercised mid-trajectory has not been tested: a
+  green health probe answers a question about the endpoint, not about your history.
+- **Capability is not spent evenly — the planner is the bottleneck.** *Plan-and-Act*
+  (arXiv:2503.09572) found that with good enough planning a relatively simple executor
+  suffices, and with a wrong decomposition every downstream executor is building on a false
+  premise; their 54% on WebArena-Lite came from improving the **planner**, not the executor.
+  So the strongest model and the most carefully written prompt go to the **manager**. It
+  also says where to look when a multi-agent system underperforms: **a weak plan is
+  invisible in every executor's transcript**, because each one did its own step correctly.
+- **Steps the agent cannot see buy nothing.** Standard agents have no budget awareness, so
+  at **300 steps** they still plateau at roughly what they achieve at **30**. A
+  max-iteration guard is the floor of this rather than the mechanism — it stops the spend
+  and never changes the behaviour that led there.
+
+**And the second displacement in two releases.** The body was at 4609/4750 after v0.19.0
+bought that headroom back; these three lines would have left **5 tokens**. §1's context
+dataclass and sub-agent base class moved to `references/patterns.md`, beside the loop
+listing that went there in v0.19.0, landing the body at **4631/4750**. That is now twice in
+a row that an addition has cost a displacement, which is the auditor's own signal — *the
+answer then is a split, not a trim* — and it is filed rather than absorbed again.
+
 ## v0.19.1 — the class the umbrella had been catching for us, twice
 
 `B-126`'s board row shipped in v0.19.0 with **nine cells against the eight its header

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.19.1",
+  "version": "0.21.0",
   "scripts": {
     "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js"
   },

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
-  "version": "0.19.1",
+  "version": "0.21.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/agent-stack/skills/agent-harness/references/tools.md

@@ -126,6 +126,48 @@ the mistake cannot be made**, rather than documenting the mistake.
 - A required `confirm: true` on a destructive action, so a partially-formed call fails
   closed.
 
+## Annotations, and the risk one tool cannot show you
+
+MCP tools carry four hints, and their defaults are asymmetric on purpose:
+
+| Hint | Default | So an unannotated tool is assumed to be |
+|---|---|---|
+| `readOnlyHint` | `false` | one that writes |
+| `destructiveHint` | `true` | destructive |
+| `idempotentHint` | `false` | unsafe to repeat |
+| `openWorldHint` | `true` | reaching outside your system |
+
+**A server author who omits annotations entirely has declared the most dangerous shape**,
+which is the correct fail-closed choice and the opposite of what most authors assume they
+are doing. Annotate to *narrow* the assumption; silence widens it.
+
+**Every one is a hint, not a contract.** The specification is explicit that a client must
+treat descriptions and annotations as untrusted unless the server itself is trusted — so an
+annotation informs a UI and a policy default, and may never be the thing that decides
+whether a destructive call runs.
+
+### The lethal trifecta — a property of the session, not of a tool
+
+Three capabilities that are individually ordinary and jointly an exfiltration path:
+
+1. access to **private data**,
+2. exposure to **untrusted content**,
+3. the ability to **communicate externally**.
+
+Any two are safe. All three in one session mean untrusted content can instruct the agent to
+read private data and send it out, and no prompt-level instruction reliably prevents it.
+
+**The reason it belongs here rather than in a permission check:** the trifecta is a property
+of *the tool set assembled in a session*, so **per-tool analysis cannot see it**. Every tool
+can pass its own review and the combination still be unsafe — which is why a review that
+walks a server's tools one at a time answers a different question than "what can this
+session do". Look at the set, and at what a gateway composes into it.
+
+**The practical consequence for a tool author:** a tool that only reads private data is
+fine; adding a `fetch(url)` beside it is what closes the triangle, and it will not look like
+a security change in review.
+
+
 ## Evaluating tools
 
 Tools deserve **thorough documentation and testing**, and testing means running the agent
@@ -150,5 +192,7 @@ Fix the interface, not the prompt, when the fault is in this table.
   can do nothing with.
 - **Treating tool output as trusted.** It is attacker-controlled input if the server is; the
   specification says descriptions and annotations are untrusted unless the server is.
+- **Reviewing tools one at a time.** The lethal trifecta is a property of the assembled set;
+  a per-tool review cannot see it by construction.
 - **Adding a tool to fix a prompt problem.** The set grows, selection degrades, and the
   original defect is still there.

package/plugins/agent-stack/skills/agent-interop/references/gateway.md

@@ -147,6 +147,25 @@ authorization** — the last being the per-tool control that a generic gateway l
 That is the strongest single argument for it in a cluster that already runs Gateway API: the
 routing objects are ones your platform team already reviews.
 
+## The gateway is where the lethal trifecta gets assembled
+
+A gateway's whole value is composing many servers into one surface, and that is exactly the
+operation that can create a risk none of its inputs had. The **lethal trifecta** — private
+data, untrusted content, and the ability to communicate externally — is a property of the
+**tool set in a session**, so every upstream can pass its own review and the composition
+still be unsafe.
+
+Two consequences for whoever runs the gateway:
+
+- **A role is a trifecta decision, not only a least-privilege one.** Granting a role one
+  more server is the moment to ask which of the three corners it just completed — not
+  whether that server is individually trustworthy.
+- **Per-tool annotations do not answer it.** They are hints about one tool (`readOnlyHint`,
+  `destructiveHint`, `idempotentHint`, `openWorldHint`, and their defaults assume the
+  dangerous shape — `agent-harness/references/tools.md`), and no combination of per-tool
+  hints computes a session-level property.
+
+
 ## Traps
 
 - **Introducing a gateway and leaving the direct paths open.** The policy is then advisory.

package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md

@@ -40,23 +40,12 @@ OrchestratorAgent.run(AgentContext)
 
 ### Shared Context Object
 
-Pass a single immutable-ish context object to all sub-agents:
+Pass a single immutable-ish context object to every sub-agent. It carries the request
+(`project_id`, `user_question`, `chat_history`), the machinery (`llm_router`, `tracker`,
+`workflow_id`), the resolved provider and model, and one `extra` dict for pipeline flags.
+The full dataclass is in [`references/patterns.md`](references/patterns.md) → *The
+orchestrator's shared context and sub-agent protocol*.
 
-```python
-@dataclass
-class AgentContext:
-    project_id: str
-    user_question: str
-    chat_history: list[Message]
-    llm_router: LLMRouter           # provider abstraction with retry/fallback
-    tracker: WorkflowTracker        # SSE event emitter for real-time UI
-    workflow_id: str                 # unique ID for this request
-    connection_config: ... | None   # external resource config
-    user_id: str | None
-    preferred_provider: str | None  # e.g. "openrouter"
-    model: str | None               # e.g. "<provider>/<model-id>"
-    extra: dict[str, Any]           # pipeline_action, flags, overrides
-```
 
 **Key principles:**
 - Sub-agents never modify context — they return typed results
@@ -65,20 +54,10 @@ class AgentContext:
 
 ### Sub-Agent Protocol
 
-Every sub-agent extends a base class:
-
-```python
-class BaseAgent(ABC):
-    @abstractmethod
-    async def run(self, context: AgentContext, **kwargs) -> AgentResult: ...
-
-    @property
-    @abstractmethod
-    def name(self) -> str: ...
+Every sub-agent extends one base class with a single abstract `run(context) -> Result`,
+so the orchestrator never learns what any of them does internally. The class is in the same
+reference section.
 
-    @staticmethod
-    def accum_usage(total, usage): ...  # merge token counters
-```
 
 Typed result subclasses per agent (e.g. `SQLAgentResult` with `query`, `results`, `attempts`).
 
@@ -107,7 +86,9 @@ was gathered rather than returning nothing. The full listing is in
 - **In-loop trimming**: At ~80% capacity, collapse older assistant+tool pairs into one-liner summaries
 - **Token limit recovery**: On `LLMTokenLimitError`, compress to 60% and retry once. If still fails, return partial answer
 - **Max iterations guard**: Always have a hard limit. On exhaustion, compose best-effort answer from data gathered so far
-- **Iteration refund**: a recoverable provider error is not charged to that guard
+- **Iteration refund**: a recoverable provider error is not charged to that guard — the
+  money still is, and accounting on the success path alone under-counts the worst runs
+- **Budget awareness**: tell the model what is left, or 300 steps performs like 30
 
 ---
 
@@ -197,6 +178,11 @@ fallback chain and per-provider retry with exponential backoff, and its
 - **Model selection has three levels** — the request, the tenant, the system
   default — and a tenant override that silently loses to a request parameter is
   how a cheap model ends up billed at a premium one's rate.
+- **Those three assume the REQUEST is portable; the trajectory is not.** Reasoning carries
+  a vendor credential — sometimes on the tool call — so mid-turn failover can 400, and
+  *strip all reasoning* is what causes it. Fail over between turns.
+- **Capability is not spent evenly**: the planner is the bottleneck, so the strongest model
+  goes to the manager, not to whichever agent does the most work.
 ## 7. Multi-Layer Memory System
 
 Four layers, each with a different lifetime and a different reason to exist:

package/plugins/agent-stack/skills/agent-orchestrator/references/llm-proxy-billing.md

@@ -24,6 +24,7 @@ the patterns hold for any upstream that issues per-tenant keys with limits.
 - [Discovering spend you do not control](#discovering-spend-you-do-not-control)
 - [Guardrails: budgets, loops, auto-pause](#guardrails-budgets-loops-auto-pause)
 - [Key lifecycle and healing](#key-lifecycle-and-healing)
+- [Refund the iteration, charge the money](#refund-the-iteration-charge-the-money)
 - [The refund waterfall](#the-refund-waterfall)
 - [Model routing and fallbacks](#model-routing-and-fallbacks)
 
@@ -226,6 +227,41 @@ handled three different ways.
 
 ---
 
+## Refund the iteration, charge the money
+
+These are two axes and it is easy to ship one rule for both.
+
+`agent-orchestrator`'s loop **refunds the iteration** on a recoverable provider error: a
+502 should not consume one of the ten attempts the agent has to finish its work. That is
+right, and it says nothing about money.
+
+**The provider still billed the call.** A response that arrived and then failed to parse
+was generated, metered and charged upstream; so was the one that arrived truncated, and the
+one whose tool call was malformed. The tempting symmetry — *the attempt did not count, so
+it did not cost* — is how a spend guard under-counts on exactly the runs that go worst.
+
+The failure mode is specific and worth naming, because it hides where nobody looks:
+
+> A harness charged `self.cost` on the **success path** of its `query()`. When parsing the
+> response raised a format error, `query()` never returned, so the cost was never added —
+> and the code compensated for it explicitly inside the exception handler. Where a cost
+> ceiling is the *primary* bound — that harness ships `cost_limit = 3.0` with the step
+> limit **off** — a leak in that accounting is a leak in the only guard there is.
+
+The rule, in one line each:
+
+- **Iteration:** refunded on a recoverable provider error, never on a misconfiguration.
+- **Money:** charged whenever the provider generated tokens, including on every path that
+  raises after the response arrived.
+- **Therefore:** accounting belongs in a `finally`, or in the exception handler as well as
+  the success path. If the only place your cost is added is the line after a successful
+  parse, the guard is quietly optimistic.
+
+A budget that under-counts is worse than one that over-counts: over-counting stops a run
+early and is visible immediately; under-counting is invisible until the invoice, and it
+biases toward the failing runs, which are the expensive ones.
+
+
 ## The refund waterfall
 
 A payment refund has to come out of somewhere, and the money has usually moved.
@@ -260,3 +296,57 @@ model" is otherwise unanswerable.
 
 See `patterns.md` for the retry, health-check and error-hierarchy patterns these
 routing calls sit inside.
+
+### What a trajectory cannot carry across a vendor
+
+The routing above assumes the **request** is portable. Mid-run failover is a different
+problem, because by then there is an accumulated history and not all of it can move.
+
+- **Tool calls and results are portable.** They differ in structure between vendors and
+  mean the same thing, so re-rendering them is enough.
+- **Reasoning is not.** It is portable *text* plus a **non-portable credential** the vendor
+  attaches to prove the reasoning is its own. Vendors disagree on what they demand: one end
+  validates nothing, the other rejects any credential it did not issue.
+- **The credential is not always attached to the reasoning.** It may sit on the *tool call*
+  — which is why the apparently safe policy *"just strip all reasoning before failing
+  over"* is exactly what fails at some vendors, and fails as a 400 rather than as
+  degradation.
+
+Design rules that follow:
+
+- Store trajectories in a **neutral internal format**: keep the text, discard the
+  credential, re-render per vendor at send time.
+- Decide the failover boundary deliberately. **Between turns** is cheap and safe; **inside
+  a turn**, after reasoning has been emitted, is where the credential problem lives.
+- A fallback chain that has never been exercised **mid-trajectory** has not been tested.
+  A green health probe answers a question about the endpoint, not about your history.
+
+### Where the capability goes — not evenly
+
+The intuitive allocation is to spend evenly across agents, or to give the strongest model
+to whichever agent does the most work. Both are wrong for a planner–executor pair.
+
+*Plan-and-Act* (arXiv:2503.09572) found the **planner is the bottleneck of the whole
+system**: with good enough planning a relatively simple executor suffices, and with a wrong
+decomposition every downstream executor is building on a false premise. Their 54% on
+WebArena-Lite came from improving the **planner's** planning, not the executor's execution.
+
+So: **give the strongest model and the most carefully written prompt to the manager**, and
+let the executors be cheaper. It also sets where to look when a multi-agent system
+underperforms — a weak plan is invisible in every executor's transcript, because each one
+did its own step correctly.
+
+### Budget awareness — steps the agent cannot see buy nothing
+
+Raising a step budget does not by itself buy more work. Google's *Budget-Aware Tool-Use
+Enables Effective Agent Scaling* reports that standard agents have **no budget awareness**,
+so at **300 steps** they still conduct shallow searches and plateau at roughly what they
+achieve at **30**.
+
+Spending a larger budget requires telling the model where it is in that budget, so it can
+shift strategy — broad exploration early, narrowing later. The multi-agent form is the
+manager allocating step budget per sub-task rather than handing every executor the same cap.
+
+A max-iteration guard that only composes a partial answer at exhaustion is the *floor* of
+this, not the mechanism: it stops the spend, and it never changes the behaviour that led
+there.

package/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md

@@ -200,6 +200,51 @@ for attempt in range(1, max_retries + 1):
 
 ---
 
+## The orchestrator's shared context and sub-agent protocol
+
+`SKILL.md` §1 states the two rules — one context object down, typed results back — and this
+is the shape they describe. It moved here in v0.20.0 for the same reason §2's loop listing
+did in v0.19.0: the body carries what is read every time, a reference carries what is read
+once.
+
+```python
+@dataclass
+class AgentContext:
+    project_id: str
+    user_question: str
+    chat_history: list[Message]
+    llm_router: LLMRouter           # provider abstraction with retry/fallback
+    tracker: WorkflowTracker        # SSE event emitter for real-time UI
+    workflow_id: str                 # unique ID for this request
+    connection_config: ... | None   # external resource config
+    user_id: str | None
+    preferred_provider: str | None  # e.g. "openrouter"
+    model: str | None               # e.g. "<provider>/<model-id>"
+    extra: dict[str, Any]           # pipeline_action, flags, overrides
+```
+
+**Key principles:**
+- Sub-agents never modify context — they return typed results
+- Provider/model preferences flow down from user → project defaults → app defaults
+- `extra` carries pipeline state, flags like `_skip_complexity`, session ids
+
+```python
+class BaseAgent(ABC):
+    @abstractmethod
+    async def run(self, context: AgentContext, **kwargs) -> AgentResult: ...
+
+    @property
+    @abstractmethod
+    def name(self) -> str: ...
+
+    @staticmethod
+    def accum_usage(total, usage): ...  # merge token counters
+```
+
+The single abstract method is what keeps the orchestrator ignorant of any sub-agent's
+internals; a second one is how that boundary starts leaking.
+
+
 ## The tool-calling loop, in full
 
 `SKILL.md` §2 states the six steps and the guard; this is the listing they describe. It