@ssheleg/agent-stack 0.21.0 → 0.23.0

package/CHANGELOG.md

@@ -1,3 +1,94 @@
+## v0.23.0 — the wire format for §7, and the three things it will not carry
+
+`agent-evals` §7 said *what* to instrument in four bullets and named no format.
+`references/otel-genai.md` is the format — and the reason it is worth its own file is not
+the field list but the three places the standard refuses to carry what this skill requires.
+
+**Read the status before encoding any of it.** The conventions have moved out of the main
+`semantic-conventions` repository into their own, whose README lists **Schema URL: `TODO`**,
+and every document is `Status: Development`. On the inference span the only `Stable`
+attributes are the ones borrowed from core semconv; **every `gen_ai.*` attribute is
+Development**. Adopt it anyway — a moving standard beats a private vocabulary nobody else's
+tooling reads — but pin the version and treat any branch on a `gen_ai.*` name as code with
+an expiry date.
+
+**The evaluation event has no field for who produced the score.** `gen_ai.evaluation.result`
+carries a name, a value, a low-cardinality label and an explanation, and nothing about the
+scorer — while §7 requires exactly that (`human` | `llm_judge` | `code_check`), because a
+score with no source cannot be calibrated or audited. OpenInference has the field under
+another name, `annotation.annotator_kind`, with our three values. The reference says to carry
+it as a documented extension rather than drop the requirement: a judge score and a human
+label that are indistinguishable on the wire get averaged by somebody downstream, which is
+what §5's calibration exists to prevent.
+
+**There is no cost attribute at all**, and usage is eleven numbers rather than one —
+`input_tokens`, `output_tokens`, `reasoning.output_tokens`, `cache_read.input_tokens`,
+`cache_write.input_tokens`, plus per-modality splits. **A cost from `input + output` alone is
+wrong in both directions**: it bills cache reads at full price and misses reasoning tokens
+and cache writes entirely. `gen_ai.client.token.usage` carries a hard **MUST NOT report**
+when counts are unobtainable — absence over a fabricated zero, the rule `audit.md` already
+states for attribution.
+
+**And two seams that decide a design rather than a field.** The content-upload hook *"SHOULD
+operate independently of the opt-in flags"* and *"SHOULD be invoked regardless of the span
+sampling decision"* — so it is the last scrubbing point and it fires on spans nobody will
+read; a masking hook meanwhile **cannot change the span name**, which is built from tool and
+MCP target names, so redaction and naming are one decision. Verified against a shipping
+implementation rather than assumed: OpenLLMetry's `is_content_tracing_enabled()` returns
+**true when its environment variable is unset**, the opposite of the spec's default, so
+prompts are captured unless you turn them off (`traceloop-sdk/traceloop/sdk/config/__init__.py`,
+read 2026-08-31).
+
+**Finally, two words that mean three things each.** *"OpenTelemetry-based"* answers a
+transport question and no semantic one: Phoenix emits OpenInference, OpenLLMetry emits
+`gen_ai.*` plus `traceloop.*`, neither is a subset of the other, and a trace assembled from
+both is two disconnected halves under one trace id. And *"replay"* is durable execution
+(recorded results, nothing re-runs), a trace playground (the call re-runs against the live
+provider), or this skill's fixture replay (an assertion over a stored run) — `audit.md` asks
+whether a run can be replayed without saying which, and the reference names the three.
+
+## v0.22.0 — the failures named from outside, and the control that is absence
+
+**`graph-engineering.md` was argued entirely from what breaks, with no citation behind it.**
+*Why Do Multi-Agent LLM Systems Fail?* (2025) measured the same ground: traces from seven
+mainstream frameworks — MetaGPT, ChatDev, AG2, Magentic-One — with human annotators
+independently analysing ~**150 traces** at **Cohen's kappa = 0.88**, yielding **14 failure
+modes in three groups**.
+
+The reason to carry it is that the three groups land on this file's own five-field node
+contract without being bent to fit: *system design flaws* on the interface and the `owner`,
+*inter-agent alignment failures* on the edge payload — §3's **Carries** column — and
+*missing task verification* on `check` and the checker node. An outside measurement arriving
+at the same joints is worth more than another argument from failure.
+
+**And its load-bearing result is the negative one.** Better prompts, more explicit roles and
+retries bought ChatDev **15.6%**, and the authors conclude the modes are **architectural
+rather than bugs** — the same claim §1 makes about the two fields nobody draws. The section
+is written as a review checklist, not a taxonomy to admire: per node, is the interface
+stated, is there exactly one owner, and does the check run on something other than the
+node's own claim of success.
+
+**`governance.md` put per-tool authorisation at the moment of invocation, and there is a
+strictly stronger control one layer earlier.** *The model cannot reason about capabilities
+it does not know exist.* A tool absent from the schema cannot be invoked, argued for, or
+probed for a bypass; a tool present and refused at call time is a negotiation, and
+negotiations are won sometimes. Sub-agent isolation needs both halves — schema filtering at
+construction **and** `message_history = None` at execution, because a sub-agent handed its
+parent's transcript has been told about every capability you removed from its schema.
+
+**Two numbers that point in opposite directions on purpose.** Eagerly loading every MCP tool
+schema at startup consumed **40% of the context budget before the first user message**; a
+metadata index with schemas fetched on selection takes it **under 5%**. Yet the same system
+builds *its own* prompt and tool schemas **eagerly**, in the constructor. The rule is not
+*lazy is better*: it is **eager for what you own and always need, lazy for what is foreign
+and might not be used.**
+
+**Approval fatigue is filed as a safety failure, not a UX complaint.** An approval system
+with no persistence makes users re-approve the same operations every session, which produces
+blanket auto-approval and defeats the safety system entirely — through the user rather than
+through a bug, so nothing in the logs looks wrong. **A control that is asked too often is a
+control on its way to being switched off.**
+
 ## v0.21.0 — the risk one tool cannot show you, and the money an iteration refund does not cover
 
 Two findings, both of them about a rule that is right on one axis and silently assumed to

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.21.0",
+  "version": "0.23.0",
   "scripts": {
     "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js"
   },

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
-  "version": "0.21.0",
+  "version": "0.23.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/agent-stack/skills/agent-evals/SKILL.md

@@ -288,6 +288,10 @@ predictive.
 
 ## 7. What to instrument first
 
+**The wire format for all of this is `references/otel-genai.md`** — and it is worth
+reading before the first span, because two of the things this section requires (a score's
+source, a trustworthy cost) are the two the standard does not carry.
+
 None of the above runs without these, and they are the part people skip:
 
 - **A durable trace store, queryable by id, filterable by score and time.** The live
@@ -330,6 +334,7 @@ None of the above runs without these, and they are the part people skip:
 | Load | When |
 |---|---|
 | [`references/statistics.md`](references/statistics.md) | a number is about to change a decision — how many runs before a difference is real, `pass@k` vs `pass^k` and why trials are not independent, paired comparison, the harness as a variable, and what a given piece of evidence authorises next |
+| [`references/otel-genai.md`](references/otel-genai.md) | you are **instrumenting an agent for someone else to read** — span-name formulas and the closed operation enum, the evaluation event that has no field for who scored, the three content tiers and the upload hook that runs even on dropped spans, the eleven token attributes and why cost is a join you can get wrong in both directions, and why "OpenTelemetry-based" is not one vocabulary |
 
 ---
 

package/plugins/agent-stack/skills/agent-evals/references/otel-genai.md

@@ -0,0 +1,177 @@
+# OpenTelemetry GenAI — the wire format for §7, and what it does not carry
+
+**Load this when instrumenting an agent for someone else to read**: choosing span names and
+attributes, deciding what of a prompt reaches a backend, wiring token usage into a bill, or
+judging what an "OpenTelemetry-based" tool actually gives you.
+
+§7 says *what* to instrument in four bullets. This is the format that carries it — and the
+three places it will not carry what this skill requires.
+
+## Contents
+
+- [Read this before encoding any of it](#read-this-before-encoding-any-of-it)
+- [Span names are formulas, and the operation list is closed](#span-names-are-formulas-and-the-operation-list-is-closed)
+- [The evaluation event, missing the field §7 requires](#the-evaluation-event-missing-the-field-7-requires)
+- [Content: three tiers, and a hook that runs when nothing else does](#content-three-tiers-and-a-hook-that-runs-when-nothing-else-does)
+- [Tokens are eleven numbers and money is none of them](#tokens-are-eleven-numbers-and-money-is-none-of-them)
+- ["OpenTelemetry-based" is not one vocabulary](#opentelemetry-based-is-not-one-vocabulary)
+- [Replay is three different guarantees](#replay-is-three-different-guarantees)
+
+---
+
+## Read this before encoding any of it
+
+**The conventions moved.** They no longer live in the main `semantic-conventions` repository
+— that page now says only that they have moved and is no longer maintained. They have their
+own repository, whose README lists **Schema URL: `TODO`**.
+
+**Nothing in them is stable.** Every document is marked `Status: Development`. On the
+inference span the only attributes marked `Stable` are the ones borrowed from core semconv —
+`error.type`, `server.address`, `server.port`, `exception.*`. **Every single `gen_ai.*`
+attribute is `Development`.**
+
+So: adopt it, because a moving standard beats a private vocabulary that will never be read
+by anyone else's tooling — and **pin the version you adopted and expect to migrate**. Treat
+any code that branches on a `gen_ai.*` attribute as code with an expiry date, and re-read
+the spec before quoting a field name from this file.
+
+## Span names are formulas, and the operation list is closed
+
+Span names are computed, not free text:
+
+| Span | Name | Kind |
+|---|---|---|
+| inference | `{gen_ai.operation.name} {gen_ai.request.model}` | `CLIENT` |
+| tool call | `execute_tool {gen_ai.tool.name}` | `INTERNAL` |
+| agent invocation | `invoke_agent {gen_ai.agent.name}` | `CLIENT` |
+| planning | `plan {gen_ai.agent.name}` | `INTERNAL` |
+| agent creation | `create_agent {gen_ai.agent.name}` | — |
+| MCP | `{mcp.method.name} {target}`, target being the tool or prompt name | — |
+
+`gen_ai.operation.name` is a **closed 17-value enum** — `chat`, `text_completion`,
+`generate_content`, `embeddings`, `retrieval`, `fetch_response`, `execute_tool`,
+`create_agent`, `invoke_agent`, `plan` and the rest. A value outside it is not an extension,
+it is a name a backend cannot group by.
+
+**Only two attributes are Required on an inference span.** Everything else that matters —
+the model that actually answered, token counts, finish reasons — is Recommended or
+Conditionally Required. A conformant instrumentation can therefore be almost empty, which is
+the reason to specify what *you* need rather than to trust conformance as a floor.
+
+One behaviour worth knowing before you count spans: **automatic retries collapse into one
+span.** A span is one logical operation, not one HTTP request, so a retry storm is invisible
+at this layer and has to be measured somewhere else.
+
+## The evaluation event, missing the field §7 requires
+
+The spec defines `gen_ai.evaluation.result`, parented to the span being evaluated — or
+carrying `gen_ai.response.id` when the span id is not available — with:
+
+- `gen_ai.evaluation.name` (**Required**)
+- `gen_ai.evaluation.score.value`
+- `gen_ai.evaluation.score.label` — low-cardinality: `pass`, `fail`, `relevant`, …
+- `gen_ai.evaluation.explanation`
+
+**There is no attribute for who or what produced the score.**
+
+That is precisely the field this skill requires: §7 says a score binds to a run with a
+`source` of `human` | `llm_judge` | `code_check`, *because a score with no source cannot be
+calibrated, audited, or trusted differently from its neighbours*. The standard omits it.
+
+OpenInference has it, under a different name: **`annotation.annotator_kind`** — `HUMAN`,
+`LLM`, `CODE`, or custom — with `evaluation.annotator_kind` beside it. The three values are
+our three values.
+
+**So carry it yourself.** Add the attribute under your own namespace and document that it is
+an extension; do not drop the requirement because the schema has no slot for it. A judge
+score and a human label that are indistinguishable on the wire will be averaged by somebody
+downstream, and that average is the thing §5's calibration exists to prevent.
+
+## Content: three tiers, and a hook that runs when nothing else does
+
+Prompt and completion content is governed by three named patterns, chosen by environment:
+
+1. **Record nothing** — the default.
+2. **Record on span attributes** — *"best suited for … pre-production environments"*.
+3. **Store externally, record a reference on the span** — *"recommended in production
+   environments where telemetry volume is a concern or sensitive data needs to be handled
+   securely. Using external storage enables separate access controls."*
+
+**The spec's own default is to capture nothing**, and it says instrumentations SHOULD NOT
+capture content by default. Verify that in your stack rather than assuming it: the most
+widely used `gen_ai.*` implementation ships the opposite. OpenLLMetry's
+`is_content_tracing_enabled()` reads
+
+```python
+return (os.getenv("TRACELOOP_TRACE_CONTENT") or "true").lower() == "true"
+```
+
+— an unset variable is **true**, so prompts are captured unless you turn them off. Measured
+2026-08-31 against `traceloop-sdk/traceloop/sdk/config/__init__.py`. An agent's trace carries
+prompts, prompts carry secrets, and the default here works against you.
+
+**The upload hook is the last scrubbing point, and it runs where nothing else does.** Two
+properties make it the mechanism rather than a convenience: it *"SHOULD operate independently
+of the opt-in flags"*, and instrumentations *"SHOULD invoke it regardless of the span sampling
+decision"*. It therefore fires on spans nobody will ever look at — which is right, because
+the content has already left the process by then, and wrong to reason about as *"we only keep
+sampled traces"*.
+
+**A masking hook cannot change the span name.** Span names are built from tool names and MCP
+targets, so a redaction design that scrubs attributes and leaves the name is a design that
+leaks through the one field it never inspected. Decide naming and redaction together.
+
+## Tokens are eleven numbers and money is none of them
+
+**There is no cost attribute anywhere in the GenAI semantic conventions.** The spec
+standardises tokens and never money, so cost is always a join against a price table living
+outside the trace — and that join is where the number goes wrong.
+
+Because usage is not one number. It is eleven: `gen_ai.usage.input_tokens`, `output_tokens`,
+`reasoning.output_tokens`, `cache_read.input_tokens`, `cache_write.input_tokens`, and
+per-modality `text.*` / `image.*` / `audio.*` splits including
+`image.cache_read.input_tokens`.
+
+**A cost computed from `input_tokens + output_tokens` alone is wrong in both directions.** It
+bills cache reads at full price — they are the cheap ones — and it misses reasoning tokens and
+cache writes entirely. `references/kv-cache.md` in `agent-orchestrator` is the other half of
+this: the cache read is the case worth getting right, because at scale it is most of the
+traffic.
+
+And `gen_ai.client.token.usage` carries a hard **MUST NOT report** when the counts are not
+obtainable. A zero is a claim; absence is the honest value. That is the same rule
+`agent-harness/references/audit.md` states for cost attribution — *missing attribution beats
+wrong attribution, and absence is a named state rather than a fabricated zero*.
+
+## "OpenTelemetry-based" is not one vocabulary
+
+Two widely used tools both describe themselves as OpenTelemetry-based and emit **disjoint**
+attribute sets:
+
+| | Emits |
+|---|---|
+| **Phoenix** | OpenInference — `openinference.span.kind` (required; `LLM`, `CHAIN`, `RETRIEVER`, `RERANKER`, `TOOL`, `AGENT`, `GUARDRAIL`, `EVALUATOR`, `PROMPT`, `EMBEDDING`), plus `llm.*`, `input.value`, `input.mime_type`, `document.*`, `annotation.*` |
+| **OpenLLMetry** | `gen_ai.*` plus `traceloop.*` |
+
+**Neither set is a subset of the other, and neither is a superset of the GenAI
+conventions.** A backend query written against one returns nothing against the other, and a
+trace assembled from two services using each is **two disconnected halves under one trace
+id** — which reads as a gap in the system rather than a gap in the vocabulary.
+
+So *"we use OpenTelemetry"* answers a transport question and no semantic one. Ask which
+attribute set, and pick one per system rather than per service.
+
+## Replay is three different guarantees
+
+The word appears in three products in this space and means three incompatible things. Our
+own `agent-harness/references/audit.md` asks *"can a past run be replayed"* without saying
+which:
+
+| Sense | What re-runs | Cost | Answers |
+|---|---|---|---|
+| **Durable execution** (Temporal) | nothing — recorded results are replayed and only the failed step retries | free, deterministic | can I resume without redoing 20 web searches |
+| **Trace playground** (Phoenix) | the model call, against the live provider, with an edited prompt | a real call | would a different prompt have done better |
+| **Fixture replay** (this skill, §2 single-step) | an assertion over a stored run | free | did the decision at this point change |
+
+They are not interchangeable, and a runbook that says *"replay the run"* has not said what it
+means. Name the sense.

package/plugins/agent-stack/skills/agent-orchestrator/references/governance.md

@@ -36,6 +36,45 @@ guardrails" so often means only the first.
 The last one is the one most designs miss: a sub-agent that inherits its caller's
 authority silently widens every permission the caller had.
 
+## The cheapest control is absence
+
+The **Tool call** row above puts per-tool authorisation at the moment of invocation. There
+is a control one layer earlier and it is strictly stronger:
+
+> **The model cannot reason about capabilities it does not know exist.**
+
+A tool absent from the schema cannot be invoked, cannot be argued for, and cannot be probed
+for a bypass — there is nothing to jailbreak toward. A tool present in the schema and
+refused at call time is a negotiation, and negotiations are won sometimes.
+
+So **filter the schema at build time**, and treat runtime authorisation as the second line
+rather than the first. Sub-agent isolation comes from exactly two mechanisms used together:
+schema filtering when the agent is constructed, and no inherited conversation
+(`message_history = None`) when it runs. The second matters as much as the first — a
+sub-agent handed its parent's transcript has been told about every capability you carefully
+removed from its schema.
+
+**Two numbers, and they point in opposite directions on purpose.** Eagerly loading every
+MCP tool schema at startup consumed **40% of the context budget before the first user
+message**; a metadata index at startup with the full schema fetched on selection takes it
+**under 5%**. But the same system builds *its own* prompt and tool schemas **eagerly**, in
+the constructor. The rule underneath is not *lazy is better*: it is **eager for what you
+own and always need, lazy for what is foreign and might not be used** — the first removes
+latency and race conditions from the hot path, the second removes a cost you cannot predict.
+
+### Approval fatigue is a safety failure, not a UX complaint
+
+> Without persistence, users must re-approve the same operations every session, causing
+> approval fatigue that leads to **blanket auto-approval, defeating the safety system
+> entirely**.
+
+An approval system with no memory converts itself into no approval system, and it does so
+through the user rather than through a bug — so nothing in the logs looks wrong. The
+remedy is on the same axis as the section above: **decide once what does not need asking,
+remove it from the question, and spend the prompts on what genuinely changes.** A control
+that is asked too often is a control on its way to being switched off.
+
+
 ## Guardrails, and their honest limit
 
 The content-layer checks worth having, roughly in order of reliability:

package/plugins/agent-stack/skills/agent-orchestrator/references/graph-engineering.md

@@ -28,6 +28,7 @@ well-measured answer to the wrong question.
 - [9. What Claude Code actually executes](#9-what-claude-code-actually-executes)
 - [10. Barrier or no barrier](#10-barrier-or-no-barrier)
 - [11. Project defaults, written once](#11-project-defaults-written-once)
+- [11a. MAST — the failures, named from outside](#11a-mast--the-failures-named-from-outside)
 - [12. The source's four diagrams, and what each one is for](#12-the-sources-four-diagrams-and-what-each-one-is-for)
 - [Where this file disagrees with its source](#where-this-file-disagrees-with-its-source)
 
@@ -375,6 +376,32 @@ The last line is `context-engineering.md`'s *filesystem as context* stated as a
 rule: an edge that carries a path costs a few tokens, and an edge that carries a
 transcript costs the window.
 
+## 11a. MAST — the failures, named from outside
+
+Everything above is argued from what breaks. *Why Do Multi-Agent LLM Systems Fail?* (2025)
+measured it: execution traces from seven mainstream frameworks — MetaGPT, ChatDev, AG2 and
+Magentic-One among them — with human annotators independently analysing roughly **150
+traces** at **Cohen's kappa = 0.88**, producing **14 failure modes in three groups**.
+
+The three groups land on this file's own node contract, which is the reason to carry them:
+
+| MAST group | Modes include | The field it lands on |
+|---|---|---|
+| **System design flaws** | unclear interfaces between agents, overlapping roles, wrong tool configuration | `input` / `output` (the interface), and `owner` (overlapping roles) |
+| **Inter-agent alignment failures** | inconsistent understanding of the objective, downstream misinterpretation, logically contradictory operations | the **edge payload** — what actually crosses, §3's *Carries* column |
+| **Missing task verification** | an agent reports *completed* and the result does not meet the requirement | `check`, and §6's checker node |
+
+**The load-bearing result is the negative one.** Straightforward fixes — better prompts,
+more explicit role descriptions, retries — bought ChatDev only **15.6%**. The authors
+conclude the modes are **architectural rather than bugs**, which is the same claim §1 makes
+about the two fields nobody draws: an owner and a completion test are not documentation of
+a graph, they are the parts that make it a graph.
+
+Read the mapping as a review checklist rather than a taxonomy to admire: for each node, is
+the interface stated, is there exactly one owner, and does the *check* run on something
+other than the node's own claim of success.
+
+
 ## 12. The source's four diagrams, and what each one is for
 
 The article carries four hand-drawn figures. They are not decoration — each one is doing