@ssheleg/agent-stack 0.16.1 → 0.17.1

package/CHANGELOG.md

@@ -1,5 +1,89 @@
 # Changelog
 
+## v0.17.1 — the evals run for the first time, and the tails go to zero
+
+Wave-3 of the 2026-08-29 family audit (rows AST-05, AST-07, AST-08, AST-09, AST-10),
+plus the two board rows wave 2 filed (AST-A1, AST-B). All fixes and docs — no
+description or trigger changes, hence a patch.
+
+- **AST-05: the eval suite is executed for the first time.** `test/evals/RESULTS.md`
+  gains two dated rows — haiku and sonnet, 2026-08-31 — each a fresh blind subagent
+  per trigger query against the family's 28 skill descriptions, plus all three
+  scenarios scored line by line. Both models: 11/12 triggers. Scenario lines: 10/12
+  (haiku), 9/12 (sonnet). Each miss is named — haiku false-triggered `agent-harness`
+  on "rewrite this one system prompt so it sounds friendlier" (q09), sonnet answered
+  `none` on the Russian harness-audit query (q04) — and the Method section states the
+  protocol and its three limits rather than presenting the rates as clean-room.
+- **AST-B: the generator-evaluator citation lands where the doctrine lives.**
+  `agent-evals` §5a now cites Anthropic's *tuning a standalone evaluator to be
+  skeptical is more tractable than making a generator self-critical* (read
+  2026-08-30) beside the sentence making the same claim, marked as convergence
+  rather than invention — beside the first executed eval run, as the board row asked.
+- **AST-A1: the nine-subsystem coverage check is done and the row closes.** The ninth
+  subsystem the filing row could not name is **Automation**; all nine map to existing
+  doctrine with a `file:line` each, and the suspected identity/approval-policy gap
+  resolves as a split verdict — approval policy covered (`governance.md`),
+  authentication mechanics a named delegation (`layers.md:78-80`), not a hole. The
+  map lives in the board row (`docs/evidence/backlog.md`), not in a new reference.
+- **AST-07: the README stops counting its references.** "Twenty references" had
+  drifted to 24 actual files within a week of being written; the aggregate count is
+  dropped in favour of the per-skill counts that are still true, with the reason
+  stated in place.
+- **AST-10: the orchestrator's reference index lists all 11 references.** The three
+  memory rows — `memory-architecture.md`, `memory-lifecycle.md`,
+  `memory-landscape.md` — join the index table, each paraphrasing its own
+  "Load this when" line.
+- **AST-08: `license: MIT` in all four skill front matters** (was 1 of 4 —
+  `agent-evals` alone carried it). Every front matter re-checked with
+  `yaml.safe_load` after the edit.
+- **AST-09: `$schema` in both manifests** — `claude-code-plugin-manifest.json` for
+  `plugin.json`, `claude-code-marketplace.json` for `marketplace.json`, the two
+  schemastore addresses that resolve (the third candidate, `claude-code-plugin.json`,
+  is a 404 and two siblings currently point at it; referred, not copied).
+
+## v0.17.0 — the descriptions stop colliding, and the claims get their dates
+
+Wave-2 of the 2026-08-29 family audit (rows AST-02, AST-03, AST-04, AST-06, AST-11), plus
+the harness-engineering externals read 2026-08-30.
+
+- **AST-02: `agent-orchestrator` gives "checker node" back to `agent-evals`.** The quoted
+  trigger appeared verbatim in both descriptions, so the router had two skills advertising
+  one phrase. `agent-evals` owns it; the orchestrator's description already carries "a
+  checker before convergence" in prose, which is the claim it actually makes.
+- **AST-11: the single-word triggers "agent" / "агент" narrow to "agent system" /
+  "агентная система".** A one-word trigger matching every sentence with the word *agent*
+  in it is how a skill teaches the router to route around the boundary its own description
+  draws ("Not for a single LLM call in a script"). The multiword triggers carry the load.
+- **`agent-harness` advertises "ReAct loop" and "react pattern" instead of bare "ReAct"** —
+  a phrase the umbrella's deferred hook trigger can carry without firing on every mention
+  of the React framework. The body prose still explains ReAct by its own name.
+- **AST-06: the harness↔orchestrator boundary names its four seams instead of claiming
+  one.** `agent-harness` §Boundaries said the two skills "meet in one place" while the
+  pack actually crosses at four: orchestrator §3 → harness `tools.md` (describing a tool
+  vs assembling the list), orchestrator §10 → harness `system-prompt.md` (what the prompt
+  says vs rebuilding it per request), harness static-or-dynamic → orchestrator
+  `graph-engineering.md`, and `context-engineering.md` (compaction) vs `system-prompt.md`
+  (filling). The rewrite states explicitly that shape-of-the-work doctrine has ONE home —
+  `agent-orchestrator/references/graph-engineering.md` — and the harness intro stops
+  claiming "the shape of the work" as its own ground, which was the sentence that made
+  both skills owners of one table.
+- **AST-03/AST-04: `agent-interop`'s undated external claims get their dates.** "Moved in
+  the last twelve months", "SDKs and blog posts still document v0.3" and "its own overview
+  page still introduces `binds`" sat undated inside the very section whose rule is *a
+  protocol claim without a date is a guess* — all three now anchor to **2026-08-13**, the
+  stamp every reference in the skill already carries. The neighbourhood table gains a
+  "Verdicts as of 2026-08-13" line, since `test/validate.py` gates stamps only under
+  `references/` and the table lives in SKILL.md.
+- **`agent-harness` cites the outside term for its ground: harness engineering.** OpenAI's
+  harness-engineering article and Anthropic's harness-design article (both read
+  2026-08-30) name the layer this skill covers, with the measured leverage — ARC-AGI-3
+  harness-level changes moved a fixed model 13.3%→38.3% at a sixth of the tokens (as
+  reported 2026-08-30). "harness engineering" joins the description triggers: users will
+  say it. The deeper work is filed, not done: board rows **AST-A1** (coverage check
+  against OpenAI's nine-subsystem taxonomy, identity/approval policy the suspected gap)
+  and **AST-B** (the dated generator-evaluator citation in `agent-evals`, deferred to
+  evals day).
+
 ## v0.16.1 — the installers refuse the shadow, and the pack stops mis-selling itself
 
 - **Both installers refuse to write plain copies over an installed plugin.** The family

package/README.md

@@ -33,8 +33,9 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills).
 
 Four skills — `agent-orchestrator` for wiring the loop, `agent-evals` for proving it
 behaves, `agent-interop` for everything it talks to outside its own process,
-`agent-harness` for what it is **told** — and twenty references they load on demand,
-plus one scanner.
+`agent-harness` for what it is **told** — and the references they load on demand,
+plus one scanner. The per-skill sections below carry their own counts; no total is
+stated here, because the last one drifted stale within a week of being written.
 
 **The orchestrator** (`SKILL.md`) — what the agent reads first:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.16.1",
+  "version": "0.17.1",
   "scripts": {
     "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js"
   },

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -1,8 +1,9 @@
 {
+  "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
-  "version": "0.16.1",
+  "version": "0.17.1",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/agent-stack/skills/agent-evals/SKILL.md

@@ -168,6 +168,13 @@ the one most systems are missing: a **checker node** sitting between a parallel
 the node that consumes it. Its only job is *usable / not usable*, and the convergence
 depends on **it** rather than on the branches — otherwise the gate has a bypass.
 
+**That split is measured practice, not only this pack's position.** Anthropic's
+harness-design guidance (`anthropic.com/engineering/harness-design-long-running-apps`,
+read 2026-08-30) reports that *tuning a standalone evaluator to be skeptical is more
+tractable than making a generator self-critical* — the same reason the verdict belongs to
+a separate node rather than to the branches grading themselves. Dated and cited so a
+reader can tell doctrine that converged with the field from doctrine invented here.
+
 It matters here because it is the same machinery as §5, positioned differently:
 
 <!-- checker-contract: missing, empty, unevidenced, malformed, contradictory, off-topic | optional: under-confident -->

package/plugins/agent-stack/skills/agent-harness/SKILL.md

@@ -8,17 +8,25 @@ description: >-
   instead of a score, plus a scanner. Carries Pi as a worked kernel implementation — SDK, RPC
   and extension seams — for embedding or extending a harness. Triggers - "system prompt",
   "tool description", "agent picks the wrong tool", "agent loops forever", "prompt
-  engineering", "ReAct", "workflow or agent", "static or dynamic", "audit this agent",
+  engineering", "harness engineering", "ReAct loop", "react pattern", "workflow or agent",
+  "static or dynamic", "audit this agent",
   "embed an agent", "agent SDK", "Pi harness", "системный промпт", "агент не вызывает тул",
   "аудит агента", "встроить агента". Not for the loop's plumbing, its evals, or its
   protocols — those are siblings.
+license: MIT
 ---
 
 # Agent harness — what the agent is told, and how to audit what someone else told theirs
 
 `agent-orchestrator` wires the loop. `agent-evals` proves it behaves. `agent-interop` gets
 it talking to other processes. **This skill is the layer between them and the model: the
-prompt, the tools, and the shape of the work.**
+prompt, the tools, and the choice of technique.** The outside term for this ground is
+**harness engineering** — OpenAI's article of that name (`openai.com/index/harness-engineering`,
+read 2026-08-30) and Anthropic's harness-design guidance
+(`anthropic.com/engineering/harness-design-long-running-apps`, read 2026-08-30) both name
+this same layer, and its leverage is measured: on ARC-AGI-3, harness-level changes alone
+moved a fixed model from 13.3% to 38.3% while spending a sixth of the tokens (as reported
+2026-08-30).
 
 It runs in both directions. Building one and auditing one are the same checklist read
 forwards and backwards, which is why they live together here.
@@ -146,10 +154,22 @@ is then unfalsifiable — including this audit. Say so first, and make it the fi
 
 **Against `agent-orchestrator`.** That skill owns the loop's *plumbing*: iteration guards,
 trimming, sub-agent dispatch, provider routing, memory layers, checkpoints. This one owns
-what the model is *told*. They meet in one place:
-`agent-orchestrator/references/context-engineering.md` covers **compaction** — what to drop
-when the window fills — while this skill's `system-prompt.md` covers what to put there in
-the first place. Filling and emptying, two files.
+what the model is *told*. They meet at four seams, each crossing in exactly one place:
+
+1. **Describing a tool** so the model picks the right one is this skill's
+   `references/tools.md`; assembling the tool *list* per request from capability flags is
+   the orchestrator's §3, which points here for the wording.
+2. **What the prompt says** — altitude, vocabulary, enumerated statuses — is this skill's
+   `references/system-prompt.md`; *rebuilding* that prompt per request, in the same pass
+   as the tools, is the orchestrator's §10.
+3. **The shape of the work** — the static/dynamic table, the fake-edge test, the checker
+   before a convergence — has ONE home, and it is not here:
+   `agent-orchestrator/references/graph-engineering.md`. This skill's static-or-dynamic
+   section stops at the decision and links there for the model.
+4. **The context window:** `agent-orchestrator/references/context-engineering.md` covers
+   **compaction** — what to drop when the window fills — while this skill's
+   `system-prompt.md` covers what to put there in the first place. Filling and emptying,
+   two files.
 
 **Against `agent-evals`.** That skill measures whether an agent behaves, from execution
 records. This one reviews how it was *built*, from its source and prompts. An audit that

package/plugins/agent-stack/skills/agent-interop/SKILL.md

@@ -12,6 +12,7 @@ description: >-
   "agentgateway", "tool federation", "MCP-сервер", "карточка агента", "интероперабельность
   агентов", "реестр MCP", "шлюз для агентов". Not for designing one server's tool set, nor for
   a skill's own construction — that is make-skill.
+license: MIT
 ---
 
 # Agent interop — the protocols an agent speaks outside its own process
@@ -24,19 +25,20 @@ registry, and the gateway between them.
 
 ## Rule zero — a protocol claim without a date is a guess
 
-Every one of these specifications moved in the last twelve months, and each moved in a way
-that silently breaks code written against the previous revision. Concretely, and each one
-measured against the live specification rather than recalled:
+Every one of these specifications moved in the twelve months before **2026-08-13** — the
+date every reference below was read — and each moved in a way that silently breaks code
+written against the previous revision. Concretely, and each one measured on that date
+against the live specification rather than recalled:
 
 - MCP replaced the `initialize` handshake with `server/discover` and made the protocol
   **stateless**; a client that opens with `initialize` is speaking a revision that is on its
   way out.
 - MCP **deprecated `sampling`, `roots` and `logging`** — three things an older model will
   reach for first, because for a year they were the interesting part of the client side.
-- A2A renamed its wire surface between v0.x and v1.0, and SDKs and blog posts still document
-  v0.3.
+- A2A renamed its wire surface between v0.x and v1.0, and SDKs and blog posts still
+  documented v0.3 as of 2026-08-13.
 - agentgateway **deprecated `binds`** in favour of `gateways` — while its own overview page
-  still introduces `binds` as a core concept.
+  still introduced `binds` as a core concept as of 2026-08-13.
 
 So: **every reference in this skill opens with a `**Spec pinned:**` line**, and
 `test/validate.py` fails the build without one. That is a mechanical check, not an
@@ -119,6 +121,8 @@ it); prompt wording; a single LLM call in a script.
 ## The neighbourhood — one verdict each
 
 Named so an agent stops guessing, with the verdict stated rather than implied.
+**Verdicts as of 2026-08-13**, the day this skill's references were pinned — this ground
+moves, so re-verify anything marked *watch* before building on it.
 
 | Thing | What it actually is | Verdict |
 |---|---|---|

package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md

@@ -7,11 +7,12 @@ description: >-
   architecture, retrieval and decay, context budgets, sub-agent coordination, error hierarchies; the
   work as a graph — parallel layers, fake edges, a checker before convergence; for resale:
   tiered wallets, one markup boundary, two-phase commit across database and provider API,
-  spend-delta polling, budget and loop guards, per-tenant keys. Triggers - "agent",
+  spend-delta polling, budget and loop guards, per-tenant keys. Triggers - "agent system",
   "orchestrator", "tool calling", "sub-agent", "LLM router", "fallback chain", "human in the
-  loop", "memory layer", "LLM billing", "token wallet", "checker node", "агент", "оркестратор",
+  loop", "memory layer", "LLM billing", "token wallet", "агентная система", "оркестратор",
   "суб-агент", "роутер моделей", "человек в цикле", "слой памяти", "биллинг LLM", "граф задач".
   Not for a single LLM call in a script, or prompt wording.
+license: MIT
 ---
 
 # Agent Orchestrator — Production Best Practices
@@ -245,9 +246,7 @@ answer built from nothing, in no error log.
 **Design a memory layer from
 [`references/memory-architecture.md`](references/memory-architecture.md)**, not from this
 table — it also carries the context-budget trap, layer 0 carryover and workspace scale.
-The write path is [`references/memory-lifecycle.md`](references/memory-lifecycle.md);
-what to build on and measure with is
-[`references/memory-landscape.md`](references/memory-landscape.md).
+The write path and the build-or-adopt question have their own rows in the index below.
 ## 8. Self-Learning Feedback Loops
 
 Three cycles feed the memory layers, and they differ by what supplies the signal: a failed
@@ -373,15 +372,18 @@ heading — the ones that were learned by getting them wrong:
 ## References
 
 Each file opens with its own **Load this when** line — the authoritative trigger lives
-there, so this table stays an index and the two cannot drift apart.
+there; this table is only an index.
 
 | File | Read it when |
 |---|---|
-| [`references/graph-engineering.md`](references/graph-engineering.md) | you are deciding the **shape of the work** — the fake-edge test, the diamond, the checker node, static versus dynamic, and what the host actually runs |
+| [`references/graph-engineering.md`](references/graph-engineering.md) | you are deciding the **shape of the work** — fake edges, the diamond, the checker node, static versus dynamic |
 | [`references/pipeline.md`](references/pipeline.md) | one loop is **not the shape** — the planned path, its checkpoints, resume, and the interrupt that asks a person |
 | [`references/patterns.md`](references/patterns.md) | you need the **data models and algorithms** under the body |
+| [`references/memory-architecture.md`](references/memory-architecture.md) | an agent is **being given memory**, or memory is misbehaving |
+| [`references/memory-lifecycle.md`](references/memory-lifecycle.md) | the memory **write path** — what enters, and what leaves |
+| [`references/memory-landscape.md`](references/memory-landscape.md) | **build or adopt** a memory layer, and what is settled practice |
 | [`references/context-engineering.md`](references/context-engineering.md) | the loop is **running out of window** |
 | [`references/runtime.md`](references/runtime.md) | the agent must **survive a crash, a pause, a second message or a schedule** |
 | [`references/governance.md`](references/governance.md) | the question is **permission, not cost** — what it may do, and how you prove it |
 | [`references/llm-proxy-billing.md`](references/llm-proxy-billing.md) | the product **resells LLM access** |
-| [`references/provider-lifecycle.md`](references/provider-lifecycle.md) | the question is the **workforce, not the loop** — where providers come from, produced-once/bound-many, knowledge packs, canary trust, workspace lifecycle, fleet budgets |
+| [`references/provider-lifecycle.md`](references/provider-lifecycle.md) | the question is the **workforce, not the loop** — provider lifecycle, canary trust, fleet budgets |