@ssheleg/agent-stack 0.7.2 → 0.9.0

package/CHANGELOG.md

@@ -1,5 +1,125 @@
 # Changelog
 
+## [0.9.0] — 2026-08-15
+
+### Added
+
+- **`agent-harness/references/pi.md` and `pi-sdk.md` — the harness doctrine as a worked
+  implementation.** Every other reference in that skill states a rule; **Pi** is small enough
+  to read and complete enough to have made each of those decisions in public. So each section
+  says what Pi does and then **which rule it is an instance of** — the second half is the
+  point, and where Pi disagrees with this pack, that is said rather than smoothed over.
+
+  Read from `pi.dev/docs/latest` on 2026-08-15: sixteen doc pages plus the package source.
+  All sixteen verified reachable (`200`); the only non-resolving URLs in either file are the
+  two deliberate placeholders inside custom-provider examples.
+
+- **`pi.md`** — the four ways to run it; **sessions as a JSONL tree** (8-hex `id`, `parentId`,
+  version 3, `BranchSummaryEntry`) matched to *time travel and forking* in
+  `agent-orchestrator/references/runtime.md`; **compaction with the real numbers**
+  (`contextTokens > contextWindow - reserveTokens`, defaults 16,384 and 20,000, tool results
+  truncated to 2,000 chars while summarizing) matched to the compaction ladder, with what the
+  ladder adds that Pi leaves to you; settings precedence that **merges rather than replaces**;
+  skills, prompt templates and packages; and the trust model.
+
+  Its sharpest section is **the deliberate absence of a sandbox**, quoted: *"prompt injection
+  from repository files … is expected local-agent risk and cannot be reliably prevented by
+  pi."* That is `layers.md`'s delegation thesis stated by the project itself — and for an
+  audit it changes the finding, because "no permission model" here is a delegation, not a
+  defect. Three containerization patterns are compared by **where credentials end up**, which
+  is the question that actually decides between them.
+
+- **`pi-sdk.md`** — `createAgentSession()`, `ModelRuntime`, `SessionManager`, `defineTool()`,
+  `DefaultResourceLoader`; the RPC protocol with its command groups, its full event lifecycle
+  and the **`\n`-only JSONL framing warning**; JSON mode's delta-only records and why;
+  the `ExtensionAPI` surface; and **the eight seams** — `tool_call` (can block),
+  `tool_result` (a middleware chain), `context`, `before_agent_start`, the three provider
+  hooks, and the compaction pair — each matched to the doctrine it lets you implement.
+  `tool_call` blocking is called out as the single most important one for an audit: it is
+  where a per-tool, per-caller policy can actually live.
+
+- Noted with its evidence and its limit: **Pi discovers skills from `~/.agents/skills/`**,
+  which on this machine is the ssheleg hub — 72 entries, every family skill carrying the
+  `name` and `description` front matter Pi requires. Stated as a fact about the path and the
+  front matter, **not** as an observed load: Pi is not installed here. The reference also
+  names Pi's documented divergence from the Agent Skills standard (a skill name may differ
+  from its directory) and warns that `make-skill`'s validator enforces the strict rule.
+
+### Changed
+
+- `references/layers.md` points at the two new files as the worked example of the kernel
+  layer it describes abstractly.
+- `agent-harness`'s description gains the embedding triggers (`agent SDK`, `embed an agent`,
+  `Pi harness`, `встроить агента`) — 807/1024, inside the family's 970 working budget. The
+  repository's own front-matter gate caught a first draft at 1066 and refused the write.
+
+
+## [0.8.0] — 2026-08-14
+
+### Added
+
+- **`agent-harness`, a fourth skill — the layer between the loop and the model.** The pack
+  could wire an agent (`agent-orchestrator`), prove it behaved (`agent-evals`) and connect
+  it to other processes (`agent-interop`), and said nothing about **what the agent is
+  told**. Five references and a scanner. It runs in both directions: building a harness and
+  auditing somebody else's are the same checklist read forwards and backwards, which is why
+  the audit lives here rather than in a sixth skill.
+
+  It is a fourth skill rather than a section because `agent-orchestrator`'s body is
+  **489 lines / ~4761 tokens** — already past the 4750 working limit — and could not absorb
+  a paragraph, let alone a layer.
+
+- **`references/system-prompt.md`** — the right altitude (hardcoded branches on one side,
+  vague hope on the other), what actually belongs in a system prompt in order of behaviour
+  bought, **enumerating the vocabulary** so an agent stops inventing `pending` and `to-do` in
+  the same run, injecting what the model cannot know, and flexible-while-learning versus
+  strict-in-production. Plus the three things reasoning models changed: **do not add
+  chain-of-thought** (it can degrade instruction-following), give goals rather than
+  procedures, and treat reasoning effort as a per-stage dial.
+
+- **`references/tools.md`** — the agent–computer interface. Fewer tools than instinct
+  suggests, namespacing, and the description as the product: a worked before/after where the
+  strong version names *when*, *what it costs*, *how to narrow*, and **the neighbouring tool
+  it is confused with** — the highest-value sentence in a tool definition and the one almost
+  nobody writes. Then meaning over identifiers, token efficiency as a correctness issue,
+  errors that teach, and **poka-yoke** — changing the interface so the wrong call cannot be
+  made.
+
+- **`references/techniques.md`** — fifteen techniques with a **verdict each for a production
+  loop**, not a benchmark score. ReAct is the agent loop and its under-quoted failure is that
+  non-informative results derail it; reflection is strong exactly where a cheap objective
+  signal exists and is a second opinion from the same source where it does not; Tree of
+  Thoughts is almost never worth its combinatorics. Ends with an ordered five-question
+  chooser.
+
+- **`references/layers.md`** — the question that resolves most framework arguments (*which
+  layer am I working at*), what a harness owns, and the design position that **permission
+  boundaries usually belong to the environment**: a harness that also claims to be a sandbox
+  is claiming a guarantee it cannot keep from inside the same process.
+
+- **`references/audit.md`** — seven tracks, three evidence tiers (**measured / documented /
+  judgement**, never inflated), computed priority, and a report shape that ends in a plan
+  rather than a score. The finding that ends most audits early is stated first: no evals
+  makes everything downstream unfalsifiable, including the audit.
+
+- **`scripts/audit_agent.py`** — the mechanical half. Five conservative detectors
+  (unbounded loop, empty tool description, swallowed error, missing timeout, duplicated
+  model literal), each requiring the file to show **two** independent signs of an agent, each
+  finding carrying `file:line`. It always prints **what it cannot see** and a **denominator**,
+  because `read: 1` alone looks like a broken pass while `1 of 4261` is itself a finding.
+  Virtualenvs are skipped by their `pyvenv.cfg` marker rather than by name — a real
+  repository met during testing kept 4249 of its 4261 files in `myenv/`, and was excluded
+  only because `site-packages` happened to be listed too.
+
+### Changed
+
+- **`PROTOCOL_PINNED` now covers `agent-harness`.** Its references document guidance that
+  moves, so each carries `**Spec pinned:** … · read <date>` and the build fails without it.
+- **CI runs the scanner's own self-test**, and asserts that a real-tree run discloses both
+  its blind-spot list and its denominator — a scanner that could stop disclosing would be a
+  scanner nobody could calibrate.
+
+
 ## v0.7.2 — the plants say whether they landed, and two of them were not
 
 Eight negative self-tests asserted inline, in Python, that their edit had happened —

package/README.md

@@ -9,9 +9,10 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills).
 
 ## What is in here
 
-Three skills — `agent-orchestrator` for building one, `agent-evals` for proving it
-behaves, `agent-interop` for everything it talks to outside its own process — and
-eleven references they load on demand.
+Four skills — `agent-orchestrator` for wiring the loop, `agent-evals` for proving it
+behaves, `agent-interop` for everything it talks to outside its own process,
+`agent-harness` for what it is **told** — and eighteen references they load on demand,
+plus one scanner.
 
 **The orchestrator** (`SKILL.md`) — what the agent reads first:
 
@@ -56,6 +57,26 @@ undoes both), `mcp-ship.md` (mounting, and the 404 that is really a double path)
 an API gateway does not). Plus a link map, and a verdict on each neighbouring standard —
 ACP, AGNTCY, AP2, Agent Skills — so an agent stops guessing.
 
+**The harness skill** (`agent-harness/SKILL.md`) — the layer between the loop and the
+model, and the one where most agent bugs actually live: *the biggest performance
+improvements often come from clearly explaining tool usage in the system prompt*, and *even
+small refinements to tool descriptions can yield dramatic improvements*. Before adding a
+retry or a sub-agent, it asks four questions about the text. Seven references —
+`system-prompt.md` (right altitude, enumerated vocabulary, and the three things reasoning
+models changed — starting with **do not add chain-of-thought**), `tools.md` (the
+agent–computer interface, with a worked before/after and poka-yoke), `techniques.md`
+(fifteen techniques, a verdict each **for production** rather than a benchmark),
+`layers.md` (which layer you are building at, and why permission boundaries are usually the
+environment's job), `audit.md` (seven tracks, evidence tiers, a plan instead of a score) —
+plus **`pi.md` and `pi-sdk.md`, the doctrine as a worked implementation**: Pi read end to
+end, each mechanism matched to the rule it instantiates, its divergences named, and the
+eight extension seams where a permission gate or a context rewrite can actually live.
+
+It runs in both directions: **building a harness and auditing somebody else's are one
+checklist read forwards and backwards.** `scripts/audit_agent.py` is the mechanical half —
+five conservative detectors, and it always prints what it *cannot* see plus a denominator,
+so its silence is never read as a pass.
+
 **`references/context-engineering.md`** — what the loop gives up when the window
 runs out: the five-rung compaction ladder and why to re-measure between rungs,
 the tool-pair boundary invariant, typed carryover blocks copied across the
@@ -139,6 +160,10 @@ audit row has to carry to prove a control was on.
 trajectory rather than a final answer, turning a production failure into a
 permanent fixture, calibrating a judge, gating a release on offline evals.
 
+`agent-harness`: writing or fixing a system prompt, shaping tools so the model picks the
+right one, choosing between ReAct, reflection, planning and voting — or auditing an agent
+system somebody else built. Not the loop's plumbing, its evals, or its protocols.
+
 `agent-interop`: building or consuming an MCP server, exposing or calling another
 agent over A2A, publishing to the MCP Registry, or putting a gateway in front of
 agent traffic. Not for designing one server's tool set — that is a design problem,
@@ -147,7 +172,7 @@ construction, which is `make-skill`. That boundary runs both ways: `make-skill`
 what changes *because you are writing a skill*, and the protocol itself is described
 here and nowhere else in the family.
 
-None of the three triggers for a single LLM call in a script or for prompt wording —
+None of the four triggers for a single LLM call in a script or for prompt wording —
 that is not an orchestrator, and pulling this much doctrine for it is how a skill
 teaches you to route around it.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.7.2",
+  "version": "0.9.0",
   "description": "Production patterns for AI agent orchestrators \u2014 tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay \u2014 plus the wallet side of reselling LLM access. This package is the installer CLI.",
   "bin": {
     "agent-stack": "bin/agent-stack.js"

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Two skills: agent-orchestrator \u2014 tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals \u2014 run/trace/thread evals, judges, and fixtures grown from production.",
-  "version": "0.7.2",
+  "version": "0.9.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/agent-stack/skills/agent-harness/SKILL.md

@@ -0,0 +1,163 @@
+---
+name: agent-harness
+description: >-
+  Use when the question is what the agent is TOLD rather than how its loop is wired — writing
+  or fixing a system prompt, shaping tools so the model picks the right one, deciding whether
+  a job wants a workflow or an agent, or choosing between ReAct, reflection and voting. Also
+  auditing an agent system somebody else built: tracks, evidence tiers and a prioritized plan
+  instead of a score, plus a scanner. Carries Pi as a worked kernel implementation — SDK, RPC
+  and extension seams — for embedding or extending a harness. Triggers - "system prompt",
+  "tool description", "agent picks the wrong tool", "agent loops forever", "prompt
+  engineering", "ReAct", "workflow or agent", "audit this agent", "embed an agent", "agent
+  SDK", "Pi harness", "системный промпт", "агент не вызывает тул", "аудит агента", "встроить
+  агента". Not for the loop's plumbing, its evals, or its protocols — those are siblings.
+---
+
+# Agent harness — what the agent is told, and how to audit what someone else told theirs
+
+`agent-orchestrator` wires the loop. `agent-evals` proves it behaves. `agent-interop` gets
+it talking to other processes. **This skill is the layer between them and the model: the
+prompt, the tools, and the shape of the work.**
+
+It runs in both directions. Building one and auditing one are the same checklist read
+forwards and backwards, which is why they live together here.
+
+---
+
+## Rule zero — most agent bugs are prompt bugs wearing a stack trace
+
+The instinct when an agent misbehaves is to change the code. The measured reality, in every
+source this skill was built from, is that the largest behavioural changes come from the
+text: **"the biggest performance improvements often come from clearly explaining tool usage
+in the system prompt"**, and **"even small refinements to tool descriptions can yield
+dramatic improvements."**
+
+Before adding a retry, a router, or a sub-agent, check in this order:
+
+1. **Does the tool description say when to use it, not just what it does?**
+2. **Does the system prompt name the vocabulary?** An agent told to track status will invent
+   `pending` and `to-do` and `done` and `completed` in the same run unless the allowed values
+   are enumerated.
+3. **Does the agent know today's date?** A model with a training cutoff will answer from
+   memory rather than search unless the current date is injected.
+4. **Is the instruction flexible where it should be strict?** *"Use the tools in the order
+   that makes most sense to you"* is right while you are learning the task and wrong in
+   production, where *"you MUST execute a web search for each task"* is what stops a step
+   from being skipped.
+
+Only then reach for architecture. Reaching for it first is how a prompt defect becomes a
+permanent structural cost.
+
+---
+
+## Workflow or agent — decide this before anything else
+
+An **agent** dynamically directs its own process. A **workflow** follows predefined code
+paths. The choice is not about sophistication; it is about whether the number of steps is
+knowable in advance.
+
+| Build a workflow when | Build an agent when |
+|---|---|
+| requirements are clear and stable | the task is open-ended or exploratory |
+| predictability and explicit control matter | flexibility outweighs predictability |
+| debugging and cost control are priorities | adaptive reasoning across variables is needed |
+| you can name every step now | step count is unpredictable and cannot be hardcoded |
+
+**Start at the simplest thing that works, and stop there.** An agent adds latency, cost and
+a class of failure a workflow does not have — it needs *trust in its own decisions*. Pay for
+that only where a fixed path genuinely cannot be written.
+
+### The five workflow patterns, before you reach for autonomy
+
+| Pattern | Shape | Reach for it when |
+|---|---|---|
+| **Prompt chaining** | sequential calls, each on the last output, with programmatic checks between | the task decomposes into fixed steps — outline then draft, draft then translate |
+| **Routing** | classify the input, send it to a specialist | categories are distinct and each wants its own prompt |
+| **Parallelization** | *sectioning* (independent subtasks at once) or *voting* (same task N times) | subtasks are independent, or confidence needs more than one sample |
+| **Orchestrator–workers** | a central model decomposes and delegates, then synthesizes | the subtasks **cannot be predefined** — this is the honest boundary with routing |
+| **Evaluator–optimizer** | one model produces, another critiques, loop | clear evaluation criteria exist and iteration measurably helps |
+
+**Orchestrator–workers versus routing is the distinction people get wrong.** Routing picks
+from a known set. Orchestration invents the set per request. If you can enumerate the
+branches, you wanted routing and it is cheaper.
+
+---
+
+## References
+
+Each opens with its own **Load this when** line and a revision stamp — this material moves,
+and `test/validate.py` fails the build on a reference that does not say when it was read.
+
+| File | Read it when |
+|---|---|
+| [`references/system-prompt.md`](references/system-prompt.md) | you are **writing or fixing the prompt** — altitude, structure, vocabulary, dynamic context, and what changes for reasoning models |
+| [`references/tools.md`](references/tools.md) | the model **picks the wrong tool, or none** — the agent–computer interface: how many, named how, described how, returning what |
+| [`references/techniques.md`](references/techniques.md) | you are choosing between **ReAct, reflection, voting, planning** and the rest — every entry carries a verdict for production, not a benchmark score |
+| [`references/layers.md`](references/layers.md) | deciding **what your harness owns** — kernel, workbench and product layers, and why permission boundaries are usually somebody else's job |
+| [`references/audit.md`](references/audit.md) | reviewing **an agent system you did not build** — seven tracks, evidence tiers, and a prioritized plan |
+| [`references/pi.md`](references/pi.md) | you want the doctrine above as a **worked implementation** — Pi's sessions, compaction, config, skills, trust and containerization, each matched to the rule it implements, and the places it deliberately disagrees |
+| [`references/pi-sdk.md`](references/pi-sdk.md) | you are **embedding or extending** a harness — the SDK, the RPC protocol, JSON mode, and the eight extension seams where a permission gate, a context rewrite or a cost hook can actually live |
+
+**`scripts/audit_agent.py`** — the mechanical half of the audit. It finds what is visible
+without understanding intent (an unbounded loop, a tool with no description, a swallowed
+tool error, a hardcoded model, a missing timeout) and **prints the list of things it cannot
+see**, so its silence is never read as a pass.
+
+---
+
+## Auditing an agent system — the short version
+
+The long version is `references/audit.md`. The shape:
+
+1. **Run the scanner first.** It is cheap, and its blind-spot list tells you what the rest of
+   the audit must cover by hand.
+2. **Walk the seven tracks** — prompt, tools, control flow, context, failure, permission,
+   evidence — and record a finding only with an observation attached.
+3. **Tier every recommendation** by what backs it: measured here, documented upstream, or
+   judgement.
+4. **Output a prioritized plan, not a score.** A number tells nobody what to change on
+   Monday. This is the same rule `agent-evals` applies to eval rubrics and
+   `seo-aeo-audit` to sites.
+
+**The finding that ends most audits early:** the system has no evals. Everything downstream
+is then unfalsifiable — including this audit. Say so first, and make it the first item.
+
+---
+
+## Boundaries
+
+**Against `agent-orchestrator`.** That skill owns the loop's *plumbing*: iteration guards,
+trimming, sub-agent dispatch, provider routing, memory layers, checkpoints. This one owns
+what the model is *told*. They meet in one place:
+`agent-orchestrator/references/context-engineering.md` covers **compaction** — what to drop
+when the window fills — while this skill's `system-prompt.md` covers what to put there in
+the first place. Filling and emptying, two files.
+
+**Against `agent-evals`.** That skill measures whether an agent behaves, from execution
+records. This one reviews how it was *built*, from its source and prompts. An audit that
+finds no evals hands over to it; an eval suite that keeps failing on the same axis hands
+back here.
+
+**Against `agent-interop`.** MCP, A2A, the registry, gateways — the wire between processes.
+Tool *descriptions* are here; tool *protocol* is there.
+
+**Not covered:** model choice and pricing (see the `claude-api` skill for Anthropic's), the
+wallet under resale (`agent-orchestrator/references/llm-proxy-billing.md`), and RAG
+retrieval quality, which is a search problem this skill only touches where it enters the
+prompt.
+
+---
+
+## Checklist — a harness worth shipping
+
+- [ ] Workflow-versus-agent decided deliberately, and the simpler option was actually tried
+- [ ] System prompt at the **right altitude** — heuristics, not hardcoded branches, not vague hope
+- [ ] Every status, category and enum the agent must produce is **enumerated in the prompt**
+- [ ] Today's date, and any other volatile context, injected rather than assumed
+- [ ] Tools: a few high-impact ones, namespaced, each described as if to a new colleague
+- [ ] Tool responses carry **meaning, not identifiers**, and are paginated or truncated by default
+- [ ] Tool errors **teach the next attempt** instead of restating a stack trace
+- [ ] One technique chosen per problem, with a reason — not ReAct because it was in a paper
+- [ ] Sub-agents return **distilled summaries**, not transcripts
+- [ ] The agent can be observed: which tool, which arguments, which observation, how many tokens
+- [ ] An eval exists before the prompt is tuned, or the tuning is folklore

package/plugins/agent-stack/skills/agent-harness/references/audit.md

@@ -0,0 +1,141 @@
+# Auditing an agent system
+
+**Load this when:** reviewing an agent system you did not build — a due-diligence pass, an
+inherited codebase, or a "why is this unreliable" investigation.
+
+**Spec pinned:** the tracks below are this pack's synthesis of Anthropic agent guidance, `promptingguide.ai` and the sibling skills · read 2026-08-14
+
+## Contents
+
+- What this audit produces
+- Run the scanner first
+- Seven tracks
+- Evidence tiers
+- Priority, computed
+- The report
+- Traps
+
+## What this audit produces
+
+**A prioritized change plan, with an observation behind every finding.** Not a score.
+
+A number compresses away the only useful information — *what to change on Monday* — and
+invites arguing about the number. This is the same rule `agent-evals` applies to eval
+rubrics and `seo-aeo-audit` applies to sites, and it is the family's position rather than a
+preference: **pass/fail with a named failure condition beats a scalar that names no fix.**
+
+## Run the scanner first
+
+```bash
+python3 scripts/audit_agent.py <path>          # human-readable
+python3 scripts/audit_agent.py <path> --json   # machine-readable
+```
+
+It finds only what is mechanically visible, and **it prints the list of what it cannot see.**
+Read that list: it is the agenda for the manual half. A scanner that goes quiet is reporting
+its own blindness, and an audit that stops there has audited the scanner.
+
+## Seven tracks
+
+Walk them in order. Later tracks assume earlier ones.
+
+### 1 — Prompt
+
+- Is there a system prompt under version control, or is it a string literal edited in place?
+- **Altitude**: hardcoded branches (brittle) or vague hope (useless)? See `system-prompt.md`.
+- Are the values the agent must emit **enumerated**, or invented per run?
+- Is volatile context — the date above all — **injected**?
+- Is tool *policy* in the prompt, or only tool *schema*?
+- Has it been pruned, or only appended to since the first incident?
+
+### 2 — Tools
+
+- How many? Can the team say, without hesitating, which applies to a borderline case?
+- Are descriptions written **for a model choosing under uncertainty**, or for a human reading docs?
+- Do responses return **meaning or identifiers**?
+- Is there a default limit on response size, or only an optional one?
+- Do errors **name the next action**?
+- Are destructive tools guarded by shape (`confirm: true`, absolute paths, enums) rather than by instruction?
+
+### 3 — Control flow
+
+- **Workflow or agent — and was that decided, or defaulted?** An agent where a chain would do is the most expensive finding on this list.
+- Is there a **bounded iteration guard**, and what happens at the bound — a partial answer, or nothing?
+- Are retries and fallbacks **multiplied**? Three providers × three retries is nine calls for one prompt.
+- Is there loop detection, or does a repeated near-identical tool call run until the budget does?
+- Do sub-agents return **distilled summaries** or transcripts?
+
+### 4 — Context
+
+- Does anything measure window usage **before** a request fails?
+- Is there a compaction strategy, and does it preserve **decisions and open questions** rather than the discussion?
+- Can a large tool result be **offloaded** and referenced, or does it land in the window whole?
+- Is memory a design, or the conversation history by default?
+
+### 5 — Failure
+
+- What happens when a tool errors — is it distinguished from a tool returning nothing?
+- Under MCP, is `isError: true` handled, or does a 200 read as success?
+- Is there a timeout on every external call?
+- Is degradation **honest** — does the user learn the answer is partial?
+- Is there a path where the agent silently does nothing and reports success?
+
+### 6 — Permission
+
+- **Which layer owns the boundary?** If the harness delegates it (`layers.md`), audit the surroundings instead of filing a finding.
+- Is tool access differentiated per caller, or is one credential shared by every path?
+- Is tool output treated as **untrusted input**?
+- Can an audit row prove a control was applied — does it carry the **policy version**?
+- Is there a deterministic limit anywhere consequential, or only probabilistic content checks?
+
+### 7 — Evidence
+
+- **Are there evals?** If not, this is finding number one and everything else is unfalsifiable.
+- Do they judge the **trajectory**, or only the final answer?
+- Has any production failure become a permanent fixture?
+- Is a judge calibrated against human labels, or trusted because it is a judge?
+- Can a past run be replayed — is the execution record durable?
+
+## Evidence tiers
+
+Every finding carries one, and the tier is part of the finding:
+
+| Tier | Means | Example |
+|---|---|---|
+| **Measured** | observed here, in this system, with the observation attached | "`agent.py:212` — the `while` has no bound; a repeated call ran 47 times in the log at `logs/2026-08-02`" |
+| **Documented** | the upstream source says so, and this system contradicts it | "descriptions restate the name; Anthropic's tool guidance calls this the highest-leverage fix" |
+| **Judgement** | experience, no measurement available here | "two tools look interchangeable to us" |
+
+**Never present judgement as measured.** A finding whose tier is honest survives the meeting
+where it is challenged; one that is inflated loses the whole report.
+
+## Priority, computed
+
+`P = blast × confidence / effort`
+
+- **blast** 3 = a user of the system is harmed · 2 = the operator · 1 = a future maintainer
+- **confidence** 3 = measured · 2 = documented · 1 = judgement
+- **effort** 1 = under an hour · 2 = a session · 3 = its own project
+
+Computed, not felt — so a dramatic finding nobody can act on ranks below a boring one that
+is fixed before lunch, and the ranking can be argued with on its inputs.
+
+## The report
+
+1. **One paragraph** — what the system is, which layer, and the single most important thing.
+2. **The scanner output**, including its blind-spot list, verbatim.
+3. **Findings by track**, each with observation, tier and priority.
+4. **The plan** — ordered by P, with the first three items sized so they can start immediately.
+5. **What was not looked at**, and why. An audit that does not say where it stopped is read
+   as complete.
+
+## Traps
+
+- **Auditing the code and not the prompt.** The prompt is the largest behavioural surface and
+  is often not in the repository at all — ask where it lives before concluding it is fine.
+- **Filing "no permission model" against a harness that delegates by design.** Check the
+  layer first (`layers.md`).
+- **Grading instead of planning.** A score ends the conversation the audit was meant to start.
+- **Confusing "no evals" with "not measured yet."** It is the root finding; put it first,
+  because every other conclusion inherits it.
+- **Reading a silent scanner as a clean system.** It is silent about what it can see.

package/plugins/agent-stack/skills/agent-harness/references/layers.md

@@ -0,0 +1,104 @@
+# Layers — what your harness owns, and what it must not
+
+**Load this when:** deciding what to build versus adopt, or comparing agent frameworks and
+finding the comparison keeps sliding.
+
+**Spec pinned:** the three-layer reading of Pi / Goose / OpenCode (gist `AIMOWAY/bd8007c8`); Anthropic agent guidance · read 2026-08-14
+
+## Contents
+
+- The question that resolves most framework arguments
+- Three layers
+- What a harness owns
+- What a harness should delegate
+- Choosing your layer
+- Traps
+
+## The question that resolves most framework arguments
+
+"Which of these agent frameworks is better" is usually unanswerable because the candidates
+sit at different heights. The question that resolves it:
+
+> **Which layer of the agent stack am I trying to work at?**
+
+Compare within a layer. Across layers, the comparison is a category error, and the argument
+will not converge no matter how long it runs.
+
+## Three layers
+
+| Layer | What it is | Shape | Best when |
+|---|---|---|---|
+| **Kernel / harness** | the agent loop itself — runtime, LLM API abstraction, tool dispatch, a terminal UI | a toolkit you build *on* | you are building an agent product, or studying how agents actually work |
+| **Workbench / orchestration** | a local environment: desktop app, CLI and API, extensions, workflows across many kinds of work | a product you *extend* | you want capability now, across coding, research and automation, without owning the loop |
+| **Product agent** | a domain agent — most visibly coding: explore, plan, edit, test, with built-in modes | a product you *use* | the domain is the one it was built for |
+
+The distinction is not quality. A kernel is *supposed* to be smaller than a workbench; that
+is what makes it legible.
+
+## What a harness owns
+
+If you are building at the kernel layer, these are yours and nobody else's:
+
+1. **The loop** — iterate, dispatch tools, decide when to stop. With a bounded iteration
+   guard, because an unbounded loop is the defect that costs money while looking like work.
+2. **The model boundary** — one abstraction over providers, so the loop does not know which
+   vendor answered. See `agent-orchestrator` §6.
+3. **Tool dispatch and the ACI** — registration, schemas, parallel versus sequential
+   execution, error surfacing. See `tools.md`.
+4. **Context accounting** — knowing how full the window is *before* the request fails, and
+   what to do about it. See `agent-orchestrator/references/context-engineering.md`.
+5. **Observability** — which tool, which arguments, which observation, how many tokens. If
+   this is missing, every other item becomes unfalsifiable.
+6. **The interrupt/resume contract** — one mechanism, not two. See
+   `agent-orchestrator/references/runtime.md`.
+
+## What a harness should delegate
+
+The interesting stance, and it is a design position rather than an omission:
+
+**Permission boundaries usually belong to the environment, not the harness.** Pi states this
+explicitly — it provides the kernel and delegates sandboxing and permissions to whatever
+surrounds it. Read as a weakness, it looks like a missing feature. Read as architecture, it
+is a clear statement: *a harness that also claims to be a sandbox is claiming a guarantee it
+cannot keep*, because it runs in the same process as the code it would be confining.
+
+**Worked out in full in `pi.md` and `pi-sdk.md`** — the kernel layer read end to end, with
+each mechanism matched to the rule it implements and its divergences named. Use it as the
+concrete example when this file's abstractions need a body.
+
+Two consequences worth stating plainly:
+
+- **A harness advertising "safe tool execution" without an OS-level or container boundary is
+  advertising a preference, not a control.** MCP says the same about roots: servers *SHOULD*
+  respect them, and real enforcement is OS permissions and sandboxing.
+- **Deciding the layer decides the audit.** If the harness delegates permission, the audit
+  looks at what surrounds it, and "the harness has no permission model" stops being a
+  finding and becomes a question about the deployment.
+
+Also usually delegated: identity and secrets (a credential store, never the loop), durable
+state (a database or a file system, not the conversation), and policy (see
+`agent-orchestrator/references/governance.md` — permission, not protocol).
+
+## Choosing your layer
+
+- **Building a product with agentic features** → adopt a workbench or embed a kernel; do not
+  write a third loop. Loops are commodity; your tools and prompt are not.
+- **Building an agent platform others build on** → the kernel layer is yours, and the ACI is
+  your product surface.
+- **Automating your own work** → a product agent, and stop. The most common expensive mistake
+  is building a kernel to solve a workbench problem.
+
+**The test that catches the mistake early:** if you cannot name a behaviour you need that the
+layer above does not provide, you are building the layer for its own sake.
+
+## Traps
+
+- **Comparing across layers**, then choosing on a benchmark that only makes sense within one.
+- **Writing a loop because the loop is the interesting part.** It is a week; the tools and
+  the prompt are the year.
+- **Claiming a security boundary the layer cannot enforce.** Say what is enforced and by
+  what; a sandbox protects the host, not the sandbox.
+- **Adopting a workbench and then fighting its opinions.** Its opinions are the product; if
+  you disagree with enough of them, you wanted the kernel.
+- **Assuming the layer is stable.** These projects move; check what the current version
+  actually owns before designing around a division of labour you read about once.