@ssheleg/agent-stack 0.7.1 → 0.8.0

package/CHANGELOG.md

@@ -1,5 +1,100 @@
 # Changelog
 
+## [0.8.0] — 2026-08-14
+
+### Added
+
+- **`agent-harness`, a fourth skill — the layer between the loop and the model.** The pack
+  could wire an agent (`agent-orchestrator`), prove it behaved (`agent-evals`) and connect
+  it to other processes (`agent-interop`), and said nothing about **what the agent is
+  told**. Five references and a scanner. It runs in both directions: building a harness and
+  auditing somebody else's are the same checklist read forwards and backwards, which is why
+  the audit lives here rather than in a sixth skill.
+
+  It is a fourth skill rather than a section because `agent-orchestrator`'s body is
+  **489 lines / ~4761 tokens** — already past the 4750 working limit — and could not absorb
+  a paragraph, let alone a layer.
+
+- **`references/system-prompt.md`** — the right altitude (hardcoded branches on one side,
+  vague hope on the other), what actually belongs in a system prompt in order of behaviour
+  bought, **enumerating the vocabulary** so an agent stops inventing `pending` and `to-do` in
+  the same run, injecting what the model cannot know, and flexible-while-learning versus
+  strict-in-production. Plus the three things reasoning models changed: **do not add
+  chain-of-thought** (it can degrade instruction-following), give goals rather than
+  procedures, and treat reasoning effort as a per-stage dial.
+
+- **`references/tools.md`** — the agent–computer interface. Fewer tools than instinct
+  suggests, namespacing, and the description as the product: a worked before/after where the
+  strong version names *when*, *what it costs*, *how to narrow*, and **the neighbouring tool
+  it is confused with** — the highest-value sentence in a tool definition and the one almost
+  nobody writes. Then meaning over identifiers, token efficiency as a correctness issue,
+  errors that teach, and **poka-yoke** — changing the interface so the wrong call cannot be
+  made.
+
+- **`references/techniques.md`** — fifteen techniques with a **verdict each for a production
+  loop**, not a benchmark score. ReAct is the agent loop and its under-quoted failure is that
+  non-informative results derail it; reflection is strong exactly where a cheap objective
+  signal exists and is a second opinion from the same source where it does not; Tree of
+  Thoughts is almost never worth its combinatorics. Ends with an ordered five-question
+  chooser.
+
+- **`references/layers.md`** — the question that resolves most framework arguments (*which
+  layer am I working at*), what a harness owns, and the design position that **permission
+  boundaries usually belong to the environment**: a harness that also claims to be a sandbox
+  is claiming a guarantee it cannot keep from inside the same process.
+
+- **`references/audit.md`** — seven tracks, three evidence tiers (**measured / documented /
+  judgement**, never inflated), computed priority, and a report shape that ends in a plan
+  rather than a score. The finding that ends most audits early is stated first: no evals
+  makes everything downstream unfalsifiable, including the audit.
+
+- **`scripts/audit_agent.py`** — the mechanical half. Five conservative detectors
+  (unbounded loop, empty tool description, swallowed error, missing timeout, duplicated
+  model literal), each requiring the file to show **two** independent signs of an agent, each
+  finding carrying `file:line`. It always prints **what it cannot see** and a **denominator**,
+  because `read: 1` alone looks like a broken pass while `1 of 4261` is itself a finding.
+  Virtualenvs are skipped by their `pyvenv.cfg` marker rather than by name — a real
+  repository met during testing kept 4249 of its 4261 files in `myenv/`, and was excluded
+  only because `site-packages` happened to be listed too.
+
+### Changed
+
+- **`PROTOCOL_PINNED` now covers `agent-harness`.** Its references document guidance that
+  moves, so each carries `**Spec pinned:** … · read <date>` and the build fails without it.
+- **CI runs the scanner's own self-test**, and asserts that a real-tree run discloses both
+  its blind-spot list and its denominator — a scanner that could stop disclosing would be a
+  scanner nobody could calibrate.
+
+
+## v0.7.2 — the plants say whether they landed, and two of them were not
+
+Eight negative self-tests asserted inline, in Python, that their edit had happened —
+which works, and is the **fifth** careful copy of an idea this family already scripted.
+`plant_guard.py`'s own docstring was written against exactly that shape: five hand-written
+variants produced five different bugs, one of which reached a pull request.
+
+Adopting the shared implementation found two plants that were doing nothing at all.
+
+### Fixed
+
+- **`cp -R . /tmp/x` into an existing `/tmp/x` nests the tree instead of replacing it.**
+  The plant then edits a file left by the previous run, and `touch` on a file that already
+  exists changes neither content nor mode — so *a reference nobody links* and *stray
+  SKILL.md* both planted nothing, the validator honestly passed, and the step would have
+  reported a healthy guard as broken. CI is always fresh, so this was invisible there and
+  only ever bit the machine the plants were written on. Every copy is now `rm -rf`'d first.
+
+### Added
+
+- **`test/plant_guard.py` and its nine fixtures**, shared with `make-skill`, the umbrella
+  and `seo-aeo-audit`. It compares content **and permission bits**, because the variant
+  that shipped compared bytes against a plant whose whole effect was `chmod`.
+- **Every plant wrapped in `snap` / `verify`**, the description being the step's own name,
+  so a refusal names which plant died.
+- **A negative for the guard itself**: an unchanged copy must be reported, not passed.
+
+All eight plants were then watched landing on the machine they were written on.
+
 All notable changes to this project are documented here.
 Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning: [SemVer](https://semver.org/spec/v2.0.0.html).

package/README.md

@@ -9,9 +9,10 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills).
 
 ## What is in here
 
-Three skills — `agent-orchestrator` for building one, `agent-evals` for proving it
-behaves, `agent-interop` for everything it talks to outside its own process — and
-eleven references they load on demand.
+Four skills — `agent-orchestrator` for wiring the loop, `agent-evals` for proving it
+behaves, `agent-interop` for everything it talks to outside its own process,
+`agent-harness` for what it is **told** — and sixteen references they load on demand,
+plus one scanner.
 
 **The orchestrator** (`SKILL.md`) — what the agent reads first:
 
@@ -56,6 +57,23 @@ undoes both), `mcp-ship.md` (mounting, and the 404 that is really a double path)
 an API gateway does not). Plus a link map, and a verdict on each neighbouring standard —
 ACP, AGNTCY, AP2, Agent Skills — so an agent stops guessing.
 
+**The harness skill** (`agent-harness/SKILL.md`) — the layer between the loop and the
+model, and the one where most agent bugs actually live: *the biggest performance
+improvements often come from clearly explaining tool usage in the system prompt*, and *even
+small refinements to tool descriptions can yield dramatic improvements*. Before adding a
+retry or a sub-agent, it asks four questions about the text. Five references —
+`system-prompt.md` (right altitude, enumerated vocabulary, and the three things reasoning
+models changed — starting with **do not add chain-of-thought**), `tools.md` (the
+agent–computer interface, with a worked before/after and poka-yoke), `techniques.md`
+(fifteen techniques, a verdict each **for production** rather than a benchmark),
+`layers.md` (which layer you are building at, and why permission boundaries are usually the
+environment's job), `audit.md` (seven tracks, evidence tiers, a plan instead of a score).
+
+It runs in both directions: **building a harness and auditing somebody else's are one
+checklist read forwards and backwards.** `scripts/audit_agent.py` is the mechanical half —
+five conservative detectors, and it always prints what it *cannot* see plus a denominator,
+so its silence is never read as a pass.
+
 **`references/context-engineering.md`** — what the loop gives up when the window
 runs out: the five-rung compaction ladder and why to re-measure between rungs,
 the tool-pair boundary invariant, typed carryover blocks copied across the
@@ -139,6 +157,10 @@ audit row has to carry to prove a control was on.
 trajectory rather than a final answer, turning a production failure into a
 permanent fixture, calibrating a judge, gating a release on offline evals.
 
+`agent-harness`: writing or fixing a system prompt, shaping tools so the model picks the
+right one, choosing between ReAct, reflection, planning and voting — or auditing an agent
+system somebody else built. Not the loop's plumbing, its evals, or its protocols.
+
 `agent-interop`: building or consuming an MCP server, exposing or calling another
 agent over A2A, publishing to the MCP Registry, or putting a gateway in front of
 agent traffic. Not for designing one server's tool set — that is a design problem,
@@ -147,7 +169,7 @@ construction, which is `make-skill`. That boundary runs both ways: `make-skill`
 what changes *because you are writing a skill*, and the protocol itself is described
 here and nowhere else in the family.
 
-None of the three triggers for a single LLM call in a script or for prompt wording —
+None of the four triggers for a single LLM call in a script or for prompt wording —
 that is not an orchestrator, and pulling this much doctrine for it is how a skill
 teaches you to route around it.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.7.1",
+  "version": "0.8.0",
   "description": "Production patterns for AI agent orchestrators \u2014 tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay \u2014 plus the wallet side of reselling LLM access. This package is the installer CLI.",
   "bin": {
     "agent-stack": "bin/agent-stack.js"

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Two skills: agent-orchestrator \u2014 tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals \u2014 run/trace/thread evals, judges, and fixtures grown from production.",
-  "version": "0.7.1",
+  "version": "0.8.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/agent-stack/skills/agent-harness/SKILL.md

@@ -0,0 +1,161 @@
+---
+name: agent-harness
+description: >-
+  Use when the question is what the agent is TOLD rather than how its loop is wired — writing
+  or fixing a system prompt, shaping tools so the model actually picks the right one,
+  deciding whether a job wants a workflow or an agent, or choosing between ReAct, reflection,
+  planning and voting. Also the audit direction: reviewing an agent system somebody else
+  built, with tracks, evidence tiers and a prioritized plan instead of a score, plus a
+  scanner for the defects that are mechanically visible. Triggers - "system prompt", "tool
+  description", "the agent picks the wrong tool", "agent loops forever", "prompt engineering",
+  "ReAct", "reflection", "workflow or agent", "audit this agent", "review our agent system",
+  "системный промпт", "агент не вызывает тул", "аудит агента", "воркфлоу или агент". Not for
+  the loop's plumbing, its evals, or the protocols it speaks — those are the sibling skills.
+---
+
+# Agent harness — what the agent is told, and how to audit what someone else told theirs
+
+`agent-orchestrator` wires the loop. `agent-evals` proves it behaves. `agent-interop` gets
+it talking to other processes. **This skill is the layer between them and the model: the
+prompt, the tools, and the shape of the work.**
+
+It runs in both directions. Building one and auditing one are the same checklist read
+forwards and backwards, which is why they live together here.
+
+---
+
+## Rule zero — most agent bugs are prompt bugs wearing a stack trace
+
+The instinct when an agent misbehaves is to change the code. The measured reality, in every
+source this skill was built from, is that the largest behavioural changes come from the
+text: **"the biggest performance improvements often come from clearly explaining tool usage
+in the system prompt"**, and **"even small refinements to tool descriptions can yield
+dramatic improvements."**
+
+Before adding a retry, a router, or a sub-agent, check in this order:
+
+1. **Does the tool description say when to use it, not just what it does?**
+2. **Does the system prompt name the vocabulary?** An agent told to track status will invent
+   `pending` and `to-do` and `done` and `completed` in the same run unless the allowed values
+   are enumerated.
+3. **Does the agent know today's date?** A model with a training cutoff will answer from
+   memory rather than search unless the current date is injected.
+4. **Is the instruction flexible where it should be strict?** *"Use the tools in the order
+   that makes most sense to you"* is right while you are learning the task and wrong in
+   production, where *"you MUST execute a web search for each task"* is what stops a step
+   from being skipped.
+
+Only then reach for architecture. Reaching for it first is how a prompt defect becomes a
+permanent structural cost.
+
+---
+
+## Workflow or agent — decide this before anything else
+
+An **agent** dynamically directs its own process. A **workflow** follows predefined code
+paths. The choice is not about sophistication; it is about whether the number of steps is
+knowable in advance.
+
+| Build a workflow when | Build an agent when |
+|---|---|
+| requirements are clear and stable | the task is open-ended or exploratory |
+| predictability and explicit control matter | flexibility outweighs predictability |
+| debugging and cost control are priorities | adaptive reasoning across variables is needed |
+| you can name every step now | step count is unpredictable and cannot be hardcoded |
+
+**Start at the simplest thing that works, and stop there.** An agent adds latency, cost and
+a class of failure a workflow does not have — it needs *trust in its own decisions*. Pay for
+that only where a fixed path genuinely cannot be written.
+
+### The five workflow patterns, before you reach for autonomy
+
+| Pattern | Shape | Reach for it when |
+|---|---|---|
+| **Prompt chaining** | sequential calls, each on the last output, with programmatic checks between | the task decomposes into fixed steps — outline then draft, draft then translate |
+| **Routing** | classify the input, send it to a specialist | categories are distinct and each wants its own prompt |
+| **Parallelization** | *sectioning* (independent subtasks at once) or *voting* (same task N times) | subtasks are independent, or confidence needs more than one sample |
+| **Orchestrator–workers** | a central model decomposes and delegates, then synthesizes | the subtasks **cannot be predefined** — this is the honest boundary with routing |
+| **Evaluator–optimizer** | one model produces, another critiques, loop | clear evaluation criteria exist and iteration measurably helps |
+
+**Orchestrator–workers versus routing is the distinction people get wrong.** Routing picks
+from a known set. Orchestration invents the set per request. If you can enumerate the
+branches, you wanted routing and it is cheaper.
+
+---
+
+## References
+
+Each opens with its own **Load this when** line and a revision stamp — this material moves,
+and `test/validate.py` fails the build on a reference that does not say when it was read.
+
+| File | Read it when |
+|---|---|
+| [`references/system-prompt.md`](references/system-prompt.md) | you are **writing or fixing the prompt** — altitude, structure, vocabulary, dynamic context, and what changes for reasoning models |
+| [`references/tools.md`](references/tools.md) | the model **picks the wrong tool, or none** — the agent–computer interface: how many, named how, described how, returning what |
+| [`references/techniques.md`](references/techniques.md) | you are choosing between **ReAct, reflection, voting, planning** and the rest — every entry carries a verdict for production, not a benchmark score |
+| [`references/layers.md`](references/layers.md) | deciding **what your harness owns** — kernel, workbench and product layers, and why permission boundaries are usually somebody else's job |
+| [`references/audit.md`](references/audit.md) | reviewing **an agent system you did not build** — seven tracks, evidence tiers, and a prioritized plan |
+
+**`scripts/audit_agent.py`** — the mechanical half of the audit. It finds what is visible
+without understanding intent (an unbounded loop, a tool with no description, a swallowed
+tool error, a hardcoded model, a missing timeout) and **prints the list of things it cannot
+see**, so its silence is never read as a pass.
+
+---
+
+## Auditing an agent system — the short version
+
+The long version is `references/audit.md`. The shape:
+
+1. **Run the scanner first.** It is cheap, and its blind-spot list tells you what the rest of
+   the audit must cover by hand.
+2. **Walk the seven tracks** — prompt, tools, control flow, context, failure, permission,
+   evidence — and record a finding only with an observation attached.
+3. **Tier every recommendation** by what backs it: measured here, documented upstream, or
+   judgement.
+4. **Output a prioritized plan, not a score.** A number tells nobody what to change on
+   Monday. This is the same rule `agent-evals` applies to eval rubrics and
+   `seo-aeo-audit` to sites.
+
+**The finding that ends most audits early:** the system has no evals. Everything downstream
+is then unfalsifiable — including this audit. Say so first, and make it the first item.
+
+---
+
+## Boundaries
+
+**Against `agent-orchestrator`.** That skill owns the loop's *plumbing*: iteration guards,
+trimming, sub-agent dispatch, provider routing, memory layers, checkpoints. This one owns
+what the model is *told*. They meet in one place:
+`agent-orchestrator/references/context-engineering.md` covers **compaction** — what to drop
+when the window fills — while this skill's `system-prompt.md` covers what to put there in
+the first place. Filling and emptying, two files.
+
+**Against `agent-evals`.** That skill measures whether an agent behaves, from execution
+records. This one reviews how it was *built*, from its source and prompts. An audit that
+finds no evals hands over to it; an eval suite that keeps failing on the same axis hands
+back here.
+
+**Against `agent-interop`.** MCP, A2A, the registry, gateways — the wire between processes.
+Tool *descriptions* are here; tool *protocol* is there.
+
+**Not covered:** model choice and pricing (see the `claude-api` skill for Anthropic's), the
+wallet under resale (`agent-orchestrator/references/llm-proxy-billing.md`), and RAG
+retrieval quality, which is a search problem this skill only touches where it enters the
+prompt.
+
+---
+
+## Checklist — a harness worth shipping
+
+- [ ] Workflow-versus-agent decided deliberately, and the simpler option was actually tried
+- [ ] System prompt at the **right altitude** — heuristics, not hardcoded branches, not vague hope
+- [ ] Every status, category and enum the agent must produce is **enumerated in the prompt**
+- [ ] Today's date, and any other volatile context, injected rather than assumed
+- [ ] Tools: a few high-impact ones, namespaced, each described as if to a new colleague
+- [ ] Tool responses carry **meaning, not identifiers**, and are paginated or truncated by default
+- [ ] Tool errors **teach the next attempt** instead of restating a stack trace
+- [ ] One technique chosen per problem, with a reason — not ReAct because it was in a paper
+- [ ] Sub-agents return **distilled summaries**, not transcripts
+- [ ] The agent can be observed: which tool, which arguments, which observation, how many tokens
+- [ ] An eval exists before the prompt is tuned, or the tuning is folklore

package/plugins/agent-stack/skills/agent-harness/references/audit.md

@@ -0,0 +1,141 @@
+# Auditing an agent system
+
+**Load this when:** reviewing an agent system you did not build — a due-diligence pass, an
+inherited codebase, or a "why is this unreliable" investigation.
+
+**Spec pinned:** the tracks below are this pack's synthesis of Anthropic agent guidance, `promptingguide.ai` and the sibling skills · read 2026-08-14
+
+## Contents
+
+- What this audit produces
+- Run the scanner first
+- Seven tracks
+- Evidence tiers
+- Priority, computed
+- The report
+- Traps
+
+## What this audit produces
+
+**A prioritized change plan, with an observation behind every finding.** Not a score.
+
+A number compresses away the only useful information — *what to change on Monday* — and
+invites arguing about the number. This is the same rule `agent-evals` applies to eval
+rubrics and `seo-aeo-audit` applies to sites, and it is the family's position rather than a
+preference: **pass/fail with a named failure condition beats a scalar that names no fix.**
+
+## Run the scanner first
+
+```bash
+python3 scripts/audit_agent.py <path>          # human-readable
+python3 scripts/audit_agent.py <path> --json   # machine-readable
+```
+
+It finds only what is mechanically visible, and **it prints the list of what it cannot see.**
+Read that list: it is the agenda for the manual half. A scanner that goes quiet is reporting
+its own blindness, and an audit that stops there has audited the scanner.
+
+## Seven tracks
+
+Walk them in order. Later tracks assume earlier ones.
+
+### 1 — Prompt
+
+- Is there a system prompt under version control, or is it a string literal edited in place?
+- **Altitude**: hardcoded branches (brittle) or vague hope (useless)? See `system-prompt.md`.
+- Are the values the agent must emit **enumerated**, or invented per run?
+- Is volatile context — the date above all — **injected**?
+- Is tool *policy* in the prompt, or only tool *schema*?
+- Has it been pruned, or only appended to since the first incident?
+
+### 2 — Tools
+
+- How many? Can the team say, without hesitating, which applies to a borderline case?
+- Are descriptions written **for a model choosing under uncertainty**, or for a human reading docs?
+- Do responses return **meaning or identifiers**?
+- Is there a default limit on response size, or only an optional one?
+- Do errors **name the next action**?
+- Are destructive tools guarded by shape (`confirm: true`, absolute paths, enums) rather than by instruction?
+
+### 3 — Control flow
+
+- **Workflow or agent — and was that decided, or defaulted?** An agent where a chain would do is the most expensive finding on this list.
+- Is there a **bounded iteration guard**, and what happens at the bound — a partial answer, or nothing?
+- Are retries and fallbacks **multiplied**? Three providers × three retries is nine calls for one prompt.
+- Is there loop detection, or does a repeated near-identical tool call run until the budget does?
+- Do sub-agents return **distilled summaries** or transcripts?
+
+### 4 — Context
+
+- Does anything measure window usage **before** a request fails?
+- Is there a compaction strategy, and does it preserve **decisions and open questions** rather than the discussion?
+- Can a large tool result be **offloaded** and referenced, or does it land in the window whole?
+- Is memory a design, or the conversation history by default?
+
+### 5 — Failure
+
+- What happens when a tool errors — is it distinguished from a tool returning nothing?
+- Under MCP, is `isError: true` handled, or does a 200 read as success?
+- Is there a timeout on every external call?
+- Is degradation **honest** — does the user learn the answer is partial?
+- Is there a path where the agent silently does nothing and reports success?
+
+### 6 — Permission
+
+- **Which layer owns the boundary?** If the harness delegates it (`layers.md`), audit the surroundings instead of filing a finding.
+- Is tool access differentiated per caller, or is one credential shared by every path?
+- Is tool output treated as **untrusted input**?
+- Can an audit row prove a control was applied — does it carry the **policy version**?
+- Is there a deterministic limit anywhere consequential, or only probabilistic content checks?
+
+### 7 — Evidence
+
+- **Are there evals?** If not, this is finding number one and everything else is unfalsifiable.
+- Do they judge the **trajectory**, or only the final answer?
+- Has any production failure become a permanent fixture?
+- Is a judge calibrated against human labels, or trusted because it is a judge?
+- Can a past run be replayed — is the execution record durable?
+
+## Evidence tiers
+
+Every finding carries one, and the tier is part of the finding:
+
+| Tier | Means | Example |
+|---|---|---|
+| **Measured** | observed here, in this system, with the observation attached | "`agent.py:212` — the `while` has no bound; a repeated call ran 47 times in the log at `logs/2026-08-02`" |
+| **Documented** | the upstream source says so, and this system contradicts it | "descriptions restate the name; Anthropic's tool guidance calls this the highest-leverage fix" |
+| **Judgement** | experience, no measurement available here | "two tools look interchangeable to us" |
+
+**Never present judgement as measured.** A finding whose tier is honest survives the meeting
+where it is challenged; one that is inflated loses the whole report.
+
+## Priority, computed
+
+`P = blast × confidence / effort`
+
+- **blast** 3 = a user of the system is harmed · 2 = the operator · 1 = a future maintainer
+- **confidence** 3 = measured · 2 = documented · 1 = judgement
+- **effort** 1 = under an hour · 2 = a session · 3 = its own project
+
+Computed, not felt — so a dramatic finding nobody can act on ranks below a boring one that
+is fixed before lunch, and the ranking can be argued with on its inputs.
+
+## The report
+
+1. **One paragraph** — what the system is, which layer, and the single most important thing.
+2. **The scanner output**, including its blind-spot list, verbatim.
+3. **Findings by track**, each with observation, tier and priority.
+4. **The plan** — ordered by P, with the first three items sized so they can start immediately.
+5. **What was not looked at**, and why. An audit that does not say where it stopped is read
+   as complete.
+
+## Traps
+
+- **Auditing the code and not the prompt.** The prompt is the largest behavioural surface and
+  is often not in the repository at all — ask where it lives before concluding it is fine.
+- **Filing "no permission model" against a harness that delegates by design.** Check the
+  layer first (`layers.md`).
+- **Grading instead of planning.** A score ends the conversation the audit was meant to start.
+- **Confusing "no evals" with "not measured yet."** It is the root finding; put it first,
+  because every other conclusion inherits it.
+- **Reading a silent scanner as a clean system.** It is silent about what it can see.

package/plugins/agent-stack/skills/agent-harness/references/layers.md

@@ -0,0 +1,100 @@
+# Layers — what your harness owns, and what it must not
+
+**Load this when:** deciding what to build versus adopt, or comparing agent frameworks and
+finding the comparison keeps sliding.
+
+**Spec pinned:** the three-layer reading of Pi / Goose / OpenCode (gist `AIMOWAY/bd8007c8`); Anthropic agent guidance · read 2026-08-14
+
+## Contents
+
+- The question that resolves most framework arguments
+- Three layers
+- What a harness owns
+- What a harness should delegate
+- Choosing your layer
+- Traps
+
+## The question that resolves most framework arguments
+
+"Which of these agent frameworks is better" is usually unanswerable because the candidates
+sit at different heights. The question that resolves it:
+
+> **Which layer of the agent stack am I trying to work at?**
+
+Compare within a layer. Across layers, the comparison is a category error, and the argument
+will not converge no matter how long it runs.
+
+## Three layers
+
+| Layer | What it is | Shape | Best when |
+|---|---|---|---|
+| **Kernel / harness** | the agent loop itself — runtime, LLM API abstraction, tool dispatch, a terminal UI | a toolkit you build *on* | you are building an agent product, or studying how agents actually work |
+| **Workbench / orchestration** | a local environment: desktop app, CLI and API, extensions, workflows across many kinds of work | a product you *extend* | you want capability now, across coding, research and automation, without owning the loop |
+| **Product agent** | a domain agent — most visibly coding: explore, plan, edit, test, with built-in modes | a product you *use* | the domain is the one it was built for |
+
+The distinction is not quality. A kernel is *supposed* to be smaller than a workbench; that
+is what makes it legible.
+
+## What a harness owns
+
+If you are building at the kernel layer, these are yours and nobody else's:
+
+1. **The loop** — iterate, dispatch tools, decide when to stop. With a bounded iteration
+   guard, because an unbounded loop is the defect that costs money while looking like work.
+2. **The model boundary** — one abstraction over providers, so the loop does not know which
+   vendor answered. See `agent-orchestrator` §6.
+3. **Tool dispatch and the ACI** — registration, schemas, parallel versus sequential
+   execution, error surfacing. See `tools.md`.
+4. **Context accounting** — knowing how full the window is *before* the request fails, and
+   what to do about it. See `agent-orchestrator/references/context-engineering.md`.
+5. **Observability** — which tool, which arguments, which observation, how many tokens. If
+   this is missing, every other item becomes unfalsifiable.
+6. **The interrupt/resume contract** — one mechanism, not two. See
+   `agent-orchestrator/references/runtime.md`.
+
+## What a harness should delegate
+
+The interesting stance, and it is a design position rather than an omission:
+
+**Permission boundaries usually belong to the environment, not the harness.** Pi states this
+explicitly — it provides the kernel and delegates sandboxing and permissions to whatever
+surrounds it. Read as a weakness, it looks like a missing feature. Read as architecture, it
+is a clear statement: *a harness that also claims to be a sandbox is claiming a guarantee it
+cannot keep*, because it runs in the same process as the code it would be confining.
+
+Two consequences worth stating plainly:
+
+- **A harness advertising "safe tool execution" without an OS-level or container boundary is
+  advertising a preference, not a control.** MCP says the same about roots: servers *SHOULD*
+  respect them, and real enforcement is OS permissions and sandboxing.
+- **Deciding the layer decides the audit.** If the harness delegates permission, the audit
+  looks at what surrounds it, and "the harness has no permission model" stops being a
+  finding and becomes a question about the deployment.
+
+Also usually delegated: identity and secrets (a credential store, never the loop), durable
+state (a database or a file system, not the conversation), and policy (see
+`agent-orchestrator/references/governance.md` — permission, not protocol).
+
+## Choosing your layer
+
+- **Building a product with agentic features** → adopt a workbench or embed a kernel; do not
+  write a third loop. Loops are commodity; your tools and prompt are not.
+- **Building an agent platform others build on** → the kernel layer is yours, and the ACI is
+  your product surface.
+- **Automating your own work** → a product agent, and stop. The most common expensive mistake
+  is building a kernel to solve a workbench problem.
+
+**The test that catches the mistake early:** if you cannot name a behaviour you need that the
+layer above does not provide, you are building the layer for its own sake.
+
+## Traps
+
+- **Comparing across layers**, then choosing on a benchmark that only makes sense within one.
+- **Writing a loop because the loop is the interesting part.** It is a week; the tools and
+  the prompt are the year.
+- **Claiming a security boundary the layer cannot enforce.** Say what is enforced and by
+  what; a sandbox protects the host, not the sandbox.
+- **Adopting a workbench and then fighting its opinions.** Its opinions are the product; if
+  you disagree with enough of them, you wanted the kernel.
+- **Assuming the layer is stable.** These projects move; check what the current version
+  actually owns before designing around a division of labour you read about once.

package/plugins/agent-stack/skills/agent-harness/references/system-prompt.md

@@ -0,0 +1,127 @@
+# The system prompt — altitude, vocabulary, and what changes for reasoning models
+
+**Load this when:** writing or fixing an agent's system prompt, or explaining why the same
+prompt behaves differently across runs.
+
+**Spec pinned:** Anthropic context-engineering and agent guidance; `promptingguide.ai` agents/* and guides/reasoning-llms · read 2026-08-14
+
+## Contents
+
+- The right altitude
+- What actually belongs in there
+- Enumerate the vocabulary
+- Inject what the model cannot know
+- Flexible while learning, strict in production
+- Structure, examples, and the cost of formatting
+- Reasoning models change three things
+- Traps
+
+## The right altitude
+
+A system prompt fails in two directions and the middle is narrower than it looks.
+
+**Too low** — hardcoded if-then branches for every case. It works on the cases you wrote and
+is brittle everywhere else, and each new case costs another branch. You are writing a
+program in prose, badly.
+
+**Too high** — vague guidance that assumes a shared understanding the model does not have.
+"Be helpful and use good judgement" tells it nothing it did not already believe.
+
+The target: **specific enough to guide behaviour, flexible enough to give the model strong
+heuristics.** A useful test — could a competent new colleague follow this without asking a
+clarifying question, and without being insulted? If they would ask, it is too high. If they
+would feel micromanaged into a corner where their judgement cannot help, it is too low.
+
+## What actually belongs in there
+
+In rough order of how much behaviour each buys:
+
+1. **Tool usage instructions.** Not the tool schema — the *policy*. When to reach for which,
+   what order usually makes sense, what to do when one fails. The largest measured gains come
+   from here, and it is the part most teams leave to the schema alone.
+2. **The role and its boundaries** — what this agent is for, and what it must hand off.
+3. **The vocabulary** it must produce (below).
+4. **Volatile context** it cannot know (below).
+5. **Failure instructions** — what to do when a tool errors, when data is missing, when the
+   task is impossible. Absent, the model invents a recovery, and inventions are not uniform.
+6. **Output contract** — shape, not prose about shape.
+
+## Enumerate the vocabulary
+
+**Be explicit about allowed values.** An agent asked to track task status will produce
+`pending` in one turn and `to-do` in the next, `completed` here and `done` there — and any
+code reading those strings now has a bug that appears intermittently and reads as
+flakiness.
+
+This generalises past status: every category, label, severity, priority or state the agent
+emits should appear as an enumerated set in the prompt, or in the tool's parameter `enum`,
+or both. It is the cheapest determinism available.
+
+## Inject what the model cannot know
+
+**Today's date is the canonical example**, and its absence has a specific failure signature:
+the agent answers from training data instead of searching, confidently and with no error.
+Inject the date and the behaviour changes without another word of instruction.
+
+The general rule: anything volatile that the model would otherwise fill from memory —
+current date, environment, tenant, available capabilities, the user's locale — is injected,
+not assumed. A capability-aware prompt that lists only the tools actually connected beats a
+static prompt describing tools that may be absent.
+
+## Flexible while learning, strict in production
+
+The same instruction should not survive the whole lifecycle.
+
+- **While you are still learning what good looks like:** *"Use the tools in the order that
+  makes most sense to you."* This surfaces what the model thinks the task is, which is the
+  information you need.
+- **Once the sequence is known and a skipped step is a defect:** *"You MUST execute a web
+  search for each task."* Flexibility here buys nothing and costs a silently missing step.
+
+Teams get stuck at the first form because it felt elegant, then debug an agent that
+"sometimes forgets" — which is not forgetting, it is permission.
+
+## Structure, examples, and the cost of formatting
+
+**Structure inputs and outputs** with delimiters, XML tags or JSON. Clear segmentation
+reduces the class of error where the model treats data as instruction.
+
+**Examples are worth more than description**, and the mistake is quantity. Curate a few
+**diverse, canonical** examples rather than an exhaustive list of edge cases — the latter
+reads as a lookup table and the model generalises from it badly.
+
+**But formatting has a cost.** Keep formats close to natural internet text where you can;
+elaborate escaping, deeply nested structures and unusual syntaxes spend the model's
+attention on parsing rather than the task. Ask whether a human writing this by hand would
+choose the format. If not, it is overhead.
+
+## Reasoning models change three things
+
+Treating a reasoning model like a completion model is now a common and expensive mistake.
+
+1. **Do not add chain-of-thought instructions.** Native reasoning already happens.
+   Explicit step-by-step prompting is redundant and **can hurt instruction-following** —
+   which is the opposite of what the person adding it intends.
+2. **Give goals, not procedures.** Be explicit about the high-level outcome and let the
+   model plan the route. Procedural micro-steps fight the thing you are paying for.
+3. **Reasoning effort is a dial that did not exist before** — low/medium/high trades cost
+   against accuracy per call, so it is a per-stage decision, not a global setting.
+
+Two more, worth knowing before you architect around them: **few-shot is still useful, but
+mainly for output *format***, not for teaching the task; and **tool-calling remains weaker
+in most reasoning models**, which is why the common shape is a reasoning model for planning
+and a different one for execution.
+
+## Traps
+
+- **Growing the prompt instead of fixing it.** Every incident adds a sentence; nothing is
+  ever removed; a year later nobody can say which line does work. Prune on the same schedule
+  you add.
+- **Describing tools twice**, in the schema and in the prompt, with the two drifting. Put
+  *policy* in the prompt and *contract* in the schema, and say which is which.
+- **A prompt that assumes a tool exists.** Capability-aware assembly, or an explicit
+  fallback; never a promise the runtime may not keep.
+- **Tuning the prompt with no eval.** You are optimising against the last thing you noticed.
+  See `agent-evals`.
+- **One prompt for planning and execution.** Separation of concerns measurably improves
+  reliability and lets a cheaper model take the mechanical half.

package/plugins/agent-stack/skills/agent-harness/references/techniques.md

@@ -0,0 +1,115 @@
+# The technique catalogue, with a verdict on each
+
+**Load this when:** choosing between ReAct, reflection, voting, planning and the rest — or
+being asked why the system does not use one that appeared in a paper.
+
+**Spec pinned:** `promptingguide.ai` techniques/* and guides/*; Anthropic agent guidance · read 2026-08-14
+
+**How to read the verdicts.** Every technique here works somewhere; the column says whether
+it earns its cost *in a production agent loop*, which is a narrower question than whether it
+raised a benchmark. Costs are real: each of these multiplies calls, latency or context, and
+several of them were measured on single-turn QA rather than a long-running agent.
+
+## Contents
+
+- The catalogue
+- ReAct, in detail
+- Reflection, in detail
+- Voting and self-consistency
+- What reasoning models made redundant
+- Choosing one
+
+## The catalogue
+
+| Technique | What it is | Verdict for a production agent loop |
+|---|---|---|
+| **Zero-shot** | instruction alone | **Default.** Start here; everything below is a cost you must justify |
+| **Few-shot** | input→output exemplars | **Yes, for format.** Curate a few diverse canonical examples. It teaches shape far better than it teaches judgement, and an exhaustive list makes the model brittle |
+| **Chain-of-thought** | "think step by step" | **Legacy on reasoning models — actively harmful there** (it can degrade instruction-following). Still useful on non-reasoning models for arithmetic and multi-constraint tasks |
+| **ReAct** | interleaved thought → action → observation | **Yes — this is the agent loop.** Most harnesses implement it without naming it. See below for what it does not fix |
+| **Reflexion / self-critique** | actor, evaluator, self-reflection, with episodic memory | **Selectively.** Real gains where a *cheap objective signal* exists — tests pass, query runs, schema validates. Without one, the model grades its own homework |
+| **Self-consistency** | sample N, take the majority | **Rarely.** N× cost and latency for a single answer; needs a well-defined answer to vote on. Use for a high-stakes classification, not for a whole trajectory |
+| **Tree of Thoughts** | explore and prune a branching search | **Almost never in production.** Combinatorial cost, and the pruning heuristic is usually the hard part you have not solved. A planner plus a bounded retry gets most of the value |
+| **ART** (automatic reasoning + tool use) | select exemplars and tools from a task library automatically | **Watch.** The idea — a library of trajectories rather than a hand-written prompt — is where prompt maintenance is heading; the tooling is not settled |
+| **Prompt chaining** | fixed sequence with checks between | **Yes — and prefer it over an agent** wherever the steps are knowable |
+| **Meta prompting** | prompt about the structure of the task, not its content | **Occasionally.** Useful for generating scaffolds; not a loop technique |
+| **Generate-knowledge** | elicit facts first, then answer | **No.** Retrieval solves the same problem with grounding; this invents plausible knowledge |
+| **RAG** | retrieve, then generate | **Yes, and it is a search problem.** Its quality lives outside this skill; what belongs here is *just-in-time* retrieval — see below |
+| **Just-in-time retrieval** | hold lightweight identifiers (paths, queries, links); load at runtime via tools | **Yes.** The scalable default: it mirrors how people work, and keeps the window for reasoning rather than for data |
+| **Structured note-taking** | write notes to durable memory outside the window, read them back | **Yes for long tasks.** Persistent memory at low overhead, and it survives compaction — which a summary of the discussion does not |
+| **Sub-agents** | specialists returning distilled summaries | **Yes where the sub-task has its own context need.** The value is the *isolated window*; a sub-agent returns a 1–2k-token distilled summary, never a transcript |
+
+## ReAct, in detail
+
+**Thought → Action → Observation**, repeated. Reasoning traces and task actions generated in
+an interleaved way, so the plan updates on what the world actually returned.
+
+What it fixes: plain chain-of-thought is isolated from external information and therefore
+suffers **fact hallucination and error propagation** — it reasons confidently past a wrong
+premise. ReAct grounds each step in an observation.
+
+**What it does not fix, and this is under-quoted:**
+
+- It **constrains reasoning flexibility** compared with free-form CoT — the format itself is
+  a cost.
+- **Non-informative results derail it.** A search returning nothing useful leaves the model
+  struggling to reformulate, and it will loop on near-identical queries. This is the failure
+  your iteration guard exists for, and it is why tool errors must teach (`tools.md`).
+- It is strongest **combined** with CoT and self-consistency rather than alone — which is
+  the honest reading of the paper and rarely the reading in a blog post.
+
+## Reflection, in detail
+
+Three roles, and naming them separates the ones people conflate:
+
+| Role | Does | In practice |
+|---|---|---|
+| **Actor** | generates text and actions, produces a trajectory | your existing loop |
+| **Evaluator** | scores the trajectory | **the part that decides whether this works at all** |
+| **Self-reflection** | turns the score into verbal guidance stored for next time | an extra call, plus memory |
+
+**The evaluator is the whole question.** Where the signal is objective and cheap — the test
+suite ran, the SQL executed, the JSON validated, the build passed — reflection is one of the
+strongest available techniques. Where the evaluator is the same model judging its own
+output with no ground truth, you have added cost and a confident second opinion.
+
+Stated limitations worth carrying: it depends on **accurate self-evaluation**, its memory is
+typically a **sliding window**, and it struggles where correctness is non-deterministic.
+
+## Voting and self-consistency
+
+Sample the same task several times and take the majority. It genuinely reduces variance —
+and it multiplies cost and latency by N, needs a discrete answer to vote on, and does
+nothing for a long trajectory where the runs diverge at step three.
+
+Reach for it on a **single high-stakes decision** — a routing classification, a safety
+judgement, an extraction that everything downstream depends on. Not on a whole agent run.
+
+## What reasoning models made redundant
+
+A live shift, and it invalidates a lot of otherwise-good advice:
+
+- **Do not instruct step-by-step thinking.** It is native, and explicit CoT can hurt
+  instruction-following.
+- **Give high-level goals rather than procedures**, and let planning happen inside the model.
+- **Reasoning effort** is now a per-call dial, which is a cheaper knob than most of the
+  techniques above.
+- But **tool-calling stays weaker** in most reasoning models — so the common production shape
+  is a reasoning model that plans and a different model that executes tools. That is
+  *separation of concerns*, and it also lets the cheap half be cheap.
+
+## Choosing one
+
+Ask in this order, and stop at the first yes:
+
+1. **Are the steps knowable?** → prompt chaining, not an agent.
+2. **Is there a cheap objective signal?** → evaluator–optimizer or reflection.
+3. **Does the sub-task need its own window?** → a sub-agent returning a distilled summary.
+4. **Is one decision disproportionately expensive to get wrong?** → voting, on that decision only.
+5. **Otherwise** → a plain ReAct loop with a bounded iteration guard, and spend the effort on
+   the tools and the prompt instead. That is where the measured gains are.
+
+**The anti-pattern this section exists to prevent:** adopting a technique because it appears
+in a paper, without naming the signal it consumes or the cost it adds. If you cannot say
+what the evaluator measures, you are not doing reflection — you are paying for a second
+opinion from the same source.

package/plugins/agent-stack/skills/agent-harness/references/tools.md

@@ -0,0 +1,154 @@
+# Tools — the agent–computer interface
+
+**Load this when:** the model picks the wrong tool, calls none, calls one with bad arguments,
+or you are deciding what to expose in the first place.
+
+**Spec pinned:** Anthropic *Writing tools for agents* and *Building effective agents*; `promptingguide.ai` agents/function-calling · read 2026-08-14
+
+Anthropic's framing is worth adopting whole: this is the **agent–computer interface**, and it
+deserves the same craft a human interface gets. Most teams spend their effort on the model
+and none on the ACI, then conclude the model is bad at tool use.
+
+## Contents
+
+- Fewer tools than you think
+- Namespacing
+- The description is the product
+- Return meaning, not identifiers
+- Token efficiency is a correctness issue
+- Errors that teach
+- Poka-yoke: make the wrong call impossible
+- Evaluating tools
+- Traps
+
+## Fewer tools than you think
+
+**More tools do not lead to better outcomes.** The reflex — wrap every API endpoint, ship
+forty tools, let the model choose — produces an agent that chooses badly, because selection
+degrades with the size of the set and every definition costs context.
+
+Build **a few thoughtful tools targeting specific high-impact workflows.** The test:
+consolidate where a human would. `search_and_summarize` beats `search` + `fetch` +
+`summarize` when the three are always used together, because it removes two decisions and
+two round trips.
+
+**The honest check, and it is brutal:** *if your engineers cannot definitively say which
+tool applies to a case, the model cannot either.* Ambiguity between two tools is a design
+defect, not a prompting problem.
+
+## Namespacing
+
+Group related tools under a common prefix — `asana_search`, `asana_create_task`,
+`jira_search`. Boundaries become visible, and the model stops crossing services by accident.
+
+This also matters at the federation layer: behind a gateway, tool names commonly gain a
+prefix from their source server, and a name that changes between sessions invalidates every
+prompt and eval that referenced it. See `agent-interop/references/gateway.md`.
+
+## The description is the product
+
+**Even small refinements to tool descriptions yield dramatic improvements.** The rule that
+makes them good: **write as if explaining to a new team member**, and make implicit context
+explicit.
+
+A description must answer **when and why**, not only what:
+
+```jsonc
+// weak — restates the name, and says nothing about choosing it
+{ "name": "search_users", "description": "Searches for users." }
+
+// strong — the model can now decide
+{ "name": "search_users",
+  "description": "Find users by name, email or team. Use this before any operation \
+that needs a user ID — IDs are never guessable. Returns at most 20 matches; narrow \
+with `team` rather than paging when you can. Do NOT use for the current user: \
+`get_current_user` is cheaper and always correct." }
+```
+
+Note what the strong version carries: **when to reach for it, what it costs, how to narrow,
+and the neighbouring tool it is confused with.** That last clause is the highest-value
+sentence in most tool descriptions and almost nobody writes it.
+
+**Parameters carry their own guidance.** Use `enum` to constrain values rather than
+describing the constraint in prose, give examples in parameter descriptions, and mark
+required versus optional honestly — an optional parameter the tool actually needs is a
+silent failure.
+
+## Return meaning, not identifiers
+
+Prioritise **contextual relevance over flexibility**. A response of
+`{"id": "u_8f3a", "gid": "1209...", "rid": 44}` gives the model nothing to reason with; it
+will echo identifiers into prose and hallucinate what they mean. Return
+`{"name": "Ada Lovelace", "team": "Platform", "id": "u_8f3a"}` — the id stays for the next
+call, the meaning arrives for the reasoning.
+
+## Token efficiency is a correctness issue
+
+Not merely a cost issue: a tool that returns 40,000 tokens of JSON has consumed the window
+the agent needed to finish the task, and no amount of history trimming recovers it (see
+`agent-orchestrator/references/context-engineering.md` → *tool-output offload*).
+
+Build in **pagination, range selection, filtering and truncation — with sensible defaults**.
+The default matters more than the capability: an agent will rarely opt into a limit it was
+not given.
+
+## Errors that teach
+
+A tool error is a turn in a conversation. Compare:
+
+```
+Error: 422 Unprocessable Entity
+```
+```
+Error: `due_date` must be ISO-8601 (e.g. 2026-08-14). You sent "next friday".
+Call `resolve_date` first, or pass an absolute date.
+```
+
+The second costs nothing extra and converts a dead end into a recovery. **Return informative
+messages that help the agent recover or try an alternative** — naming the alternative is the
+part that gets skipped.
+
+Two structural notes: under MCP a failed tool arrives as a *successful* response carrying
+`isError: true`, so code that only catches transport exceptions treats every tool failure as
+a success containing an apology (`agent-interop/references/mcp.md`). And in code-mode
+harnesses, generated wrappers should convert that into a thrown exception so model-authored
+code can `try`/`catch`.
+
+## Poka-yoke: make the wrong call impossible
+
+Borrowed from manufacturing, and the highest-leverage idea here: **change the interface so
+the mistake cannot be made**, rather than documenting the mistake.
+
+- Absolute paths instead of relative ones, when relative paths get resolved against a
+  directory the agent guessed.
+- An `enum` instead of a free-text field with a list of valid values in the description.
+- One tool that does the two-step correctly instead of two tools that must be ordered.
+- A required `confirm: true` on a destructive action, so a partially-formed call fails
+  closed.
+
+## Evaluating tools
+
+Tools deserve **thorough documentation and testing**, and testing means running the agent
+against real tasks and reading which tool it picked, with what arguments, and what came
+back. Enable intermediate-step visibility and look for the three recurring faults:
+
+| Symptom | Almost always |
+|---|---|
+| wrong tool chosen | two descriptions do not distinguish themselves; add the "do NOT use for…" clause |
+| bad arguments | the parameter description assumes context the model does not have, or the type is too loose |
+| result misread | the response returned identifiers, or too much, or both |
+
+Fix the interface, not the prompt, when the fault is in this table.
+
+## Traps
+
+- **Wrapping the API you have** instead of designing the tools the agent needs.
+- **A description written for a human reading docs** rather than a model choosing under
+  uncertainty.
+- **Unbounded responses** with an optional `limit` nobody sets.
+- **Errors that are true and useless.** `null is not an object` names the symptom the model
+  can do nothing with.
+- **Treating tool output as trusted.** It is attacker-controlled input if the server is; the
+  specification says descriptions and annotations are untrusted unless the server is.
+- **Adding a tool to fix a prompt problem.** The set grows, selection degrades, and the
+  original defect is still there.

package/plugins/agent-stack/skills/agent-harness/scripts/audit_agent.py

@@ -0,0 +1,299 @@
+#!/usr/bin/env python3
+"""Mechanical half of an agent-system audit.
+
+Finds only what is visible WITHOUT understanding intent, and prints what it cannot see —
+because a scanner that goes quiet is reporting its own blindness, and an audit that stops
+at a silent scanner has audited the scanner.
+
+    python3 audit_agent.py <path>            human-readable
+    python3 audit_agent.py <path> --json     machine-readable
+    python3 audit_agent.py --self-test       plant each defect, require each to be found
+
+Zero dependencies. Python 3.9+.
+
+Design rule, and the reason this file is short: every detector is CONSERVATIVE. A false
+positive costs more than a miss here, because an audit report that cries wolf is discarded
+whole — and the seven tracks in `references/audit.md` cover by hand everything this cannot
+reach. Detectors therefore require corroboration (the file must look agent-related) and
+each finding carries `file:line` so a human can disagree with it in one click.
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+
+# A file is "agent-related" only if it shows two independent signs. One is a coincidence:
+# plenty of code says "message" or "prompt" without being an agent loop.
+AGENTISH = [
+    re.compile(r"\btool[_ ]?call", re.I),
+    re.compile(r"\btools\s*=|\"tools\"\s*:|'tools'\s*:"),
+    re.compile(r"\bsystem[_ ]?prompt", re.I),
+    re.compile(r"\b(anthropic|openai|litellm|langchain|langgraph|bedrock|mistral)\b", re.I),
+    re.compile(r"\bfunction[_ ]?call", re.I),
+    re.compile(r"\bmessages\s*=\s*\[|\"messages\"\s*:"),
+]
+CODE_EXT = {".py", ".js", ".mjs", ".cjs", ".ts", ".tsx", ".jsx"}
+SKIP_DIRS = {".git", "node_modules", "venv", ".venv", "__pycache__", "dist", "build",
+             ".next", "target", "vendor", ".tox", "site-packages"}
+MAX_BYTES = 400_000          # a generated bundle is not worth reading, and skews everything
+
+# Model ids that are usually hardcoded by accident. Deliberately not exhaustive: this is a
+# smell detector, and the finding says "pin it deliberately", not "this id is wrong".
+MODEL_LITERAL = re.compile(
+    r"[\"']((?:claude|gpt|gemini|llama|mistral|deepseek|qwen)[-\w.]*\d[\w.-]*)[\"']", re.I)
+
+FINDINGS = []
+
+
+def add(kind, path, line, detail, fix):
+    FINDINGS.append({"check": kind, "file": path, "line": line, "detail": detail, "fix": fix})
+
+
+def agentish(text):
+    return sum(1 for p in AGENTISH if p.search(text)) >= 2
+
+
+def walk(root):
+    for dirpath, dirnames, filenames in os.walk(root):
+        # Skip a virtualenv by its MARKER, not by its name. `venv`/`.venv` in SKIP_DIRS
+        # only catches the conventional names; a real repository met during testing used
+        # `myenv/`, holding 4249 of its 4261 code files, and was excluded only because
+        # `site-packages` happened to be listed too. Right by accident is not right.
+        dirnames[:] = [d for d in dirnames
+                       if d not in SKIP_DIRS and not d.startswith(".")
+                       and not os.path.exists(os.path.join(dirpath, d, "pyvenv.cfg"))]
+        for fn in filenames:
+            if os.path.splitext(fn)[1] not in CODE_EXT:
+                continue
+            full = os.path.join(dirpath, fn)
+            try:
+                if os.path.getsize(full) > MAX_BYTES:
+                    continue
+                with open(full, encoding="utf-8", errors="replace") as fh:
+                    text = fh.read()
+            except OSError:
+                continue
+            yield os.path.relpath(full, root), text
+
+
+# ------------------------------------------------------------------ detectors
+
+def check_unbounded_loop(rel, text, lines):
+    """`while True` in an agent file with no visible iteration bound.
+
+    Conservative twice over: the file must be agent-related, AND the file must not mention
+    any bound at all. A loop with `max_iter` somewhere else in the file is left alone.
+    """
+    if re.search(r"max[_ ]?iter|max[_ ]?steps|max[_ ]?turns|iteration_limit|for\s+\w+\s+in\s+range\(",
+                 text, re.I):
+        return
+    for i, l in enumerate(lines, 1):
+        if re.search(r"^\s*while\s+(True|true|1)\s*[:)]|^\s*while\s*\(\s*true\s*\)", l):
+            add("unbounded-loop", rel, i,
+                "`while True` in an agent file with no iteration bound anywhere in it",
+                "Add a max-iteration guard that composes a partial answer at the bound, "
+                "rather than returning nothing")
+
+
+def check_tool_without_description(rel, text, lines):
+    """A tool declared with an empty or missing description.
+
+    Only fires on an explicit empty string — a missing key is too easy to get wrong across
+    frameworks, and a wrong finding here is worse than a missed one.
+    """
+    for i, l in enumerate(lines, 1):
+        if re.search(r"[\"']description[\"']\s*:\s*[\"']\s*[\"']", l) or \
+           re.search(r"\bdescription\s*=\s*[\"']\s*[\"']", l):
+            add("tool-no-description", rel, i,
+                "a tool description is the empty string",
+                "Describe WHEN and WHY to use it, and name the neighbouring tool it is "
+                "confused with — the highest-leverage sentence in a tool definition")
+
+
+def check_swallowed_error(rel, text, lines):
+    """An exception caught and discarded inside an agent file."""
+    for i, l in enumerate(lines, 1):
+        nxt = lines[i] if i < len(lines) else ""
+        if re.search(r"^\s*except[^\n]*:\s*$", l) and re.search(r"^\s*pass\s*$", nxt):
+            add("swallowed-error", rel, i,
+                "`except: pass` — the failure is invisible to the loop and to the model",
+                "Return an error the agent can act on; a tool error is a turn in the "
+                "conversation, not a silence")
+        if re.search(r"catch\s*\([^)]*\)\s*\{\s*\}", l):
+            add("swallowed-error", rel, i,
+                "empty `catch` block — the failure is discarded",
+                "Surface it to the loop; an error that teaches the next attempt costs "
+                "nothing extra")
+
+
+def check_no_timeout(rel, text, lines):
+    """An outbound HTTP call with no timeout, in an agent file."""
+    for i, l in enumerate(lines, 1):
+        if re.search(r"\brequests\.(get|post|put|patch|delete)\s*\(", l) and "timeout" not in l:
+            add("no-timeout", rel, i,
+                "`requests` call with no `timeout=` — a hung provider hangs the agent",
+                "Set an explicit timeout on every external call, and decide what the loop "
+                "does when it fires")
+        if re.search(r"\burllib\.request\.urlopen\s*\(", l) and "timeout" not in l:
+            add("no-timeout", rel, i, "`urlopen` with no `timeout=`",
+                "Set an explicit timeout on every external call")
+
+
+def check_hardcoded_model(rel, text, lines):
+    """A model id as a literal, in more than one place — the smell is duplication."""
+    hits = []
+    for i, l in enumerate(lines, 1):
+        if l.lstrip().startswith(("#", "//", "*")):
+            continue
+        m = MODEL_LITERAL.search(l)
+        if m:
+            hits.append((i, m.group(1)))
+    if len(hits) >= 2:
+        i, name = hits[0]
+        add("hardcoded-model", rel, i,
+            f"model id {name!r} appears as a literal {len(hits)}× in this file",
+            "Resolve the model from configuration at one boundary; three levels — request, "
+            "tenant, system default — is the shape that bills correctly")
+
+
+CHECKS = [check_unbounded_loop, check_tool_without_description, check_swallowed_error,
+          check_no_timeout, check_hardcoded_model]
+
+# What no static pass can reach. Printed every run, never suppressed.
+BLIND = [
+    "whether the SYSTEM PROMPT is at the right altitude — or whether it is in this repo at all",
+    "whether two tool descriptions actually distinguish themselves to a model",
+    "whether the workflow/agent choice was made deliberately or defaulted to an agent",
+    "whether retries and fallbacks MULTIPLY (three providers x three retries is nine calls)",
+    "whether compaction preserves decisions and open questions, or keeps the discussion",
+    "whether tool output is treated as untrusted input",
+    "whether evals exist, judge trajectories, and are calibrated",
+    "whether an audit row could prove a control was applied (policy version)",
+    "everything in a language this pass does not read, and everything in configuration",
+]
+
+
+def scan(root):
+    seen_files = considered = 0
+    for rel, text in walk(root):
+        considered += 1
+        if not agentish(text):
+            continue
+        seen_files += 1
+        lines = text.splitlines()
+        for c in CHECKS:
+            c(rel, text, lines)
+    return seen_files, considered
+
+
+def report_text(root, seen, considered):
+    # The denominator is not decoration. "read: 1" alone looks like a broken pass; "1 of
+    # 4261" says the repository is mostly not an agent, which is a finding in itself when
+    # somebody called it one.
+    out = [f"agent-audit: {root}",
+           f"  code files considered: {considered}",
+           f"  of those, agent-related: {seen}"]
+    if not seen:
+        out.append("  NOTHING READ — no file showed two independent signs of an agent loop.")
+        out.append("  That is a fact about this pass, not about the system. Check the path,")
+        out.append("  and whether the agent lives in a language or a config this cannot read.")
+    out.append("")
+    if FINDINGS:
+        out.append(f"FINDINGS ({len(FINDINGS)}) — each is a smell with a location, not a verdict:")
+        for f in FINDINGS:
+            out.append(f"  {f['file']}:{f['line']}  [{f['check']}]")
+            out.append(f"      {f['detail']}")
+            out.append(f"      fix: {f['fix']}")
+    else:
+        out.append("FINDINGS (0) — nothing mechanically visible.")
+    out.append("")
+    out.append("THIS PASS CANNOT SEE — the manual half of the audit, and it is the larger half:")
+    for b in BLIND:
+        out.append(f"  - {b}")
+    out.append("")
+    out.append("Walk the seven tracks in references/audit.md. Silence above is not a pass.")
+    return "\n".join(out)
+
+
+def self_test():
+    """Plant each defect and require the matching check to fire.
+
+    A detector nobody has watched fire is not evidence that it works, and every plant
+    asserts it changed something so a reworded fixture fails HERE rather than reporting a
+    healthy checker as broken.
+    """
+    import tempfile
+    header = ("import requests\n"
+              "system_prompt = 'x'\n"
+              "tools = [{'name': 't', 'description': 'does a thing'}]\n"
+              "messages = []\n")
+    cases = {
+        "unbounded-loop": header + "while True:\n    pass\n",
+        "tool-no-description": header + "T = [{'name': 'a', 'description': ''}]\n",
+        "swallowed-error": header + "try:\n    x = 1\nexcept Exception:\n    pass\n",
+        "no-timeout": header + "r = requests.get('https://example.com')\n",
+        "hardcoded-model": header + "a = 'claude-opus-4'\nb = 'claude-opus-4'\n",
+    }
+    failures = 0
+    for kind, body in cases.items():
+        FINDINGS.clear()
+        with tempfile.TemporaryDirectory() as d:
+            p = os.path.join(d, "agent.py")
+            with open(p, "w", encoding="utf-8") as fh:
+                fh.write(body)
+            assert agentish(body), f"PLANT DID NOT LAND: fixture for {kind} is not agent-related"
+            scan(d)
+        got = {f["check"] for f in FINDINGS}
+        if kind in got:
+            print(f"  OK   {kind}: detected")
+        else:
+            print(f"  FAIL {kind}: NOT detected (found {sorted(got) or 'nothing'})")
+            failures += 1
+    # and a clean file must produce nothing, or every finding above is noise
+    FINDINGS.clear()
+    with tempfile.TemporaryDirectory() as d:
+        with open(os.path.join(d, "agent.py"), "w", encoding="utf-8") as fh:
+            fh.write(header + "for _ in range(10):\n    pass\n"
+                     "r = requests.get('https://example.com', timeout=5)\n")
+        scan(d)
+    if FINDINGS:
+        print(f"  FAIL clean file produced {len(FINDINGS)} finding(s): "
+              f"{[f['check'] for f in FINDINGS]}")
+        failures += 1
+    else:
+        print("  OK   clean file: silent")
+    print(f"\nself-test: {len(cases) + 1 - failures}/{len(cases) + 1} passed")
+    return 1 if failures else 0
+
+
+def main(argv):
+    ap = argparse.ArgumentParser(description="Mechanical half of an agent-system audit.")
+    ap.add_argument("path", nargs="?", default=".")
+    ap.add_argument("--json", action="store_true")
+    ap.add_argument("--self-test", action="store_true")
+    a = ap.parse_args(argv)
+
+    if a.self_test:
+        return self_test()
+
+    root = os.path.abspath(a.path)
+    if not os.path.isdir(root):
+        print(f"error: {a.path} is not a directory", file=sys.stderr)
+        return 2
+    seen, considered = scan(root)
+    if a.json:
+        print(json.dumps({"root": root, "files_considered": considered,
+                          "files_agent_related": seen, "findings": FINDINGS,
+                          "cannot_see": BLIND}, indent=2))
+    else:
+        print(report_text(root, seen, considered))
+    # Findings are smells, not failures: exit 0 so this composes in a pipeline, and let the
+    # human decide. A non-zero exit here would turn an audit into a gate it was never
+    # calibrated to be.
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))