@ssheleg/agent-stack 0.4.0 → 0.6.0

package/CHANGELOG.md

@@ -4,6 +4,90 @@ All notable changes to this project are documented here.
 Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning: [SemVer](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.0] — 2026-08-12
+
+### Added
+
+- **`references/runtime.md`** — the layer most orchestrators assume rather than
+  specify, and the one this skill was quietly missing. **Checkpoint every iteration,
+  not just the stages a human reviews**: the multi-stage path persisted and could
+  resume, the simple tool-calling path persisted nothing, so a crash lost the run that
+  executes most often — an asymmetry, not a design. Then one interrupt/resume contract
+  instead of the two mechanisms the body had for one idea (`ask_user` and a stage
+  checkpoint); the four double-texting policies and why interrupt and rollback differ
+  in what the transcript looks like afterwards; streaming with event ids so a dropped
+  connection rejoins instead of watching nothing for ninety seconds; forking a past
+  checkpoint, which debugs through the real loop rather than a reconstruction that may
+  not share the bug; stateful versus stateless schedules; and the seven cross-cutting
+  concerns welded into the loop pulled out as ordered interceptors — where **order is
+  semantics**, because redaction after summarisation redacts a summary that already
+  leaked.
+
+- **`references/governance.md`** — permission, where `llm-proxy-billing.md` is cost.
+  **The greatest risk is usually not what the model says but what the agent can do**,
+  so the four boundaries get four control sets: model call, tool call, external server,
+  and agent-to-agent — the last being the one designs miss, since a sub-agent
+  inheriting its caller's authority silently widens every permission. The guardrail
+  taxonomy in order of reliability, and its honest limit: every content check is
+  probabilistic, so anything consequential takes a deterministic limit or a human, never
+  a classifier's confidence. Why an audit row without a **policy version** cannot prove
+  a control was applied. Cost attribution as a hierarchy, because "which team's agent
+  did this" is unanswerable from a flat tenant id. Failover that must be
+  policy-equivalent rather than merely available — a chain that silently fails into
+  another jurisdiction does it precisely when nobody is reading logs. Fail-open versus
+  fail-closed per workload. And blast radius: a sandbox protects the host, not the
+  sandbox, and credentials never enter it.
+
+### Changed
+
+- **The References table is an index again.** Each reference now opens with its own
+  `Load this when` line, so the trigger has exactly one home and the table cannot drift
+  from the files it points at. Compressing it returned ~100 tokens of body budget, which
+  is what paid for two new rows: the body sits at 489 lines / ~4883 tokens against
+  500 / 5000.
+- README describes two skills and five references, and its trigger section covers the
+  evals skill and the permission surface, not only the orchestrator and the wallet.
+
+## [0.5.0] — 2026-08-12
+
+### Added
+
+- **A second skill: `agent-evals`** — how you know the thing built by
+  `agent-orchestrator` actually behaves. There was no evaluation doctrine anywhere in
+  this family; `grep -ril "llm-as-judge\|eval dataset\|regression fixture\|trace id"`
+  across four plugins returned nothing.
+
+  An agent's behaviour is not in its source — the code says what it is allowed to do,
+  only a run says what it did — so the artifact under test is the execution record.
+  Three primitives (run, trace, thread) crossed with three granularities (single-step,
+  full-turn, multi-turn), each with its own fixture shape and its own precondition:
+  step assertions need a stable architecture or they die at the next refactor; turn
+  assertions cover trajectory **and** response **and** state change, because an agent
+  that says it saved the preference and did not passes two axes out of three; thread
+  scripts checkpoint after every turn and fail fast, or turn 3 derails and turns 4–10
+  assert nothing while still reporting a result.
+
+  Then the parts that decide whether any of it is trustworthy: the offline/online/ad-hoc
+  timing axis and why offline is necessary and not sufficient; pass-fail rubrics with
+  enumerated failure conditions instead of scalar scores that name no fix; code checks
+  before model judges; judging the trajectory, not just the answer; **calibrating a judge
+  against human labels before trusting it**, because an uncalibrated judge is an opinion
+  with a number attached; the classes of output no general judge can grade; a corpus
+  grown from production failures where every fixed failure stays a fixture permanently;
+  annotation queues with the two reviewer roles kept apart; and simulated users made
+  deliberately worse so offline results predict production.
+
+  Body 214 lines / ~2500 tokens; description 911/1024 with paired RU triggers.
+
+### Changed
+
+- **Both installers iterate over `skills/` instead of naming one skill.** `install.sh`
+  and `bin/agent-stack.js` each hardcoded `agent-orchestrator`, so a second skill would
+  have shipped in the package and reached nobody. Verified by running both against a
+  clean `HOME` and listing what arrived. The CI smoke test now asserts both skills.
+- README and both manifests describe two skills; the marketplace entry's description is
+  the manifest's, rather than a second one drifting beside it.
+
 ## [0.4.0] — 2026-08-12
 
 ### Added

package/README.md

@@ -9,7 +9,8 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills).
 
 ## What is in here
 
-One skill, `agent-orchestrator`, and two references it loads on demand.
+Two skills — `agent-orchestrator` for building one, `agent-evals` for proving it
+behaves — and five references the first loads on demand.
 
 **The orchestrator** (`SKILL.md`) — what the agent reads first:
 
@@ -29,6 +30,39 @@ One skill, `agent-orchestrator`, and two references it loads on demand.
 - context budget allocation by priority
 - self-learning feedback loops
 
+**The evals skill** (`agent-evals/SKILL.md`) — how you know any of it works. An
+agent's behaviour is not in its source, so the artifact under test is the
+execution record: three primitives (run, trace, thread) crossed with three
+granularities (single-step, full-turn, multi-turn), the offline/online/ad-hoc
+timing axis, pass-fail rubrics instead of scalar scores that name no fix, cheap
+code checks before model judges, judges calibrated against human labels before
+they are trusted, and a corpus grown from production failures rather than
+authored up front — where every fixed failure stays a fixture permanently.
+
+**`references/context-engineering.md`** — what the loop gives up when the window
+runs out: the five-rung compaction ladder and why to re-measure between rungs,
+the tool-pair boundary invariant, typed carryover blocks copied across the
+boundary rather than summarized, tool-output offload to a file, token estimation
+and the direction it errs, the compaction circuit breaker, sub-agent context
+isolation, and how to choose constants for your own window.
+
+**`references/runtime.md`** — what keeps an agent alive between requests, which
+most orchestrators assume rather than specify: checkpointing every iteration and
+not just the stages a human reviews, one interrupt/resume contract instead of two
+mechanisms for one idea, the four double-texting policies, streaming a dropped
+connection can rejoin, forking a past checkpoint to debug through the real loop,
+stateful versus stateless schedules, and the seven cross-cutting concerns pulled
+out of the loop into ordered interceptors — where order is semantics.
+
+**`references/governance.md`** — permission rather than cost. The four boundaries
+an agent crosses (model, tool, external server, agent-to-agent), each with its own
+control set; the guardrail taxonomy and its honest limit — every content check is
+probabilistic, so anything consequential gets a deterministic limit or a human;
+why an audit row without a policy version cannot prove a control was applied;
+cost attribution as a hierarchy; failover that must land somewhere approved rather
+than merely available; fail-open versus fail-closed as a per-workload decision;
+and blast radius — a sandbox protects the host, not the sandbox.
+
 **`references/patterns.md`** — the data models and algorithms underneath:
 message and result protocols, pipeline models, the SQL validation loop,
 context-window sizes and token estimation, learning-extraction heuristics,
@@ -54,7 +88,7 @@ waterfall, and model-routing precedence.
 /plugin install agent-stack@agent-stack
 ```
 
-**npm installer** — copies the skill into `~/.claude/skills/`:
+**npm installer** — copies both skills into `~/.claude/skills/`:
 
 ```bash
 npx @ssheleg/agent-stack
@@ -78,13 +112,19 @@ Restart your agent afterwards — skills load at session start.
 
 ## When it triggers
 
-Building an agent system, an orchestrator, an LLM-powered tool, a chatbot with
-tool use, or an AI pipeline. Also when the work is the money side: metering
-usage, per-tenant keys, spend tracking, budget limits, loop detection.
+`agent-orchestrator`: building an agent system, an orchestrator, an LLM-powered
+tool, a chatbot with tool use, or an AI pipeline. Also the money side — metering
+usage, per-tenant keys, spend tracking, budget limits, loop detection — and the
+permission side: what a tool may reach, what leaves the boundary, and what an
+audit row has to carry to prove a control was on.
+
+`agent-evals`: measuring whether the result behaves. Building a suite, judging a
+trajectory rather than a final answer, turning a production failure into a
+permanent fixture, calibrating a judge, gating a release on offline evals.
 
-It does **not** trigger for a single LLM call in a script, or for prompt
-wording — that is not an orchestrator, and pulling 1200 lines of doctrine for it
-is how a skill teaches you to route around it.
+Neither triggers for a single LLM call in a script or for prompt wording — that
+is not an orchestrator, and pulling this much doctrine for it is how a skill
+teaches you to route around it.
 
 ---
 

package/bin/agent-stack.js

@@ -2,7 +2,7 @@
 /*
  * agent-stack installer CLI.
  *
- * Installs the agent-orchestrator skill into ~/.claude/skills/agent-orchestrator
+ * Installs every skill this plugin ships into ~/.claude/skills/<name>
  * (same layout as install.sh). Idempotent: an existing install is skipped unless
  * --force. Zero dependencies.
  *
@@ -21,7 +21,7 @@ function usage() {
   console.log(`agent-stack installer
 
 Usage:
-  npx @ssheleg/agent-stack [--force]   install the agent-orchestrator skill
+  npx @ssheleg/agent-stack [--force]   install every skill this plugin ships
                                        into ~/.claude (skip existing unless --force)
   npx @ssheleg/agent-stack --help
 
@@ -67,20 +67,34 @@ function main(argv) {
     return 2;
   }
 
-  const skillSrc = path.join(ROOT, 'plugins/agent-stack/skills/agent-orchestrator');
-  if (!fs.existsSync(skillSrc)) {
-    console.error(`error: skill sources missing at ${skillSrc} — corrupted package?`);
+  const skillRoot = path.join(ROOT, 'plugins/agent-stack/skills');
+  if (!fs.existsSync(skillRoot)) {
+    console.error(`error: skill sources missing at ${skillRoot} — corrupted package?`);
+    return 1;
+  }
+
+  // Iterate rather than name one skill: a skill added to the plugin must not
+  // require an installer change to reach anybody.
+  const names = fs
+    .readdirSync(skillRoot, { withFileTypes: true })
+    .filter((e) => e.isDirectory())
+    .map((e) => e.name)
+    .sort();
+  if (!names.length) {
+    console.error(`error: no skills found under ${skillRoot} — corrupted package?`);
     return 1;
   }
 
   const home = os.homedir();
-  installOne(
-    'agent-orchestrator skill',
-    skillSrc,
-    path.join(home, '.claude', 'skills', 'agent-orchestrator'),
-    true,
-    force
-  );
+  for (const name of names) {
+    installOne(
+      `${name} skill`,
+      path.join(skillRoot, name),
+      path.join(home, '.claude', 'skills', name),
+      true,
+      force
+    );
+  }
   return 0;
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.4.0",
+  "version": "0.6.0",
   "description": "Production patterns for AI agent orchestrators — tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay — plus the wallet side of reselling LLM access. This package is the installer CLI.",
   "bin": {
     "agent-stack": "bin/agent-stack.js"

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -1,8 +1,8 @@
 {
   "name": "agent-stack",
   "displayName": "Agent Stack",
-  "description": "Production patterns for AI agent orchestrators: tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay — plus the wallet side of reselling LLM access.",
-  "version": "0.4.0",
+  "description": "Two skills: agent-orchestrator \u2014 tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals \u2014 run/trace/thread evals, judges, and fixtures grown from production.",
+  "version": "0.6.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"
@@ -19,6 +19,9 @@
     "openrouter",
     "llm-billing",
     "claude-code",
-    "cursor"
+    "cursor",
+    "evals",
+    "llm-judge",
+    "observability"
   ]
 }

package/plugins/agent-stack/skills/agent-evals/SKILL.md

@@ -0,0 +1,230 @@
+---
+name: agent-evals
+description: >-
+  Use when measuring whether an agent actually works — building an eval suite, judging a
+  trajectory rather than a final answer, turning production traces into regression
+  fixtures, calibrating an LLM judge against human labels, or gating a release on offline
+  evals. Covers the three observability primitives (run, trace, thread) crossed with three
+  eval granularities (single-step, full-turn, multi-turn), the offline/online/ad-hoc
+  timing axis, pass-fail rubrics over scalar scores, cheap code checks before model
+  judges, simulated users with adversarial personas, annotation queues, and what to
+  instrument so any of it is possible. Triggers - "agent eval", "eval suite", "LLM judge",
+  "regression fixture", "trajectory eval", "is the agent getting better", "эвалы агента",
+  "оценка агента", "LLM-судья", "регрессионный набор", "как проверить агента". Not for
+  unit tests of ordinary code, or for benchmarking a model.
+license: MIT
+---
+
+# Agent evals — proving the thing behaves
+
+An agent's behaviour does not exist in its source. The code says what it is *allowed* to
+do; only a run says what it *did*. So the artifact under test is the execution record,
+and the suite is grown from production rather than authored up front.
+
+Three claims follow, and they are what makes this different from testing ordinary code:
+
+- **You are testing reasoning, not code paths**, so one granularity is never enough.
+- **Every natural-language input is unique**, so the edge cases cannot be enumerated
+  offline. Production is not only where you catch what you missed — it is where you
+  discover what to test for.
+- **Traces become test cases.** The suite grows from what actually happened.
+
+---
+
+## 1. Three primitives
+
+| Primitive | Is | Carries |
+|---|---|---|
+| **Run** | one model call | the complete prompt — instructions, tools, context — and what came back |
+| **Trace** | one full execution | every run, every tool call with arguments and results, nested to show how steps relate |
+| **Thread** | many traces in one session | multi-turn context in order, **state evolution** (memory, files, artifacts), and elapsed time |
+
+The thread level is the one most systems lack, and it is where a whole class of defect
+lives: a bad memory write at turn 6 surfaces as a wrong answer at turn 11, and neither
+the run nor the trace view can localise it.
+
+**Precondition for all of this: traces are durable and queryable.** A live event stream
+renders a progress bar and evaporates. If you cannot fetch last week's low-scoring runs
+by id, nothing below is available to you — see §7.
+
+---
+
+## 2. Three granularities
+
+Each primitive gets its own kind of assertion.
+
+### Single-step — validates a run
+
+Fixture is a serialized run: prompt, tool schemas, context. Assert the decision at that
+point — tool name, argument shape.
+
+> "Schedule a meeting with Harrison tomorrow morning", with `find_meeting_times`,
+> `schedule_meeting` and `send_email` available, must call `find_meeting_times` first.
+
+Cheap, deterministic, CI-blocking. **Precondition: a stable agent architecture.** These
+break on a graph refactor, and a suite that fails on every refactor gets deleted.
+
+### Full-turn — validates a trace
+
+Assert on three axes at once, with three different mechanisms:
+
+| Axis | Assert | With |
+|---|---|---|
+| Trajectory | tool-call sequence — `read_file` → `edit_file` → `run_tests` | set/subset/order matchers |
+| Final response | quality, tone, policy compliance | rubric or judge |
+| **State change** | the memory row exists, the file was written, the artifact is there | direct inspection of the side effect |
+
+The third axis is the one people forget. **Assert on side effects, not only on prose** —
+an agent that says it saved the preference and did not is a pass on two axes out of three.
+
+Easiest inputs to generate, hardest outputs to validate automatically.
+
+### Multi-turn — validates a thread
+
+A scripted turn sequence with a **checkpoint after every turn and fail-fast on
+deviation**. Without that, turn 3 goes off the rails and turns 4–10 assert nothing while
+still reporting a result.
+
+> Turn 1: "I prefer Python over JavaScript." Turn 3's output must still be Python.
+
+Hardest to implement well. Start here only for behaviour that is genuinely about memory
+across turns.
+
+**Production suites combine all three.** One vendor reports about half of theirs sitting
+at single-step — recorded as their observation, not as a target to hit.
+
+---
+
+## 3. Offline, online, ad-hoc
+
+| When | Reference | Blocks a release? | Answers |
+|---|---|---|---|
+| **Offline** | a dataset, ground truth optional | **yes** — this is the gate | did my change break what used to work |
+| **Online** | none — **definitionally reference-free** | no | is production drifting |
+| **Ad-hoc** | none | no | what is actually happening out there |
+
+**Offline is necessary and not sufficient.** A green suite proves you did not regress the
+cases you already know about. It cannot prove the agent handles what nobody thought of,
+because that input is not in the dataset — which is what the online tier is for.
+
+Online evaluators fire on trace ingestion and check what needs no expected answer:
+trajectory anomalies, step-count and latency trends, judge scores, error rates. Route
+them over all traces, a sample, or a filtered subset; the sampling rate is a cost
+decision, not a correctness one.
+
+Ad-hoc is exploratory analysis over stored traces — clustering to surface failure modes
+nobody predefined. A dashboard tracks metrics you chose in advance; this finds the ones
+you did not.
+
+---
+
+## 4. Rubrics beat scores
+
+Generic metrics — helpfulness, naturalness, completeness — produce numbers and no
+decision. A 3.4 out of 5 on "helpfulness" tells you nothing about what to change.
+
+**Write narrow, behaviour-specific pass/fail rubrics, and write them with the people who
+own the behaviour.** Each failure must point at one thing: a prompt, a tool description,
+a workflow step, a missing capability.
+
+A rubric that works, in full:
+
+> **Escalation.** On a request for a human: push back once, escalate on the repeat.
+> **Fails if** it escalates immediately · refuses after the second request · escalates
+> before providing information it already had · continues several turns past the point it
+> is clearly not helping.
+
+Note the shape: one behaviour, an explicit pass condition, and an enumerated failure list.
+That is what makes a judge reproducible and a disagreement resolvable.
+
+---
+
+## 5. Judges
+
+**Cheap checks first.** Schema validation, exact match, format conformity, business-rule
+assertions, tool-call correctness — all deterministic, all faster and cheaper than a model
+call. Send to a judge only what cannot be decided by code.
+
+**Judge the trajectory, not just the answer.** Right tools, right order, right arguments.
+An agent that reaches a correct answer through three wrong tool calls is a latent outage.
+
+**Calibrate the judge before trusting it.** Collect human labels on the same traces,
+measure agreement, iterate the judge prompt until agreement is high — *then* let it score
+unattended. An uncalibrated judge is an opinion with a number attached, and shipping on it
+is exactly the failure of grading instead of measuring.
+
+**Some things a judge cannot do.** Plausible-but-wrong domain output — an invented legal
+citation, a subtly wrong SQL join — reads as correct to a general judge. Route those to a
+domain expert and accept that this tier stays human.
+
+---
+
+## 6. The corpus grows from production
+
+Never author the suite up front. Every production failure and every thumbs-down becomes a
+fixture:
+
+1. Capture the state at the failure point.
+2. Minimise it to the smallest input that still reproduces.
+3. Add it to the tier that isolates it — step, turn or thread.
+4. **It stays in the suite permanently.** A fixed bug that silently returns is the defect
+   this rule exists to prevent.
+
+Two dataset shapes come out of review:
+
+- **Ground truth** — the reviewer writes the correct output; the suite asserts equality.
+- **Criteria-based** — for open-ended work, the reviewer labels dimensions (relevance,
+  completeness, tone) instead of an exact answer.
+
+**The annotation queue** is what feeds both: filters route a subset of traces to humans —
+low automated score, thumbs-down, a feature area, a cluster. Two reviewer roles, and
+mixing them wastes both: generalists judge surface quality, domain experts judge
+correctness only they can see.
+
+**Simulated users, if you generate inputs.** A model playing a customer is articulate,
+patient and cooperative, and inflates every pass rate. Fine-tune it on real user
+transcripts and add adversarial personas — the refund-seeker, the AI-sceptic, the one who
+wants a human immediately. Making the simulated user worse makes the offline result
+predictive.
+
+---
+
+## 7. What to instrument first
+
+None of the above runs without these, and they are the part people skip:
+
+- **A durable trace store, queryable by id, filterable by score and time.** The live
+  stream is a view over it, never the record itself.
+- **Scores as first-class records bound to a run**: `run_id`, key, value, and a `source`
+  of `human` | `llm_judge` | `code_check`. A score with no source cannot be calibrated,
+  audited, or trusted differently from its neighbours.
+- **Whole prompts, not just messages** — instructions, tool schemas and context as they
+  were sent. A fixture cannot be replayed from a summary.
+- **State snapshots at turn boundaries**, so a thread test can assert what carried.
+
+---
+
+## Checklist
+
+- [ ] Traces durable and queryable by id, not only streamed
+- [ ] Scores bound to runs, each with a source
+- [ ] Single-step fixtures for the decisions that must not drift
+- [ ] Full-turn assertions on trajectory **and** final response **and** state change
+- [ ] Multi-turn scripts with a checkpoint after every turn, failing fast
+- [ ] Offline suite as the release gate; online evaluators reference-free and non-blocking
+- [ ] Pass/fail rubrics with enumerated failure conditions, written with behaviour owners
+- [ ] Code checks before model judges
+- [ ] Judge calibrated against human labels before it is trusted
+- [ ] Domain-expert review for output a general judge cannot grade
+- [ ] Every production failure minimised into a permanent fixture
+- [ ] Annotation queue with filters, and the two reviewer roles kept separate
+- [ ] Simulated users trained on real transcripts, with adversarial personas
+
+---
+
+## Related
+
+Building the agent this measures is the **`agent-orchestrator`** skill, shipped in the
+same plugin — named rather than linked, because a packager may ship either directory
+alone. Its context-engineering reference covers the context-pressure behaviour that
+trajectory assertions most often catch drifting.

package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md

@@ -490,11 +490,14 @@ transcript.
 
 ## References
 
-Load these when the task reaches them — the checklist above is the map, these
-are the territory.
+The checklist above is the map, these are the territory. Each file opens with its
+own **Load this when** line — the authoritative trigger lives there, so this table
+stays an index and the two cannot drift apart.
 
 | File | Read it when |
 |---|---|
-| [`references/patterns.md`](references/patterns.md) | you need the **data models and algorithms**: message and result protocols, pipeline models, the SQL validation loop, context-window sizes and token estimation, learning-extraction heuristics, confidence lifecycle, fuzzy dedup, conflict resolution, cross-resource transfer, the no-LLM suggestion engine |
-| [`references/context-engineering.md`](references/context-engineering.md) | the loop is **running out of window**: the five-rung compaction ladder, the tool-pair boundary invariant, typed carryover attachments, tool-output offload, token estimation, the compaction circuit breaker, sub-agent isolation, and how to pick your own constants |
-| [`references/llm-proxy-billing.md`](references/llm-proxy-billing.md) | the product **resells LLM access**: tiered wallets and where markup applies, two-phase commit against a provider API with compensating transactions, advisory locking, optimistic concurrency for reclaims, spend-delta polling and its three cases, budget/loop/auto-pause guardrails, per-tenant key lifecycle and healing, the refund waterfall, model routing |
+| [`references/patterns.md`](references/patterns.md) | you need the **data models and algorithms** under the body |
+| [`references/context-engineering.md`](references/context-engineering.md) | the loop is **running out of window** |
+| [`references/runtime.md`](references/runtime.md) | the agent must **survive a crash, a pause, a second message or a schedule** |
+| [`references/governance.md`](references/governance.md) | the question is **permission, not cost** — what it may do, and how you prove it |
+| [`references/llm-proxy-billing.md`](references/llm-proxy-billing.md) | the product **resells LLM access** |

package/plugins/agent-stack/skills/agent-orchestrator/references/governance.md

@@ -0,0 +1,138 @@
+# Governance — what the agent is allowed to do
+
+**Load this when** the question is permission rather than money: which model may see
+which data, which tool may run against production, what leaves the infrastructure
+boundary, and how you prove afterwards that the control was on. `llm-proxy-billing.md`
+answers *what did this cost and who pays* — this answers *should it have happened at
+all*, and the two are separate systems that share an audit row.
+
+The claim worth keeping: **for an agent, the greatest risk is usually not what the model
+says, but what the agent can do.** Content filters are aimed at the first. Most real
+damage comes through the second.
+
+## Contents
+
+- Four boundaries, four control sets
+- Guardrails, and their honest limit
+- Where a control runs: before or after
+- The audit row
+- Cost attribution as a hierarchy
+- Failover must land somewhere approved
+- Fail-open or fail-closed, decided by risk
+- Blast radius: sandboxes and credentials
+
+## Four boundaries, four control sets
+
+An agent crosses four kinds of boundary, and treating them as one is why "we have
+guardrails" so often means only the first.
+
+| Boundary | The risk | Controls that fit |
+|---|---|---|
+| **Model call** | cost; prompt content reaching a provider's logs | spend limits, redaction, provider routing, data-residency choice |
+| **Tool call** | an unintended action on a real system | per-tool authorisation, argument validation, an audit row per invocation |
+| **External server call** (MCP and similar) | data leaving your infrastructure boundary | allowlist of servers, logging, explicit scope per server |
+| **Agent-to-agent** | errors compounding down a chain; context passed on without authority | tracing across hops, and **policy enforced at each hop**, not only at the entrance |
+
+The last one is the one most designs miss: a sub-agent that inherits its caller's
+authority silently widens every permission the caller had.
+
+## Guardrails, and their honest limit
+
+The content-layer checks worth having, roughly in order of reliability:
+
+- **Structured secrets and identifiers** — API keys, tokens, card numbers, national ids.
+  Pattern-matched, high precision, cheap. Run these first.
+- **Unstructured personal data** — names, locations, affiliations. Needs entity
+  recognition; precision drops.
+- **Injection and jailbreak attempts** — classifier-based, adversarial by nature, and
+  the arms race is not winnable by pattern alone.
+- **Groundedness** — does the answer follow from the retrieved material. Runs on output,
+  costs a model call, and is the least reliable of the four.
+
+**Then the rule that makes the list honest: guardrails reduce risk, they do not
+eliminate it.** Every item above is probabilistic. So for anything consequential —
+money moving, data deleted, a message sent to a customer, a deploy — the control is a
+**deterministic limit or a human**, never a classifier's confidence. A guardrail is a
+filter on the way to a decision, not the decision.
+
+A useful signal on top: **a sudden spike in guardrail violations is usually the first
+sign that something upstream is wrong** — a prompt change, a new data source, an agent
+in a loop. Alert on the rate, not just on the individual hit.
+
+## Where a control runs: before or after
+
+Almost everything belongs **before** the call: redaction, secret detection, provider
+routing, rate and spend limits, tool authorisation. A control that runs after the
+request has left has already failed at the thing it was for.
+
+**After** the call, only what needs the output: groundedness, moderation of generated
+text, structured-output validation.
+
+Two consequences: pre-call controls sit on the latency path, so they must be cheap
+enough to run every time; and a control that can only run post-call must be paired with
+something that can undo or withhold the result.
+
+## The audit row
+
+An audit row exists to answer a question months later, when the person who ran the agent
+is unavailable. It needs:
+
+- **who** — the identity that ran the workload, and separately the identity that last
+  changed the policy
+- **what** — the action, its arguments in redacted form, and the outcome
+- **which policy version applied** — this is the field everyone omits and the one that
+  makes the record evidence. "The control was on" is unprovable without it; a policy
+  that changed twice since is unfalsifiable without it.
+- **which model and which tools were reached**, including through sub-agents
+- **when**, at a precision that survives clock skew between services
+
+The rule to hold: **an audit trail written for reconciliation answers "where did the
+money go"; an audit trail written for governance answers "prove the control was
+applied".** They are different queries and the second needs the policy version.
+
+## Cost attribution as a hierarchy
+
+The billing reference tracks spend per tenant. Governance needs it resolvable up a
+chain: **organisation → business unit → team → credential → individual**. Not because
+finance asks, but because the question that actually gets asked in an incident is "which
+team's agent did this", and a flat tenant id cannot answer it.
+
+Limits belong at more than one level too — a per-credential cap does not stop twenty
+credentials in one team from draining a budget together.
+
+## Failover must land somewhere approved
+
+The router in the body falls back to the next healthy provider. Governance adds one
+constraint: **the fallback must be policy-equivalent, not merely available.**
+
+A chain that silently fails over to a provider with different data handling, a different
+jurisdiction, or a different retention policy has moved the data somewhere nobody
+approved — and it does it precisely during an incident, when nobody is reading logs. Tag
+each provider with the policy it satisfies, and let the fallback chain filter on the tag
+before it filters on health.
+
+## Fail-open or fail-closed, decided by risk
+
+When the control plane itself is unavailable — the guardrail service times out, the
+policy store is unreachable — the system either proceeds without the check or refuses.
+**Both answers are correct for different workloads, and neither is a default.**
+
+- Fail-**open** for a low-risk, high-volume path where refusing is the bigger harm.
+- Fail-**closed** for anything consequential.
+
+Write the choice down per workload, and make the control plane itself redundant enough
+that the choice is rarely exercised: timeouts, load balancing, and a health check that
+distinguishes "slow" from "gone".
+
+## Blast radius: sandboxes and credentials
+
+When an agent runs code, two rules carry most of the weight:
+
+- **A sandbox protects the host, not the sandbox.** Anything the agent can reach *from
+  inside* is still reachable — network egress, mounted paths, environment. Restrict
+  egress explicitly and allowlist commands rather than denylisting.
+- **Credentials never enter the sandbox.** Put a proxy in front that injects them per
+  request, so a prompt injection that dumps the environment gets nothing worth having.
+
+Ephemeral is the default: create on demand, tear down after, never reuse across tenants.
+A long-lived sandbox accumulates state that nobody audits.

package/plugins/agent-stack/skills/agent-orchestrator/references/llm-proxy-billing.md

@@ -1,5 +1,12 @@
 # Reselling LLM access — metering, wallets and guardrails
 
+**Load this when** the product resells LLM access: tiered wallets and the single
+boundary where markup applies, two-phase commit across a database and a provider API
+with compensating transactions, advisory locking, optimistic concurrency for reclaims,
+spend-delta polling and its three cases, budget / loop / auto-pause guardrails,
+per-tenant key lifecycle and healing, the refund waterfall, and model-routing
+precedence.
+
 When your product gives users LLM access and bills for it, you are running a
 proxy with a wallet behind it. The failure modes are not model failures: they
 are **double-credited transfers**, **spend you discovered after it happened**,

package/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md

@@ -1,6 +1,10 @@
 # Agent Orchestrator — Reference Guide
 
-Extended patterns, data models, and implementation details.
+**Load this when** you need the data models and algorithms under the body: message
+and result protocols, pipeline models, the SQL validation loop, context-window sizes
+and token estimation, learning-extraction heuristics, the confidence lifecycle, fuzzy
+deduplication, conflict resolution, cross-resource transfer, and the suggestion engine
+that costs no LLM call.
 
 ## Contents
 

package/plugins/agent-stack/skills/agent-orchestrator/references/runtime.md

@@ -0,0 +1,151 @@
+# Runtime — what keeps an agent alive between requests
+
+**Load this when** the agent must survive things a single request does not: a crash
+mid-run, a human who has to approve before it continues, a user who sends a second
+message while the first is still working, a dropped connection, a schedule. The body's
+loop is the *harness* — what the model is given to work with. This is the layer beneath
+it, and most orchestrators assume it exists rather than specify it.
+
+The split is worth keeping in mind while reading: **a good harness makes an agent
+capable, a good runtime makes it deployable.** They fail differently, and a team that
+has only built the first one discovers the second in production.
+
+## Contents
+
+- Durability, and where our own asymmetry was
+- The interrupt/resume contract
+- Double-texting: four policies
+- Streaming that survives a dropped connection
+- Time travel and forking
+- Scheduled and sleep-time work
+- Middleware: the seven concerns, unwelded
+
+## Durability, and where our own asymmetry was
+
+**Checkpoint every iteration of the loop, not just the stages a human reviews.**
+
+The body's multi-stage pipeline persists at each `stage.checkpoint` and can resume from
+a `pipeline_run_id`. The simple tool-calling path persists nothing: a crash, a deploy or
+a killed worker loses the entire run, including the tool calls that already cost money
+and time. That asymmetry is a defect, not a design — the simple path is the one that
+runs most often.
+
+What a checkpoint holds: the message array, the iteration counter, accumulated token
+usage, the carryover state (see `context-engineering.md`), and whatever the sub-agents
+have returned so far. Keyed by a thread id that acts as a cursor into the run.
+
+Two properties earn their cost:
+
+- **Resume at the point of failure**, not at the last human review. The difference is
+  whole minutes of re-executed tool calls.
+- **A pause frees the worker.** An agent waiting for a human should hold no process. If
+  waiting costs a worker, long approvals are quietly expensive and teams stop using them.
+
+## The interrupt/resume contract
+
+The body has two mechanisms for one idea: `ask_user` raises a clarification error, and a
+pipeline checkpoint returns a paused result. **They should be one contract.**
+
+- **Interrupt** — the run stops at a named point, persists its state, and surfaces a
+  payload describing what it needs: a question, a plan to approve, a destructive action
+  to confirm.
+- **Resume** — the caller returns a decision, and execution continues *from that point*
+  with the decision in scope. Not a fresh run that re-derives its way back.
+
+One contract means one persistence format, one place a UI has to understand, and one
+answer to "what happens if nobody replies for a day".
+
+**When an interrupt is mandatory** rather than optional: any action that is
+hard to reverse or outward-facing. Content-level guardrails are probabilistic — see
+`governance.md` — so consequential actions need a deterministic limit or a human, not a
+classifier's opinion.
+
+## Double-texting: four policies
+
+A user sends a second message while the first is still running. This has four possible
+answers, and a system that never chose one has chosen the worst by accident:
+
+| Policy | Behaviour | Fits |
+|---|---|---|
+| **Enqueue** | finish the current run, then start the new one | a task where the first answer is still wanted |
+| **Reject** | refuse the second message while busy | expensive or transactional runs |
+| **Interrupt** | stop the current run, start the new one, keep what was produced | conversational agents — the usual default |
+| **Rollback** | discard the current run *including its input*, start clean | the user is correcting themselves |
+
+The difference between interrupt and rollback is what the transcript looks like
+afterwards, and it is worth deciding deliberately: interrupt leaves a half-finished turn
+in history that the next prompt will see.
+
+## Streaming that survives a dropped connection
+
+Four things are worth streaming, and they are not the same thing:
+
+1. **State snapshots** after each step — for a UI that renders the whole picture.
+2. **State deltas** — the same, cheaper.
+3. **Tokens** — the typing effect.
+4. **Custom events** — domain progress: "queried 3 of 7 sources".
+
+The body's tracker emits an in-memory feed. Two properties turn it into something a
+client can rely on:
+
+- **Every event carries a monotonic id**, and a client reconnecting sends the last id it
+  saw. The server replays from there. Without this, a dropped connection during a
+  ninety-second run means the user watches nothing and then gets an answer from nowhere.
+- **The feed is a view over the durable trace, not the record itself.** If the only copy
+  of what happened is a stream nobody stored, evaluation is impossible — see the
+  `agent-evals` skill, which cannot function without it.
+
+## Time travel and forking
+
+Once every iteration is checkpointed, one capability follows nearly free: **pick a past
+checkpoint, modify the state, and resume from it.** The original history stays; the
+modified run forks.
+
+This is the debugging tool the loop otherwise lacks. "Why did it call that tool?" is
+answerable by rewinding to the step before, changing one thing, and running forward
+again — through the real loop, with real model calls and real tools, rather than a
+reconstruction that may not share the bug.
+
+It is also how a failed production run becomes a regression fixture: fork at the failure
+point, minimise, save the state as the fixture's input.
+
+## Scheduled and sleep-time work
+
+Not all agent work starts with a user. Two shapes, and the distinction matters:
+
+- **Stateful schedule** — each run appends to an existing thread, so the agent remembers
+  the previous ones. A daily briefing that should not repeat itself.
+- **Stateless schedule** — each run starts a fresh thread. A monitor that must not drift
+  on yesterday's context.
+
+Scheduled runs need the same retry and tracing as interactive ones, and one extra rule:
+**a schedule that fails silently is worse than no schedule.** Failures must reach a human
+through something other than the absence of a result.
+
+**Sleep-time compute** is the useful pattern on top: work done between conversations —
+consolidating memory, refreshing an index, pre-computing what tomorrow's first question
+will need. It is also where memory consolidation belongs when the hot path is too busy
+for it.
+
+## Middleware: the seven concerns, unwelded
+
+The body's loop hand-codes seven cross-cutting concerns inside itself: retry, provider
+fallback, summarisation, human-in-the-loop, tool-call limits, redaction, and moderation.
+Each is correct and none is separable — changing the retry policy means editing the loop.
+
+The alternative is ordered interceptors at four points:
+
+| Hook | Runs | Typical use |
+|---|---|---|
+| `before_model` | before the request is built | inject context, redact, enforce a budget |
+| `wrap_model_call` | around the call | retry, fallback, timing, cost accounting |
+| `wrap_tool_call` | around each tool | authorisation, rate limits, argument validation |
+| `after_model` | on the response | moderation, structured-output repair, guardrails |
+
+The hook names are borrowed vocabulary; the shape is generic. What it buys is
+composition — a tool-call limit is one interceptor, not a counter threaded through three
+functions — and testability: an interceptor is a unit, the loop is not.
+
+The trap: **order is semantics.** Redaction after summarisation redacts a summary that
+already leaked. Write the order down where the list is defined, not in the head of
+whoever wrote it.