@ssheleg/agent-stack 0.5.0 → 0.6.1

package/CHANGELOG.md

@@ -4,6 +4,70 @@ All notable changes to this project are documented here.
 Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning: [SemVer](https://semver.org/spec/v2.0.0.html).
 
+## v0.6.1 — 2026-08-13
+
+Two CI fixes that had been sitting on `main` unreleased ship here, and the half of
+standing instruction #6 that was missing arrives with them.
+
+### Fixed
+
+- **Two plants used `sed -i` and were no-ops on macOS.** BSD sed requires an argument to
+  `-i`, so they errored and changed nothing; they could only ever be exercised in CI.
+  Converted to Python — the rule `task-pipeline` has enforced on itself for months.
+- **Every plant now asserts that it changed the file.** The 2026-08-13 commit
+  *anchor the description plant on the file's shape, not its wording* applied one half of
+  instruction #6; this applies the corollary to the rest, so a plant that stops landing
+  says `PLANT DID NOT LAND: <why>` rather than reporting a healthy guard as broken.
+- Also shipping, previously merged and unreleased: *one CHANGELOG-extraction pattern for
+  the whole family* (B-11), which is why this release can be cut at all.
+
+All six plants verified by running them locally: each lands, and each makes the
+validator fail.
+
+## [0.6.0] — 2026-08-12
+
+### Added
+
+- **`references/runtime.md`** — the layer most orchestrators assume rather than
+  specify, and the one this skill was quietly missing. **Checkpoint every iteration,
+  not just the stages a human reviews**: the multi-stage path persisted and could
+  resume, the simple tool-calling path persisted nothing, so a crash lost the run that
+  executes most often — an asymmetry, not a design. Then one interrupt/resume contract
+  instead of the two mechanisms the body had for one idea (`ask_user` and a stage
+  checkpoint); the four double-texting policies and why interrupt and rollback differ
+  in what the transcript looks like afterwards; streaming with event ids so a dropped
+  connection rejoins instead of watching nothing for ninety seconds; forking a past
+  checkpoint, which debugs through the real loop rather than a reconstruction that may
+  not share the bug; stateful versus stateless schedules; and the seven cross-cutting
+  concerns welded into the loop pulled out as ordered interceptors — where **order is
+  semantics**, because redaction after summarisation redacts a summary that already
+  leaked.
+
+- **`references/governance.md`** — permission, where `llm-proxy-billing.md` is cost.
+  **The greatest risk is usually not what the model says but what the agent can do**,
+  so the four boundaries get four control sets: model call, tool call, external server,
+  and agent-to-agent — the last being the one designs miss, since a sub-agent
+  inheriting its caller's authority silently widens every permission. The guardrail
+  taxonomy in order of reliability, and its honest limit: every content check is
+  probabilistic, so anything consequential takes a deterministic limit or a human, never
+  a classifier's confidence. Why an audit row without a **policy version** cannot prove
+  a control was applied. Cost attribution as a hierarchy, because "which team's agent
+  did this" is unanswerable from a flat tenant id. Failover that must be
+  policy-equivalent rather than merely available — a chain that silently fails into
+  another jurisdiction does it precisely when nobody is reading logs. Fail-open versus
+  fail-closed per workload. And blast radius: a sandbox protects the host, not the
+  sandbox, and credentials never enter it.
+
+### Changed
+
+- **The References table is an index again.** Each reference now opens with its own
+  `Load this when` line, so the trigger has exactly one home and the table cannot drift
+  from the files it points at. Compressing it returned ~100 tokens of body budget, which
+  is what paid for two new rows: the body sits at 489 lines / ~4883 tokens against
+  500 / 5000.
+- README describes two skills and five references, and its trigger section covers the
+  evals skill and the permission surface, not only the orchestrator and the wallet.
+
 ## [0.5.0] — 2026-08-12
 
 ### Added

package/README.md

@@ -10,7 +10,7 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills).
 ## What is in here
 
 Two skills — `agent-orchestrator` for building one, `agent-evals` for proving it
-behaves — and three references the first loads on demand.
+behaves — and five references the first loads on demand.
 
 **The orchestrator** (`SKILL.md`) — what the agent reads first:
 
@@ -46,6 +46,23 @@ boundary rather than summarized, tool-output offload to a file, token estimation
 and the direction it errs, the compaction circuit breaker, sub-agent context
 isolation, and how to choose constants for your own window.
 
+**`references/runtime.md`** — what keeps an agent alive between requests, which
+most orchestrators assume rather than specify: checkpointing every iteration and
+not just the stages a human reviews, one interrupt/resume contract instead of two
+mechanisms for one idea, the four double-texting policies, streaming a dropped
+connection can rejoin, forking a past checkpoint to debug through the real loop,
+stateful versus stateless schedules, and the seven cross-cutting concerns pulled
+out of the loop into ordered interceptors — where order is semantics.
+
+**`references/governance.md`** — permission rather than cost. The four boundaries
+an agent crosses (model, tool, external server, agent-to-agent), each with its own
+control set; the guardrail taxonomy and its honest limit — every content check is
+probabilistic, so anything consequential gets a deterministic limit or a human;
+why an audit row without a policy version cannot prove a control was applied;
+cost attribution as a hierarchy; failover that must land somewhere approved rather
+than merely available; fail-open versus fail-closed as a per-workload decision;
+and blast radius — a sandbox protects the host, not the sandbox.
+
 **`references/patterns.md`** — the data models and algorithms underneath:
 message and result protocols, pipeline models, the SQL validation loop,
 context-window sizes and token estimation, learning-extraction heuristics,
@@ -71,7 +88,7 @@ waterfall, and model-routing precedence.
 /plugin install agent-stack@agent-stack
 ```
 
-**npm installer** — copies the skill into `~/.claude/skills/`:
+**npm installer** — copies both skills into `~/.claude/skills/`:
 
 ```bash
 npx @ssheleg/agent-stack
@@ -95,13 +112,19 @@ Restart your agent afterwards — skills load at session start.
 
 ## When it triggers
 
-Building an agent system, an orchestrator, an LLM-powered tool, a chatbot with
-tool use, or an AI pipeline. Also when the work is the money side: metering
-usage, per-tenant keys, spend tracking, budget limits, loop detection.
+`agent-orchestrator`: building an agent system, an orchestrator, an LLM-powered
+tool, a chatbot with tool use, or an AI pipeline. Also the money side — metering
+usage, per-tenant keys, spend tracking, budget limits, loop detection — and the
+permission side: what a tool may reach, what leaves the boundary, and what an
+audit row has to carry to prove a control was on.
+
+`agent-evals`: measuring whether the result behaves. Building a suite, judging a
+trajectory rather than a final answer, turning a production failure into a
+permanent fixture, calibrating a judge, gating a release on offline evals.
 
-It does **not** trigger for a single LLM call in a script, or for prompt
-wording — that is not an orchestrator, and pulling 1200 lines of doctrine for it
-is how a skill teaches you to route around it.
+Neither triggers for a single LLM call in a script or for prompt wording — that
+is not an orchestrator, and pulling this much doctrine for it is how a skill
+teaches you to route around it.
 
 ---
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.5.0",
+  "version": "0.6.1",
   "description": "Production patterns for AI agent orchestrators — tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay — plus the wallet side of reselling LLM access. This package is the installer CLI.",
   "bin": {
     "agent-stack": "bin/agent-stack.js"

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Two skills: agent-orchestrator \u2014 tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals \u2014 run/trace/thread evals, judges, and fixtures grown from production.",
-  "version": "0.5.0",
+  "version": "0.6.1",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md

@@ -490,11 +490,14 @@ transcript.
 
 ## References
 
-Load these when the task reaches them — the checklist above is the map, these
-are the territory.
+The checklist above is the map, these are the territory. Each file opens with its
+own **Load this when** line — the authoritative trigger lives there, so this table
+stays an index and the two cannot drift apart.
 
 | File | Read it when |
 |---|---|
-| [`references/patterns.md`](references/patterns.md) | you need the **data models and algorithms**: message and result protocols, pipeline models, the SQL validation loop, context-window sizes and token estimation, learning-extraction heuristics, confidence lifecycle, fuzzy dedup, conflict resolution, cross-resource transfer, the no-LLM suggestion engine |
-| [`references/context-engineering.md`](references/context-engineering.md) | the loop is **running out of window**: the five-rung compaction ladder, the tool-pair boundary invariant, typed carryover attachments, tool-output offload, token estimation, the compaction circuit breaker, sub-agent isolation, and how to pick your own constants |
-| [`references/llm-proxy-billing.md`](references/llm-proxy-billing.md) | the product **resells LLM access**: tiered wallets and where markup applies, two-phase commit against a provider API with compensating transactions, advisory locking, optimistic concurrency for reclaims, spend-delta polling and its three cases, budget/loop/auto-pause guardrails, per-tenant key lifecycle and healing, the refund waterfall, model routing |
+| [`references/patterns.md`](references/patterns.md) | you need the **data models and algorithms** under the body |
+| [`references/context-engineering.md`](references/context-engineering.md) | the loop is **running out of window** |
+| [`references/runtime.md`](references/runtime.md) | the agent must **survive a crash, a pause, a second message or a schedule** |
+| [`references/governance.md`](references/governance.md) | the question is **permission, not cost** — what it may do, and how you prove it |
+| [`references/llm-proxy-billing.md`](references/llm-proxy-billing.md) | the product **resells LLM access** |

package/plugins/agent-stack/skills/agent-orchestrator/references/governance.md

@@ -0,0 +1,138 @@
+# Governance — what the agent is allowed to do
+
+**Load this when** the question is permission rather than money: which model may see
+which data, which tool may run against production, what leaves the infrastructure
+boundary, and how you prove afterwards that the control was on. `llm-proxy-billing.md`
+answers *what did this cost and who pays* — this answers *should it have happened at
+all*, and the two are separate systems that share an audit row.
+
+The claim worth keeping: **for an agent, the greatest risk is usually not what the model
+says, but what the agent can do.** Content filters are aimed at the first. Most real
+damage comes through the second.
+
+## Contents
+
+- Four boundaries, four control sets
+- Guardrails, and their honest limit
+- Where a control runs: before or after
+- The audit row
+- Cost attribution as a hierarchy
+- Failover must land somewhere approved
+- Fail-open or fail-closed, decided by risk
+- Blast radius: sandboxes and credentials
+
+## Four boundaries, four control sets
+
+An agent crosses four kinds of boundary, and treating them as one is why "we have
+guardrails" so often means only the first.
+
+| Boundary | The risk | Controls that fit |
+|---|---|---|
+| **Model call** | cost; prompt content reaching a provider's logs | spend limits, redaction, provider routing, data-residency choice |
+| **Tool call** | an unintended action on a real system | per-tool authorisation, argument validation, an audit row per invocation |
+| **External server call** (MCP and similar) | data leaving your infrastructure boundary | allowlist of servers, logging, explicit scope per server |
+| **Agent-to-agent** | errors compounding down a chain; context passed on without authority | tracing across hops, and **policy enforced at each hop**, not only at the entrance |
+
+The last one is the one most designs miss: a sub-agent that inherits its caller's
+authority silently widens every permission the caller had.
+
+## Guardrails, and their honest limit
+
+The content-layer checks worth having, roughly in order of reliability:
+
+- **Structured secrets and identifiers** — API keys, tokens, card numbers, national ids.
+  Pattern-matched, high precision, cheap. Run these first.
+- **Unstructured personal data** — names, locations, affiliations. Needs entity
+  recognition; precision drops.
+- **Injection and jailbreak attempts** — classifier-based, adversarial by nature, and
+  the arms race is not winnable by pattern alone.
+- **Groundedness** — does the answer follow from the retrieved material. Runs on output,
+  costs a model call, and is the least reliable of the four.
+
+**Then the rule that makes the list honest: guardrails reduce risk, they do not
+eliminate it.** Every item above is probabilistic. So for anything consequential —
+money moving, data deleted, a message sent to a customer, a deploy — the control is a
+**deterministic limit or a human**, never a classifier's confidence. A guardrail is a
+filter on the way to a decision, not the decision.
+
+A useful signal on top: **a sudden spike in guardrail violations is usually the first
+sign that something upstream is wrong** — a prompt change, a new data source, an agent
+in a loop. Alert on the rate, not just on the individual hit.
+
+## Where a control runs: before or after
+
+Almost everything belongs **before** the call: redaction, secret detection, provider
+routing, rate and spend limits, tool authorisation. A control that runs after the
+request has left has already failed at the thing it was for.
+
+**After** the call, only what needs the output: groundedness, moderation of generated
+text, structured-output validation.
+
+Two consequences: pre-call controls sit on the latency path, so they must be cheap
+enough to run every time; and a control that can only run post-call must be paired with
+something that can undo or withhold the result.
+
+## The audit row
+
+An audit row exists to answer a question months later, when the person who ran the agent
+is unavailable. It needs:
+
+- **who** — the identity that ran the workload, and separately the identity that last
+  changed the policy
+- **what** — the action, its arguments in redacted form, and the outcome
+- **which policy version applied** — this is the field everyone omits and the one that
+  makes the record evidence. "The control was on" is unprovable without it; a policy
+  that changed twice since is unfalsifiable without it.
+- **which model and which tools were reached**, including through sub-agents
+- **when**, at a precision that survives clock skew between services
+
+The rule to hold: **an audit trail written for reconciliation answers "where did the
+money go"; an audit trail written for governance answers "prove the control was
+applied".** They are different queries and the second needs the policy version.
+
+## Cost attribution as a hierarchy
+
+The billing reference tracks spend per tenant. Governance needs it resolvable up a
+chain: **organisation → business unit → team → credential → individual**. Not because
+finance asks, but because the question that actually gets asked in an incident is "which
+team's agent did this", and a flat tenant id cannot answer it.
+
+Limits belong at more than one level too — a per-credential cap does not stop twenty
+credentials in one team from draining a budget together.
+
+## Failover must land somewhere approved
+
+The router in the body falls back to the next healthy provider. Governance adds one
+constraint: **the fallback must be policy-equivalent, not merely available.**
+
+A chain that silently fails over to a provider with different data handling, a different
+jurisdiction, or a different retention policy has moved the data somewhere nobody
+approved — and it does it precisely during an incident, when nobody is reading logs. Tag
+each provider with the policy it satisfies, and let the fallback chain filter on the tag
+before it filters on health.
+
+## Fail-open or fail-closed, decided by risk
+
+When the control plane itself is unavailable — the guardrail service times out, the
+policy store is unreachable — the system either proceeds without the check or refuses.
+**Both answers are correct for different workloads, and neither is a default.**
+
+- Fail-**open** for a low-risk, high-volume path where refusing is the bigger harm.
+- Fail-**closed** for anything consequential.
+
+Write the choice down per workload, and make the control plane itself redundant enough
+that the choice is rarely exercised: timeouts, load balancing, and a health check that
+distinguishes "slow" from "gone".
+
+## Blast radius: sandboxes and credentials
+
+When an agent runs code, two rules carry most of the weight:
+
+- **A sandbox protects the host, not the sandbox.** Anything the agent can reach *from
+  inside* is still reachable — network egress, mounted paths, environment. Restrict
+  egress explicitly and allowlist commands rather than denylisting.
+- **Credentials never enter the sandbox.** Put a proxy in front that injects them per
+  request, so a prompt injection that dumps the environment gets nothing worth having.
+
+Ephemeral is the default: create on demand, tear down after, never reuse across tenants.
+A long-lived sandbox accumulates state that nobody audits.

package/plugins/agent-stack/skills/agent-orchestrator/references/llm-proxy-billing.md

@@ -1,5 +1,12 @@
 # Reselling LLM access — metering, wallets and guardrails
 
+**Load this when** the product resells LLM access: tiered wallets and the single
+boundary where markup applies, two-phase commit across a database and a provider API
+with compensating transactions, advisory locking, optimistic concurrency for reclaims,
+spend-delta polling and its three cases, budget / loop / auto-pause guardrails,
+per-tenant key lifecycle and healing, the refund waterfall, and model-routing
+precedence.
+
 When your product gives users LLM access and bills for it, you are running a
 proxy with a wallet behind it. The failure modes are not model failures: they
 are **double-credited transfers**, **spend you discovered after it happened**,

package/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md

@@ -1,6 +1,10 @@
 # Agent Orchestrator — Reference Guide
 
-Extended patterns, data models, and implementation details.
+**Load this when** you need the data models and algorithms under the body: message
+and result protocols, pipeline models, the SQL validation loop, context-window sizes
+and token estimation, learning-extraction heuristics, the confidence lifecycle, fuzzy
+deduplication, conflict resolution, cross-resource transfer, and the suggestion engine
+that costs no LLM call.
 
 ## Contents
 

package/plugins/agent-stack/skills/agent-orchestrator/references/runtime.md

@@ -0,0 +1,151 @@
+# Runtime — what keeps an agent alive between requests
+
+**Load this when** the agent must survive things a single request does not: a crash
+mid-run, a human who has to approve before it continues, a user who sends a second
+message while the first is still working, a dropped connection, a schedule. The body's
+loop is the *harness* — what the model is given to work with. This is the layer beneath
+it, and most orchestrators assume it exists rather than specify it.
+
+The split is worth keeping in mind while reading: **a good harness makes an agent
+capable, a good runtime makes it deployable.** They fail differently, and a team that
+has only built the first one discovers the second in production.
+
+## Contents
+
+- Durability, and where our own asymmetry was
+- The interrupt/resume contract
+- Double-texting: four policies
+- Streaming that survives a dropped connection
+- Time travel and forking
+- Scheduled and sleep-time work
+- Middleware: the seven concerns, unwelded
+
+## Durability, and where our own asymmetry was
+
+**Checkpoint every iteration of the loop, not just the stages a human reviews.**
+
+The body's multi-stage pipeline persists at each `stage.checkpoint` and can resume from
+a `pipeline_run_id`. The simple tool-calling path persists nothing: a crash, a deploy or
+a killed worker loses the entire run, including the tool calls that already cost money
+and time. That asymmetry is a defect, not a design — the simple path is the one that
+runs most often.
+
+What a checkpoint holds: the message array, the iteration counter, accumulated token
+usage, the carryover state (see `context-engineering.md`), and whatever the sub-agents
+have returned so far. Keyed by a thread id that acts as a cursor into the run.
+
+Two properties earn their cost:
+
+- **Resume at the point of failure**, not at the last human review. The difference is
+  whole minutes of re-executed tool calls.
+- **A pause frees the worker.** An agent waiting for a human should hold no process. If
+  waiting costs a worker, long approvals are quietly expensive and teams stop using them.
+
+## The interrupt/resume contract
+
+The body has two mechanisms for one idea: `ask_user` raises a clarification error, and a
+pipeline checkpoint returns a paused result. **They should be one contract.**
+
+- **Interrupt** — the run stops at a named point, persists its state, and surfaces a
+  payload describing what it needs: a question, a plan to approve, a destructive action
+  to confirm.
+- **Resume** — the caller returns a decision, and execution continues *from that point*
+  with the decision in scope. Not a fresh run that re-derives its way back.
+
+One contract means one persistence format, one place a UI has to understand, and one
+answer to "what happens if nobody replies for a day".
+
+**When an interrupt is mandatory** rather than optional: any action that is
+hard to reverse or outward-facing. Content-level guardrails are probabilistic — see
+`governance.md` — so consequential actions need a deterministic limit or a human, not a
+classifier's opinion.
+
+## Double-texting: four policies
+
+A user sends a second message while the first is still running. This has four possible
+answers, and a system that never chose one has chosen the worst by accident:
+
+| Policy | Behaviour | Fits |
+|---|---|---|
+| **Enqueue** | finish the current run, then start the new one | a task where the first answer is still wanted |
+| **Reject** | refuse the second message while busy | expensive or transactional runs |
+| **Interrupt** | stop the current run, start the new one, keep what was produced | conversational agents — the usual default |
+| **Rollback** | discard the current run *including its input*, start clean | the user is correcting themselves |
+
+The difference between interrupt and rollback is what the transcript looks like
+afterwards, and it is worth deciding deliberately: interrupt leaves a half-finished turn
+in history that the next prompt will see.
+
+## Streaming that survives a dropped connection
+
+Four things are worth streaming, and they are not the same thing:
+
+1. **State snapshots** after each step — for a UI that renders the whole picture.
+2. **State deltas** — the same, cheaper.
+3. **Tokens** — the typing effect.
+4. **Custom events** — domain progress: "queried 3 of 7 sources".
+
+The body's tracker emits an in-memory feed. Two properties turn it into something a
+client can rely on:
+
+- **Every event carries a monotonic id**, and a client reconnecting sends the last id it
+  saw. The server replays from there. Without this, a dropped connection during a
+  ninety-second run means the user watches nothing and then gets an answer from nowhere.
+- **The feed is a view over the durable trace, not the record itself.** If the only copy
+  of what happened is a stream nobody stored, evaluation is impossible — see the
+  `agent-evals` skill, which cannot function without it.
+
+## Time travel and forking
+
+Once every iteration is checkpointed, one capability follows nearly free: **pick a past
+checkpoint, modify the state, and resume from it.** The original history stays; the
+modified run forks.
+
+This is the debugging tool the loop otherwise lacks. "Why did it call that tool?" is
+answerable by rewinding to the step before, changing one thing, and running forward
+again — through the real loop, with real model calls and real tools, rather than a
+reconstruction that may not share the bug.
+
+It is also how a failed production run becomes a regression fixture: fork at the failure
+point, minimise, save the state as the fixture's input.
+
+## Scheduled and sleep-time work
+
+Not all agent work starts with a user. Two shapes, and the distinction matters:
+
+- **Stateful schedule** — each run appends to an existing thread, so the agent remembers
+  the previous ones. A daily briefing that should not repeat itself.
+- **Stateless schedule** — each run starts a fresh thread. A monitor that must not drift
+  on yesterday's context.
+
+Scheduled runs need the same retry and tracing as interactive ones, and one extra rule:
+**a schedule that fails silently is worse than no schedule.** Failures must reach a human
+through something other than the absence of a result.
+
+**Sleep-time compute** is the useful pattern on top: work done between conversations —
+consolidating memory, refreshing an index, pre-computing what tomorrow's first question
+will need. It is also where memory consolidation belongs when the hot path is too busy
+for it.
+
+## Middleware: the seven concerns, unwelded
+
+The body's loop hand-codes seven cross-cutting concerns inside itself: retry, provider
+fallback, summarisation, human-in-the-loop, tool-call limits, redaction, and moderation.
+Each is correct and none is separable — changing the retry policy means editing the loop.
+
+The alternative is ordered interceptors at four points:
+
+| Hook | Runs | Typical use |
+|---|---|---|
+| `before_model` | before the request is built | inject context, redact, enforce a budget |
+| `wrap_model_call` | around the call | retry, fallback, timing, cost accounting |
+| `wrap_tool_call` | around each tool | authorisation, rate limits, argument validation |
+| `after_model` | on the response | moderation, structured-output repair, guardrails |
+
+The hook names are borrowed vocabulary; the shape is generic. What it buys is
+composition — a tool-call limit is one interceptor, not a counter threaded through three
+functions — and testability: an interceptor is a unit, the loop is not.
+
+The trap: **order is semantics.** Redaction after summarisation redacts a summary that
+already leaked. Write the order down where the list is defined, not in the head of
+whoever wrote it.