@ssheleg/agent-stack 0.3.0 → 0.5.0

package/CHANGELOG.md

@@ -4,6 +4,79 @@ All notable changes to this project are documented here.
 Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning: [SemVer](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.0] — 2026-08-12
+
+### Added
+
+- **A second skill: `agent-evals`** — how you know the thing built by
+  `agent-orchestrator` actually behaves. There was no evaluation doctrine anywhere in
+  this family; `grep -ril "llm-as-judge\|eval dataset\|regression fixture\|trace id"`
+  across four plugins returned nothing.
+
+  An agent's behaviour is not in its source — the code says what it is allowed to do,
+  only a run says what it did — so the artifact under test is the execution record.
+  Three primitives (run, trace, thread) crossed with three granularities (single-step,
+  full-turn, multi-turn), each with its own fixture shape and its own precondition:
+  step assertions need a stable architecture or they die at the next refactor; turn
+  assertions cover trajectory **and** response **and** state change, because an agent
+  that says it saved the preference and did not passes two axes out of three; thread
+  scripts checkpoint after every turn and fail fast, or turn 3 derails and turns 4–10
+  assert nothing while still reporting a result.
+
+  Then the parts that decide whether any of it is trustworthy: the offline/online/ad-hoc
+  timing axis and why offline is necessary and not sufficient; pass-fail rubrics with
+  enumerated failure conditions instead of scalar scores that name no fix; code checks
+  before model judges; judging the trajectory, not just the answer; **calibrating a judge
+  against human labels before trusting it**, because an uncalibrated judge is an opinion
+  with a number attached; the classes of output no general judge can grade; a corpus
+  grown from production failures where every fixed failure stays a fixture permanently;
+  annotation queues with the two reviewer roles kept apart; and simulated users made
+  deliberately worse so offline results predict production.
+
+  Body 214 lines / ~2500 tokens; description 911/1024 with paired RU triggers.
+
+### Changed
+
+- **Both installers iterate over `skills/` instead of naming one skill.** `install.sh`
+  and `bin/agent-stack.js` each hardcoded `agent-orchestrator`, so a second skill would
+  have shipped in the package and reached nobody. Verified by running both against a
+  clean `HOME` and listing what arrived. The CI smoke test now asserts both skills.
+- README and both manifests describe two skills; the marketplace entry's description is
+  the manifest's, rather than a second one drifting beside it.
+
+## [0.4.0] — 2026-08-12
+
+### Added
+
+- **`references/context-engineering.md`** — what the loop gives up when the window
+  runs out, which the skill named a threshold for and never answered. The five-rung
+  compaction ladder, cheapest first, with a re-measure between rungs so a small
+  overage never buys a model call; the rung that exists only for the case where the
+  compaction request itself does not fit; the **tool-pair invariant** — one boundary
+  finder used by every rung, because a truncation that orphans a `tool_use` fails the
+  *next* request with a 400 in the middle of a task; **typed carryover blocks** copied
+  across the boundary rather than summarized, since a summarizer keeps the discussion
+  and drops the state, including the flag that said not to write; **tool-output
+  offload** to a file with the path left in context, because trimming history cannot
+  save a window one result already filled; token estimation and the direction it errs;
+  the compaction circuit breaker; sub-agent context isolation; the filesystem as
+  context; and how to pick constants for your own window.
+
+  Ladder structure and the attachment taxonomy are adapted from
+  [HKUDS/OpenHarness](https://github.com/HKUDS/OpenHarness) (MIT),
+  `services/compact/__init__.py` and `services/tool_outputs.py`. The constants are
+  deliberately ours — and deliberately absent, with anchors for choosing them, because
+  a threshold copied without its window is a number nobody can defend.
+
+### Changed
+
+- **§12 Context Engineering** in the body: five traps an agent cannot know to look up,
+  because it does not know they exist. **§7 gains Layer 0 — carryover state**, the one
+  memory layer whose survival is deterministic rather than a model's choice. §2 gains
+  the iteration refund. Body lands at 486 lines / ~4985 tokens against the 500 / 5000
+  budget — every insertion was cut twice to fit, and the depth is in the reference
+  where it belongs.
+
 ## [0.3.0] — 2026-08-12
 
 ### Changed

package/README.md

@@ -9,7 +9,8 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills).
 
 ## What is in here
 
-One skill, `agent-orchestrator`, and two references it loads on demand.
+Two skills — `agent-orchestrator` for building one, `agent-evals` for proving it
+behaves — and three references the first loads on demand.
 
 **The orchestrator** (`SKILL.md`) — what the agent reads first:
 
@@ -29,6 +30,22 @@ One skill, `agent-orchestrator`, and two references it loads on demand.
 - context budget allocation by priority
 - self-learning feedback loops
 
+**The evals skill** (`agent-evals/SKILL.md`) — how you know any of it works. An
+agent's behaviour is not in its source, so the artifact under test is the
+execution record: three primitives (run, trace, thread) crossed with three
+granularities (single-step, full-turn, multi-turn), the offline/online/ad-hoc
+timing axis, pass-fail rubrics instead of scalar scores that name no fix, cheap
+code checks before model judges, judges calibrated against human labels before
+they are trusted, and a corpus grown from production failures rather than
+authored up front — where every fixed failure stays a fixture permanently.
+
+**`references/context-engineering.md`** — what the loop gives up when the window
+runs out: the five-rung compaction ladder and why to re-measure between rungs,
+the tool-pair boundary invariant, typed carryover blocks copied across the
+boundary rather than summarized, tool-output offload to a file, token estimation
+and the direction it errs, the compaction circuit breaker, sub-agent context
+isolation, and how to choose constants for your own window.
+
 **`references/patterns.md`** — the data models and algorithms underneath:
 message and result protocols, pipeline models, the SQL validation loop,
 context-window sizes and token estimation, learning-extraction heuristics,

package/bin/agent-stack.js

@@ -2,7 +2,7 @@
 /*
  * agent-stack installer CLI.
  *
- * Installs the agent-orchestrator skill into ~/.claude/skills/agent-orchestrator
+ * Installs every skill this plugin ships into ~/.claude/skills/<name>
  * (same layout as install.sh). Idempotent: an existing install is skipped unless
  * --force. Zero dependencies.
  *
@@ -21,7 +21,7 @@ function usage() {
   console.log(`agent-stack installer
 
 Usage:
-  npx @ssheleg/agent-stack [--force]   install the agent-orchestrator skill
+  npx @ssheleg/agent-stack [--force]   install every skill this plugin ships
                                        into ~/.claude (skip existing unless --force)
   npx @ssheleg/agent-stack --help
 
@@ -67,20 +67,34 @@ function main(argv) {
     return 2;
   }
 
-  const skillSrc = path.join(ROOT, 'plugins/agent-stack/skills/agent-orchestrator');
-  if (!fs.existsSync(skillSrc)) {
-    console.error(`error: skill sources missing at ${skillSrc} — corrupted package?`);
+  const skillRoot = path.join(ROOT, 'plugins/agent-stack/skills');
+  if (!fs.existsSync(skillRoot)) {
+    console.error(`error: skill sources missing at ${skillRoot} — corrupted package?`);
+    return 1;
+  }
+
+  // Iterate rather than name one skill: a skill added to the plugin must not
+  // require an installer change to reach anybody.
+  const names = fs
+    .readdirSync(skillRoot, { withFileTypes: true })
+    .filter((e) => e.isDirectory())
+    .map((e) => e.name)
+    .sort();
+  if (!names.length) {
+    console.error(`error: no skills found under ${skillRoot} — corrupted package?`);
     return 1;
   }
 
   const home = os.homedir();
-  installOne(
-    'agent-orchestrator skill',
-    skillSrc,
-    path.join(home, '.claude', 'skills', 'agent-orchestrator'),
-    true,
-    force
-  );
+  for (const name of names) {
+    installOne(
+      `${name} skill`,
+      path.join(skillRoot, name),
+      path.join(home, '.claude', 'skills', name),
+      true,
+      force
+    );
+  }
   return 0;
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.3.0",
+  "version": "0.5.0",
   "description": "Production patterns for AI agent orchestrators — tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay — plus the wallet side of reselling LLM access. This package is the installer CLI.",
   "bin": {
     "agent-stack": "bin/agent-stack.js"

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -1,8 +1,8 @@
 {
   "name": "agent-stack",
   "displayName": "Agent Stack",
-  "description": "Production patterns for AI agent orchestrators: tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay — plus the wallet side of reselling LLM access.",
-  "version": "0.3.0",
+  "description": "Two skills: agent-orchestrator \u2014 tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals \u2014 run/trace/thread evals, judges, and fixtures grown from production.",
+  "version": "0.5.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"
@@ -19,6 +19,9 @@
     "openrouter",
     "llm-billing",
     "claude-code",
-    "cursor"
+    "cursor",
+    "evals",
+    "llm-judge",
+    "observability"
   ]
 }

package/plugins/agent-stack/skills/agent-evals/SKILL.md

@@ -0,0 +1,230 @@
+---
+name: agent-evals
+description: >-
+  Use when measuring whether an agent actually works — building an eval suite, judging a
+  trajectory rather than a final answer, turning production traces into regression
+  fixtures, calibrating an LLM judge against human labels, or gating a release on offline
+  evals. Covers the three observability primitives (run, trace, thread) crossed with three
+  eval granularities (single-step, full-turn, multi-turn), the offline/online/ad-hoc
+  timing axis, pass-fail rubrics over scalar scores, cheap code checks before model
+  judges, simulated users with adversarial personas, annotation queues, and what to
+  instrument so any of it is possible. Triggers - "agent eval", "eval suite", "LLM judge",
+  "regression fixture", "trajectory eval", "is the agent getting better", "эвалы агента",
+  "оценка агента", "LLM-судья", "регрессионный набор", "как проверить агента". Not for
+  unit tests of ordinary code, or for benchmarking a model.
+license: MIT
+---
+
+# Agent evals — proving the thing behaves
+
+An agent's behaviour does not exist in its source. The code says what it is *allowed* to
+do; only a run says what it *did*. So the artifact under test is the execution record,
+and the suite is grown from production rather than authored up front.
+
+Three claims follow, and they are what makes this different from testing ordinary code:
+
+- **You are testing reasoning, not code paths**, so one granularity is never enough.
+- **Every natural-language input is unique**, so the edge cases cannot be enumerated
+  offline. Production is not only where you catch what you missed — it is where you
+  discover what to test for.
+- **Traces become test cases.** The suite grows from what actually happened.
+
+---
+
+## 1. Three primitives
+
+| Primitive | Is | Carries |
+|---|---|---|
+| **Run** | one model call | the complete prompt — instructions, tools, context — and what came back |
+| **Trace** | one full execution | every run, every tool call with arguments and results, nested to show how steps relate |
+| **Thread** | many traces in one session | multi-turn context in order, **state evolution** (memory, files, artifacts), and elapsed time |
+
+The thread level is the one most systems lack, and it is where a whole class of defect
+lives: a bad memory write at turn 6 surfaces as a wrong answer at turn 11, and neither
+the run nor the trace view can localise it.
+
+**Precondition for all of this: traces are durable and queryable.** A live event stream
+renders a progress bar and evaporates. If you cannot fetch last week's low-scoring runs
+by id, nothing below is available to you — see §7.
+
+---
+
+## 2. Three granularities
+
+Each primitive gets its own kind of assertion.
+
+### Single-step — validates a run
+
+Fixture is a serialized run: prompt, tool schemas, context. Assert the decision at that
+point — tool name, argument shape.
+
+> "Schedule a meeting with Harrison tomorrow morning", with `find_meeting_times`,
+> `schedule_meeting` and `send_email` available, must call `find_meeting_times` first.
+
+Cheap, deterministic, CI-blocking. **Precondition: a stable agent architecture.** These
+break on a graph refactor, and a suite that fails on every refactor gets deleted.
+
+### Full-turn — validates a trace
+
+Assert on three axes at once, with three different mechanisms:
+
+| Axis | Assert | With |
+|---|---|---|
+| Trajectory | tool-call sequence — `read_file` → `edit_file` → `run_tests` | set/subset/order matchers |
+| Final response | quality, tone, policy compliance | rubric or judge |
+| **State change** | the memory row exists, the file was written, the artifact is there | direct inspection of the side effect |
+
+The third axis is the one people forget. **Assert on side effects, not only on prose** —
+an agent that says it saved the preference and did not is a pass on two axes out of three.
+
+Easiest inputs to generate, hardest outputs to validate automatically.
+
+### Multi-turn — validates a thread
+
+A scripted turn sequence with a **checkpoint after every turn and fail-fast on
+deviation**. Without that, turn 3 goes off the rails and turns 4–10 assert nothing while
+still reporting a result.
+
+> Turn 1: "I prefer Python over JavaScript." Turn 3's output must still be Python.
+
+Hardest to implement well. Start here only for behaviour that is genuinely about memory
+across turns.
+
+**Production suites combine all three.** One vendor reports about half of theirs sitting
+at single-step — recorded as their observation, not as a target to hit.
+
+---
+
+## 3. Offline, online, ad-hoc
+
+| When | Reference | Blocks a release? | Answers |
+|---|---|---|---|
+| **Offline** | a dataset, ground truth optional | **yes** — this is the gate | did my change break what used to work |
+| **Online** | none — **definitionally reference-free** | no | is production drifting |
+| **Ad-hoc** | none | no | what is actually happening out there |
+
+**Offline is necessary and not sufficient.** A green suite proves you did not regress the
+cases you already know about. It cannot prove the agent handles what nobody thought of,
+because that input is not in the dataset — which is what the online tier is for.
+
+Online evaluators fire on trace ingestion and check what needs no expected answer:
+trajectory anomalies, step-count and latency trends, judge scores, error rates. Route
+them over all traces, a sample, or a filtered subset; the sampling rate is a cost
+decision, not a correctness one.
+
+Ad-hoc is exploratory analysis over stored traces — clustering to surface failure modes
+nobody predefined. A dashboard tracks metrics you chose in advance; this finds the ones
+you did not.
+
+---
+
+## 4. Rubrics beat scores
+
+Generic metrics — helpfulness, naturalness, completeness — produce numbers and no
+decision. A 3.4 out of 5 on "helpfulness" tells you nothing about what to change.
+
+**Write narrow, behaviour-specific pass/fail rubrics, and write them with the people who
+own the behaviour.** Each failure must point at one thing: a prompt, a tool description,
+a workflow step, a missing capability.
+
+A rubric that works, in full:
+
+> **Escalation.** On a request for a human: push back once, escalate on the repeat.
+> **Fails if** it escalates immediately · refuses after the second request · escalates
+> before providing information it already had · continues several turns past the point it
+> is clearly not helping.
+
+Note the shape: one behaviour, an explicit pass condition, and an enumerated failure list.
+That is what makes a judge reproducible and a disagreement resolvable.
+
+---
+
+## 5. Judges
+
+**Cheap checks first.** Schema validation, exact match, format conformity, business-rule
+assertions, tool-call correctness — all deterministic, all faster and cheaper than a model
+call. Send to a judge only what cannot be decided by code.
+
+**Judge the trajectory, not just the answer.** Right tools, right order, right arguments.
+An agent that reaches a correct answer through three wrong tool calls is a latent outage.
+
+**Calibrate the judge before trusting it.** Collect human labels on the same traces,
+measure agreement, iterate the judge prompt until agreement is high — *then* let it score
+unattended. An uncalibrated judge is an opinion with a number attached, and shipping on it
+is exactly the failure of grading instead of measuring.
+
+**Some things a judge cannot do.** Plausible-but-wrong domain output — an invented legal
+citation, a subtly wrong SQL join — reads as correct to a general judge. Route those to a
+domain expert and accept that this tier stays human.
+
+---
+
+## 6. The corpus grows from production
+
+Never author the suite up front. Every production failure and every thumbs-down becomes a
+fixture:
+
+1. Capture the state at the failure point.
+2. Minimise it to the smallest input that still reproduces.
+3. Add it to the tier that isolates it — step, turn or thread.
+4. **It stays in the suite permanently.** A fixed bug that silently returns is the defect
+   this rule exists to prevent.
+
+Two dataset shapes come out of review:
+
+- **Ground truth** — the reviewer writes the correct output; the suite asserts equality.
+- **Criteria-based** — for open-ended work, the reviewer labels dimensions (relevance,
+  completeness, tone) instead of an exact answer.
+
+**The annotation queue** is what feeds both: filters route a subset of traces to humans —
+low automated score, thumbs-down, a feature area, a cluster. Two reviewer roles, and
+mixing them wastes both: generalists judge surface quality, domain experts judge
+correctness only they can see.
+
+**Simulated users, if you generate inputs.** A model playing a customer is articulate,
+patient and cooperative, and inflates every pass rate. Fine-tune it on real user
+transcripts and add adversarial personas — the refund-seeker, the AI-sceptic, the one who
+wants a human immediately. Making the simulated user worse makes the offline result
+predictive.
+
+---
+
+## 7. What to instrument first
+
+None of the above runs without these, and they are the part people skip:
+
+- **A durable trace store, queryable by id, filterable by score and time.** The live
+  stream is a view over it, never the record itself.
+- **Scores as first-class records bound to a run**: `run_id`, key, value, and a `source`
+  of `human` | `llm_judge` | `code_check`. A score with no source cannot be calibrated,
+  audited, or trusted differently from its neighbours.
+- **Whole prompts, not just messages** — instructions, tool schemas and context as they
+  were sent. A fixture cannot be replayed from a summary.
+- **State snapshots at turn boundaries**, so a thread test can assert what carried.
+
+---
+
+## Checklist
+
+- [ ] Traces durable and queryable by id, not only streamed
+- [ ] Scores bound to runs, each with a source
+- [ ] Single-step fixtures for the decisions that must not drift
+- [ ] Full-turn assertions on trajectory **and** final response **and** state change
+- [ ] Multi-turn scripts with a checkpoint after every turn, failing fast
+- [ ] Offline suite as the release gate; online evaluators reference-free and non-blocking
+- [ ] Pass/fail rubrics with enumerated failure conditions, written with behaviour owners
+- [ ] Code checks before model judges
+- [ ] Judge calibrated against human labels before it is trusted
+- [ ] Domain-expert review for output a general judge cannot grade
+- [ ] Every production failure minimised into a permanent fixture
+- [ ] Annotation queue with filters, and the two reviewer roles kept separate
+- [ ] Simulated users trained on real transcripts, with adversarial personas
+
+---
+
+## Related
+
+Building the agent this measures is the **`agent-orchestrator`** skill, shipped in the
+same plugin — named rather than linked, because a packager may ship either directory
+alone. Its context-engineering reference covers the context-pressure behaviour that
+trajectory assertions most often catch drifting.

package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md

@@ -137,6 +137,7 @@ else:
 - **In-loop trimming**: At ~80% capacity, collapse older assistant+tool pairs into one-liner summaries
 - **Token limit recovery**: On `LLMTokenLimitError`, compress to 60% and retry once. If still fails, return partial answer
 - **Max iterations guard**: Always have a hard limit. On exhaustion, compose best-effort answer from data gathered so far
+- **Iteration refund**: a recoverable provider error is not charged to that guard
 
 ---
 
@@ -323,6 +324,9 @@ context window, so allocation has to be decided per call rather than per layer:
 a session that trims chat history to fit a large set of learnings has quietly
 chosen old generalities over what the user said sixty seconds ago. Give layer 1
 a floor.
+
+**Layer 0 — carryover state.** Goal, artifacts, verified work and restrictive
+mode cross a compaction boundary as copied typed blocks, not prose (§12).
 ## 8. Self-Learning Feedback Loops
 
 ### Cycle 1: Automatic (Validation Loop)
@@ -436,6 +440,27 @@ async def handle_ask_user(tc, context, wf_id):
 
 ---
 
+## 12. Context Engineering
+
+**Compaction is a ladder, not a call.** Clear old tool results, collapse
+oversized blocks, condense messages, and only then pay a summarizer —
+re-measuring between rungs. Most pressure resolves before the first call.
+
+**Never orphan a `tool_use`.** Every truncation point lands *between* an
+assistant+tool pair, or the next request is a 400 in the middle of a task.
+
+**Carry state across the boundary as typed blocks, not prose** — goal,
+artifacts, verified work, restrictive mode. A summarizer keeps the discussion
+and drops the state, including the flag that said not to write anything.
+
+**Offload a large tool result to a file, keep the path.** Trimming history
+cannot save a window one tool output already filled.
+
+**A sub-agent's value is its own window**: it returns a typed summary, not a
+transcript.
+
+---
+
 ## Checklist — Building a New Orchestrator
 
 - [ ] Shared `AgentContext` dataclass with all sub-agents
@@ -459,6 +484,7 @@ async def handle_ask_user(tc, context, wf_id):
 - [ ] Multi-stage pipeline with checkpoints and resume
 - [ ] `ask_user` clarification mechanism
 - [ ] Graceful degradation (partial answers on context overflow or max iterations)
+- [ ] Compaction ladder, tool-pair-safe boundaries, typed carryover, output offload
 
 ---
 
@@ -470,4 +496,5 @@ are the territory.
 | File | Read it when |
 |---|---|
 | [`references/patterns.md`](references/patterns.md) | you need the **data models and algorithms**: message and result protocols, pipeline models, the SQL validation loop, context-window sizes and token estimation, learning-extraction heuristics, confidence lifecycle, fuzzy dedup, conflict resolution, cross-resource transfer, the no-LLM suggestion engine |
+| [`references/context-engineering.md`](references/context-engineering.md) | the loop is **running out of window**: the five-rung compaction ladder, the tool-pair boundary invariant, typed carryover attachments, tool-output offload, token estimation, the compaction circuit breaker, sub-agent isolation, and how to pick your own constants |
 | [`references/llm-proxy-billing.md`](references/llm-proxy-billing.md) | the product **resells LLM access**: tiered wallets and where markup applies, two-phase commit against a provider API with compensating transactions, advisory locking, optimistic concurrency for reclaims, spend-delta polling and its three cases, budget/loop/auto-pause guardrails, per-tenant key lifecycle and healing, the refund waterfall, model routing |

package/plugins/agent-stack/skills/agent-orchestrator/references/context-engineering.md

@@ -0,0 +1,201 @@
+# Context engineering — what survives the window, and how
+
+**Load this when** the loop is running out of window: choosing a compaction strategy,
+deciding what crosses the boundary, bounding a tool result that will not fit, or
+splitting message history without breaking the next request.
+
+`SKILL.md` §2 gives the loop and its two thresholds — trim at ~80% of the window,
+inject the wrap-up at ~70%. This file is what happens *at* those thresholds. The
+difference matters: §2 decides **when** to act, this decides **what to give up**.
+
+Ladder structure and the attachment taxonomy are adapted from
+[HKUDS/OpenHarness](https://github.com/HKUDS/OpenHarness) (MIT),
+`src/openharness/services/compact/__init__.py` and `services/tool_outputs.py`. The
+constants below are ours; theirs are tuned for a different window and a different tool
+mix, and a threshold copied without its context is a number nobody can defend.
+
+## Contents
+
+- The ladder — five rungs, cheapest first
+- Re-measure between rungs
+- The tool-pair invariant
+- Carryover attachments — what crosses the boundary
+- Tool-output offload
+- Estimating what you have left
+- The circuit breaker
+- Sub-agent context isolation
+- The filesystem as context
+- Choosing your own constants
+
+## The ladder — five rungs, cheapest first
+
+Compaction is not one call. It is an ordered set of strategies, and the expensive one
+is last. A loop that summarizes at the first sign of pressure pays a model call and a
+round-trip for what a string operation would have solved.
+
+| Rung | What it does | Cost | Loses |
+|---|---|---|---|
+| 1. Microcompact | Replace old tool results with a tombstone: `[tool result cleared]`. Keep the N most recent. | free | old observations, kept recent ones |
+| 2. Head/tail collapse | For an oversized text block, keep a head and a tail, elide the middle with a marker | free | the middle of long outputs |
+| 3. Session condensation | Collapse each older message to a one-line summary, capped in total | free | phrasing, keeps the thread of events |
+| 4. LLM compaction | One model call summarizes the transcript into a structured brief | a call + latency | anything the summarizer does not think to keep |
+| 5. Prompt-round truncation | Drop the oldest whole prompt rounds, boundary-aligned | free | the earliest history entirely |
+
+Rung 5 exists for one case: **the compaction request itself does not fit.** When rung 4
+fails because the transcript it must summarize is over the limit, summarizing harder is
+not available — you drop oldest rounds and retry, bounded by a small retry count.
+
+**Eligibility, not just age.** A tool result becomes a rung-1 candidate by size as well
+as position: anything past a few thousand characters, and every result from an external
+tool server, whose outputs are the usual window hog and the least likely to be re-read.
+
+## Re-measure between rungs
+
+After every rung, measure again and stop if you are under the threshold. Two failure
+modes hide here:
+
+- **Running the whole ladder every time** turns a 200-character overage into a model
+  call. Rung 1 usually settles it.
+- **Trusting a rung that did nothing.** Head/tail collapse on a transcript with no
+  oversized blocks changes nothing. If the estimate did not drop, reject the result and
+  escalate rather than recording a compaction that never happened.
+
+## The tool-pair invariant
+
+**A tool call and its result are one unit. Never split them.**
+
+Every provider rejects a request where an assistant message announces a tool call whose
+result is missing, or a tool result with no matching call. It is not a soft error — the
+next request fails, mid-task, with a 400 that names nothing useful.
+
+This makes every boundary computation in this file conditional: a truncation point, a
+collapse range, a dropped round must all land **between** pairs. Write the boundary
+finder once, use it from every rung, and give it a test that plants an orphan and
+requires rejection.
+
+The same invariant governs parallel dispatch: if one tool in a batch raises, its result
+block must still be emitted — as an error result. A sibling's crash is not a reason to
+send a malformed request.
+
+## Carryover attachments — what crosses the boundary
+
+A summarizer keeps prose and drops state. Ask one to compress a transcript and it will
+faithfully preserve the discussion while losing the fact that you are in a read-only
+mode, the path of the file you verified, and what the user actually asked for.
+
+So state does not go through the summarizer. It is accumulated during normal execution
+and re-attached after compaction as **typed blocks**:
+
+| Block | Carries | Why it is lost otherwise |
+|---|---|---|
+| `task_focus` | the goal, recent sub-goals, active artifacts, verified state, next step | the summarizer rewrites the goal into its own words and drifts |
+| `recent_files` | paths read or written, most recent first | the agent re-reads what it already has |
+| `verified_work` | what was checked and how it was checked | re-verification, or worse, a claim of verification that was never re-run |
+| `plan` | the active plan and any mode that restricts what may be done | **a safety mode forgotten across a boundary is how a read-only run starts writing** |
+| `invoked_skills` | which doctrine is already loaded | re-loading a large file that is already in effect |
+| `work_log` | one line per significant action | the agent repeats a step it already completed |
+
+Accumulate them in bounded buckets during the loop — a small cap per bucket, oldest
+evicted — so the attachment cost is known in advance and does not itself grow into a
+context problem. Cap the number of attached blocks too.
+
+The property that matters: this is **deterministic**. Structured state survives because
+it is copied, not because a model chose to keep it.
+
+## Tool-output offload
+
+One large tool result poisons a window before any history trimming is relevant. Trimming
+the conversation does not help when the problem is a single 200 KB response sitting in
+the current turn.
+
+The rule: **above an inline limit, write the full output to a file and put a preview
+plus the path into the context.** The model reads the preview, and when it needs the
+rest it reads the file with the tools it already has.
+
+- **Inline limit** — the size above which offload happens.
+- **Preview size** — enough to decide whether the rest is needed; head, or head and tail
+  when the tail carries the verdict (test runs, build logs).
+- **Path** — stable and unique per call, so two calls to the same tool do not collide.
+- **Location** — a scratch directory scoped to the run, not the project tree; artifacts
+  of a loop are not deliverables and must never arrive in a diff.
+
+Both limits belong in configuration. The right value depends on the window and the tool
+mix, and the only way to find it is to measure a real run.
+
+## Estimating what you have left
+
+Every threshold in this file depends on an estimate, and estimates are optimistic in the
+direction that hurts:
+
+- **Character-based estimation runs low.** A divisor calibrated on prose under-counts
+  code, JSON and non-Latin text. Apply a padding factor, and prefer being early over
+  being right.
+- **Non-text content costs more than its representation suggests.** An image is worth
+  hundreds to thousands of tokens depending on its dimensions; a reference to a file is
+  worth nothing until it is read.
+- **Reserve the output.** The threshold is not "the window" — it is the window minus the
+  space the answer needs, minus a buffer for the next tool result. Express the trigger
+  as a floor in absolute tokens, not only as a percentage: at a small window a
+  percentage silently reserves too little.
+
+## The circuit breaker
+
+Compaction can fail — the summarizer errors, the provider is down, the result does not
+shrink. **After a small number of consecutive failures, disable compaction for the run
+and degrade deliberately**: stop adding to the context, compose the best answer
+available, and say that compaction is unavailable.
+
+Without this, a failing compaction is retried at the top of every iteration, and the
+loop spends its remaining budget on the one operation that is not working.
+
+**Related: refund the iteration.** When an iteration ends in a recoverable provider
+error — a token limit that can be clamped and retried, a rate limit — do not charge it
+to the max-iteration guard. A misconfiguration should not consume the budget that exists
+to stop a runaway agent.
+
+## Sub-agent context isolation
+
+A sub-agent's value is not that it is a different prompt. **It is that it has its own
+window.**
+
+The contract: the sub-agent receives a task and the context it needs, works in a window
+the parent never sees, and returns a typed result — a summary, not a transcript. The
+parent's window grows by the size of the result, not by the size of the work.
+
+This makes delegation the strongest context tool available: a search across forty files
+costs the parent one paragraph. It also sets the design rule — if a sub-agent's return
+value is proportional to its input rather than its conclusion, it is a function call
+wearing a costume, and it will fill the parent's window anyway.
+
+## The filesystem as context
+
+Anything durable belongs in a file, not in the transcript:
+
+- **It survives the boundary.** Compaction cannot delete what was never in the window.
+- **It survives the process.** A crash loses the conversation and keeps the work.
+- **It is shared.** Two agents on one task coordinate through files; a transcript is
+  private to one loop.
+- **It is addressable.** A path is a few tokens; the content it names can be any size.
+
+The pattern is the same in each case: do the work, write the artifact, keep the path.
+Where the runtime offers no real filesystem, the same contract works over any keyed
+store — what matters is that the address is cheap and the content is retrievable, not
+that it is POSIX.
+
+## Choosing your own constants
+
+Every number in this file is deliberately absent, because the useful value depends on
+the window, the tool mix and the cost of a summarizer call in your stack. Pick them
+against these anchors:
+
+1. **Reserve output before you reserve anything else.** Start from window minus expected
+   answer minus one worst-case tool result.
+2. **Rung 1's keep-count** is the smallest number of recent tool results the loop
+   actually re-reads. Measure it; the intuition is always too high.
+3. **The inline limit for offload** should be crossed only by outputs that are genuinely
+   large — if half of all tool calls offload, the limit is too low and the model is
+   reading files instead of working.
+4. **The circuit-breaker count** is small. Three consecutive failures is a pattern, not
+   a coincidence.
+5. **Write them down where the loop reads them**, not inline at the call site. A
+   threshold nobody can find is a threshold nobody will tune.