@ssheleg/agent-stack 0.10.1 → 0.11.0

package/CHANGELOG.md

@@ -1,5 +1,50 @@
 # Changelog
 
+## [0.11.0] — 2026-08-16
+
+### Changed
+
+- **`agent-orchestrator`'s body is under its own budget for the first time: 5670 →
+  4728 tokens** against the 4750 the pack set itself, and by **splitting rather than
+  trimming**. Three layers moved to where they belong instead of every section losing a
+  sentence:
+
+  - **`references/pipeline.md`** is new — the planned path and the interrupt that asks a
+    person, which are one suspend-and-resume seen from two sides rather than two features.
+  - **Observability folded into `references/runtime.md`**, beside the streaming contract it
+    was the concrete half of. The body had the API and the reference had the two properties
+    that decide whether it is a feed or a decoration; they are one thing now.
+  - **Sub-agent retry and the learning cycles went to `references/patterns.md`**, whose
+    mechanisms they were the surface of.
+
+  The body keeps the decisions and the checklist keeps only what a heading cannot say. The
+  v0.8.0 notes had already made this argument to justify a fourth skill and then the body
+  absorbed a layer anyway; the budget is now stated in the file itself.
+
+### Added
+
+- **A check for one home per fact.** Every reference was checked for *existence* in both
+  directions and nothing checked whether two of them **say the same thing**. On 2026-08-15
+  the same six-row decision table was written into `agent-harness/SKILL.md` and into the
+  graph-engineering reference in one afternoon — 50 shared twelve-word runs, found by
+  measuring rather than by review. The floor is set above the legitimate maximum, measured
+  after that duplication was removed: 12 runs is a skill quoting the rule it defers to, and
+  20 leaves headroom for a longer citation while still catching a restated section.
+
+  **It caught its own author within the minute.** Moving two sections into `patterns.md`
+  left one rule in both homes; the gate refused, and the copy was deleted.
+
+- **A seventh scanner detector, `declared-deps-ignored`.** A model with a `depends_on`
+  field, and a loop over the collection in the order it happens to be stored: the plan says
+  it need not be serialised, and then is. This pack shipped exactly that in its own
+  reference until yesterday. Conservative — any sign of a topological pass anywhere in the
+  file (`layers`, `kahn`, `toposort`, `in_degree`, a `ready` set) and it says nothing. Its
+  entry in the blind-spot list is retired, because it is no longer blind.
+  `self-test: 9/9` → **`11/11`** (eight plants, three clean fixtures).
+
+- **`agent-harness` cites the static-versus-dynamic model instead of restating it.** The
+  six-row table has one home, and it is the reference.
+
 ## [0.10.1] — 2026-08-15
 
 ### Changed

package/README.md

@@ -11,7 +11,7 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills).
 
 Four skills — `agent-orchestrator` for wiring the loop, `agent-evals` for proving it
 behaves, `agent-interop` for everything it talks to outside its own process,
-`agent-harness` for what it is **told** — and nineteen references they load on demand,
+`agent-harness` for what it is **told** — and twenty references they load on demand,
 plus one scanner.
 
 **The orchestrator** (`SKILL.md`) — what the agent reads first:
@@ -78,7 +78,7 @@ eight extension seams where a permission gate or a context rewrite can actually
 
 It runs in both directions: **building a harness and auditing somebody else's are one
 checklist read forwards and backwards.** `scripts/audit_agent.py` is the mechanical half —
-six conservative detectors, and it always prints what it *cannot* see plus a denominator,
+seven conservative detectors, and it always prints what it *cannot* see plus a denominator,
 so its silence is never read as a pass.
 
 **`references/graph-engineering.md`** — deciding the shape of the work before

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.10.1",
+  "version": "0.11.0",
   "description": "Production patterns for AI agent orchestrators \u2014 tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay \u2014 plus the wallet side of reselling LLM access. This package is the installer CLI.",
   "bin": {
     "agent-stack": "bin/agent-stack.js"

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Two skills: agent-orchestrator \u2014 tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals \u2014 run/trace/thread evals, judges, and fixtures grown from production.",
-  "version": "0.10.1",
+  "version": "0.11.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/agent-stack/skills/agent-harness/SKILL.md

@@ -85,27 +85,20 @@ branches, you wanted routing and it is cheaper.
 ### Static or dynamic — the second question, and it is not the same one
 
 Having chosen a workflow, one thing is still open: **is its shape known before it runs?**
-A **static** graph has every node and edge decided up front. A **dynamic** one grows — a
-node finishes, reads what it found, and decides what comes next.
-
-| Reach for | When |
-|---|---|
-| **static** | the task repeats and the structure is the same every time |
-| **static** | predictability and speed matter more than flexibility |
-| **static** | **always first** — go dynamic only after the static version hits a wall you can name |
-| dynamic | the scope of the work depends on what is discovered along the way |
-| dynamic | a node must choose its successors from its own output |
-| **never dynamic** | **the run has to be auditable** — see below |
-
-**The audit rule is hard, not a preference.** A dynamic graph's executed shape is not the
-shape anybody drew, so *"here is the design"* and *"here is what happened"* stop being the
-same document and every claim about the run becomes unfalsifiable from outside. Most
-workflows that feel like they need a dynamic graph need a better static one.
-
-The rest of the model — the fake-edge test, the diamond, the checker node before a
-convergence, and what a host actually executes when it fans out — is
-`agent-orchestrator/references/graph-engineering.md`.
-
+A **static** graph has every node and edge decided up front; a **dynamic** one grows as
+nodes read their own output and decide what comes next.
+
+**Static first, always** — go dynamic only after the static version hits a wall you can
+name, because dynamic is more powerful and much harder to control. And one row of that
+decision is hard rather than preferential: **a run that has to be auditable is static.**
+A dynamic graph's executed shape is not the shape anybody drew, so *"here is the design"*
+and *"here is what happened"* stop being the same document, and every claim about the run
+becomes unfalsifiable from outside.
+
+The six-row table, the rest of the model — the fake-edge test, the diamond, the checker
+node before a convergence — and what a host actually executes when it fans out are one
+home away: `agent-orchestrator/references/graph-engineering.md`. It is not restated here,
+because a decision table with two homes is one that will disagree with itself.
 ---
 
 ## References

package/plugins/agent-stack/skills/agent-harness/scripts/audit_agent.py

@@ -190,8 +190,40 @@ def check_unguarded_fanout(rel, text, lines):
                 "a checker that can see which branch failed")
 
 
+def check_declared_deps_ignored(rel, text, lines):
+    """A plan that declares dependencies and is then walked in list order.
+
+    A model with a `depends_on` (or `dependsOn`, or `depends`) field has gone to the trouble
+    of saying which steps need which — and then a loop over the collection in the order it
+    happens to be stored serialises the whole thing anyway. The declaration is not wrong and
+    the loop is not wrong; together they are a plan that says it need not be serialised,
+    serialised. This pack shipped exactly that in its own reference until 2026-08-15.
+
+    Conservative: the file must declare a dependency field AND iterate the collection that
+    holds it, and it must contain no sign of a topological pass anywhere — `layers`, `kahn`,
+    `toposort`, `in_degree` or a `ready`/`runnable` set. Any of those and this says nothing.
+    """
+    if re.search(r"\b(layers?|kahn|toposort|topological|in_degree|indegree|runnable|ready_set)\b",
+                 text, re.I):
+        return
+    if not re.search(r"\bdepends?(_on|On)?\b\s*[:=]", text):
+        return
+    for i, l in enumerate(lines, 1):
+        m = re.search(r"for\s+\w+\s+in\s+(\w+)\.(stages|steps|nodes|tasks|plan)\b", l) or \
+            re.search(r"for\s+\w+\s+in\s+(plan|stages|steps|nodes|tasks)\b", l)
+        if m:
+            add("declared-deps-ignored", rel, i,
+                "a dependency field is declared and the collection is walked in list order — "
+                "the plan says it need not be serialised, and then is",
+                "Execute in dependency layers (Kahn over the declared edges); a cycle fails "
+                "the plan rather than deadlocking the run, and a layer of more than one gets "
+                "a checker before anything downstream consumes it")
+            return
+
+
 CHECKS = [check_unbounded_loop, check_tool_without_description, check_swallowed_error,
-          check_no_timeout, check_hardcoded_model, check_unguarded_fanout]
+          check_no_timeout, check_hardcoded_model, check_unguarded_fanout,
+          check_declared_deps_ignored]
 
 # What no static pass can reach. Printed every run, never suppressed.
 BLIND = [
@@ -200,8 +232,6 @@ BLIND = [
     "whether the workflow/agent choice was made deliberately or defaulted to an agent",
     "whether a fan-out has a CHECKER between it and the node that consumes it — this pass "
     "sees an unguarded gather, never a missing gate",
-    "whether a declared dependency graph is actually executed in dependency order, or in "
-    "the order the stages happen to be listed in",
     "whether retries and fallbacks MULTIPLY (three providers x three retries is nine calls)",
     "whether compaction preserves decisions and open questions, or keeps the discussion",
     "whether tool output is treated as untrusted input",
@@ -277,6 +307,9 @@ PLANTS = [
      PY_HEADER + "out = await asyncio.gather(*(run(t) for t in tasks))\n"),
     ("unguarded-fanout (promise)", "unguarded-fanout", "agent.js",
      JS_HEADER + "const out = await Promise.all(tasks.map(t => run(t)));\n"),
+    ("declared-deps-ignored", "declared-deps-ignored", "agent.py",
+     PY_HEADER + "class Stage:\n    depends_on = []\n"
+     "for stage in plan.stages:\n    run(stage)\n"),
 ]
 
 # A detector that fires on the defect AND on its fix has no discriminating power. Each
@@ -288,6 +321,10 @@ CLEAN = [
     ("a fan-out that DOES capture its branches", "agent.py",
      PY_HEADER + "for _ in range(10):\n    pass\n"
      "out = await asyncio.gather(*(run(t) for t in tasks), return_exceptions=True)\n"),
+    ("a plan that DOES execute in dependency layers", "agent.py",
+     PY_HEADER + "class Stage:\n    depends_on = []\n"
+     "for layer in plan.layers():\n    run_layer(layer)\n"
+     "r = requests.get('https://example.com', timeout=5)\n"),
 ]
 
 

package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md

@@ -17,31 +17,25 @@ description: >-
 
 # Agent Orchestrator — Production Best Practices
 
-Battle-tested patterns from a production multi-agent system. Apply these when building any agent
-orchestrator, LLM-powered tool system, or agentic workflow.
+Patterns from a production multi-agent system. **The body is decisions; the mechanisms are
+one file away**, and it is held under a 4750-token budget — a body that grows absorbs the
+layer that should have been split, and this one did until 2026-08-16.
 
 ## Architecture Overview
 
 ```
 User Question
     ↓
-ConversationalAgent (thin wrapper, backward-compat)
-    ↓
 OrchestratorAgent.run(AgentContext)
-    ├─ Complexity check → simple (tool loop) or complex (pipeline)
-    ├─ Context loading (parallel: staleness, MCP sources, KB check)
-    ├─ History trimming
-    ├─ Context budget allocation
-    ├─ System prompt construction (dynamic, capability-aware)
-    └─ Execution:
-        ├─ SIMPLE: iterative LLM tool-calling loop
-        │    LLM → tool calls → sub-agent dispatch → results → LLM → ... → final text
-        └─ COMPLEX: multi-stage pipeline
-             QueryPlanner → ExecutionPlan → StageExecutor → checkpoints → final
+    ├─ Shape check → one loop, or a planned path (§5, references/pipeline.md)
+    ├─ Context loading (a parallel layer: staleness, sources, KB — §13)
+    ├─ History trimming, then context budget allocation
+    ├─ System prompt built from the live capabilities (§10)
+    └─ Execute:
+        ├─ SIMPLE: LLM → tools → sub-agents → results → LLM → … → answer   (§2)
+        └─ PLANNED: plan → dependency layers → checker → checkpoints → done (§5)
 ```
 
----
-
 ## 1. The Orchestrator Pattern
 
 ### Shared Context Object
@@ -170,112 +164,42 @@ when to use it, and the neighbour it is confused with — is
 
 ## 4. Sub-Agent Retry and Validation
 
-Wrap every sub-agent call in retry + validation:
-
-```python
-MAX_SUB_AGENT_RETRIES = 2
-
-for attempt in range(MAX_SUB_AGENT_RETRIES + 1):
-    try:
-        result = await sub_agent.run(context, question=q)
-        validation = validator.validate(result)
-        if validation.passed or attempt == MAX_SUB_AGENT_RETRIES:
-            return format_for_llm(result, validation.warnings), result
-        continue  # retry on validation failure
-    except AgentRetryableError:
-        if attempt < MAX_SUB_AGENT_RETRIES: continue
-        return "Failed after retries", None
-    except AgentFatalError as e:
-        return f"Fatal: {e}", None  # no retry
-```
+Wrap every sub-agent call in retry **and** validation, and keep the two apart: a call that
+threw and a call that returned something unusable need different answers. Retry the first,
+re-prompt or fail the second.
 
-**Error hierarchy:**
-```
-AgentError (base)
-├── AgentRetryableError    → orchestrator retries with adjusted context
-├── AgentFatalError        → unrecoverable (bad config, auth failure)
-├── AgentTimeoutError      → retry with smaller context
-└── AgentValidationError   → sub-agent result failed quality checks
-```
+Three decisions the rest follows from:
 
-**Result validation** (check before returning to user):
-- SQL: query present? execution error? zero rows (warn)? slow query >30s (warn)?
-- Viz: valid chart type? appropriate for data shape? (pie with 100 slices → bar)
-- Knowledge: non-empty answer? source citations present?
+- **Split errors into retryable and fatal at the type level**, not at the call site. A bad
+  credential and an overloaded provider are both exceptions and only one is worth a second
+  attempt.
+- **Validate the result before it reaches the user**, against the shape the caller
+  expects — rows present, columns named, a citation attached. A confident wrong answer
+  passes every check that only looks for an exception.
+- **Cap the attempts and return the best partial**, because the alternative to a partial
+  answer is not a better answer, it is no answer and a spent budget.
 
----
+The hierarchy, the loop and the per-domain validators:
+[`references/patterns.md`](references/patterns.md).
 
 ## 5. Multi-Stage Pipeline (Complex Path)
 
-For complex queries requiring multiple data steps:
+When one loop is not the shape — several data steps that depend on each other, a person
+who has to approve something in the middle, a run that must survive the gap between two
+messages — the orchestrator plans first and executes stages instead of tools.
 
-### Complexity Detection
-
-Two-tier: fast heuristic + optional LLM check.
-
-```python
-COMPLEXITY_KEYWORDS = ["summary table", "pivot", "cross-reference", "compare",
-                       "for each", "step 1", "first find", "then"]
-
-def detect_complexity(question, history) -> bool:
-    return any(kw in question.lower() for kw in COMPLEXITY_KEYWORDS)
-
-async def detect_complexity_adaptive(question, llm, history) -> bool:
-    # Lightweight LLM call: "Is this simple or complex? Reply 'simple' or 'complex'."
-    resp = await llm.complete([...], max_tokens=10)
-    return "complex" in resp.content.lower()
-```
-
-### Pipeline Components
-
-```
-QueryPlanner   → (single LLM call) → ExecutionPlan (stages + their depends_on)
-StageExecutor  → runs stages in DEPENDENCY LAYERS with validation + retry (§13)
-StageValidator → checks data shape, row bounds, cross-stage consistency
-StageContext   → in-memory state (plan, results per stage, user feedback)
-PipelineRun    → DB-persisted state for resume/retry across requests
-```
-
-### Checkpoint Pattern (Human-in-the-Loop)
-
-```python
-for layer in plan.layers():                       # Kahn over depends_on — never list order
-    results = await run_layer(layer, context)     # execute_with_retries per stage, together
-
-    for i, (stage, result) in enumerate(zip(layer, results)):
-        validation = validator.validate(stage, result, stage_ctx)
-        if not validation.passed:
-            results[i] = await retry_failed_validation(stage, context, validation)
-            if results[i] is None:
-                return StageFailedResult(stage, validation)   # ask user
-
-    if len(layer) > 1:                            # cheap per-stage checks ran first; this
-        verdict = checker.check(results)          # one is the cross-item gate (§13)
-        if not verdict.passed:
-            return StageFailedResult(layer, verdict)  # nothing converges on a flagged output
-
-    for stage, result in zip(layer, results):
-        stage_ctx.set_result(stage.id, result)
-
-    if any(s.checkpoint for s in layer):
-        persist_to_db(pipeline_run_id, stage_ctx)
-        return CheckpointResult(layer, results)   # pause for user review
-        # User responds: "continue" | "modify" | "retry"
-```
+Three decisions belong here; the mechanism is
+[`references/pipeline.md`](references/pipeline.md).
 
-### Pipeline Resume
-
-```python
-async def resume_pipeline(resume_info, context):
-    pipeline_run = load_from_db(resume_info["pipeline_run_id"])
-    plan = ExecutionPlan.from_json(pipeline_run.plan_json)
-    stage_ctx = StageContext.from_persistence(...)
-
-    resume_from = current_idx + 1 if action == "continue" else current_idx
-    return await executor.execute(plan, context, resume_from=resume_from, stage_ctx=stage_ctx)
-```
-
----
+- **Detect complexity in two tiers**, cheap first: a keyword heuristic, then one small
+  model call only where the heuristic is unsure. Paying a model to classify every question
+  is a tax on the common case.
+- **Execute in dependency layers, never in list order** (§13). A plan that declares
+  `depends_on` and is then walked down the list has serialised itself, and a layer of more
+  than one stage gets a checker before anything consumes it.
+- **A checkpoint is a pause that frees the worker.** If waiting for a human costs a
+  process, long approvals get quietly designed out — which is how a human-in-the-loop
+  system stops having one.
 
 ## 6. LLM Provider Routing
 
@@ -324,51 +248,24 @@ a floor.
 mode cross a compaction boundary as copied typed blocks, not prose (§12).
 ## 8. Self-Learning Feedback Loops
 
-Three cycles feed layers 3 and 4, and they differ by what supplies the signal:
-
-| Cycle | Signal | Produces |
-|---|---|---|
-| **Validation** | the attempt sequence of a call that failed and was then fixed | a learning, extracted by heuristic — the wrong table, a renamed column, a unit divisor, a soft-delete filter, a missing `LIMIT`. Deep LLM analysis only past 3 attempts, on a cooldown |
-| **User feedback** | a thumbs-down, or a data verdict of confirmed / approximate / rejected | a benchmark, a session note with the deviation, or a learning plus a flag on the now-stale benchmark |
-| **Lifecycle** | time, and contradiction | decay, conflict resolution by negation flip, and promotion of a pattern seen on two independent resources |
-
-The extractors, the exact confidence arithmetic and the promotion query live in
-`references/patterns.md` — **Learning Extraction Heuristics**, **Confidence Management**
-and **Cross-Resource Learning Transfer** — and not here, because a decay rate is a
-constant to tune and a constant with two homes is one that will disagree with itself.
+Three cycles feed the memory layers, and they differ by what supplies the signal: a failed
+attempt that was then fixed, a user's verdict, and time.
 
 **The rule the whole section exists for:** a learning is written from a **contrast** — the
-attempt that failed beside the attempt that worked — never from a single successful run.
-A system that learns from its successes learns its own habits.
-
----
-
-## 9. Observability (SSE Event Streaming)
+attempt that failed beside the attempt that worked — never from a single successful run. A
+system that learns from its successes learns its own habits.
 
-Real-time progress via `WorkflowTracker`:
+The extractors, the confidence arithmetic and the promotion query:
+[`references/patterns.md`](references/patterns.md).
 
-```python
-class WorkflowTracker:
-    # In-memory event bus with asyncio.Queue subscribers
-    async def begin(pipeline, context) -> workflow_id
-    async def emit(wf_id, step, status, detail)
-    async def end(wf_id, agent, status, detail)
-
-    @asynccontextmanager
-    async def step(wf_id, step_name, description):
-        # Emits started/completed/failed with elapsed_ms
-
-# Event types:
-# pipeline_start/end, thinking, token (streaming), orchestrator:llm_call,
-# orchestrator:sql_agent, orchestrator:llm_retry, orchestrator:warning
-```
-
-The final answer streams in chunks as `token` events on the same bus — a typing effect is
-a chunked emit, not a second mechanism. What makes the feed reliable rather than decorative
-is in `references/runtime.md`: a monotonic id per event so a reconnecting client can resume,
-and the feed being a **view over the durable trace** rather than the record itself.
+## 9. Observability
 
----
+One bus, an event per step, and the answer streamed as chunks on the same bus. Two
+properties decide whether it is a feed or a decoration: every event carries a **monotonic
+id**, so a reconnecting client resumes rather than missing the run, and the feed is a
+**view over a durable trace**, never the record itself — a stream nobody stored is a run
+`agent-evals` cannot evaluate. The tracker's shape:
+[`references/runtime.md`](references/runtime.md).
 
 ## 10. Dynamic System Prompts
 
@@ -394,18 +291,8 @@ list is built from, so the two can never disagree.
 
 ## 11. Clarification Requests (ask_user)
 
-Interrupt the tool loop to ask the user:
-
-```python
-async def handle_ask_user(tc, context, wf_id):
-    payload = {"question": ..., "question_type": "multiple_choice",
-               "options": [...], "context": "why I'm asking"}
-    raise _ClarificationRequestError(json.dumps(payload))
-    # Caught in orchestrator.run() → returns AgentResponse(response_type="clarification_request")
-    # Frontend renders special UI, user responds, next message continues flow
-```
-
----
+Stopping to ask is the same suspend-and-resume as a checkpoint with a different caller —
+one contract, not two ([`references/pipeline.md`](references/pipeline.md)).
 
 ## 12. Context Engineering
 
@@ -457,44 +344,38 @@ Four rules, and these are the ones that change code:
 
 ## Checklist — Building a New Orchestrator
 
-- [ ] Shared `AgentContext` dataclass with all sub-agents
-- [ ] `BaseAgent` protocol with typed results + `accum_usage()`
-- [ ] Tool-calling loop with max iterations guard
-- [ ] In-loop context trimming (80% compress, 70% wrap-up)
-- [ ] Parallel tool dispatch where independent, sequential where stateful
-- [ ] Sub-agent retry with validation (retryable vs fatal errors)
-- [ ] Multi-provider LLM router with fallback chain + health checks
-- [ ] Per-provider retry with exponential backoff (respect `retry_after`)
-- [ ] Unified LLM error hierarchy with `user_message` property
-- [ ] Context budget manager (priority-based allocation)
-- [ ] Dynamic system prompt (capability-aware, learning-injected)
-- [ ] Chat history trimming (tool condensing, LLM summarization)
-- [ ] Working memory (session notes, fuzzy dedup, confidence decay)
-- [ ] Long-term learnings (heuristic extraction, conflict resolution, global patterns)
-- [ ] Insight memory (lifecycle, trust scoring, decay)
-- [ ] Feedback pipeline (thumbs, data validation → learnings/notes/benchmarks)
-- [ ] SSE event streaming for real-time progress
-- [ ] Complexity detection (heuristic + adaptive LLM)
-- [ ] Multi-stage pipeline with checkpoints and resume
-- [ ] `ask_user` clarification mechanism
-- [ ] Graceful degradation (partial answers on context overflow or max iterations)
-- [ ] Compaction ladder, tool-pair-safe boundaries, typed carryover, output offload
-- [ ] Every declared dependency names the data it carries — the fake-edge test run once
-- [ ] Plans executed in dependency layers, not in the order the stages were listed
-- [ ] A checker between every parallel layer and the node that consumes it, and that
-      checker watched refusing a planted bad input at least once
-
----
+The sections above are the map. These are the items a reader **cannot** derive from a
+heading — the ones that were learned by getting them wrong:
+
+- [ ] In-loop trimming at ~80% of the window, wrap-up injected at ~70%, and a max-iteration
+      guard that composes a partial answer rather than returning nothing
+- [ ] A recoverable provider error **refunds** its iteration; a misconfiguration must not
+      spend the budget that exists to stop a runaway
+- [ ] Retries and fallbacks are capped **in total** — three providers × three retries is
+      nine calls for one prompt
+- [ ] A provider marked unhealthy is probed on a schedule; a health check that only runs on
+      failure never recovers, and the chain runs one short with nobody seeing it
+- [ ] Chat history has a **floor** — a session that trims it to fit old learnings has chosen
+      generalities over what the user said a minute ago
+- [ ] Every declared dependency names the data it carries; plans execute in layers, and a
+      layer of more than one gets a checker before anything consumes it (§13)
+- [ ] That checker has been watched refusing a planted bad input, and its verdicts are
+      stored as scores — one that has never rejected anything is a finding
+- [ ] Sub-agents return **distilled summaries**, not transcripts; a return value proportional
+      to the input is a function call wearing a costume
+- [ ] Model, window and price are resolved at one boundary from configuration or the
+      provider — never from a table of vendor ids in source
+- [ ] An eval exists before the prompt is tuned, or the tuning is folklore
 
 ## References
 
-The checklist above is the map, these are the territory. Each file opens with its
-own **Load this when** line — the authoritative trigger lives there, so this table
-stays an index and the two cannot drift apart.
+Each file opens with its own **Load this when** line — the authoritative trigger lives
+there, so this table stays an index and the two cannot drift apart.
 
 | File | Read it when |
 |---|---|
 | [`references/graph-engineering.md`](references/graph-engineering.md) | you are deciding the **shape of the work** — the fake-edge test, the diamond, the checker node, static versus dynamic, and what the host actually runs |
+| [`references/pipeline.md`](references/pipeline.md) | one loop is **not the shape** — the planned path, its checkpoints, resume, and the interrupt that asks a person |
 | [`references/patterns.md`](references/patterns.md) | you need the **data models and algorithms** under the body |
 | [`references/context-engineering.md`](references/context-engineering.md) | the loop is **running out of window** |
 | [`references/runtime.md`](references/runtime.md) | the agent must **survive a crash, a pause, a second message or a schedule** |

package/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md

@@ -12,6 +12,8 @@ that costs no LLM call.
 - [Pipeline Data Models](#pipeline-data-models)
 - [Validation Loop (SQL Execution)](#validation-loop-sql-execution)
 - [Context Window Sizes](#context-window-sizes)
+- [Sub-agent retry, and the error hierarchy under it](#sub-agent-retry-and-the-error-hierarchy-under-it)
+- [The three learning cycles, and what each one consumes](#the-three-learning-cycles-and-what-each-one-consumes)
 - [Learning Extraction Heuristics](#learning-extraction-heuristics)
 - [Confidence Management](#confidence-management)
 - [Fuzzy Deduplication Pattern](#fuzzy-deduplication-pattern)
@@ -391,3 +393,64 @@ def generate_followups(query, columns, row_count) -> list[str]:
     random.shuffle(pool)
     return pool[:3]
 ```
+
+## Sub-agent retry, and the error hierarchy under it
+
+Wrap every sub-agent call in retry + validation:
+
+```python
+MAX_SUB_AGENT_RETRIES = 2
+
+for attempt in range(MAX_SUB_AGENT_RETRIES + 1):
+    try:
+        result = await sub_agent.run(context, question=q)
+        validation = validator.validate(result)
+        if validation.passed or attempt == MAX_SUB_AGENT_RETRIES:
+            return format_for_llm(result, validation.warnings), result
+        continue  # retry on validation failure
+    except AgentRetryableError:
+        if attempt < MAX_SUB_AGENT_RETRIES: continue
+        return "Failed after retries", None
+    except AgentFatalError as e:
+        return f"Fatal: {e}", None  # no retry
+```
+
+**Error hierarchy:**
+```
+AgentError (base)
+├── AgentRetryableError    → orchestrator retries with adjusted context
+├── AgentFatalError        → unrecoverable (bad config, auth failure)
+├── AgentTimeoutError      → retry with smaller context
+└── AgentValidationError   → sub-agent result failed quality checks
+```
+
+**Result validation** (check before returning to user):
+- SQL: query present? execution error? zero rows (warn)? slow query >30s (warn)?
+- Viz: valid chart type? appropriate for data shape? (pie with 100 slices → bar)
+- Knowledge: non-empty answer? source citations present?
+
+---
+
+
+## The three learning cycles, and what each one consumes
+
+Three cycles feed layers 3 and 4, and they differ by what supplies the signal:
+
+| Cycle | Signal | Produces |
+|---|---|---|
+| **Validation** | the attempt sequence of a call that failed and was then fixed | a learning, extracted by heuristic — the wrong table, a renamed column, a unit divisor, a soft-delete filter, a missing `LIMIT`. Deep LLM analysis only past 3 attempts, on a cooldown |
+| **User feedback** | a thumbs-down, or a data verdict of confirmed / approximate / rejected | a benchmark, a session note with the deviation, or a learning plus a flag on the now-stale benchmark |
+| **Lifecycle** | time, and contradiction | decay, conflict resolution by negation flip, and promotion of a pattern seen on two independent resources |
+
+The extractors, the exact confidence arithmetic and the promotion query live in
+`references/patterns.md` — **Learning Extraction Heuristics**, **Confidence Management**
+and **Cross-Resource Learning Transfer** — and not here, because a decay rate is a
+constant to tune and a constant with two homes is one that will disagree with itself.
+
+
+---
+
+
+Both moved out of `SKILL.md` on 2026-08-16. The mechanisms they describe were already
+in this file — the validation loop, the extractors, the confidence arithmetic — so the
+body was holding a second copy of their surface. One home; the body keeps the decision.

package/plugins/agent-stack/skills/agent-orchestrator/references/pipeline.md

@@ -0,0 +1,110 @@
+# The pipeline path — planning, checkpoints, resume, and the human in the middle
+
+**Load this when** one tool-calling loop is not the shape: the question needs several data
+steps that depend on each other, a person has to approve something in the middle, or a run
+must survive the gap between two of the user's messages.
+
+**Spec pinned:** this pack's own patterns, from a production multi-agent system · read 2026-08-16
+
+`SKILL.md` §2 owns the simple path — one loop, tools, an iteration guard — and §13 owns the
+question that comes before both: what shape is this work. This file is the **complex**
+path, and it was in the body until 2026-08-16, when the body went 920 tokens over the
+budget this pack set itself and the honest fix was to split a layer rather than trim
+sentences off every section.
+
+Two things live here because they are one mechanism seen from two sides: a pipeline that
+**pauses** at a checkpoint and a loop that **interrupts** to ask a question are the same
+suspend-and-resume with different callers. `references/runtime.md` states that contract in
+the abstract; this is what it looks like in the orchestrator.
+
+## Contents
+
+- The complex path, end to end
+- Clarification requests — interrupting the loop to ask
+
+## The complex path, end to end
+
+For complex queries requiring multiple data steps:
+
+### Complexity Detection
+
+Two-tier: fast heuristic + optional LLM check.
+
+```python
+COMPLEXITY_KEYWORDS = ["summary table", "pivot", "cross-reference", "compare",
+                       "for each", "step 1", "first find", "then"]
+
+def detect_complexity(question, history) -> bool:
+    return any(kw in question.lower() for kw in COMPLEXITY_KEYWORDS)
+
+async def detect_complexity_adaptive(question, llm, history) -> bool:
+    # Lightweight LLM call: "Is this simple or complex? Reply 'simple' or 'complex'."
+    resp = await llm.complete([...], max_tokens=10)
+    return "complex" in resp.content.lower()
+```
+
+### Pipeline Components
+
+```
+QueryPlanner   → (single LLM call) → ExecutionPlan (stages + their depends_on)
+StageExecutor  → runs stages in DEPENDENCY LAYERS with validation + retry (§13)
+StageValidator → checks data shape, row bounds, cross-stage consistency
+StageContext   → in-memory state (plan, results per stage, user feedback)
+PipelineRun    → DB-persisted state for resume/retry across requests
+```
+
+### Checkpoint Pattern (Human-in-the-Loop)
+
+```python
+for layer in plan.layers():                       # Kahn over depends_on — never list order
+    results = await run_layer(layer, context)     # execute_with_retries per stage, together
+
+    for i, (stage, result) in enumerate(zip(layer, results)):
+        validation = validator.validate(stage, result, stage_ctx)
+        if not validation.passed:
+            results[i] = await retry_failed_validation(stage, context, validation)
+            if results[i] is None:
+                return StageFailedResult(stage, validation)   # ask user
+
+    if len(layer) > 1:                            # cheap per-stage checks ran first; this
+        verdict = checker.check(results)          # one is the cross-item gate (§13)
+        if not verdict.passed:
+            return StageFailedResult(layer, verdict)  # nothing converges on a flagged output
+
+    for stage, result in zip(layer, results):
+        stage_ctx.set_result(stage.id, result)
+
+    if any(s.checkpoint for s in layer):
+        persist_to_db(pipeline_run_id, stage_ctx)
+        return CheckpointResult(layer, results)   # pause for user review
+        # User responds: "continue" | "modify" | "retry"
+```
+
+### Pipeline Resume
+
+```python
+async def resume_pipeline(resume_info, context):
+    pipeline_run = load_from_db(resume_info["pipeline_run_id"])
+    plan = ExecutionPlan.from_json(pipeline_run.plan_json)
+    stage_ctx = StageContext.from_persistence(...)
+
+    resume_from = current_idx + 1 if action == "continue" else current_idx
+    return await executor.execute(plan, context, resume_from=resume_from, stage_ctx=stage_ctx)
+```
+
+---
+
+## Clarification requests — interrupting the loop to ask
+
+Interrupt the tool loop to ask the user:
+
+```python
+async def handle_ask_user(tc, context, wf_id):
+    payload = {"question": ..., "question_type": "multiple_choice",
+               "options": [...], "context": "why I'm asking"}
+    raise _ClarificationRequestError(json.dumps(payload))
+    # Caught in orchestrator.run() → returns AgentResponse(response_type="clarification_request")
+    # Frontend renders special UI, user responds, next message continues flow
+```
+
+---

package/plugins/agent-stack/skills/agent-orchestrator/references/runtime.md

@@ -16,6 +16,7 @@ has only built the first one discovers the second in production.
 - The interrupt/resume contract
 - Double-texting: four policies
 - Streaming that survives a dropped connection
+- The tracker, concretely — what the feed above is made of
 - Time travel and forking
 - Scheduled and sleep-time work
 - Middleware: the seven concerns, unwelded
@@ -95,6 +96,38 @@ client can rely on:
   of what happened is a stream nobody stored, evaluation is impossible — see the
   `agent-evals` skill, which cannot function without it.
 
+## The tracker, concretely — what the feed above is made of
+
+Real-time progress via `WorkflowTracker`:
+
+```python
+class WorkflowTracker:
+    # In-memory event bus with asyncio.Queue subscribers
+    async def begin(pipeline, context) -> workflow_id
+    async def emit(wf_id, step, status, detail)
+    async def end(wf_id, agent, status, detail)
+
+    @asynccontextmanager
+    async def step(wf_id, step_name, description):
+        # Emits started/completed/failed with elapsed_ms
+
+# Event types:
+# pipeline_start/end, thinking, token (streaming), orchestrator:llm_call,
+# orchestrator:sql_agent, orchestrator:llm_retry, orchestrator:warning
+```
+
+The final answer streams in chunks as `token` events on the same bus — a typing effect is
+a chunked emit, not a second mechanism. What makes the feed reliable rather than decorative
+is in `references/runtime.md`: a monotonic id per event so a reconnecting client can resume,
+and the feed being a **view over the durable trace** rather than the record itself.
+
+---
+
+This moved out of `SKILL.md` on 2026-08-16. It was the concrete half of the section
+two headings up, in a different file: *streaming that survives a dropped connection*
+stated the two properties that matter and the body stated the API without them. One
+home, and the properties now sit beside the thing they are properties of.
+
 ## Time travel and forking
 
 Once every iteration is checkpointed, one capability follows nearly free: **pick a past