@multileaf/ai-workflow 0.7.0 → 0.9.0

package/docs/token-efficiency-research.md

@@ -0,0 +1,135 @@
+# Token Efficiency Research
+
+Research notes and external references for the open problem: **guaranteeing low token consumption without sacrificing output quality**. This document consolidates published methodologies, open-source implementations, and a recommended path for AI Workflow.
+
+> Methodology note: sources were collected via GitHub Search API and web search. Descriptions and savings figures come from the projects' own READMEs and blog posts and have not been independently validated. Last reviewed: October 2026.
+
+## The core problem
+
+Token optimization without a quality constraint is indistinguishable from degradation. Every credible source converges on the same inversion of the problem:
+
+1. **Fix quality first** as a measurable constraint (benchmark of real work).
+2. **Then minimize cost subject to that constraint.**
+
+"Guaranteeing" low consumption is only possible once quality is pinned to a benchmark that an optimization cannot silently regress.
+
+## Reference publications
+
+| Publication                                                                                                                                                                       | Key contribution                                                                                                                                                                                                 | Relevance to AI Workflow                                                                   |
+| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
+| Uber Engineering — [Running a Software Factory Efficiently at Uber Scale](https://www.uber.com/blog/efficient-software-factory/)                                                  | Cost equation decomposed into six multiplicative, independently measurable terms; sequential driver decomposition with no unexplained residual; outcome-denominated cost (cost per merged PR, per triaged alert) | The measurement model to adopt: decompose, attribute, and denominate cost by unit of value |
+| Anthropic — [Prompt Caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) and [Token-Saving Updates](https://www.anthropic.com/news/token-saving-updates) | Cache reads at ~0.1x input price; TTL economics (5 min at 1.25x write, 1 h at 2x); thinking blocks cached in tool-use loops                                                                                      | Cache-friendly ordering of system prompt, rules, and project profile                       |
+| Microsoft Azure — [The Economics of Agent Optimization](https://azure.microsoft.com/en-us/blog/the-economics-of-agent-optimization-context-engineering-for-enterprise-ai-agents/) | Context engineering framed as the primary economic lever for enterprise agents                                                                                                                                   | Aligns with product principle 6 (just-in-time context)                                     |
+| LMSYS — [RouteLLM](https://www.lmsys.org/blog/2024-07-01-routellm/)                                                                                                               | Formal cost/quality trade-off for model routing; 2x+ cost reduction with quality held                                                                                                                            | Routing strategy for the provider-neutral `AiProvider` interface                           |
+| GetBeam — [AI Agent Token Cost Optimization](https://getbeam.dev/blog/ai-agent-token-cost-optimization.html)                                                                      | Practical case study with spend decomposition                                                                                                                                                                    | Cross-check on decomposition methodology                                                   |
+
+### Common pattern across publications
+
+Savings never come from a single technique. The pattern is:
+
+1. Measure per task, not per aggregate.
+2. Pin quality with a benchmark built from real work.
+3. Optimize each cost term separately (price/token, tokens/request, requests/turn).
+4. Hold the model constant when attributing savings (behavior shifts with every upgrade).
+
+## Repositories by category
+
+### Context engineering and token reduction
+
+| Repository                                                                                                          | Stars | Notes                                                                                                                                                                 |
+| ------------------------------------------------------------------------------------------------------------------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [microsoft/LLMLingua](https://github.com/microsoft/LLMLingua)                                                       | ~6.8k | Prompt compression (EMNLP'23, ACL'24); 5–20x compression. `gladehq/claude-shorthand` applies it to Claude Code (~55% claimed). Last-resort layer due to quality risk. |
+| [ratel-ai/ratel](https://github.com/ratel-ai/ratel)                                                                 | ~484  | Context engineering for agents: ~80% fewer tokens via fixing tool overload; in-process BM25 skills and memory. Architecturally the closest peer.                      |
+| [jia-gao/leanctx](https://github.com/jia-gao/leanctx)                                                               | ~323  | Drop-in prompt compression, 40–60% claimed without app changes.                                                                                                       |
+| [rossoclt/context-guru](https://github.com/rossoclt/context-guru)                                                   | ~70   | Context optimization for coding agents: lower cost and latency "without sacrificing accuracy".                                                                        |
+| [pleasedodisturb/awesome-llm-token-optimization](https://github.com/pleasedodisturb/awesome-llm-token-optimization) | ~99   | **The reference map.** Quick-wins table (cache 90% input, routing 60–95%, batch 50%, compression 5–20x), papers, and tools by category. Read first.                   |
+| [wai-21/llm-token-optimization](https://github.com/wai-21/llm-token-optimization)                                   | ~2    | Decision Distillation, skill routing; 60–99% claimed. Unvalidated.                                                                                                    |
+| [8b-is/smart-tree](https://github.com/8b-is/smart-tree)                                                             | ~271  | Context-aware filesystem tooling replacing 20+ tools; token reduction via tool design.                                                                                |
+
+### Telemetry across harnesses
+
+| Repository                                                                        | Stars | Notes                                                                                                                                                                                            |
+| --------------------------------------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [pulsemcp/switchboard](https://github.com/pulsemcp/switchboard)                   | new   | Self-hosted capture of agent transcripts (Claude Code, Codex, others). Cross-harness data source without vendor lock-in — the piece Uber gets from its gateway but a portable system must build. |
+| [DeepAgentLabs/agenticlens](https://github.com/DeepAgentLabs/agenticlens)         | ~58   | Open-source profiler: token usage, cost, latency, and optimization opportunities per session.                                                                                                    |
+| [traceloop/openllmetry](https://github.com/traceloop/openllmetry)                 | ~7.5k | GenAI observability on OpenTelemetry. The interoperable schema to adopt if telemetry grows beyond local use.                                                                                     |
+| [gocenalper/agent-optimization](https://github.com/gocenalper/agent-optimization) | ~12   | Live token monitor for Claude Code / Codex: cost tracking, analysis, LLM-powered suggestions.                                                                                                    |
+| [BerriAI/litellm](https://github.com/BerriAI/litellm)                             | high  | Proxy with routing and cost tracking for 100+ providers; candidate neutral measurement layer.                                                                                                    |
+| [timharris707/modeldeck](https://github.com/timharris707/modeldeck)               | ~116  | Usage/quota tracking across Claude Code and Codex accounts.                                                                                                                                      |
+
+### Harnesses, workflows, and SDD
+
+| Repository                                                                          | Stars | Notes                                                                                                           |
+| ----------------------------------------------------------------------------------- | ----- | --------------------------------------------------------------------------------------------------------------- |
+| [github/spec-kit](https://github.com/github/spec-kit)                               | ~142k | De-facto standard for spec-driven development; direct philosophical competitor.                                 |
+| [cisco-open/ai-harness-toolkit](https://github.com/cisco-open/ai-harness-toolkit)   | ~15   | Skills for assessing AI-native engineering workflows in repositories.                                           |
+| [Zoria-Lind/dsh-token-optimizer](https://github.com/Zoria-Lind/dsh-token-optimizer) | ~20   | Layered pipeline: output ladder, MCP lazy loading, compaction driver. Small but architecturally interesting.    |
+| [ComposioHQ/composio](https://github.com/ComposioHQ/composio)                       | ~30k  | 1000+ toolkits with tool search and context management — the "tool search" pattern Uber describes, productized. |
+
+### Benchmarks with quality data (fix quality before optimizing)
+
+| Repository                                                                      | Stars | Notes                                                                                                                                                                                                            |
+| ------------------------------------------------------------------------------- | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [HumphreySun98/repoagentbench](https://github.com/HumphreySun98/repoagentbench) | ~27   | Mines merged PRs into a local, contamination-free coding-agent benchmark. This is Uber's uReview methodology, open-sourced.                                                                                      |
+| [mupt-ai/self-bench](https://github.com/mupt-ai/self-bench)                     | ~33   | Benchmarks agents on your own repository, Harbor-formatted evals.                                                                                                                                                |
+| [jettison-ai/jettison-parity](https://github.com/jettison-ai/jettison-parity)   | new   | Benchmark harness for context-optimization middleware: task parity, session cost, harness-agnostic. Defines exactly the experiment needed: same task, different context configuration, measure cost and quality. |
+
+## Savings reference table
+
+From the awesome-list and Uber's post, ordered by effort-to-savings ratio:
+
+| Lever                                        | Claimed savings        | Quality risk               | Applies at              |
+| -------------------------------------------- | ---------------------- | -------------------------- | ----------------------- |
+| Prompt prefix caching                        | ~90% of cached input   | None (transparent)         | Provider layer          |
+| Tool schema lazy loading / tool search       | 50–70K tokens/session  | Low                        | Harness / MCP layer     |
+| Model routing (cheap model for simple tasks) | 60–95% on routed calls | Medium — needs benchmark   | Provider / orchestrator |
+| Batch APIs for non-urgent work               | ~50%                   | None (same output)         | Provider layer          |
+| Code-mode / batched tool calls               | 55–100% per workflow   | Low                        | Harness layer           |
+| Compaction budget                            | Context-dependent      | Medium                     | Harness layer           |
+| Prompt compression (LLMLingua)               | 5–20x                  | High — degrades edge cases | Last resort, middleware |
+
+Combined pipelines claim 95–99% versus a naive baseline, but every high-risk row requires the quality gate below.
+
+## Recommended path for AI Workflow
+
+A four-step guarantee pipeline, reusing existing pieces:
+
+### Step 1 — Fix quality with a self-benchmark
+
+Mine real SDD tasks (specs, plans, verification results from `.aiw/checkpoints/` and `.aiw/generated/`) into an evaluable set, following the `repoagentbench` / `self-bench` pattern. Define the acceptance rule: **an optimization ships only if outcome does not fall below threshold X% on the benchmark.** AI Workflow already has the ingredients: `evidence.ts`, `traceability.ts`, quality gates, and `src/fixtures`.
+
+### Step 2 — Decompose cost per term
+
+Extend `token-usage.ts` toward Uber-style decomposition, per SDD stage and per skill:
+
+- input vs output vs cached tokens per request;
+- requests per turn and turns per task;
+- cost per **verified requirement** (outcome-denominated), not per session.
+
+Data sources: agent JSONL logs (Claude Code, Codex — the `switchboard` approach) plus AIW checkpoints. Deliver through the existing `dashboard.ts`, evolving it toward anti-pattern detection (Uber reports 16 distinct session anti-patterns; the traces to detect them are already captured).
+
+### Step 3 — Apply levers in ROI order
+
+1. Stable cacheable prefix: system prompt + confirmed profile + rules first; dynamic context last.
+2. Routing: subagents and mechanical skills to cheaper models, manual override preserved (AIW's provider neutrality makes this a configuration, not a rewrite).
+3. Tool overload: lazy loading / on-demand tool schemas for MCP surfaces.
+4. Stage-budgeted compaction (already designed in `token-efficiency.md`).
+5. Prompt compression only behind the quality gate.
+
+### Step 4 — Continuous guardrail
+
+A CI gate that reruns the self-benchmark on changes to models, skills, or configuration and reports Δcost and Δquality. This is the "model held constant" discipline from Uber: savings claims are only attributed when the model is fixed. Fail the gate when quality regresses or cost exceeds threshold.
+
+### Suggested study order
+
+1. `pleasedodisturb/awesome-llm-token-optimization` — full map.
+2. `jettison-ai/jettison-parity` — how to design the cost/quality experiment.
+3. `HumphreySun98/repoagentbench` — how to build the benchmark.
+4. `pulsemcp/switchboard` and `DeepAgentLabs/agenticlens` — cross-harness telemetry layer.
+
+## Gaps and open questions
+
+- **Absolute cost data:** none of the sources publish per-task absolute costs in a form transferable to another codebase; only relative deltas. The self-benchmark must establish its own baseline.
+- **Quality drift across model upgrades:** Uber acknowledges behavior shifts with every upgrade; savings measured against one model do not transfer. The CI guardrail (step 4) is the only mitigation found.
+- **Compression safety:** no source provides a formal guarantee that prompt compression preserves behavior on edge cases; treat it as untrusted until benchmarked.
+- **Telemetry privacy:** most third-party trackers are local-first, but AI Workflow's telemetry consent model (opt-in, minimal payload) should gate any integration with observability backends.

package/docs/token-efficiency-study-notes.md

@@ -0,0 +1,128 @@
+# Token Efficiency Study Notes
+
+Deep-dive notes from studying the five reference repositories recommended in [token-efficiency-research.md](token-efficiency-research.md). Local clones live outside the repository at `../token-research/` (sibling of this repo, not committed).
+
+| Repository                       | Role in the plan                             | Maturity                                                                    |
+| -------------------------------- | -------------------------------------------- | --------------------------------------------------------------------------- |
+| `awesome-llm-token-optimization` | Complete map of levers, tools, and papers    | Curated list; strong curation discipline                                    |
+| `jettison-ai/jettison-parity`    | Cost/quality experiment design               | Small but rigorous: offline, deterministic, 29 tests, honest-labeling rules |
+| `HumphreySun98/repoagentbench`   | How to build a benchmark from real work      | Published on PyPI, real leaderboards (~$11 API spend for 12 runs)           |
+| `pulsemcp/switchboard`           | Cross-harness telemetry architecture         | Early (docs-first repo); architecture doc is the value                      |
+| `DeepAgentLabs/agenticlens`      | Evaluation, diagnosis, and model-fit scoring | Large active codebase (400+ files), versioned releases, research program    |
+
+## 1. awesome-llm-token-optimization — the map
+
+Beyond the levers already captured in the research doc, the full read surfaced:
+
+- **GitHub's own engineering result** ([github.blog](https://github.blog/ai-and-ml/github-copilot/improving-token-efficiency-in-github-agentic-workflows/)): pruning unused MCP tool schemas and swapping MCP calls for the `gh` CLI cut agentic-CI token spend **43–62%** across real workflows. This independently replicates Uber's CLI-resolved MCP lever.
+- **Cockroach Labs**: re-sent context can be **62% of agent inference bills** — quantifies why per-turn context, not generation, is the dominant cost.
+- **ProjectDiscovery**: raising cache hit rate from 7% to 84% across 9.8B cached tokens — the single largest documented saving, achieved purely by prompt-structure discipline (stable prefix).
+- **Anthropic tool search**: 85% token reduction for large tool libraries; token-efficient tool use: 70% output reduction. Both are official provider features, not middleware.
+- **Browser tool efficiency**: accessibility-tree approaches are 10–50x smaller than raw HTML; AI-summarized fetch ~20x better than Playwright MCP (1.5 KB vs 10–33 KB per page). Relevant to any web-driven skill.
+- **Lossless rule-based compression achieves 3–4:1** with no model: strip prose/hedging, preserve numbers/entities/constraints/risks, dense bullets, self-contained 3–5K token sections. This is essentially AI Workflow's existing context strategy, externally validated.
+- **Tokenizer warning**: newer Anthropic tokenizers produce ~30% more tokens for identical text at unchanged per-token prices — model upgrades can silently raise cost without any price change. Any cost attribution must be tokenizer-aware (Uber's "hold the model constant" rule exists for the same reason).
+- **Router research caveat**: a 2026 game-theoretic analysis (arxiv:2602.09902) argues optimal routing is usually _static_ — dynamism mostly adds failure modes. Prefer a small fixed routing table per task category over clever per-request routing.
+
+## 2. jettison-parity — the experiment design
+
+The most directly transferable artifact. It answers cost and quality **together**, deterministically and offline.
+
+### Four benchmark families
+
+| Family           | Measures                                                                                                                                                                     | Transferable idea                                                                                                   |
+| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
+| Standing context | Tokens per turn of persistent context (tool schemas, skill bodies, instruction files), per category, byte-reproducible                                                       | Measure AIW's rendered context (AGENTS.md/CLAUDE.md, skills, rules) the same way — before/after any resource change |
+| Task parity      | Same task run baseline vs optimized; scores completion, correct tool + params, **critical-fact retention**, rounds, tokens. **Exits nonzero on any regression**              | The quality gate: `critical_facts` containment checks that facts which must survive compression actually survive    |
+| Session cost     | Dollars of an N-turn session pricing cache-read / fresh-input / cache-write **separately**, in cache-hit and cache-miss regimes                                              | Naive `tokens_saved × price` overstates savings ~10x on cache-heavy agents — AIW cost math must be tier-aware       |
+| Holdout RCT      | Randomized trial over recorded request logs: sha256 arm assignment, mean tokens/request, 95% CI. Synthetic output is always labeled "synthetic demonstration — not evidence" | The standard for claiming a real-world saving once logs exist                                                       |
+
+### Honest-measurement rules (adopt verbatim)
+
+1. Every number carries a label: `measured` (exact tokenizer / provider price) vs `estimated`; labels propagate pessimistically.
+2. Synthetic is never evidence.
+3. Disclose the config alongside any number (content hash).
+4. Cache-aware dollars (per billing tier).
+5. No network, no clocks — byte-identical reruns.
+
+### Adapter surface
+
+A single method makes any optimizer scoreable: `optimize_request(body, provider) -> body`. AIW's equivalent boundary is the context assembly that produces a session's initial payload.
+
+### Task file schema (quality pin)
+
+```json
+"expected": {
+  "tool": "pg_describe_table",
+  "arguments": {"table": "orders"},
+  "critical_facts": ["Never run a migration", "db-staging.internal.example.com"],
+  "final_answer_contains": "DONE"
+}
+```
+
+Six shipped scenarios cover: tool selection among 20 tools, search→load→call chains, numeric constraints, security rules, output-format requirements, mixed turns.
+
+## 3. repoagentbench — building the benchmark
+
+### Mining pipeline (PR → task)
+
+1. PR title/body/base SHA/diff via `gh`.
+2. Clone at base commit.
+3. Split diff into `solution_tests.patch` + `solution_source.patch`; apply **tests only**, so the starting state is "post-PR tests vs pre-PR source".
+4. Auto-generate `verify.sh` for the detected framework.
+
+### Run protocol (the part to copy)
+
+- Fresh workdir + per-task venv per run.
+- `pre_verify` must **FAIL** (proves the task is broken), agent runs, `post_verify` must **PASS** (proves the fix).
+- Artifacts: `diff.patch`, `agent.log`, `events.jsonl` — replayable and diffable.
+- `mock-fix` oracle agent (applies the known PR diff) validates the harness itself: 3/3 pass.
+
+### Findings that matter
+
+- **The harness is part of the system under test**: same Anthropic model via aider (no-op fix, FAIL) vs native Claude Code (canonical fix, PASS) on the identical PR. Portability claims must be measured per-harness, not assumed.
+- Contamination control: mine PRs post-dating the model's training cutoff (cites arxiv:2512.10218 — frontier models score 3x better on potentially-contaminated benchmarks).
+- Cheap: ~$11 for 12 real-agent runs. A self-benchmark is not expensive.
+
+**AIW translation**: our unit is not a PR but an SDD task — spec → plan → implementation → verification. The analog of "PR tests" is the AIW quality gate / verification evidence; the analog of `pre_verify` is "the requirement has no passing evidence yet".
+
+## 4. switchboard — cross-harness telemetry architecture
+
+Early-stage, docs-first. The architectural pattern is the takeaway:
+
+- **Canonical format ("OpenTranscripts")**: every agent's transcript is converted by a deterministic, versioned, per-source converter into one schema. New agent → new converter; spec change → version bump.
+- **One hard determinism boundary**: raw → canonical is deterministic ETL; only segmentation and outcome scoring are LLM-driven, downstream and optional (MVP makes zero LLM calls).
+- **Hooks, not interception**: a small separately-distributed package per agent pipes sessions to user-owned storage (S3/GCS/local/HTTP behind one interface). The platform never owns the ingestion edge.
+- Sidecar metadata per session: who ran it, trigger, model, token usage, harness + version. Deterministic rollups over the canonical table feed org-level dashboards.
+
+**AIW translation**: AIW checkpoints are already a deterministic, versioned artifact per harness-agnostic run. The missing piece is per-session token/cost sidecar metadata recorded at checkpoint time (model, harness, tokens in/out/cached), which turns existing checkpoints into the `open_transcripts` equivalent without any new ingestion machinery.
+
+## 5. agenticlens — evaluation and model-fit
+
+Large active project (v0.10.0). Most relevant constructs:
+
+- **Evaluator versioning**: every score carries its evaluator's version; evidence audits tie recommendations to the step they concern.
+- **Experiments**: pass@k, pass^k, confidence intervals, significance tests, factor comparison — statistics done properly for agent evals.
+- **Failure taxonomy + attribution**: deterministic diagnosis rules attribute failures to faulty step/agent, validated on controlled failures.
+- **ModelFit**: task taxonomy, **hard constraints on quality/cost/latency/safety**, weighted scoring, per-category **Pareto frontiers with cost-per-success rankings**, recommendation confidence + explanations.
+- **Advisory routing**: turns saved ModelFit evidence into explainable runtime decisions — advisory by default, execution requires explicit opt-in, with a local audit journal. Aligns with the static-routing finding from §1.
+
+**AIW translation**: ModelFit's constraint-then-Pareto structure is exactly step 1–3 of our recommended path; "cost per success" is the outcome-denominated metric from the Uber post, productized.
+
+## Synthesis: implementation blueprint for steps 1–2
+
+Combining the five sources, the sequence with concrete designs:
+
+1. **Standing-context benchmark** (jettison-parity, family 1): a command that renders AIW context for a fixture project and reports tokens per category (root context file, skills, rules, tool schemas) with a content hash. Deterministic, offline, zero API cost. First deliverable.
+2. **Task parity suite** (jettison-parity, family 2 + repoagentbench protocol): SDD tasks from `src/fixtures` with `expected` blocks (required artifacts, `critical_facts` that must survive context filtering, completion markers). Baseline vs optimized context; **exit nonzero on regression**. `pre_verify must fail / post_verify must pass` discipline via existing quality gates.
+3. **Checkpoint sidecar metadata** (switchboard pattern): record model, harness+version, token tiers (cache read / fresh input / cache write / output) into every `.aiw/checkpoints/` entry. Tier-aware, `measured` vs `estimated` labeled.
+4. **Cost-per-verified-requirement** (Uber + agenticlens ModelFit): join step-3 metadata with `evidence.ts`/`traceability.ts` to report cost per unit of value, per SDD stage and per skill.
+5. **Static routing table** (research consensus): a small per-task-category model matrix with hard quality constraints from step 2's benchmark; advisory with manual override, audit-logged.
+6. **Holdout RCT** (jettison-parity, family 4): once real session logs accumulate, upgrade claims from "benchmark delta" to CI-backed RCT evidence.
+
+### Guardrails to adopt wholesale
+
+- Honest labeling (`measured`/`estimated`, pessimistic propagation).
+- Synthetic results are never evidence.
+- Config hash disclosed with every published number.
+- Model (and tokenizer) held constant when attributing savings.
+- Oracle baseline (mock-fix equivalent) to validate the harness itself.

package/docs/token-efficiency.md

@@ -14,3 +14,160 @@ Token efficiency is a runtime concern, not only a prompt-writing concern.
 - Measure tokens per agent, package, task, and verification cycle.
 
 The runtime should prefer references such as `AUTH-REQ-04` and retrieve their content only when needed.
+
+## Standing-context benchmark
+
+`aiw context-bench` measures the persistent context AI Workflow renders into every agent session — deterministically, offline, and with zero API cost. It reports, per category (root context file, skills, rules, agents, hooks, templates, context layers): file count, measured characters, and estimated tokens (characters/4, always labeled `estimated`). A content hash pins the exact fixture behind every number, and the output contains no timestamps, so unchanged projects produce byte-identical reports.
+
+```bash
+aiw context-bench --save                      # record a checkpoint under .aiw/checkpoints/
+aiw context-bench --baseline=<checkpoint.yml> # exit 1 on any token regression
+```
+
+Any token increase against the baseline is a regression and fails the command (the parity-harness rule). This is the quality gate for context changes: optimize freely, but standing context may not silently grow.
+
+## Task parity suite
+
+`aiw parity` pins quality constraints on context assembly. Each parity task is one YAML file under `.aiw/parity/` (or `--tasks=directory`, relative to the project root) describing a scenario and its expectations:
+
+```yaml
+schema: 1
+id: deploy-guardrails
+description: Deployment constraints must survive just-in-time retrieval
+layers:
+  global: "Team conventions: trunk-based development."
+  project: "Never deploy on Fridays. Staging endpoint is db-staging.internal."
+  task: "Ship the migration with a rollback plan."
+critical_facts:
+  - Never deploy on Fridays
+  - db-staging.internal
+completion_markers:
+  - rollback plan
+```
+
+Every task runs twice against the same layers: the full composed context (baseline) and the just-in-time task-retrieval assembler. A task passes only when every critical fact and completion marker survives retrieval — exact containment checks, not similarity. If a fact lives only in a layer that retrieval drops (for example, `session`), the task fails and names the fact: that is a detected context loss, not a benchmark artifact. The command exits nonzero on any failure, so it can gate CI.
+
+### Inter-stage parity tasks
+
+Two optional fields turn a parity task into a check on the real handoff between SDD stages:
+
+```yaml
+schema: 1
+id: spec-to-plan
+description: Every requirement id from the approved specification must reach the planning context
+layers:
+  task: "Break the specification into ordered implementation tasks."
+sources:
+  project: .aiw/generated/specs/specification.md
+required_patterns:
+  - 'REQ-\d+'
+```
+
+`sources` loads a project-relative artifact into a layer (project, task, global, or session) before assembly, and `required_patterns` extracts every literal match from the loaded sources — each extracted identifier must survive assembly, or the task fails naming the exact loss (`missing_pattern: REQ-\d+ -> REQ-003`). This guards the identifiers traceability depends on: if a stage's context assembly drops a `REQ-###` or `TASK-###`, `aiw trace` cannot link it later. A missing source artifact is an explicit error, not a silent pass.
+
+Cost is reported (estimated tokens, baseline vs assembled) but never gates here — this suite is the quality gate; `aiw context-bench` is the cost gate. The two compose: optimize context freely, but neither standing cost nor critical facts may regress silently.
+
+## Usage sidecar
+
+`aiw usage-sidecar` records per-run token usage with billing tiers, measurement labels, and harness attribution, stored as a versioned ledger at `.aiw/checkpoints/usage-sidecar.yml` (sequence-ordered, no timestamps, capped at 500 entries). Four tiers are kept separate because naive `tokens x price` accounting overstates savings on cache-heavy agents:
+
+```bash
+# record a run (measured by default; pass --estimated when values are derived)
+aiw usage-sidecar record --command=session.plan --model=sonnet --harness=claude-code \
+  --cache-read=48000 --fresh-input=6000 --cache-write=2000 --output=3100
+
+aiw usage-sidecar report   # tier totals, cache hit rate, attribution by model and harness
+```
+
+Every entry carries `measured` or `estimated`, and aggregates propagate the label pessimistically: one estimated entry makes the reported total estimated. The report's `cache_hit_rate` (cache-read over billable input) is the single most economically meaningful ratio, since cached reads bill at a fraction of fresh input. Dollar conversion is deliberately not built in — provider prices rot, and unlabeled dollars would be dishonest by default.
+
+`aiw scan` auto-records sidecar entries for interpreter and embedding usage with their real measurement label, so the ledger fills without manual effort.
+
+### Importing harness session logs
+
+`aiw usage-sidecar import` turns local harness session logs into measured entries — the same lifecycle coverage as the manual `record`, without friction. Formats were validated against real logs:
+
+```bash
+aiw usage-sidecar import --from=claude-code   # ~/.claude/projects/<this-project>
+aiw usage-sidecar import --from=codex         # ~/.codex/sessions (sessions scoped by cwd)
+aiw usage-sidecar import --from=gemini        # ~/.gemini/antigravity-cli/brain transcripts
+```
+
+- **Claude Code**: per-message usage with all four tiers (including cache), model and CLI version, deduplicated by message id.
+- **Codex**: `token_usage_record` entries; cached input is separated from fresh; sessions are scoped to this project via `session_meta.cwd`.
+- **Gemini**: planner responses with input/cache-read/output. Transcripts do not record a working directory, so Gemini imports cannot be scoped per project (the command says so).
+
+Import is idempotent: every event derives a stable identity (`import.<source>.<hash>`) and already-present events are skipped, so reruns never double-count. Entries land with label `measured` and full harness attribution. Cursor and GitHub Copilot remain outside this coverage: they do not expose per-request usage to the user, so for them the estimate stays the ceiling.
+
+## Session heartbeat (cache-TTL management)
+
+Provider prompt caches expire on a TTL (typically 5 minutes); the cost lands when a session is resumed after expiry, because the next request rebuilds the whole prefix at full input price. Keeping a cache alive with provider pings has its own economics and AIW is not in the call path — but the gap can be measured for free on every event:
+
+```bash
+aiw hooks install --target=claude-code   # wires Stop + UserPromptSubmit to `aiw heartbeat`
+aiw heartbeat --session=<id>              # what the hooks call; also accepts hook JSON on stdin
+```
+
+Each beat measures the gap since the previous one and decides:
+
+- gap < warn (default 4 min): active session, nothing to do;
+- warn ≤ gap < TTL: the cache is still warm (this request just read it at cache price) — a compact checkpoint is saved now, so an idle past the TTL resumes cheap;
+- gap ≥ TTL: the cache expired and this request paid the rebuild — a compact checkpoint is saved so the _next_ gap hurts an order of magnitude less.
+
+No timers and no keep-alive pings: the decision always happens on the next beat, which costs nothing locally. The guarantee is that a session never pays the full-price prefix rebuild twice in a row on the same context. Thresholds are configurable (`--warn=seconds --ttl=seconds`), state lives in `.aiw/heartbeats.yml`, and every action writes an inspectable checkpoint under `.aiw/checkpoints/`. Hook installers are project-scoped, merge without touching foreign hook entries, and are idempotent; Codex (`notify`) and Gemini (`hooks.json`) adapters follow the same command.
+
+## Harness support matrix
+
+AI Workflow's core — the SDD workflow, quality and cost gates, token ledger, cost attribution, task-lifecycle usage, and routing — is harness-agnostic and behaves identically everywhere. What varies per harness is the edge: measured log import and live beats. The matrix declares exactly what is and is not available, because silent degradation is worse than an explicit limitation.
+
+| Capability                                      | Pi                                                               | Claude Code                          | Codex                                | Gemini                              | Cursor                     | Copilot CLI                |
+| ----------------------------------------------- | ---------------------------------------------------------------- | ------------------------------------ | ------------------------------------ | ----------------------------------- | -------------------------- | -------------------------- |
+| Neutral core (gates, ledger, reports, routing)  | ✅                                                               | ✅                                   | ✅                                   | ✅                                  | ✅                         | ✅                         |
+| Manual usage recording (`usage-sidecar record`) | ✅                                                               | ✅                                   | ✅                                   | ✅                                  | ✅                         | ✅                         |
+| Measured log import                             | ✅ all four tiers + provider per message, project-scoped         | ✅ all four tiers + model            | ✅ four tiers, project-scoped by cwd | ✅ three tiers, no project scoping  | ⚠️ local database, fragile | ❌ estimate is the ceiling |
+| Live beats (heartbeat)                          | ✅✅ extensions run in-process (event handlers, commands, tools) | ✅ hooks (Stop, UserPromptSubmit, …) | ⚠️ `notify` per turn                 | ✅ hooks (Pre/PostInvocation, Stop) | ⚠️ limited                 | ❌                         |
+| Multi-provider sessions                         | ✅ provider recorded per message                                 | —                                    | —                                    | —                                   | —                          | —                          |
+
+**Recommendation: use [Pi](https://pi.dev) for the best experience.** It is the only harness where every capability is native: per-message usage with all four billing tiers _and_ the provider, sessions grouped by project, compaction recorded with token counts, and an in-process extension API rich enough that AI Workflow can eventually run inside the harness itself. Claude Code is the strongest alternative; Codex and Gemini cover measurement fully with partial beat granularity. Where a capability is missing, AI Workflow degrades honestly: manual recording with `estimated` labels, and explicit errors instead of silent gaps.
+
+Notes:
+
+- Codex beats fire per turn (`notify`), not per event — enough for gap detection, coarser than Claude Code or Pi.
+- Gemini transcripts do not record a working directory, so their imports cover all sessions on the machine.
+- Cursor and Copilot do not expose per-request usage to the user; for them the estimate stays the ceiling, and every number derived from their sessions carries the `estimated` label.
+
+## Cost per verified requirement
+
+`aiw cost-report` joins the usage sidecar with the traceability graph to produce the outcome-denominated cost metric — tokens per verified requirement, where verified means the graph links the requirement to evidence:
+
+```bash
+aiw usage-sidecar record --command=session.verification --requirement=REQ-004 --fresh-input=9000 --output=2100
+aiw trace          # builds .aiw/generated/artifacts/traceability.yml
+aiw cost-report
+```
+
+Attributed and unattributed tokens are reported side by side so the ratio cannot be gamed by leaving expensive work unattributed, and the denominator counts only requirements with linked evidence — the same discipline as cost-per-merged-PR metrics, on AI Workflow's unit of value.
+
+## Model routing table
+
+`aiw routing` maintains a static, evidence-linked routing table at `.aiw/routing.yml`. Research consensus is that optimal routing is usually static — per-request dynamism mostly adds failure modes — so the table maps task categories (dot-separated kebab-case, e.g. `scan.interpreter`) to models deterministically:
+
+```bash
+aiw routing init                              # scaffold the table
+aiw routing resolve --category=scan.interpreter
+aiw routing show
+```
+
+Every rule carries an `evidence` field naming the benchmark, parity run, or ADR behind it, and the resolver states its advisory level honestly: `evidenced`, `unevidenced-rule` (benchmark it before trusting), `default-fallback`, or `default-unset` (a human must choose). Routing never invokes a model and humans can always override — the same advisory discipline as ModelFit.
+
+## Task-lifecycle token usage
+
+`aiw task-usage` measures the full life of one task — from `aiw task start` to the merged PR — in tokens, not dollars. Tokens are the model-invariant metric: routing part of a task to a frontier model and part to an entry-level model changes the price per token but not the input/output consumption, so lifecycle token totals stay comparable across model routing changes and harness migrations.
+
+```bash
+aiw usage-sidecar record --workflow=TEAM-123 --command=session.plan --model=sonnet \
+  --requirement=REQ-001 --cache-read=6000 --fresh-input=3000 --output=800
+aiw task-usage --workflow=TEAM-123   # or omit --workflow to use the active SDD task
+```
+
+All input tiers count as input tokens processed (cached reads still consume tokens; they just bill less), with the tier split and `cache_hit_rate` kept visible for cache-behavior analysis. The report shows totals, per-stage breakdown with the models each stage used, requirements touched, and harness attribution.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@multileaf/ai-workflow",
-  "version": "0.7.0",
+  "version": "0.9.0",
   "description": "Portable spec-driven development workflow for AI coding agents",
   "license": "MIT",
   "repository": {