pi-dev-team 0.2.0

package/docs/upstream/eval-maintenance.md

@@ -0,0 +1,95 @@
+# How eval testing works & keeping it current
+
+The conceptual model behind the agent evals and the discipline for keeping the
+corpus honest over time. For the architecture see [`eval-system.md`](eval-system.md);
+for the operational run procedure see [`eval-running-guide.md`](eval-running-guide.md).
+
+## The pieces
+
+| Piece | Where | Role |
+|---|---|---|
+| Fixtures | `evals/fixtures/` | Input code (deliberately good or bad) the agents review. |
+| Expectations | `evals/expected/*.json` | The **contract**: what a correct verdict looks like per fixture/agent. |
+| Grader | `scripts/eval_grade.py` | Deterministic, model-free: compares recorded actuals to expectations. |
+| Regression diff | `scripts/compare_eval_results.py` | Diffs two `--actuals` result files against `evals/expected/*.json`, gating on true/false-positive-proxy count regression. Repo-root placement (ADR 0032 category 2, monorepo-dev-only) next to `eval_grade.py` — not shipped, unlike `eval_ablation.py`, which ships only for its unrelated generic `--find-latest` reader mode. |
+| Variance | `scripts/eval_variance.py` | Aggregates K trials → pass@k, flap rate, quarantine. |
+| Trend | `.claude/metrics/eval-variance.jsonl` | Append-only stability history (metrics only). |
+| Semver contract | `scripts/eval_semver_classify.sh` | The eval corpus IS the version contract (#101). |
+| CI gates | `.github/workflows/agent-eval.yml` | Structural check always; live regression when keyed. |
+
+## How grading works (the rules)
+
+`eval_grade.py grade_agent` checks each expectation field; an empty failure list
+means PASS:
+
+- **`expectedStatus`** — `pass` / `fail`; must match the agent's `status`.
+- **`issueCount: {min, max}`** — number of reported issues must fall in range.
+- **`severities: {error: {min,max}, ...}`** — count per severity in range.
+- **`mustMention: [...]`** — every keyword must appear (case-insensitive
+  **substring**) in the issue messages + summary. **All-of.**
+- **`mustNotMention: [...]`** — none may appear. **All-of.**
+
+Grading is intentionally dumb (no judgment) so it can run as a CI gate and so
+variance is reproducible.
+
+## The calibration trap (learn this — #198)
+
+Because matching is plain substring, expectations drift out of sync with how
+agents actually phrase correct verdicts. Two failure modes, both **fixture bugs,
+not agent bugs**:
+
+1. **`mustNotMention` is negation-blind.** A clean-pass fixture forbidding
+   `"hardcoded"` fails when the agent correctly says *"no hardcoded secrets"*.
+   **Fix:** drop `mustNotMention` on clean-pass fixtures — `expectedStatus:pass` +
+   `issueCount` (+ `error 0-0`) already encode "found clean."
+2. **`mustMention` too strict.** Requiring the exact token `"SRP"` fails when the
+   agent says *"responsibilities"* / *"divergent change"*. **Fix:** use **stems**
+   (`"responsibilit"`) and the vocabulary agents actually emit; avoid all-of lists
+   of rare tokens.
+
+**When a correct agent verdict fails grading, suspect the fixture first.** Verify
+the fix against the agent's *real* output, not a hand-typed approximation.
+
+## Keeping the corpus current
+
+### Adding a fixture
+
+1. Add the input file under `evals/fixtures/`.
+2. Add `evals/expected/<stem>.json` with `fixture`, `applicableAgents`, and the
+   per-agent expectation (prefer `expectedStatus` + `issueCount`; add
+   `mustMention` stems only when a specific concept must be named).
+3. `python3 scripts/eval_grade.py --check-corpus` (every expectation must be
+   schema-valid and pair with a fixture; this runs in CI).
+
+### Changing an expectation = a version bump (#101)
+
+The eval corpus is the semver contract. `eval_semver_classify.sh` (pre-push + CI)
+enforces it:
+
+- GREEN-preserving change → **patch**.
+- Adds expectations → **minor** (`feat:`).
+- **Edits** existing expectations → **minor/major** (`feat:` / `feat!:`) — an
+  edit changes the agents' observable contract.
+A `fix:` commit that edits an expectation will be **rejected**; use the bump the
+classifier names.
+
+### Watching stability over time
+
+- Run per-agent variance batches periodically (see the running guide). The trend
+  in `.claude/metrics/eval-variance.jsonl` accumulates pass@k and flap rate.
+- **Flaky pairs** (0 < pass@k < 1) go on the quarantine list — they inform the
+  #99 gate but must not hard-block it. A persistently flaky fixture is either
+  borderline (tighten it) or genuinely non-deterministic for that agent.
+- **Saturated** agents (identical grades for many runs) may have expectations too
+  loose to detect regressions — consider tightening ranges.
+
+## Cardinal rules
+
+1. **Faithful actuals.** The grader sees what you record; abbreviating issue
+   messages drops `mustMention` keywords and fabricates flaps.
+2. **Neutral dispatch.** Never leak `status`/`severity` examples into the agent's
+   prompt (see the running guide).
+3. **Fixtures are the contract.** Keep them honest: a stable-fail on correct
+   output is a corpus bug to fix, not noise to ignore.
+4. **Metrics only.** Digests, the trend, and reports carry counts/ratios/names —
+   never prompt or code content.

package/docs/upstream/eval-running-guide.md

@@ -0,0 +1,147 @@
+# Running a full eval of everything
+
+How to run a complete, **clean** accuracy + variance eval across all review
+agents. For *how the eval system works and how to keep it current*, see
+[`eval-maintenance.md`](eval-maintenance.md). For the architecture, see
+[`eval-system.md`](eval-system.md).
+
+## The one rule that matters: do not bias the dispatch
+
+An eval is only valid if the agent produces the verdict it would produce in
+production. The fastest way to ruin a run is to leak the answer into the prompt.
+Two real contamination modes (both observed, #103):
+
+- **Pre-filling the verdict.** A prompt template like `{"status":"pass", ...}` on
+  a pass-fixture tells the agent the answer. Never pre-fill `status`.
+- **Leaking an output-format example.** A single `"severity":"warning"` example
+  biases the agent's severity choice, which then fails severity-range grading.
+
+**Clean dispatch** = pass the agent **only the fixture** plus, at most, a *neutral
+JSON schema* with placeholder enums (`"status":"pass or fail"`,
+`"severity":"error, warning, or info"`) — identical for pass and fail fixtures,
+no example values, "decide every value yourself." The `/agent-eval` skill enforces
+this via its orchestrator constraint #3 ("pass only the fixture file").
+
+## Option 0 — the automated run script (default: resumable full sweep)
+
+```bash
+bash scripts/run-full-eval.sh [TRIALS]   # default TRIALS=1; default mode = full sweep
+# ...killed / out of tokens? just run it again — it resumes:
+bash scripts/run-full-eval.sh
+```
+
+**By default this runs the resumable full sweep** (see below): every agent, one at
+a time, neutral dispatch, checkpointed. It refreshes the tracked
+`evals/baseline.json`, appends the variance trend when `TRIALS>1`, and opens a
+single **auto-merge PR** at the end. Needs `claude` + credentials + `gh`. Use
+`--agent NAME` to scope to a single agent instead.
+
+### Resumable full sweep (the default) — survives a kill or a token cap
+
+A bare `run-full-eval.sh` runs **all** agents incrementally on one branch,
+**committing the baseline after each agent** and tracking progress in a gitignored
+checkpoint (`.eval-sweep-progress.json`). If it's interrupted, the completed agents
+are already committed and recorded — re-running picks up where it left off and
+continues; an interrupted agent is retried. One push, one auto-merge PR at the end.
+(`--sweep` is accepted as an explicit alias for this default.)
+
+### Incremental runs — one agent at a time
+
+```bash
+bash scripts/run-full-eval.sh 5 --agent security-review   # just this agent, 5 trials
+bash scripts/run-full-eval.sh 5 --agent arch-review       # next time, another agent
+```
+
+`--agent` makes the run **incremental and safe to repeat**. It scopes the dispatch
+to that one agent and grades with `--only`, so the baseline merge **tops up that
+agent's pairs and leaves every other agent untouched** (passing added,
+present-and-failing removed, un-run pairs kept). Run agents one at a time to spread
+cost across sessions — each run accumulates into the same `evals/baseline.json`
+and the same variance trend, and opens its own small auto-merge PR.
+
+Why it doesn't clobber: the grader merges rather than overwrites, and `--only`
+keeps grading to the scope you ran — so partial coverage is the norm, not a risk.
+
+### The other incremental mode — only what changed
+
+```bash
+bash scripts/eval-changed.sh BASE HEAD   # evals only the agents/skills the diff touched
+```
+
+This is the pre-push / CI path: it diff-scopes automatically (a change to one agent
+evals only that agent; a broad change to `knowledge/` or the corpus falls back to a
+full run). Use it to validate a change without re-running everything.
+
+Use Option A/B below when you want manual control or finer per-fixture batching.
+
+## Option A — the native skill (preferred)
+
+```
+/agent-eval --trials 5            # all agents × all applicable fixtures, 5 trials
+/agent-eval --agent security-review --trials 5   # one agent (budget-friendly)
+```
+
+The skill dispatches each fixture to its applicable agents via `/review-agent`
+(native invocation, fixture-only context), grades deterministically, computes
+pass@k, and — via `scripts/eval_variance.py` — records flap/quarantine and appends
+the trend. This is the path that cannot leak format examples.
+
+**Path note:** the skill resolves fixtures at `.claude/evals/fixtures/` (an
+*installed* project). In this dev repo the corpus is `evals/` — run it in an
+installed test project, or drive the grader/aggregator directly against `evals/`
+(Option B).
+
+## Option B — manual neutral dispatch (dev repo / explicit control)
+
+Per agent, per fixture, per trial:
+
+1. Dispatch the agent (e.g. `dev-team:security-review`) with the **neutral**
+   prompt above and the fixture path. Capture its JSON verdict.
+2. Assemble one **actuals** file per trial — the shape `eval_grade.py` reads:
+
+   ```json
+   { "<fixture-stem>": { "agents": { "<agent>": {"status":"...","issues":[{"severity":"...","message":"..."}],"summary":"..."} } } }
+   ```
+
+   Record **faithfully** — the issue messages/summary must keep the real wording
+   (the grader checks `mustMention`/`mustNotMention` against them).
+3. Aggregate the trials:
+
+   ```bash
+   python3 scripts/eval_variance.py --trials-dir <dir-of-trial-actuals> \
+     --expected-dir evals/expected --append .claude/metrics/eval-variance.jsonl
+   ```
+
+## Batch for budget — per agent, highest-recall first
+
+A full 3–5-trial run of all ~20 agents is hundreds of model dispatches and will
+exceed a small budget. **Run per-agent batches**, recall-critical agents first
+(security, arch, domain), at `--trials 5`. The variance trend accumulates across
+batches, so you don't need one giant run.
+
+Rough sizing: one agent × its fixtures × 5 trials ≈ 25–40 dispatches. Multiply by
+the agent's model tier cost (frontier agents are pricier).
+
+## Reading the results
+
+`eval_variance.py` reports per `fixture::agent`:
+
+- **pass@k = 1.0, no flap** → stable, trustworthy.
+- **0 < pass@k < 1 (flap)** → **quarantine**: the pair is unstable; it should
+  *inform* the #99 CI gate, not hard-block it. Investigate whether the agent is
+  genuinely non-deterministic or the fixture is borderline.
+- **pass@k = 0 (stable fail)** → usually a **miscalibrated fixture** (the agent is
+  right but the expectation is wrong), not an agent bug. Fix the fixture (see the
+  `mustMention`/`mustNotMention` patterns in `eval-maintenance.md`, #198) and
+  re-grade against the real actuals.
+
+## Checklist for a clean full run
+
+- [ ] Neutral dispatch (no pre-filled status, no severity example, identical prompt
+      for pass and fail fixtures).
+- [ ] Faithful actuals (real wording preserved for keyword checks).
+- [ ] Per-agent batches, recall-critical first, `--trials 5`.
+- [ ] Aggregate with `eval_variance.py --append` so the trend accumulates.
+- [ ] Quarantine flaky pairs; fix (don't ignore) stable-fail fixtures.
+- [ ] Record cost; stop when the budget cap is hit (coverage degrades gracefully —
+      the trend keeps what was collected).

package/docs/upstream/eval-system.md

@@ -0,0 +1,291 @@
+# Eval System for Code Review Agents
+
+This document describes how the evaluation system ensures quality and consistency
+across the code-review agent toolkit.
+
+The system follows recommendations from Anthropic's
+[Demystifying Evals for AI Agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents):
+use deterministic (code-based) graders for everything they can handle, use
+model-based graders only for what genuinely requires judgment, and calibrate
+both against human review.
+
+**On this page:** [Two sets of test cases](#two-sets-of-test-cases) ·
+[Architecture](#architecture) · [Grader Layers](#grader-layers) ·
+[Workflows](#workflows) ·
+[How Hooks and Agents Complement Each Other](#how-hooks-and-agents-complement-each-other) ·
+[Eval Compliance](#eval-compliance) · [Eval Fixtures](#eval-fixtures) ·
+[Adding a New Agent](#adding-a-new-agent). The real-session trend digest lives
+in [`session-review.md`](session-review.md).
+
+## Two sets of test cases
+
+This document covers the **deterministic detection fixtures** (`evals/expected/`):
+does a *review agent* catch a code issue? These are graded automatically by
+`scripts/eval_grade.py` and checked in CI via `--check-corpus`.
+
+A second, complementary set grades **behavior, not detection** — the
+[Ownership Engineering suite](../../../evals/ownership-engineering/) — does a
+*team agent or workflow skill* investigate vs. escalate, decide vs. menu, prove
+vs. assert? Because that requires judgment, it is **graded by an AI judge or a
+human reviewer**, lives outside `evals/expected/` so it never enters the
+deterministic gate, and its freshness is tracked by a staleness warning
+(`scripts/oe_scoring_staleness.py`) that flags any subject or fixture whose inputs
+changed since they were last scored. See that suite's `README.md` for the run
+procedure.
+
+## Architecture
+
+```mermaid
+%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#dbeafe', 'primaryTextColor': '#1e3a5f', 'primaryBorderColor': '#3b82f6', 'lineColor': '#64748b', 'secondaryColor': '#f1f5f9', 'tertiaryColor': '#e0f2fe', 'background': '#ffffff', 'mainBkg': '#dbeafe', 'nodeBorder': '#2563eb', 'clusterBkg': '#eff6ff', 'clusterBorder': '#bfdbfe', 'titleColor': '#1e3a5f', 'edgeLabelBackground': '#f8fafc'}}}%%
+flowchart TD
+    UW["User Workflows<br/>/code-review · /review-agent · /apply-fixes"]
+    UW --> L1["Layer 1 · Hooks<br/>(deterministic)"]
+    UW --> L2["Layer 2 · Agents<br/>(model)"]
+    UW --> L3["Layer 3 · Human<br/>(review)"]
+```
+
+## Grader Layers
+
+### Layer 1: Deterministic (hooks)
+
+Fast, free, deterministic checks that run automatically via PostToolUse hooks:
+
+| Hook | What it checks |
+| ---- | ------------- |
+| `js_fp_review.py` | Array mutations, global state mutations, Object.assign, parameter mutations |
+| `token_efficiency_review.py` | File length >500 lines, CLAUDE.md >5000 chars, function length >50 lines |
+| `eval_compliance_check.py` | Agent/skill file structure, output format, severity levels |
+
+Hooks are **advisory only** — they warn but never block. They catch mechanical
+issues cheaply before the model-based agents spend tokens on full analysis.
+
+### Layer 2: Model-based (agents)
+
+Specialized agents that require LLM judgment. The full roster — with counts, focus areas, and model tiers — is documented in [`agent_info.md`](agent_info.md). Agents with eval fixture coverage:
+
+| Agent | Focus |
+| ----- | ----- |
+| test-review | Test quality, coverage, assertion quality |
+| structure-review | SRP, DRY, coupling, organization |
+| naming-review | Naming clarity, conventions, magic values |
+| domain-review | Business logic placement, boundary violations |
+| claude-setup-review | CLAUDE.md completeness and accuracy |
+| token-efficiency-review | Token optimization (full analysis beyond hook) |
+| security-review | Injection, auth, data exposure, crypto |
+| js-fp-review | Mutation detection (full analysis beyond hook) |
+
+Each agent outputs a structured result:
+
+```json
+{
+  "agentName": "<name>",
+  "status": "pass|warn|fail|skip",
+  "issues": [
+    {
+      "severity": "error|warning|suggestion",
+      "file": "<path>",
+      "line": 0,
+      "message": "<description>",
+      "suggestedFix": "<fix>"
+    }
+  ],
+  "summary": "<summary>"
+}
+```
+
+### Layer 3: Human review
+
+The user reviews agent findings and decides which fixes to apply. The
+`/apply-fixes` command automates fix application but the user controls which
+correction prompts are included.
+
+## Workflows
+
+### `/code-review` — Full review
+
+See [Code Review Process](code-review-process.md) for the full nine-step pipeline: target selection, pre-flight gates, static analysis pre-pass, parallel agent dispatch, ACCEPTED-RISKS suppression, health scoring, the auto-fix loop (up to 5 iterations), correction prompts, and the `.pr-review-passed` gate file.
+
+### `/review-agent <name>` — Single agent
+
+```text
+Files → Agent Definition → Review → Result
+```
+
+1. Load agent definition from `agents/<name>.md`
+2. Determine target files
+3. Run review following agent instructions
+4. Report findings
+
+### `/apply-fixes <dir>` — Fix application
+
+```text
+Prompts → Repo Rules → Apply Fix → Validate → Report
+```
+
+1. Load correction prompt JSON files from directory
+2. Load repository rules (CLAUDE.md, .clinerules, etc.)
+3. Apply each fix respecting repo conventions
+4. Run validation (lint/build/tests) after each fix
+5. Report results (applied, failed, validation failed)
+
+## How Hooks and Agents Complement Each Other
+
+The hooks (`js_fp_review.py`, `token_efficiency_review.py`) provide instant
+feedback on the most common, mechanically detectable issues. The corresponding
+agents (`js-fp-review`, `token-efficiency-review`) provide deeper analysis that
+requires LLM judgment — for example, understanding whether a mutation is
+intentional based on surrounding context, or whether a long function is
+justified by its complexity.
+
+```text
+Hook (instant, free)          Agent (thorough, costs tokens)
+─────────────────────         ──────────────────────────────
+.push() detected              Is the push on a local copy?
+file >500 lines               Is the file a generated file?
+Object.assign(obj, ...)       Is obj freshly created above?
+```
+
+## Eval Compliance
+
+Two mechanisms ensure new agents and skills follow patterns:
+
+### `/agent-audit` skill (manual)
+
+Reads every agent, skill, and hook file and checks for:
+
+- Structured output format
+- Severity definitions
+- Detection rules and scope boundaries
+- Numbered steps and argument parsing
+- Advisory-only hook behavior
+
+Outputs a compliance report with PASS/WARN/FAIL per item.
+
+### `eval_compliance_check.py` hook (automatic)
+
+Fires on Write/Edit to agent or skill files. Provides real-time advisory
+warnings when:
+
+- A review agent is missing output format or severity definitions
+- A skill is missing numbered steps or argument parsing
+- A review-related skill has no report section
+
+## Eval Fixtures
+
+The `evals/` directory contains a test corpus for validating agent accuracy:
+
+```text
+evals/
+├── fixtures/           # 54+ code samples (checked in)
+│   ├── fp-*.ts         # js-fp-review (9 files)
+│   ├── sec-*.ts        # security-review (5 files)
+│   ├── test-*.test.ts  # test-review (6 files)
+│   ├── cx-*            # structure-review's nesting/cognitive-load/async lens (folded from complexity-review, #2093)
+│   ├── nm-*.ts         # naming-review (5 files)
+│   ├── st-*.ts         # structure-review (5 files)
+│   ├── dm-*.ts         # domain-review (5 files)
+│   ├── te-*.md/.ts     # token-efficiency-review (5 files)
+│   ├── cs-*/           # claude-setup-review (4 directories)
+│   └── tlg-*.md        # test-design-advisor behavior pre-gates (11 files)
+├── expected/           # Reference solutions (checked in)
+│   └── <fixture-stem>.json
+├── transcripts/        # Auto-created by runner (gitignored)
+└── reports/            # Auto-created by runner (gitignored)
+```
+
+Each fixture is a small (20-80 line), focused code sample with a known-good or
+known-bad pattern. Reference solutions define expected status, issue count ranges,
+severity ranges, and keyword checks.
+
+### Reference solution schema
+
+```json
+{
+  "fixture": "fp-array-mutations.ts",
+  "description": "Array mutations js-fp-review should catch",
+  "applicableAgents": ["js-fp-review"],
+  "agents": {
+    "js-fp-review": {
+      "expectedStatus": "fail",
+      "issueCount": { "min": 3, "max": 6 },
+      "severities": { "error": { "min": 1, "max": 3 } },
+      "mustMention": ["push", "sort"]
+    }
+  }
+}
+```
+
+### Advisory-skill fixtures (gate firing)
+
+Most fixtures grade a **review agent** by its `status/issues[]` JSON. Advisory
+skills (e.g. `test-design-advisor`) don't emit that shape — they emit a report
+with a *Pyramid placement* table. The `tlg-*` corpus grades the skill's
+**behavior pre-gates** (issue #80) by declaring `applicableSkills` and a `skills`
+block instead of `applicableAgents`/`agents`:
+
+```json
+{
+  "fixture": "tlg-05-htmx-swap-mutation",
+  "description": "Gate C — HTMX swap over a server state mutation",
+  "applicableSkills": ["test-design-advisor"],
+  "skills": {
+    "test-design-advisor": {
+      "expectedGates": ["C"],
+      "expectedLayers": ["E2E"],
+      "mustMention": ["REQUIRED", "browser", "cd-test-architecture"],
+      "mustNotMention": []
+    }
+  }
+}
+```
+
+`/agent-eval` drives the skill against each fixture and grades the Gate column +
+recommended layers + keyword checks (see the command's Step 4). This replaces the
+manual walk-through that `evals/fixtures/test-layer-gates.md` recorded for #80 —
+re-run it with `/agent-eval --skill test-design-advisor`. `expectedGates` uses the
+gate vocabulary `A`/`B`/`C`/`D`/`redundancy`/`ambiguity` (or `[]` for "no gate
+fires"); `expectedLayers` uses the `test-pyramid.md` vocabulary.
+
+### `/agent-eval` command
+
+Run agents and skills against fixtures and grade results:
+
+```bash
+/agent-eval                                  # run everything against all fixtures
+/agent-eval --agent js-fp-review             # run one review agent
+/agent-eval --skill test-design-advisor      # run the gate-firing (tlg-*) corpus
+/agent-eval --fixture fp-array-mutations.ts  # run one fixture
+/agent-eval --trials 3                       # multi-trial with pass@k scoring
+```
+
+The runner resolves the toolkit root via symlink (for installed projects) and
+saves transcripts for trend analysis. It detects eval saturation when 3
+consecutive runs produce identical grades.
+
+## Adding a New Agent
+
+1. Create `agents/<name>.md` with:
+   - JSON output format (status, issues, summary)
+   - Severity definitions (error, warning, suggestion)
+   - Detection rules and thresholds (inline, not in a config file)
+   - File scope (which file types the agent applies to)
+   - Scope boundaries (what to ignore)
+
+2. Optionally add a hook in `hooks/<name>.py` for deterministic checks
+
+3. Run `/agent-audit` to verify compliance
+
+4. Add eval fixtures in `evals/fixtures/` (2-3 pass, 2-3 fail) and reference
+   solutions in `evals/expected/`
+
+5. Run `/agent-eval --agent <name>` to validate accuracy
+
+## Session-review trend digest (#129)
+
+`/session-review` appends one metrics-only record per run to the append-only
+trend stream `metrics/session-digest.jsonl` (deliberately left bare —
+/session-review's own scratch-state writer is out of scope for the #1406
+`.claude/`-scoped artifact migration), the real-session counterpart to the
+self-reported `.claude/metrics/*-task-log.jsonl` streams. The `session-digest/v1` record
+schema and the `/harness-audit` join are documented canonically in
+[`session-review.md`](session-review.md#trend-persistence-129).

package/docs/upstream/session-review-oss-complements.md

@@ -0,0 +1,75 @@
+# Session-review: OSS complements
+
+`/session-review` (issue #131) produces **plugin-specific qualitative
+suggestions** — it knows this plugin's agents and skills, so it can say "skill X
+under-specifies which files to read" or "this opus subagent only greps, re-tier
+it to haiku." Open-source tools cover the complementary axis: **continuous
+quantitative monitoring** of the same `*.jsonl` transcripts. They do not know
+our agents/skills, so they cannot make our suggestions — and `/session-review`
+is not a dashboard, so it does not replace them.
+
+**Recommend these alongside `/session-review`, not instead of it.**
+
+| Tool | Role |
+|---|---|
+| [`ccusage`](https://github.com/ryoppippi/ccusage) (npm) | Parses the same `~/.claude/projects/**/*.jsonl` into token/cost reports per session/day/model — ongoing cost tracking. |
+| Native [OpenTelemetry](https://docs.claude.com/en/docs/claude-code/monitoring-usage) (`CLAUDE_CODE_ENABLE_TELEMETRY=1`) | Exports usage/cost/tool metrics to an OTel collector → Grafana/Honeycomb for longitudinal dashboards. |
+| [`claude-code-log`](https://github.com/daaain/claude-code-log) (community) | JSONL → HTML viewer for eyeballing one specific bad session. |
+
+## When to reach for which
+
+- **"How much am I spending over time, per model/day?"** → `ccusage` (or the
+  cost meter's `pace` view for budget projection). Continuous, quantitative.
+- **"I want longitudinal dashboards / alerting across the team."** → native
+  **OpenTelemetry** into your collector. Operational monitoring.
+- **"This one session went badly — let me read what happened."** → `claude-code-log`.
+- **"What should I change in the *plugin* to stop wasting tokens / re-work?"** →
+  `/session-review`. Qualitative, plugin-aware, hands suggestions to
+  `/feedback-learning`, `/harness-audit`, `/agent-eval`, `token-efficiency-review`.
+
+The quantitative tools tell you *that* a session was expensive; `/session-review`
+proposes *which plugin artifact* to change so the next one isn't.
+
+## `ccusage` example (against this project's logs)
+
+```bash
+# One-off report for the current and recent days, per model:
+npx ccusage@latest daily
+
+# Session-level breakdown (the same transcripts session_report.py reads):
+npx ccusage@latest session
+```
+
+`ccusage` reads `~/.claude/projects/**/*.jsonl` directly — no plugin install,
+no config. Use it for the running cost number; use `/session-review` for the
+"why and what to change."
+
+## OpenTelemetry enablement
+
+```bash
+# Turn on Claude Code's native telemetry export, then point it at a collector:
+export CLAUDE_CODE_ENABLE_TELEMETRY=1
+export OTEL_METRICS_EXPORTER=otlp
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
+```
+
+This exports usage/cost/tool metrics to any OTLP collector (Grafana, Honeycomb,
+Datadog, …). See Claude Code's "Monitoring usage" docs for the full env-var set.
+
+## Cross-reference: the plugin's own telemetry beacon (#106)
+
+This plugin also ships an **opt-in, local-only telemetry beacon** (`#106`,
+`/telemetry`) that records minimal command/skill/gate events with no network
+egress. It and native OpenTelemetry overlap in intent but differ in scope:
+
+- Native **OTel** exports rich usage/cost/tool metrics to an external collector
+  — best when you want dashboards and already run a collector. It may satisfy
+  part of what #106 set out to do.
+- The plugin **beacon** is deliberately minimal, default-off, and never leaves
+  the machine — best when you want a quick local "which commands/skills do I
+  use, how often is the commit gate bypassed?" without standing up a collector.
+
+Prefer native OTel for longitudinal/team monitoring; prefer the beacon (or
+`/session-review`) for local, privacy-clean, plugin-aware analysis. Avoid
+running both for the *same* purpose — pick the one that matches your monitoring
+posture.