baldart 3.35.2 → 3.37.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,45 @@ All notable changes to BALDART will be documented in this file.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [3.37.0] - 2026-05-30
9
+
10
+ **Two-tier review for `/new`: fast `light` per-card during the loop, one guaranteed FULL pass over the whole batch before merge.** This activates the dormant v3.35.0 Review Profile Selector AND re-introduces an unconditional final full review — together they cut tokens/wall-clock on low-risk cards *without* weakening the merge gate. **No new `baldart.config.yml` keys** — both are changes inside the skill, so the schema-propagation rule does not apply.
11
+
12
+ > **The design.** `light` (per-card Phase 3.7) is now an **early-feedback optimization**, not the safety gate. The safety gate is the **Final review**, which since this release ALWAYS runs a single FULL `/codexreview` (full agent set) over the **entire batch diff** before merge — no N=1 skip, no cross-card scope reduction. So every line of every card, including any reviewed at `light`, gets a full-depth Codex review at least once before merge. This is what makes activating `light` safe: it shifts the breadth passes (qa-sentinel, api-perf-cost-auditor, doc-reviewer) + CoVe from the per-card loop to the single final pass, rather than dropping them from the merge gate. (The v3.35.0 data gate — which had kept the selector hard-coded to `full` pending consumer telemetry — is lifted by explicit maintainer decision; an adversarial review showed that gate's intended post-hoc telemetry could never have validated it anyway, since the Fix Application Log is blind by construction to blockers `light` *misses*. The final full gate makes that validation question moot.)
13
+
14
+ ### Changed — Review Profile Selector live (per-card early feedback)
15
+
16
+ - **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)**: Phase 3.7 Step C (sequential) and Team Mode D.4b no longer hard-code `profile = full`. The selector runs as designed — `light` ⟺ Step A matched **0** high-risk triggers **AND** the card's QA profile ∈ {`skip`, `light`}; `full` otherwise. **BUG-0530 invariant preserved**: `light` ≠ skip — Codex adversarial + `code-reviewer` + false-positive gate ALWAYS run per-card; `light` drops only the breadth passes + CoVe.
17
+
18
+ ### Changed — Final review is now an unconditional FULL batch-diff gate (supersedes v3.35.0 scope-reduction)
19
+
20
+ - **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)** (Final review / Step F.1): the v3.35.0 final-review scope gate (N=1 → skip F.1–F.4; N>1 → reduce to the cross-card surface) is **removed**. The final review now runs Steps F.1–F.5 for **every batch including N=1**, with `review_scope_files` = the **full union** of all touched files, never reduced. F.3 reviews the entire batch diff at full depth (Codex + doc-reviewer + api-perf-cost-auditor + qa-sentinel); F.5 runs the final build. Team mode reaches the same gate via "Post-batch — same as sequential mode" (its prior "cross-card-only scope" note is updated accordingly).
21
+ - **Rationale**: v3.35.0 de-duplicated the final full pass on the assumption that Phase 3.7 had already full-reviewed every card. Selecting `light` breaks that assumption, so the post-batch full pass is re-introduced as the guaranteed merge gate.
22
+
23
+ ### Cost & trade-offs (documented)
24
+
25
+ - **Net cost**: one full `/codexreview` per batch is re-introduced (the deliberate price of the per-card `light` speed-up). The saving comes from cheaper per-card reviews during the loop; the final pass is paid once per batch, not per card.
26
+ - **Later feedback on `light` cards**: a defect on a `light`-reviewed card may now surface at the final pass rather than at the per-card step — possibly a later fix-cycle. Acceptable: it is still caught before merge.
27
+ - **Revert**: to make Phase 3.7 always-full again, restore `profile = full` (no config kill-switch by design). The final full gate is independent and stays regardless.
28
+
29
+ ## [3.36.0] - 2026-05-30
30
+
31
+ Adds **opt-in session telemetry** to `/new` and `/prd`: append `-stats` (or `--stats`) to a run and, at the end, the skill writes a per-**agent-role** breakdown of REAL token consumption and wall-clock time. This finally answers "what took the most time and tokens in a `/new` run" — and feeds the v3.35.0 Review Profile Selector **data gate** (still pending precisely because `docs/metrics/` carried no per-role cost data). **No new `baldart.config.yml` keys** — the opt-in is a flag, not a config feature, so the schema-propagation rule does not apply.
32
+
33
+ > **Zero model-token overhead.** The measurement is a **post-hoc Node script**, not a hook and not LLM self-reporting (the model has no reliable token counter). It reads the Claude Code session transcripts — the orchestrator's `<session>.jsonl` plus every `subagents/agent-*.jsonl` — which already carry `message.usage` (the four token counters) and per-message `timestamp`. On a real `/new` run the bulk of cost lives in the subagents (measured ~53M cumulative subagent tokens vs ~47k orchestrator output), and it is all captured. Cache-read tokens are kept **separate** from fresh input (they bill ~10×cheaper) so the breakdown doesn't mislead.
34
+
35
+ ### Added — `-stats` / `--stats` opt-in session telemetry
36
+
37
+ - **[framework/scripts/analyze-session-tokens.js](framework/scripts/analyze-session-tokens.js)** (new): post-hoc aggregator. Locates the live transcripts from `$CLAUDE_CODE_SESSION_ID` + the cwd slug, sums the four `usage` counters (`fresh_input` / `cache_read` / `cache_creation` / `output`) **kept separate**, computes per-file wall-clock spans, and maps each subagent file → its role by matching the subagent's first prompt to the `Agent` tool_use `subagent_type` in the main transcript (verified 20/20 on a real `/new` batch; unmatched files bucket as `unknown`). Emits a human-readable `docs/metrics/sessions/<run_id>.md` (per-role table sorted by a weighted `cost_units` figure + totals + caveats) and a machine-readable `TELEMETRY_JSON=` line. **Fail-safe**: any error prints `stats: SKIPPED (<reason>)` and exits 0 — it can never abort a run.
38
+ - **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)**: arg parser strips `-stats` / `--stats` like `-full` and sets an internal `STATS` flag (composes with `-full`). **Phase 8 — Metrics Log** gains a `STATS`-gated step that invokes the script and enriches the same `skill-runs.jsonl` row with a `"cost"` object (`by_role` + `totals` + `run_wall_ms`). Without `-stats`, Phase 8 is byte-for-byte unchanged.
39
+ - **[framework/.claude/skills/prd/SKILL.md](framework/.claude/skills/prd/SKILL.md)**: new **HARD RULE 18** — since `/prd` is conversational (no flag parser), the opt-in is detected once at **Step 1 (Kickoff)** from `$ARGUMENTS` and persisted as `stats_enabled` in the session state file (survives context compression / Step 0 resume). The **Step 7 Metrics Log** reads it and runs the same script (`--skill prd`), enriching its JSONL row identically.
40
+ - **[framework/.claude/skills/prd/assets/state-template.md](framework/.claude/skills/prd/assets/state-template.md)**: new `stats_enabled` header field.
41
+
42
+ ### Notes & caveats
43
+
44
+ - **Accurate in sequential mode.** Team-mode L0/L1 orchestrators can spawn background / nested sub-orchestrators in *separate* session dirs; their subagents are not under the top-level `<session>/subagents/`. The script detects the gap (Agent spawns in main with no local subagent file) and prints an explicit caveat with the excluded count rather than silently undercounting.
45
+ - The per-role `wall(sum)` reflects active compute time; the headline run wall-clock is the orchestrator span and **includes human idle gaps** — both are labelled in the report.
46
+
8
47
  ## [3.35.2] - 2026-05-30
9
48
 
10
49
  Resolves the **3.35.1 known follow-up**: a flag introduced in a newer release (e.g. `--on-divergence`, v3.29.0) was rejected by an older global CLI with a cryptic `error: unknown option` **before** the `npx baldart@latest` auto-relaunch could self-upgrade — commander validates options during `program.parse()`, which runs *before* the action handler where the relaunch lives. The `update` command now parses **permissively**, so an unknown flag is captured instead of crashing the parser; the action then transparently relaunches under `@latest` (forwarding the raw argv so the unrecognized flag survives) or, when that isn't possible, emits an **actionable** error instead of commander's cryptic one. **No new `baldart.config.yml` keys** — the schema-propagation rule does not apply.
package/VERSION CHANGED
@@ -1 +1 @@
1
- 3.35.2
1
+ 3.37.0
@@ -37,6 +37,8 @@ Parse the card IDs from the arguments. Cards can be specified as:
37
37
 
38
38
  If no card IDs are provided, ask the user which cards to implement.
39
39
 
40
+ **Opt-in session telemetry (`-stats` / `--stats`)**: strip `-stats` / `--stats` from the args list exactly like `-full` (it is NOT a card ID), and set an internal flag `STATS=true` for this run. When present, Phase 8 additionally measures REAL token + wall-clock cost per agent role (coder, code-reviewer, …) by post-processing the session transcripts — zero model-token overhead, runs in Bash after the work is done. When absent, Phase 8 behaves exactly as before. The flag is batch-scoped and composes with `-full` (`/new FEAT-005 -full -stats`).
41
+
40
42
  ---
41
43
 
42
44
  ## Context Tracking (CRITICAL)
@@ -1111,7 +1113,7 @@ skill's Phase 1 falls back to deriving Gherkin scenarios from
1111
1113
 
1112
1114
  ### Phase 3 — Doc Review & Sync (code review handled by mandatory Phase 3.7 gate)
1113
1115
 
1114
- > **Note**: Code review is NOT performed in this phase — it is handled by the **mandatory unconditional `/codexreview` gate in Phase 3.7**, which runs per-card BEFORE the Phase 4 commit. The post-batch `/codexreview` in Phase 7 remains as a final cross-card sweep. This Phase 3 focuses exclusively on documentation sync, which must happen per-card (tied to the specific commit).
1116
+ > **Note**: Code review is NOT performed in this phase — it is handled by the **mandatory unconditional `/codexreview` gate in Phase 3.7**, which runs per-card BEFORE the Phase 4 commit. The post-batch `/codexreview` in the Final review remains as a final FULL-diff sweep over the entire batch (since v3.37.0 — the guaranteed full-depth merge gate). This Phase 3 focuses exclusively on documentation sync, which must happen per-card (tied to the specific commit).
1115
1117
 
1116
1118
  12. **Update tracker**: phase = "3-doc-review".
1117
1119
  13. Build a **Doc Sync Context** block:
@@ -1285,10 +1287,15 @@ For EVERY card (no conditional skip — the gate ALWAYS runs; only its DEPTH var
1285
1287
  doc-reviewer) and Step 3.5 CoVe — exactly the passes that yield ~nothing on a 0-trigger, QA-light
1286
1288
  diff. The BUG-0530 invariant ("never miss a blocker on a low-risk path") holds because the Codex
1287
1289
  adversarial pass — the one that catches those — always runs.
1288
- **DATA GATE (BLOCKING for `light`)**: until the consumer's telemetry (`docs/metrics/` Fix
1289
- Application Log) confirms that `light`-eligible cards have NOT historically produced verified
1290
- BLOCKER/HIGH at FULL depth, hard-code `profile = full` here. Flip to the selector only after the
1291
- data passes (no config keychange the line).
1290
+ **ACTIVE since v3.37.0**: the selector above runs as written. Here `light` is an **early-feedback
1291
+ optimization** for the per-card loop NOT the merge-gate. Safety is guaranteed downstream: since
1292
+ v3.37.0 the **Final review ALWAYS runs a single FULL `/codexreview` over the entire batch diff before
1293
+ merge** (no N=1 skip, no scope reduction see "Final review → Final-review FULL gate"). So every card
1294
+ reviewed at `light` here still receives a full-depth review before merge; `light` shifts the breadth
1295
+ passes (qa-sentinel, api-perf-cost-auditor, doc-reviewer) + CoVe from the per-card loop to that single
1296
+ final full pass, it does not drop them from the merge gate. The BUG-0530 invariant still holds per-card
1297
+ (Codex adversarial + `code-reviewer` + FP-gate always run in `light`). To make Phase 3.7 always-full
1298
+ again: restore `profile = full` here (the final full gate is independent and stays regardless).
1292
1299
 
1293
1300
  Then dump the card diff (reviewer grounding) and write the lean contract `/codexreview` consumes:
1294
1301
  ```bash
@@ -1400,20 +1407,24 @@ Never block the pipeline indefinitely — recover and continue per the rules abo
1400
1407
 
1401
1408
  Once ALL cards are committed in the worktree:
1402
1409
 
1403
- > **Final-review scope gate (deterministic, since v3.35.0)** — the per-card Phase 3.7 `/codexreview`
1404
- > gate has ALREADY reviewed every card individually before its commit. The final review exists to
1405
- > catch what a per-card pass structurally cannot: **cross-card interactions**. Branch on batch size:
1410
+ > **Final-review FULL gate (since v3.37.0 — supersedes the v3.35.0 scope-reduction)** because
1411
+ > Phase 3.7 may now run a card at `light` depth, the per-card pass can NO LONGER be assumed to have
1412
+ > full-reviewed every card. The final review is therefore the **unconditional safety gate that makes
1413
+ > the Phase 3.7 `light` profile safe**: it ALWAYS runs a single FULL `/codexreview` (full agent set)
1414
+ > over the **ENTIRE batch diff** before merge — **no N=1 skip, no cross-card scope reduction**. Every
1415
+ > line of every card — including any reviewed at `light` in Phase 3.7 — receives a full-depth Codex
1416
+ > review at least once before merge.
1417
+ >
1418
+ > - Run Steps F.1–F.5 for **EVERY batch, including N=1**. Nothing here is skipped.
1419
+ > - `review_scope_files` = the **FULL union** of all touched files across all cards (F.1 step 4 —
1420
+ > NEVER reduced to the cross-card subset).
1421
+ > - F.3 invokes the full reviewer set (Codex + doc-reviewer + api-perf-cost-auditor + qa-sentinel)
1422
+ > over that union; F.5 runs the final build.
1406
1423
  >
1407
- > - **Single card (N=1)**: SKIP Steps F.1–F.4 (the lone card is byte-identical to what Phase 3.7
1408
- > already passedre-review adds nothing). Log:
1409
- > `"Final review (F.1–F.4) skipped single card already passed /codexreview at Phase 3.7 (PASS), code unchanged since commit"`.
1410
- > Then jump to **Step F.5 and RUN the final build** (`npm run lint && npx tsc --noEmit && npm run build`)
1411
- > — that gate is NOT skipped.
1412
- > - **Multiple cards (N>1)**: run F.1–F.5 but **scope the review to the cross-card surface only**
1413
- > (computed, not discretionary) — see F.1 step 4. If that surface is empty, log
1414
- > `"cross-card review: no shared surface (intersection=∅, depends_on=∅) — F.3 limited to integration sanity"`
1415
- > and run a minimal integration pass (build + type-check across the union) instead of a full
1416
- > per-line re-review. This is a computed scope reduction, NEVER a model-decided skip.
1424
+ > Rationale: this re-introduces the post-batch full pass that v3.35.0 de-duplicated away. That
1425
+ > de-dup assumed Phase 3.7 had already full-reviewed every card an assumption broken the moment
1426
+ > `light` became selectable. The cost of one full batch-diff review is the deliberate price of the
1427
+ > per-card `light` speed-up (explicit maintainer decision, v3.37.0).
1417
1428
 
1418
1429
  ### Step F.1 — Resolve scope
1419
1430
 
@@ -1424,11 +1435,11 @@ Once ALL cards are committed in the worktree:
1424
1435
  git diff --name-only <base-branch>...HEAD
1425
1436
  ```
1426
1437
  3. Read each card's backlog YAML to collect `acceptance_criteria`, `files_likely_touched`.
1427
- 4. Build `review_scope_files` as union of card-indicated files + git-touched files. **For N>1 (per
1428
- the Final-review scope gate above), then REDUCE `review_scope_files` to the cross-card subset**:
1429
- files appearing in ≥2 cards' File Ownership Map entries files of any `depends_on`-linked card
1430
- pair. Record both the full union and the reduced set in the tracker; downstream F.3 reviews the
1431
- reduced set.
1438
+ 4. Build `review_scope_files` as the **FULL union** of card-indicated files + git-touched files
1439
+ across ALL cards in the batch. **Since v3.37.0 this union is NEVER reduced** (the v3.35.0
1440
+ cross-card scope-reduction is removed) the final review is a full-depth pass over the entire
1441
+ batch diff, so cards reviewed at `light` in Phase 3.7 still get a guaranteed full review before
1442
+ merge. Record the union in the tracker; downstream F.3 reviews it in full.
1432
1443
 
1433
1444
  ### Step F.2 — Architecture baseline
1434
1445
 
@@ -2191,7 +2202,7 @@ After ALL agents in the group complete successfully:
2191
2202
 
2192
2203
  4a. **D.4a — Per-card doc gate (consumes D.2, since v3.35.0)** — Do NOT re-spawn `doc-reviewer`. For EACH card in the group, take that card's doc findings from D.2's per-card-attributed output; if any exist, spawn a targeted fix-coder for that card only. (Previously D.4a re-ran doc-reviewer per card — a duplicate of D.2; collapsed to a single attributed D.2 pass. D.4b's per-card `/codexreview` also skips its doc-reviewer via the lean contract, so doc-review now runs once per group instead of ~3× per card.)
2193
2204
 
2194
- 4b. **D.4b — Pre-Merge Codex Review Gate (per-card, MANDATORY — UNCONDITIONAL)** — Invoke `/codexreview` for EACH card in the group, one at a time, BEFORE any commit. This mirrors Phase 3.7 of the sequential path and is non-skippable regardless of file paths or perceived risk. Apply the same fix sub-loop: if the consolidated report shows verified BLOCKER/HIGH findings, spawn a fix-coder and re-run `/codexreview <CARD-ID>` (max 2 retries per card). If still BLOCKER/HIGH after retries, ask the user before proceeding to D.5. The D.5 commits MUST NOT happen until every card in the group has a PASS verdict (or explicit user override via `AskUserQuestion`). Log results in the tracker under `## Pre-Merge Codex Review` per card. **Lean + profile (since v3.35.0)**: before each card's `/codexreview`, apply the same Review Profile Selector and write the same `/tmp/codexreview-lean-<CARD-ID>.json` contract as sequential Phase 3.7 Step C — `arch_baseline_path` pointing at `/tmp/arch-baseline-group-<FIRST-CARD-ID>.md`, `skip_doc_reviewer: true`, and `profile: light|full` from (0 triggers + QA ∈ {skip,light}). LIGHT still runs Codex adversarial + code-reviewer + FP-gate (subject to the same DATA GATE: keep `full` until telemetry confirms).
2205
+ 4b. **D.4b — Pre-Merge Codex Review Gate (per-card, MANDATORY — UNCONDITIONAL)** — Invoke `/codexreview` for EACH card in the group, one at a time, BEFORE any commit. This mirrors Phase 3.7 of the sequential path and is non-skippable regardless of file paths or perceived risk. Apply the same fix sub-loop: if the consolidated report shows verified BLOCKER/HIGH findings, spawn a fix-coder and re-run `/codexreview <CARD-ID>` (max 2 retries per card). If still BLOCKER/HIGH after retries, ask the user before proceeding to D.5. The D.5 commits MUST NOT happen until every card in the group has a PASS verdict (or explicit user override via `AskUserQuestion`). Log results in the tracker under `## Pre-Merge Codex Review` per card. **Lean + profile (since v3.35.0)**: before each card's `/codexreview`, apply the same Review Profile Selector and write the same `/tmp/codexreview-lean-<CARD-ID>.json` contract as sequential Phase 3.7 Step C — `arch_baseline_path` pointing at `/tmp/arch-baseline-group-<FIRST-CARD-ID>.md`, `skip_doc_reviewer: true`, and `profile: light|full` from (0 triggers + QA ∈ {skip,light}). LIGHT still runs Codex adversarial + code-reviewer + FP-gate. The Review Profile Selector is **ACTIVE since v3.37.0** — same status as sequential Phase 3.7 Step C above: `light` is per-card early feedback, and the merge-gate safety is the post-batch **Final-review FULL gate** (a single FULL `/codexreview` over the entire batch diff, which team mode reaches via "Post-batch — same as sequential mode"). To make per-card always-full again, restore `profile = full`; the final full gate stays regardless.
2195
2206
 
2196
2207
  5. **D.5 — Commit** — One commit per card using explicit staging from the ownership map. **NO stash in worktrees** (stashes are globally shared via `refs/stash` — see Phase 4 WORKTREE COMMIT RULE):
2197
2208
  ```bash
@@ -2243,7 +2254,7 @@ Before starting group N, verify ALL cards in groups 0..N-1 are status: done. If
2243
2254
 
2244
2255
  ### Post-batch (same as sequential mode)
2245
2256
 
2246
- After all groups are complete, run the same Final Review, Phase 6 (merge), and Phase 7 (production readiness) as documented above. Team mode is always N>1, so the Final Review applies its deterministic **cross-card-only scope** (per the Final-review scope gate) — not a full per-line re-review.
2257
+ After all groups are complete, run the same Final Review, Phase 6 (merge), and Phase 7 (production readiness) as documented above. Since v3.37.0 the Final Review runs a **single FULL `/codexreview` over the entire batch diff** (per the Final-review FULL gate) — no scope reduction — so every team-mode card, including any reviewed at `light` in D.4b, receives a guaranteed full review before merge.
2247
2258
 
2248
2259
  ---
2249
2260
 
@@ -2287,6 +2298,19 @@ and QA quality over time.
2287
2298
 
2288
2299
  5. Note in tracker: `## Metrics Log: WRITTEN (run_id: batch-<FIRST-CARD-ID>)`
2289
2300
 
2301
+ 6. **Session token telemetry (ONLY when `STATS=true`)** — if the run was invoked with `-stats` / `--stats`:
2302
+
2303
+ Measure REAL per-role token + wall-clock cost by post-processing the session transcripts (zero model-token cost — runs in Bash). Invoke:
2304
+
2305
+ ```bash
2306
+ node .framework/framework/scripts/analyze-session-tokens.js \
2307
+ --skill new --run-id "batch-<FIRST-CARD-ID>" --out-dir docs/metrics
2308
+ ```
2309
+
2310
+ The script reads `$CLAUDE_CODE_SESSION_ID` from the environment and derives the transcript paths itself (do NOT pass `--session` / `--project-dir` — those are test-only overrides). It writes a human-readable breakdown to `docs/metrics/sessions/batch-<FIRST-CARD-ID>.md` and prints a summary plus a final `TELEMETRY_JSON=<json>` line.
2311
+
2312
+ Parse the `TELEMETRY_JSON=` line and **enrich the same skill-runs.jsonl row** you wrote in step 3 by adding a `"cost"` key holding that JSON object (`by_role` + `totals` + `run_wall_ms`). If the script printed `stats: SKIPPED (...)` instead, add `"cost":{"skipped":"<reason>"}` and continue. Echo the script's summary to the user so they see the per-role breakdown.
2313
+
2290
2314
  **If `docs/metrics/skill-runs.jsonl` does not exist**: create it first with `touch docs/metrics/skill-runs.jsonl`.
2291
2315
  **If batch tracker is missing or unreadable**: log "Metrics Log: SKIPPED (tracker not found)" and proceed without blocking.
2292
- **This phase is NON-BLOCKING** — if it fails for any reason, do not abort the run.
2316
+ **This phase is NON-BLOCKING** — if it fails for any reason, do not abort the run. The `-stats` step in particular is best-effort: the script is fail-safe (always exits 0), so never let it block the commit.
@@ -199,6 +199,16 @@ message. You ask questions, wait for answers, and iterate.
199
199
  repository, or the consumer has explicitly disabled docs worktrees via
200
200
  overlay), STOP and report the error to the user. Do NOT silently fall back
201
201
  to writing on the main repo — that defeats the entire purpose of this rule.
202
+ 18. **Opt-in session telemetry (`-stats` / `--stats`).** Unlike `/new`, this skill
203
+ has no flag parser — it is conversational. So detect the opt-in ONCE at **Step 1
204
+ (Kickoff)**: inspect `$ARGUMENTS` for the token `-stats` or `--stats`. If present,
205
+ set `stats_enabled: true` in the state-file header (the field exists in
206
+ `assets/state-template.md`); otherwise `false`. Persisting it in the state file is
207
+ what makes it survive context compression and Step 0 resume. The Metrics Log step
208
+ (inside Step 7) reads this field and, when `true`, post-processes the session
209
+ transcripts for REAL per-role token + wall-clock cost. The token itself is NOT part
210
+ of the feature description — strip it before recording the user's verbatim feature
211
+ text in `## Feature Description`.
202
212
 
203
213
  ---
204
214
 
@@ -372,6 +382,15 @@ exactly the correct semantics). No special handling needed.
372
382
  {"ts":"<ISO-8601-UTC>","skill":"prd","run_id":"prd-<slug>","prd_slug":"<slug>","cards_created":0,"ac_coverage_ratio":0.0,"isa_count":0,"has_test_plan_pct":0.0,"has_db_indexes_pct":0.0,"discovery_iterations":1,"prd_path":"${paths.prd_dir}/<slug>/PRD.md"}
373
383
  ```
374
384
 
375
- 3. Add the file to the explicit stage list in validation-phase.md Step 7 point 6 (the stage list is at point 6; point 5 is the invocation of this very Metrics Log step).
385
+ 3. **Session token telemetry (ONLY when `stats_enabled: true` in the state file)** measure REAL per-role token + wall-clock cost by post-processing the session transcripts (zero model-token cost runs in Bash). Invoke:
376
386
 
377
- **This step is NON-BLOCKING** — if it fails, do not abort. Log "Metrics Log: SKIPPED" in the progress bar. The PRD merge proceeds regardless.
387
+ ```bash
388
+ node .framework/framework/scripts/analyze-session-tokens.js \
389
+ --skill prd --run-id "prd-<slug>" --out-dir "$WORKTREE_PATH/docs/metrics"
390
+ ```
391
+
392
+ The script reads `$CLAUDE_CODE_SESSION_ID` and derives the transcript paths itself (do NOT pass `--session` / `--project-dir` — test-only overrides). It writes `$WORKTREE_PATH/docs/metrics/sessions/prd-<slug>.md` and prints a summary plus a final `TELEMETRY_JSON=<json>` line. Parse that line and add its object under a `"cost"` key on the same JSONL row from step 2; on `stats: SKIPPED (...)` add `"cost":{"skipped":"<reason>"}`. Echo the summary to the user. Add the new `sessions/prd-<slug>.md` file to the explicit stage list too (point 4 below).
393
+
394
+ 4. Add the file(s) to the explicit stage list in validation-phase.md Step 7 point 6 (the stage list is at point 6; point 5 is the invocation of this very Metrics Log step).
395
+
396
+ **This step is NON-BLOCKING** — if it fails, do not abort. Log "Metrics Log: SKIPPED" in the progress bar. The PRD merge proceeds regardless. The `-stats` script is itself fail-safe (always exits 0).
@@ -1,6 +1,7 @@
1
1
  # PRD Session: {{slug}}
2
2
  Created: {{YYYY-MM-DD HH:MM}}
3
3
  Status: discovery
4
+ stats_enabled: {{true if invoked with -stats/--stats, else false}}
4
5
 
5
6
  ## Worktree
6
7
  - path: {{absolute worktree path returned by worktree-manager nw-docs}}
@@ -0,0 +1,411 @@
1
+ #!/usr/bin/env node
2
+ // Post-hoc session telemetry for /new and /prd (opt-in via `-stats`).
3
+ //
4
+ // Reads the live Claude Code transcripts — the orchestrator's main transcript
5
+ // plus every subagent transcript — and aggregates REAL token usage and wall-clock
6
+ // time by AGENT ROLE (coder, code-reviewer, qa-sentinel, doc-reviewer, …). The
7
+ // data already exists in the transcripts (`message.usage` + `timestamp`), so this
8
+ // costs ZERO model tokens: it runs in Node, after the run, outside the model.
9
+ //
10
+ // Transcript layout (verified):
11
+ // ~/.claude/projects/<slug>/<session_id>.jsonl ← orchestrator (main)
12
+ // ~/.claude/projects/<slug>/<session_id>/subagents/agent-*.jsonl ← one file per subagent
13
+ // where slug = cwd with every "/" replaced by "-".
14
+ //
15
+ // Role mapping: each `Agent` tool_use in the main transcript carries
16
+ // `input.subagent_type` (= role) and `input.prompt`. Every subagent file's first
17
+ // user message IS that prompt, so we link file→role by prompt-prefix match
18
+ // (verified 20/20 on a real /new run). Unlinkable files fall back to role
19
+ // "unknown" with a slug-derived label — their token/time data stays accurate.
20
+ //
21
+ // Cache accounting: the four usage counters are kept SEPARATE on purpose.
22
+ // `cache_read` is billed ~10% of fresh input; conflating them would overstate
23
+ // cost and mislead "what consumed the most". A single headline `cost_units`
24
+ // figure weights them: output + fresh_input + cache_creation×1.25 + cache_read×0.1.
25
+ //
26
+ // Usage:
27
+ // node analyze-session-tokens.js --skill new --run-id batch-FEAT-0001
28
+ // node analyze-session-tokens.js --skill prd --run-id prd-checkout --out-dir docs/metrics
29
+ // # test/override (bypass env + cwd derivation):
30
+ // node analyze-session-tokens.js --skill new --run-id X --session <id> --project-dir <path>
31
+ //
32
+ // Output:
33
+ // - <out-dir>/sessions/<run_id>.md — human-readable per-role breakdown + totals
34
+ // - stdout: same summary, PLUS a final line `TELEMETRY_JSON=<json>` carrying the
35
+ // totals object for the skill to embed in docs/metrics/skill-runs.jsonl.
36
+ //
37
+ // Fail-safe: ANY error prints `stats: SKIPPED (<reason>)` and exits 0. This script
38
+ // must NEVER abort a /new or /prd run.
39
+
40
+ 'use strict';
41
+
42
+ const fs = require('fs');
43
+ const os = require('os');
44
+ const path = require('path');
45
+
46
+ // ---- arg parsing -----------------------------------------------------------
47
+
48
+ function getArg(name, fallback) {
49
+ const i = process.argv.indexOf(`--${name}`);
50
+ return i !== -1 && process.argv[i + 1] ? process.argv[i + 1] : fallback;
51
+ }
52
+
53
+ // Bail out softly — telemetry is never allowed to break a run.
54
+ function skip(reason) {
55
+ console.log(`stats: SKIPPED (${reason})`);
56
+ process.exit(0);
57
+ }
58
+
59
+ const skill = getArg('skill', 'new');
60
+ const runId = getArg('run-id', null);
61
+ const outDir = getArg('out-dir', path.join('docs', 'metrics'));
62
+ const sessionId = getArg('session', process.env.CLAUDE_CODE_SESSION_ID || null);
63
+ const projectDir = getArg('project-dir', null);
64
+
65
+ if (!runId) skip('no --run-id provided');
66
+ if (!sessionId) skip('no session id ($CLAUDE_CODE_SESSION_ID unset, no --session)');
67
+
68
+ // ---- locate transcripts ----------------------------------------------------
69
+
70
+ function projectSlug(cwd) {
71
+ // Claude Code stores transcripts under a slug = cwd with "/" → "-".
72
+ return cwd.replace(/\//g, '-');
73
+ }
74
+
75
+ let baseDir;
76
+ if (projectDir) {
77
+ baseDir = projectDir;
78
+ } else {
79
+ baseDir = path.join(os.homedir(), '.claude', 'projects', projectSlug(process.cwd()));
80
+ }
81
+
82
+ const mainTranscript = path.join(baseDir, `${sessionId}.jsonl`);
83
+ const subDir = path.join(baseDir, sessionId, 'subagents');
84
+
85
+ if (!fs.existsSync(mainTranscript)) skip(`main transcript not found: ${mainTranscript}`);
86
+
87
+ // ---- jsonl helpers ---------------------------------------------------------
88
+
89
+ function* readLines(file) {
90
+ let raw;
91
+ try {
92
+ raw = fs.readFileSync(file, 'utf8');
93
+ } catch (err) {
94
+ return;
95
+ }
96
+ for (const line of raw.split(/\r?\n/)) {
97
+ if (!line.trim()) continue;
98
+ let obj;
99
+ try {
100
+ obj = JSON.parse(line);
101
+ } catch (err) {
102
+ continue; // tolerate partial / malformed lines
103
+ }
104
+ yield obj;
105
+ }
106
+ }
107
+
108
+ function contentBlocks(rec) {
109
+ const c = rec && rec.message && rec.message.content;
110
+ if (Array.isArray(c)) return c;
111
+ if (typeof c === 'string') return [{ type: 'text', text: c }];
112
+ return [];
113
+ }
114
+
115
+ function blocksText(blocks) {
116
+ return blocks
117
+ .map((b) => (typeof b === 'string' ? b : b && typeof b.text === 'string' ? b.text : ''))
118
+ .join('')
119
+ .trim();
120
+ }
121
+
122
+ // Aggregate the four usage counters + wall-clock span of one transcript file.
123
+ function aggregateFile(file) {
124
+ const acc = {
125
+ fresh_input: 0,
126
+ cache_read: 0,
127
+ cache_creation: 0,
128
+ output: 0,
129
+ turns: 0,
130
+ firstTs: null,
131
+ lastTs: null,
132
+ };
133
+ for (const rec of readLines(file)) {
134
+ const ts = rec.timestamp ? Date.parse(rec.timestamp) : NaN;
135
+ if (!Number.isNaN(ts)) {
136
+ if (acc.firstTs === null || ts < acc.firstTs) acc.firstTs = ts;
137
+ if (acc.lastTs === null || ts > acc.lastTs) acc.lastTs = ts;
138
+ }
139
+ if (rec.type !== 'assistant') continue;
140
+ const u = (rec.message && rec.message.usage) || null;
141
+ if (!u) continue;
142
+ acc.fresh_input += u.input_tokens || 0;
143
+ acc.cache_read += u.cache_read_input_tokens || 0;
144
+ acc.cache_creation += u.cache_creation_input_tokens || 0;
145
+ acc.output += u.output_tokens || 0;
146
+ acc.turns += 1;
147
+ }
148
+ return acc;
149
+ }
150
+
151
+ function costUnits(a) {
152
+ return Math.round(a.output + a.fresh_input + a.cache_creation * 1.25 + a.cache_read * 0.1);
153
+ }
154
+
155
+ function wallMs(a) {
156
+ return a.firstTs !== null && a.lastTs !== null ? a.lastTs - a.firstTs : 0;
157
+ }
158
+
159
+ // ---- role map from main transcript -----------------------------------------
160
+
161
+ // Collect every Agent spawn: { role, promptPrefix } in spawn order.
162
+ const agentSpawns = [];
163
+ for (const rec of readLines(mainTranscript)) {
164
+ for (const blk of contentBlocks(rec)) {
165
+ if (blk && blk.type === 'tool_use' && blk.name === 'Agent' && blk.input) {
166
+ const role = blk.input.subagent_type || 'unknown';
167
+ const prompt = (blk.input.prompt || '').trim();
168
+ agentSpawns.push({ role, prompt });
169
+ }
170
+ }
171
+ }
172
+
173
+ function matchRole(firstPrompt) {
174
+ if (!firstPrompt) return null;
175
+ for (const s of agentSpawns) {
176
+ if (!s.prompt) continue;
177
+ // bidirectional prefix match tolerates truncation on either side
178
+ if (firstPrompt.startsWith(s.prompt.slice(0, 50)) || s.prompt.startsWith(firstPrompt.slice(0, 50))) {
179
+ return s.role;
180
+ }
181
+ }
182
+ return null;
183
+ }
184
+
185
+ // ---- aggregate subagents + main --------------------------------------------
186
+
187
+ const byRole = new Map(); // role → { agg, agents }
188
+ function addToRole(role, agg) {
189
+ if (!byRole.has(role)) {
190
+ byRole.set(role, {
191
+ fresh_input: 0,
192
+ cache_read: 0,
193
+ cache_creation: 0,
194
+ output: 0,
195
+ turns: 0,
196
+ agents: 0,
197
+ wallSumMs: 0,
198
+ });
199
+ }
200
+ const r = byRole.get(role);
201
+ r.fresh_input += agg.fresh_input;
202
+ r.cache_read += agg.cache_read;
203
+ r.cache_creation += agg.cache_creation;
204
+ r.output += agg.output;
205
+ r.turns += agg.turns;
206
+ r.agents += 1;
207
+ r.wallSumMs += wallMs(agg);
208
+ }
209
+
210
+ let subFiles = [];
211
+ if (fs.existsSync(subDir)) {
212
+ try {
213
+ subFiles = fs
214
+ .readdirSync(subDir)
215
+ .filter((f) => f.endsWith('.jsonl'))
216
+ .map((f) => path.join(subDir, f));
217
+ } catch (err) {
218
+ subFiles = [];
219
+ }
220
+ }
221
+
222
+ let unmatchedFiles = 0;
223
+ for (const f of subFiles) {
224
+ const agg = aggregateFile(f);
225
+ if (agg.turns === 0 && wallMs(agg) === 0) continue; // empty/aborted subagent
226
+ // role: first user prompt of the subagent file
227
+ let firstPrompt = '';
228
+ for (const rec of readLines(f)) {
229
+ if (rec.type === 'user') {
230
+ firstPrompt = blocksText(contentBlocks(rec));
231
+ break;
232
+ }
233
+ }
234
+ const role = matchRole(firstPrompt);
235
+ if (!role) unmatchedFiles += 1;
236
+ addToRole(role || 'unknown', agg);
237
+ }
238
+
239
+ // orchestrator (main transcript) as its own row
240
+ const mainAgg = aggregateFile(mainTranscript);
241
+ addToRole('orchestrator', mainAgg);
242
+
243
+ // team-mode nesting caveat: Agent calls in main with no local subagent file
244
+ // (background / nested orchestrators write to a different session dir).
245
+ const localSubCount = subFiles.length;
246
+ const spawnedCount = agentSpawns.length;
247
+ const missingNested = Math.max(0, spawnedCount - localSubCount);
248
+
249
+ // ---- totals ----------------------------------------------------------------
250
+
251
+ const totals = {
252
+ fresh_input: 0,
253
+ cache_read: 0,
254
+ cache_creation: 0,
255
+ output: 0,
256
+ turns: 0,
257
+ agents: 0,
258
+ };
259
+ for (const r of byRole.values()) {
260
+ totals.fresh_input += r.fresh_input;
261
+ totals.cache_read += r.cache_read;
262
+ totals.cache_creation += r.cache_creation;
263
+ totals.output += r.output;
264
+ totals.turns += r.turns;
265
+ totals.agents += r.agents;
266
+ }
267
+ totals.cost_units = costUnits(totals);
268
+
269
+ // run wall-clock = span of the main transcript (first→last timestamp)
270
+ const runWallMs = wallMs(mainAgg);
271
+
272
+ // ---- formatting helpers -----------------------------------------------------
273
+
274
+ function fmtTokens(n) {
275
+ if (n >= 1e6) return `${(n / 1e6).toFixed(1)}M`;
276
+ if (n >= 1e3) return `${(n / 1e3).toFixed(1)}k`;
277
+ return String(n);
278
+ }
279
+ function fmtDuration(ms) {
280
+ if (!ms || ms < 0) return '-';
281
+ const s = Math.round(ms / 1000);
282
+ const m = Math.floor(s / 60);
283
+ const rs = s % 60;
284
+ return m > 0 ? `${m}m${rs.toString().padStart(2, '0')}s` : `${rs}s`;
285
+ }
286
+ function pad(s, w) {
287
+ s = String(s);
288
+ return s.length >= w ? s : s + ' '.repeat(w - s.length);
289
+ }
290
+ function padL(s, w) {
291
+ s = String(s);
292
+ return s.length >= w ? s : ' '.repeat(w - s.length) + s;
293
+ }
294
+
295
+ // rows sorted by cost_units desc
296
+ const roleRows = [...byRole.entries()]
297
+ .map(([role, r]) => ({ role, ...r, cost_units: costUnits(r) }))
298
+ .sort((a, b) => b.cost_units - a.cost_units);
299
+
300
+ // ---- write per-run markdown summary ----------------------------------------
301
+
302
+ const lines = [];
303
+ lines.push(`# Session telemetry — ${runId}`);
304
+ lines.push('');
305
+ lines.push(`- **skill**: ${skill}`);
306
+ lines.push(`- **session**: ${sessionId}`);
307
+ lines.push(`- **run wall-clock** (orchestrator span, includes idle): ${fmtDuration(runWallMs)}`);
308
+ lines.push(`- **agents**: ${totals.agents - 1} subagent(s) + 1 orchestrator`);
309
+ lines.push(`- **total cost units**: ${fmtTokens(totals.cost_units)} _(output + fresh_in + cache_creation×1.25 + cache_read×0.1)_`);
310
+ lines.push('');
311
+ lines.push('## By role');
312
+ lines.push('');
313
+ const H = ['role', 'agents', 'wall(sum)', 'output', 'fresh_in', 'cache_cr', 'cache_rd', 'cost_units', '% cost'];
314
+ const W = [20, 7, 10, 9, 9, 9, 9, 11, 7];
315
+ lines.push(H.map((h, i) => pad(h, W[i])).join(' | '));
316
+ lines.push(W.map((w) => '-'.repeat(w)).join('-|-'));
317
+ for (const r of roleRows) {
318
+ const pct = totals.cost_units ? `${((r.cost_units / totals.cost_units) * 100).toFixed(0)}%` : '-';
319
+ lines.push(
320
+ [
321
+ pad(r.role, W[0]),
322
+ padL(r.agents, W[1]),
323
+ padL(fmtDuration(r.wallSumMs), W[2]),
324
+ padL(fmtTokens(r.output), W[3]),
325
+ padL(fmtTokens(r.fresh_input), W[4]),
326
+ padL(fmtTokens(r.cache_creation), W[5]),
327
+ padL(fmtTokens(r.cache_read), W[6]),
328
+ padL(fmtTokens(r.cost_units), W[7]),
329
+ padL(pct, W[8]),
330
+ ].join(' | ')
331
+ );
332
+ }
333
+ lines.push(W.map((w) => '-'.repeat(w)).join('-|-'));
334
+ lines.push(
335
+ [
336
+ pad('TOTAL', W[0]),
337
+ padL(totals.agents, W[1]),
338
+ padL(fmtDuration(runWallMs), W[2]),
339
+ padL(fmtTokens(totals.output), W[3]),
340
+ padL(fmtTokens(totals.fresh_input), W[4]),
341
+ padL(fmtTokens(totals.cache_creation), W[5]),
342
+ padL(fmtTokens(totals.cache_read), W[6]),
343
+ padL(fmtTokens(totals.cost_units), W[7]),
344
+ padL('100%', W[8]),
345
+ ].join(' | ')
346
+ );
347
+ lines.push('');
348
+ lines.push('> wall(sum) is the sum of per-agent spans and reflects ACTIVE compute time; the');
349
+ lines.push('> run wall-clock is the orchestrator first→last span and INCLUDES human idle gaps.');
350
+ lines.push('> Subagents may run in parallel, so wall(sum) can exceed the run wall-clock.');
351
+ lines.push('> cache_read tokens are ~10% the cost of fresh input — kept separate on purpose.');
352
+ if (unmatchedFiles > 0) {
353
+ lines.push('>');
354
+ lines.push(`> ⚠ ${unmatchedFiles} subagent file(s) could not be mapped to a role (bucketed as "unknown").`);
355
+ }
356
+ if (missingNested > 0) {
357
+ lines.push('>');
358
+ lines.push(
359
+ `> ⚠ team-mode caveat: ${missingNested} Agent spawn(s) in the main transcript have no local`
360
+ );
361
+ lines.push(
362
+ `> subagent file (likely background / nested orchestrators in separate sessions) — their`
363
+ );
364
+ lines.push(`> tokens are NOT included in these totals. Sequential-mode runs are fully accurate.`);
365
+ }
366
+ lines.push('');
367
+
368
+ // ---- emit ------------------------------------------------------------------
369
+
370
+ const sessionsDir = path.join(outDir, 'sessions');
371
+ try {
372
+ fs.mkdirSync(sessionsDir, { recursive: true });
373
+ fs.writeFileSync(path.join(sessionsDir, `${runId}.md`), lines.join('\n'));
374
+ } catch (err) {
375
+ skip(`could not write summary: ${err.message}`);
376
+ }
377
+
378
+ // human summary to stdout
379
+ console.log(lines.join('\n'));
380
+
381
+ // machine-readable totals for the skill to embed in skill-runs.jsonl
382
+ const jsonOut = {
383
+ run_wall_ms: runWallMs,
384
+ by_role: Object.fromEntries(
385
+ roleRows.map((r) => [
386
+ r.role,
387
+ {
388
+ agents: r.agents,
389
+ output: r.output,
390
+ fresh_input: r.fresh_input,
391
+ cache_creation: r.cache_creation,
392
+ cache_read: r.cache_read,
393
+ cost_units: r.cost_units,
394
+ wall_sum_ms: r.wallSumMs,
395
+ },
396
+ ])
397
+ ),
398
+ totals: {
399
+ agents: totals.agents,
400
+ output: totals.output,
401
+ fresh_input: totals.fresh_input,
402
+ cache_creation: totals.cache_creation,
403
+ cache_read: totals.cache_read,
404
+ cost_units: totals.cost_units,
405
+ turns: totals.turns,
406
+ },
407
+ unmatched_subagents: unmatchedFiles,
408
+ nested_agents_excluded: missingNested,
409
+ };
410
+ console.log(`TELEMETRY_JSON=${JSON.stringify(jsonOut)}`);
411
+ process.exit(0);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "baldart",
3
- "version": "3.35.2",
3
+ "version": "3.37.0",
4
4
  "description": "Claude Agent Framework - Reusable framework for coordinating AI agents and humans in software projects",
5
5
  "bin": {
6
6
  "baldart": "./bin/baldart.js"