baldart 3.35.2 → 3.37.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +39 -0
- package/VERSION +1 -1
- package/framework/.claude/skills/new/SKILL.md +50 -26
- package/framework/.claude/skills/prd/SKILL.md +21 -2
- package/framework/.claude/skills/prd/assets/state-template.md +1 -0
- package/framework/scripts/analyze-session-tokens.js +411 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,45 @@ All notable changes to BALDART will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [3.37.0] - 2026-05-30
|
|
9
|
+
|
|
10
|
+
**Two-tier review for `/new`: fast `light` per-card during the loop, one guaranteed FULL pass over the whole batch before merge.** This activates the dormant v3.35.0 Review Profile Selector AND re-introduces an unconditional final full review — together they cut tokens/wall-clock on low-risk cards *without* weakening the merge gate. **No new `baldart.config.yml` keys** — both are changes inside the skill, so the schema-propagation rule does not apply.
|
|
11
|
+
|
|
12
|
+
> **The design.** `light` (per-card Phase 3.7) is now an **early-feedback optimization**, not the safety gate. The safety gate is the **Final review**, which since this release ALWAYS runs a single FULL `/codexreview` (full agent set) over the **entire batch diff** before merge — no N=1 skip, no cross-card scope reduction. So every line of every card, including any reviewed at `light`, gets a full-depth Codex review at least once before merge. This is what makes activating `light` safe: it shifts the breadth passes (qa-sentinel, api-perf-cost-auditor, doc-reviewer) + CoVe from the per-card loop to the single final pass, rather than dropping them from the merge gate. (The v3.35.0 data gate — which had kept the selector hard-coded to `full` pending consumer telemetry — is lifted by explicit maintainer decision; an adversarial review showed that gate's intended post-hoc telemetry could never have validated it anyway, since the Fix Application Log is blind by construction to blockers `light` *misses*. The final full gate makes that validation question moot.)
|
|
13
|
+
|
|
14
|
+
### Changed — Review Profile Selector live (per-card early feedback)
|
|
15
|
+
|
|
16
|
+
- **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)**: Phase 3.7 Step C (sequential) and Team Mode D.4b no longer hard-code `profile = full`. The selector runs as designed — `light` ⟺ Step A matched **0** high-risk triggers **AND** the card's QA profile ∈ {`skip`, `light`}; `full` otherwise. **BUG-0530 invariant preserved**: `light` ≠ skip — Codex adversarial + `code-reviewer` + false-positive gate ALWAYS run per-card; `light` drops only the breadth passes + CoVe.
|
|
17
|
+
|
|
18
|
+
### Changed — Final review is now an unconditional FULL batch-diff gate (supersedes v3.35.0 scope-reduction)
|
|
19
|
+
|
|
20
|
+
- **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)** (Final review / Step F.1): the v3.35.0 final-review scope gate (N=1 → skip F.1–F.4; N>1 → reduce to the cross-card surface) is **removed**. The final review now runs Steps F.1–F.5 for **every batch including N=1**, with `review_scope_files` = the **full union** of all touched files, never reduced. F.3 reviews the entire batch diff at full depth (Codex + doc-reviewer + api-perf-cost-auditor + qa-sentinel); F.5 runs the final build. Team mode reaches the same gate via "Post-batch — same as sequential mode" (its prior "cross-card-only scope" note is updated accordingly).
|
|
21
|
+
- **Rationale**: v3.35.0 de-duplicated the final full pass on the assumption that Phase 3.7 had already full-reviewed every card. Selecting `light` breaks that assumption, so the post-batch full pass is re-introduced as the guaranteed merge gate.
|
|
22
|
+
|
|
23
|
+
### Cost & trade-offs (documented)
|
|
24
|
+
|
|
25
|
+
- **Net cost**: one full `/codexreview` per batch is re-introduced (the deliberate price of the per-card `light` speed-up). The saving comes from cheaper per-card reviews during the loop; the final pass is paid once per batch, not per card.
|
|
26
|
+
- **Later feedback on `light` cards**: a defect on a `light`-reviewed card may now surface at the final pass rather than at the per-card step — possibly a later fix-cycle. Acceptable: it is still caught before merge.
|
|
27
|
+
- **Revert**: to make Phase 3.7 always-full again, restore `profile = full` (no config kill-switch by design). The final full gate is independent and stays regardless.
|
|
28
|
+
|
|
29
|
+
## [3.36.0] - 2026-05-30
|
|
30
|
+
|
|
31
|
+
Adds **opt-in session telemetry** to `/new` and `/prd`: append `-stats` (or `--stats`) to a run and, at the end, the skill writes a per-**agent-role** breakdown of REAL token consumption and wall-clock time. This finally answers "what took the most time and tokens in a `/new` run" — and feeds the v3.35.0 Review Profile Selector **data gate** (still pending precisely because `docs/metrics/` carried no per-role cost data). **No new `baldart.config.yml` keys** — the opt-in is a flag, not a config feature, so the schema-propagation rule does not apply.
|
|
32
|
+
|
|
33
|
+
> **Zero model-token overhead.** The measurement is a **post-hoc Node script**, not a hook and not LLM self-reporting (the model has no reliable token counter). It reads the Claude Code session transcripts — the orchestrator's `<session>.jsonl` plus every `subagents/agent-*.jsonl` — which already carry `message.usage` (the four token counters) and per-message `timestamp`. On a real `/new` run the bulk of cost lives in the subagents (measured ~53M cumulative subagent tokens vs ~47k orchestrator output), and it is all captured. Cache-read tokens are kept **separate** from fresh input (they bill ~10×cheaper) so the breakdown doesn't mislead.
|
|
34
|
+
|
|
35
|
+
### Added — `-stats` / `--stats` opt-in session telemetry
|
|
36
|
+
|
|
37
|
+
- **[framework/scripts/analyze-session-tokens.js](framework/scripts/analyze-session-tokens.js)** (new): post-hoc aggregator. Locates the live transcripts from `$CLAUDE_CODE_SESSION_ID` + the cwd slug, sums the four `usage` counters (`fresh_input` / `cache_read` / `cache_creation` / `output`) **kept separate**, computes per-file wall-clock spans, and maps each subagent file → its role by matching the subagent's first prompt to the `Agent` tool_use `subagent_type` in the main transcript (verified 20/20 on a real `/new` batch; unmatched files bucket as `unknown`). Emits a human-readable `docs/metrics/sessions/<run_id>.md` (per-role table sorted by a weighted `cost_units` figure + totals + caveats) and a machine-readable `TELEMETRY_JSON=` line. **Fail-safe**: any error prints `stats: SKIPPED (<reason>)` and exits 0 — it can never abort a run.
|
|
38
|
+
- **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)**: arg parser strips `-stats` / `--stats` like `-full` and sets an internal `STATS` flag (composes with `-full`). **Phase 8 — Metrics Log** gains a `STATS`-gated step that invokes the script and enriches the same `skill-runs.jsonl` row with a `"cost"` object (`by_role` + `totals` + `run_wall_ms`). Without `-stats`, Phase 8 is byte-for-byte unchanged.
|
|
39
|
+
- **[framework/.claude/skills/prd/SKILL.md](framework/.claude/skills/prd/SKILL.md)**: new **HARD RULE 18** — since `/prd` is conversational (no flag parser), the opt-in is detected once at **Step 1 (Kickoff)** from `$ARGUMENTS` and persisted as `stats_enabled` in the session state file (survives context compression / Step 0 resume). The **Step 7 Metrics Log** reads it and runs the same script (`--skill prd`), enriching its JSONL row identically.
|
|
40
|
+
- **[framework/.claude/skills/prd/assets/state-template.md](framework/.claude/skills/prd/assets/state-template.md)**: new `stats_enabled` header field.
|
|
41
|
+
|
|
42
|
+
### Notes & caveats
|
|
43
|
+
|
|
44
|
+
- **Accurate in sequential mode.** Team-mode L0/L1 orchestrators can spawn background / nested sub-orchestrators in *separate* session dirs; their subagents are not under the top-level `<session>/subagents/`. The script detects the gap (Agent spawns in main with no local subagent file) and prints an explicit caveat with the excluded count rather than silently undercounting.
|
|
45
|
+
- The per-role `wall(sum)` reflects active compute time; the headline run wall-clock is the orchestrator span and **includes human idle gaps** — both are labelled in the report.
|
|
46
|
+
|
|
8
47
|
## [3.35.2] - 2026-05-30
|
|
9
48
|
|
|
10
49
|
Resolves the **3.35.1 known follow-up**: a flag introduced in a newer release (e.g. `--on-divergence`, v3.29.0) was rejected by an older global CLI with a cryptic `error: unknown option` **before** the `npx baldart@latest` auto-relaunch could self-upgrade — commander validates options during `program.parse()`, which runs *before* the action handler where the relaunch lives. The `update` command now parses **permissively**, so an unknown flag is captured instead of crashing the parser; the action then transparently relaunches under `@latest` (forwarding the raw argv so the unrecognized flag survives) or, when that isn't possible, emits an **actionable** error instead of commander's cryptic one. **No new `baldart.config.yml` keys** — the schema-propagation rule does not apply.
|
package/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
3.
|
|
1
|
+
3.37.0
|
|
@@ -37,6 +37,8 @@ Parse the card IDs from the arguments. Cards can be specified as:
|
|
|
37
37
|
|
|
38
38
|
If no card IDs are provided, ask the user which cards to implement.
|
|
39
39
|
|
|
40
|
+
**Opt-in session telemetry (`-stats` / `--stats`)**: strip `-stats` / `--stats` from the args list exactly like `-full` (it is NOT a card ID), and set an internal flag `STATS=true` for this run. When present, Phase 8 additionally measures REAL token + wall-clock cost per agent role (coder, code-reviewer, …) by post-processing the session transcripts — zero model-token overhead, runs in Bash after the work is done. When absent, Phase 8 behaves exactly as before. The flag is batch-scoped and composes with `-full` (`/new FEAT-005 -full -stats`).
|
|
41
|
+
|
|
40
42
|
---
|
|
41
43
|
|
|
42
44
|
## Context Tracking (CRITICAL)
|
|
@@ -1111,7 +1113,7 @@ skill's Phase 1 falls back to deriving Gherkin scenarios from
|
|
|
1111
1113
|
|
|
1112
1114
|
### Phase 3 — Doc Review & Sync (code review handled by mandatory Phase 3.7 gate)
|
|
1113
1115
|
|
|
1114
|
-
> **Note**: Code review is NOT performed in this phase — it is handled by the **mandatory unconditional `/codexreview` gate in Phase 3.7**, which runs per-card BEFORE the Phase 4 commit. The post-batch `/codexreview` in
|
|
1116
|
+
> **Note**: Code review is NOT performed in this phase — it is handled by the **mandatory unconditional `/codexreview` gate in Phase 3.7**, which runs per-card BEFORE the Phase 4 commit. The post-batch `/codexreview` in the Final review remains as a final FULL-diff sweep over the entire batch (since v3.37.0 — the guaranteed full-depth merge gate). This Phase 3 focuses exclusively on documentation sync, which must happen per-card (tied to the specific commit).
|
|
1115
1117
|
|
|
1116
1118
|
12. **Update tracker**: phase = "3-doc-review".
|
|
1117
1119
|
13. Build a **Doc Sync Context** block:
|
|
@@ -1285,10 +1287,15 @@ For EVERY card (no conditional skip — the gate ALWAYS runs; only its DEPTH var
|
|
|
1285
1287
|
doc-reviewer) and Step 3.5 CoVe — exactly the passes that yield ~nothing on a 0-trigger, QA-light
|
|
1286
1288
|
diff. The BUG-0530 invariant ("never miss a blocker on a low-risk path") holds because the Codex
|
|
1287
1289
|
adversarial pass — the one that catches those — always runs.
|
|
1288
|
-
**
|
|
1289
|
-
|
|
1290
|
-
|
|
1291
|
-
|
|
1290
|
+
**ACTIVE since v3.37.0**: the selector above runs as written. Here `light` is an **early-feedback
|
|
1291
|
+
optimization** for the per-card loop — NOT the merge-gate. Safety is guaranteed downstream: since
|
|
1292
|
+
v3.37.0 the **Final review ALWAYS runs a single FULL `/codexreview` over the entire batch diff before
|
|
1293
|
+
merge** (no N=1 skip, no scope reduction — see "Final review → Final-review FULL gate"). So every card
|
|
1294
|
+
reviewed at `light` here still receives a full-depth review before merge; `light` shifts the breadth
|
|
1295
|
+
passes (qa-sentinel, api-perf-cost-auditor, doc-reviewer) + CoVe from the per-card loop to that single
|
|
1296
|
+
final full pass, it does not drop them from the merge gate. The BUG-0530 invariant still holds per-card
|
|
1297
|
+
(Codex adversarial + `code-reviewer` + FP-gate always run in `light`). To make Phase 3.7 always-full
|
|
1298
|
+
again: restore `profile = full` here (the final full gate is independent and stays regardless).
|
|
1292
1299
|
|
|
1293
1300
|
Then dump the card diff (reviewer grounding) and write the lean contract `/codexreview` consumes:
|
|
1294
1301
|
```bash
|
|
@@ -1400,20 +1407,24 @@ Never block the pipeline indefinitely — recover and continue per the rules abo
|
|
|
1400
1407
|
|
|
1401
1408
|
Once ALL cards are committed in the worktree:
|
|
1402
1409
|
|
|
1403
|
-
> **Final-review
|
|
1404
|
-
>
|
|
1405
|
-
>
|
|
1410
|
+
> **Final-review FULL gate (since v3.37.0 — supersedes the v3.35.0 scope-reduction)** — because
|
|
1411
|
+
> Phase 3.7 may now run a card at `light` depth, the per-card pass can NO LONGER be assumed to have
|
|
1412
|
+
> full-reviewed every card. The final review is therefore the **unconditional safety gate that makes
|
|
1413
|
+
> the Phase 3.7 `light` profile safe**: it ALWAYS runs a single FULL `/codexreview` (full agent set)
|
|
1414
|
+
> over the **ENTIRE batch diff** before merge — **no N=1 skip, no cross-card scope reduction**. Every
|
|
1415
|
+
> line of every card — including any reviewed at `light` in Phase 3.7 — receives a full-depth Codex
|
|
1416
|
+
> review at least once before merge.
|
|
1417
|
+
>
|
|
1418
|
+
> - Run Steps F.1–F.5 for **EVERY batch, including N=1**. Nothing here is skipped.
|
|
1419
|
+
> - `review_scope_files` = the **FULL union** of all touched files across all cards (F.1 step 4 —
|
|
1420
|
+
> NEVER reduced to the cross-card subset).
|
|
1421
|
+
> - F.3 invokes the full reviewer set (Codex + doc-reviewer + api-perf-cost-auditor + qa-sentinel)
|
|
1422
|
+
> over that union; F.5 runs the final build.
|
|
1406
1423
|
>
|
|
1407
|
-
> -
|
|
1408
|
-
>
|
|
1409
|
-
>
|
|
1410
|
-
>
|
|
1411
|
-
> — that gate is NOT skipped.
|
|
1412
|
-
> - **Multiple cards (N>1)**: run F.1–F.5 but **scope the review to the cross-card surface only**
|
|
1413
|
-
> (computed, not discretionary) — see F.1 step 4. If that surface is empty, log
|
|
1414
|
-
> `"cross-card review: no shared surface (intersection=∅, depends_on=∅) — F.3 limited to integration sanity"`
|
|
1415
|
-
> and run a minimal integration pass (build + type-check across the union) instead of a full
|
|
1416
|
-
> per-line re-review. This is a computed scope reduction, NEVER a model-decided skip.
|
|
1424
|
+
> Rationale: this re-introduces the post-batch full pass that v3.35.0 de-duplicated away. That
|
|
1425
|
+
> de-dup assumed Phase 3.7 had already full-reviewed every card — an assumption broken the moment
|
|
1426
|
+
> `light` became selectable. The cost of one full batch-diff review is the deliberate price of the
|
|
1427
|
+
> per-card `light` speed-up (explicit maintainer decision, v3.37.0).
|
|
1417
1428
|
|
|
1418
1429
|
### Step F.1 — Resolve scope
|
|
1419
1430
|
|
|
@@ -1424,11 +1435,11 @@ Once ALL cards are committed in the worktree:
|
|
|
1424
1435
|
git diff --name-only <base-branch>...HEAD
|
|
1425
1436
|
```
|
|
1426
1437
|
3. Read each card's backlog YAML to collect `acceptance_criteria`, `files_likely_touched`.
|
|
1427
|
-
4. Build `review_scope_files` as union of card-indicated files + git-touched files
|
|
1428
|
-
the
|
|
1429
|
-
|
|
1430
|
-
|
|
1431
|
-
|
|
1438
|
+
4. Build `review_scope_files` as the **FULL union** of card-indicated files + git-touched files
|
|
1439
|
+
across ALL cards in the batch. **Since v3.37.0 this union is NEVER reduced** (the v3.35.0
|
|
1440
|
+
cross-card scope-reduction is removed) — the final review is a full-depth pass over the entire
|
|
1441
|
+
batch diff, so cards reviewed at `light` in Phase 3.7 still get a guaranteed full review before
|
|
1442
|
+
merge. Record the union in the tracker; downstream F.3 reviews it in full.
|
|
1432
1443
|
|
|
1433
1444
|
### Step F.2 — Architecture baseline
|
|
1434
1445
|
|
|
@@ -2191,7 +2202,7 @@ After ALL agents in the group complete successfully:
|
|
|
2191
2202
|
|
|
2192
2203
|
4a. **D.4a — Per-card doc gate (consumes D.2, since v3.35.0)** — Do NOT re-spawn `doc-reviewer`. For EACH card in the group, take that card's doc findings from D.2's per-card-attributed output; if any exist, spawn a targeted fix-coder for that card only. (Previously D.4a re-ran doc-reviewer per card — a duplicate of D.2; collapsed to a single attributed D.2 pass. D.4b's per-card `/codexreview` also skips its doc-reviewer via the lean contract, so doc-review now runs once per group instead of ~3× per card.)
|
|
2193
2204
|
|
|
2194
|
-
4b. **D.4b — Pre-Merge Codex Review Gate (per-card, MANDATORY — UNCONDITIONAL)** — Invoke `/codexreview` for EACH card in the group, one at a time, BEFORE any commit. This mirrors Phase 3.7 of the sequential path and is non-skippable regardless of file paths or perceived risk. Apply the same fix sub-loop: if the consolidated report shows verified BLOCKER/HIGH findings, spawn a fix-coder and re-run `/codexreview <CARD-ID>` (max 2 retries per card). If still BLOCKER/HIGH after retries, ask the user before proceeding to D.5. The D.5 commits MUST NOT happen until every card in the group has a PASS verdict (or explicit user override via `AskUserQuestion`). Log results in the tracker under `## Pre-Merge Codex Review` per card. **Lean + profile (since v3.35.0)**: before each card's `/codexreview`, apply the same Review Profile Selector and write the same `/tmp/codexreview-lean-<CARD-ID>.json` contract as sequential Phase 3.7 Step C — `arch_baseline_path` pointing at `/tmp/arch-baseline-group-<FIRST-CARD-ID>.md`, `skip_doc_reviewer: true`, and `profile: light|full` from (0 triggers + QA ∈ {skip,light}). LIGHT still runs Codex adversarial + code-reviewer + FP-gate (
|
|
2205
|
+
4b. **D.4b — Pre-Merge Codex Review Gate (per-card, MANDATORY — UNCONDITIONAL)** — Invoke `/codexreview` for EACH card in the group, one at a time, BEFORE any commit. This mirrors Phase 3.7 of the sequential path and is non-skippable regardless of file paths or perceived risk. Apply the same fix sub-loop: if the consolidated report shows verified BLOCKER/HIGH findings, spawn a fix-coder and re-run `/codexreview <CARD-ID>` (max 2 retries per card). If still BLOCKER/HIGH after retries, ask the user before proceeding to D.5. The D.5 commits MUST NOT happen until every card in the group has a PASS verdict (or explicit user override via `AskUserQuestion`). Log results in the tracker under `## Pre-Merge Codex Review` per card. **Lean + profile (since v3.35.0)**: before each card's `/codexreview`, apply the same Review Profile Selector and write the same `/tmp/codexreview-lean-<CARD-ID>.json` contract as sequential Phase 3.7 Step C — `arch_baseline_path` pointing at `/tmp/arch-baseline-group-<FIRST-CARD-ID>.md`, `skip_doc_reviewer: true`, and `profile: light|full` from (0 triggers + QA ∈ {skip,light}). LIGHT still runs Codex adversarial + code-reviewer + FP-gate. The Review Profile Selector is **ACTIVE since v3.37.0** — same status as sequential Phase 3.7 Step C above: `light` is per-card early feedback, and the merge-gate safety is the post-batch **Final-review FULL gate** (a single FULL `/codexreview` over the entire batch diff, which team mode reaches via "Post-batch — same as sequential mode"). To make per-card always-full again, restore `profile = full`; the final full gate stays regardless.
|
|
2195
2206
|
|
|
2196
2207
|
5. **D.5 — Commit** — One commit per card using explicit staging from the ownership map. **NO stash in worktrees** (stashes are globally shared via `refs/stash` — see Phase 4 WORKTREE COMMIT RULE):
|
|
2197
2208
|
```bash
|
|
@@ -2243,7 +2254,7 @@ Before starting group N, verify ALL cards in groups 0..N-1 are status: done. If
|
|
|
2243
2254
|
|
|
2244
2255
|
### Post-batch (same as sequential mode)
|
|
2245
2256
|
|
|
2246
|
-
After all groups are complete, run the same Final Review, Phase 6 (merge), and Phase 7 (production readiness) as documented above.
|
|
2257
|
+
After all groups are complete, run the same Final Review, Phase 6 (merge), and Phase 7 (production readiness) as documented above. Since v3.37.0 the Final Review runs a **single FULL `/codexreview` over the entire batch diff** (per the Final-review FULL gate) — no scope reduction — so every team-mode card, including any reviewed at `light` in D.4b, receives a guaranteed full review before merge.
|
|
2247
2258
|
|
|
2248
2259
|
---
|
|
2249
2260
|
|
|
@@ -2287,6 +2298,19 @@ and QA quality over time.
|
|
|
2287
2298
|
|
|
2288
2299
|
5. Note in tracker: `## Metrics Log: WRITTEN (run_id: batch-<FIRST-CARD-ID>)`
|
|
2289
2300
|
|
|
2301
|
+
6. **Session token telemetry (ONLY when `STATS=true`)** — if the run was invoked with `-stats` / `--stats`:
|
|
2302
|
+
|
|
2303
|
+
Measure REAL per-role token + wall-clock cost by post-processing the session transcripts (zero model-token cost — runs in Bash). Invoke:
|
|
2304
|
+
|
|
2305
|
+
```bash
|
|
2306
|
+
node .framework/framework/scripts/analyze-session-tokens.js \
|
|
2307
|
+
--skill new --run-id "batch-<FIRST-CARD-ID>" --out-dir docs/metrics
|
|
2308
|
+
```
|
|
2309
|
+
|
|
2310
|
+
The script reads `$CLAUDE_CODE_SESSION_ID` from the environment and derives the transcript paths itself (do NOT pass `--session` / `--project-dir` — those are test-only overrides). It writes a human-readable breakdown to `docs/metrics/sessions/batch-<FIRST-CARD-ID>.md` and prints a summary plus a final `TELEMETRY_JSON=<json>` line.
|
|
2311
|
+
|
|
2312
|
+
Parse the `TELEMETRY_JSON=` line and **enrich the same skill-runs.jsonl row** you wrote in step 3 by adding a `"cost"` key holding that JSON object (`by_role` + `totals` + `run_wall_ms`). If the script printed `stats: SKIPPED (...)` instead, add `"cost":{"skipped":"<reason>"}` and continue. Echo the script's summary to the user so they see the per-role breakdown.
|
|
2313
|
+
|
|
2290
2314
|
**If `docs/metrics/skill-runs.jsonl` does not exist**: create it first with `touch docs/metrics/skill-runs.jsonl`.
|
|
2291
2315
|
**If batch tracker is missing or unreadable**: log "Metrics Log: SKIPPED (tracker not found)" and proceed without blocking.
|
|
2292
|
-
**This phase is NON-BLOCKING** — if it fails for any reason, do not abort the run.
|
|
2316
|
+
**This phase is NON-BLOCKING** — if it fails for any reason, do not abort the run. The `-stats` step in particular is best-effort: the script is fail-safe (always exits 0), so never let it block the commit.
|
|
@@ -199,6 +199,16 @@ message. You ask questions, wait for answers, and iterate.
|
|
|
199
199
|
repository, or the consumer has explicitly disabled docs worktrees via
|
|
200
200
|
overlay), STOP and report the error to the user. Do NOT silently fall back
|
|
201
201
|
to writing on the main repo — that defeats the entire purpose of this rule.
|
|
202
|
+
18. **Opt-in session telemetry (`-stats` / `--stats`).** Unlike `/new`, this skill
|
|
203
|
+
has no flag parser — it is conversational. So detect the opt-in ONCE at **Step 1
|
|
204
|
+
(Kickoff)**: inspect `$ARGUMENTS` for the token `-stats` or `--stats`. If present,
|
|
205
|
+
set `stats_enabled: true` in the state-file header (the field exists in
|
|
206
|
+
`assets/state-template.md`); otherwise `false`. Persisting it in the state file is
|
|
207
|
+
what makes it survive context compression and Step 0 resume. The Metrics Log step
|
|
208
|
+
(inside Step 7) reads this field and, when `true`, post-processes the session
|
|
209
|
+
transcripts for REAL per-role token + wall-clock cost. The token itself is NOT part
|
|
210
|
+
of the feature description — strip it before recording the user's verbatim feature
|
|
211
|
+
text in `## Feature Description`.
|
|
202
212
|
|
|
203
213
|
---
|
|
204
214
|
|
|
@@ -372,6 +382,15 @@ exactly the correct semantics). No special handling needed.
|
|
|
372
382
|
{"ts":"<ISO-8601-UTC>","skill":"prd","run_id":"prd-<slug>","prd_slug":"<slug>","cards_created":0,"ac_coverage_ratio":0.0,"isa_count":0,"has_test_plan_pct":0.0,"has_db_indexes_pct":0.0,"discovery_iterations":1,"prd_path":"${paths.prd_dir}/<slug>/PRD.md"}
|
|
373
383
|
```
|
|
374
384
|
|
|
375
|
-
3.
|
|
385
|
+
3. **Session token telemetry (ONLY when `stats_enabled: true` in the state file)** — measure REAL per-role token + wall-clock cost by post-processing the session transcripts (zero model-token cost — runs in Bash). Invoke:
|
|
376
386
|
|
|
377
|
-
|
|
387
|
+
```bash
|
|
388
|
+
node .framework/framework/scripts/analyze-session-tokens.js \
|
|
389
|
+
--skill prd --run-id "prd-<slug>" --out-dir "$WORKTREE_PATH/docs/metrics"
|
|
390
|
+
```
|
|
391
|
+
|
|
392
|
+
The script reads `$CLAUDE_CODE_SESSION_ID` and derives the transcript paths itself (do NOT pass `--session` / `--project-dir` — test-only overrides). It writes `$WORKTREE_PATH/docs/metrics/sessions/prd-<slug>.md` and prints a summary plus a final `TELEMETRY_JSON=<json>` line. Parse that line and add its object under a `"cost"` key on the same JSONL row from step 2; on `stats: SKIPPED (...)` add `"cost":{"skipped":"<reason>"}`. Echo the summary to the user. Add the new `sessions/prd-<slug>.md` file to the explicit stage list too (point 4 below).
|
|
393
|
+
|
|
394
|
+
4. Add the file(s) to the explicit stage list in validation-phase.md Step 7 point 6 (the stage list is at point 6; point 5 is the invocation of this very Metrics Log step).
|
|
395
|
+
|
|
396
|
+
**This step is NON-BLOCKING** — if it fails, do not abort. Log "Metrics Log: SKIPPED" in the progress bar. The PRD merge proceeds regardless. The `-stats` script is itself fail-safe (always exits 0).
|
|
@@ -0,0 +1,411 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Post-hoc session telemetry for /new and /prd (opt-in via `-stats`).
|
|
3
|
+
//
|
|
4
|
+
// Reads the live Claude Code transcripts — the orchestrator's main transcript
|
|
5
|
+
// plus every subagent transcript — and aggregates REAL token usage and wall-clock
|
|
6
|
+
// time by AGENT ROLE (coder, code-reviewer, qa-sentinel, doc-reviewer, …). The
|
|
7
|
+
// data already exists in the transcripts (`message.usage` + `timestamp`), so this
|
|
8
|
+
// costs ZERO model tokens: it runs in Node, after the run, outside the model.
|
|
9
|
+
//
|
|
10
|
+
// Transcript layout (verified):
|
|
11
|
+
// ~/.claude/projects/<slug>/<session_id>.jsonl ← orchestrator (main)
|
|
12
|
+
// ~/.claude/projects/<slug>/<session_id>/subagents/agent-*.jsonl ← one file per subagent
|
|
13
|
+
// where slug = cwd with every "/" replaced by "-".
|
|
14
|
+
//
|
|
15
|
+
// Role mapping: each `Agent` tool_use in the main transcript carries
|
|
16
|
+
// `input.subagent_type` (= role) and `input.prompt`. Every subagent file's first
|
|
17
|
+
// user message IS that prompt, so we link file→role by prompt-prefix match
|
|
18
|
+
// (verified 20/20 on a real /new run). Unlinkable files fall back to role
|
|
19
|
+
// "unknown" with a slug-derived label — their token/time data stays accurate.
|
|
20
|
+
//
|
|
21
|
+
// Cache accounting: the four usage counters are kept SEPARATE on purpose.
|
|
22
|
+
// `cache_read` is billed ~10% of fresh input; conflating them would overstate
|
|
23
|
+
// cost and mislead "what consumed the most". A single headline `cost_units`
|
|
24
|
+
// figure weights them: output + fresh_input + cache_creation×1.25 + cache_read×0.1.
|
|
25
|
+
//
|
|
26
|
+
// Usage:
|
|
27
|
+
// node analyze-session-tokens.js --skill new --run-id batch-FEAT-0001
|
|
28
|
+
// node analyze-session-tokens.js --skill prd --run-id prd-checkout --out-dir docs/metrics
|
|
29
|
+
// # test/override (bypass env + cwd derivation):
|
|
30
|
+
// node analyze-session-tokens.js --skill new --run-id X --session <id> --project-dir <path>
|
|
31
|
+
//
|
|
32
|
+
// Output:
|
|
33
|
+
// - <out-dir>/sessions/<run_id>.md — human-readable per-role breakdown + totals
|
|
34
|
+
// - stdout: same summary, PLUS a final line `TELEMETRY_JSON=<json>` carrying the
|
|
35
|
+
// totals object for the skill to embed in docs/metrics/skill-runs.jsonl.
|
|
36
|
+
//
|
|
37
|
+
// Fail-safe: ANY error prints `stats: SKIPPED (<reason>)` and exits 0. This script
|
|
38
|
+
// must NEVER abort a /new or /prd run.
|
|
39
|
+
|
|
40
|
+
'use strict';
|
|
41
|
+
|
|
42
|
+
const fs = require('fs');
|
|
43
|
+
const os = require('os');
|
|
44
|
+
const path = require('path');
|
|
45
|
+
|
|
46
|
+
// ---- arg parsing -----------------------------------------------------------
|
|
47
|
+
|
|
48
|
+
function getArg(name, fallback) {
|
|
49
|
+
const i = process.argv.indexOf(`--${name}`);
|
|
50
|
+
return i !== -1 && process.argv[i + 1] ? process.argv[i + 1] : fallback;
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
// Bail out softly — telemetry is never allowed to break a run.
|
|
54
|
+
function skip(reason) {
|
|
55
|
+
console.log(`stats: SKIPPED (${reason})`);
|
|
56
|
+
process.exit(0);
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
const skill = getArg('skill', 'new');
|
|
60
|
+
const runId = getArg('run-id', null);
|
|
61
|
+
const outDir = getArg('out-dir', path.join('docs', 'metrics'));
|
|
62
|
+
const sessionId = getArg('session', process.env.CLAUDE_CODE_SESSION_ID || null);
|
|
63
|
+
const projectDir = getArg('project-dir', null);
|
|
64
|
+
|
|
65
|
+
if (!runId) skip('no --run-id provided');
|
|
66
|
+
if (!sessionId) skip('no session id ($CLAUDE_CODE_SESSION_ID unset, no --session)');
|
|
67
|
+
|
|
68
|
+
// ---- locate transcripts ----------------------------------------------------
|
|
69
|
+
|
|
70
|
+
function projectSlug(cwd) {
|
|
71
|
+
// Claude Code stores transcripts under a slug = cwd with "/" → "-".
|
|
72
|
+
return cwd.replace(/\//g, '-');
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
let baseDir;
|
|
76
|
+
if (projectDir) {
|
|
77
|
+
baseDir = projectDir;
|
|
78
|
+
} else {
|
|
79
|
+
baseDir = path.join(os.homedir(), '.claude', 'projects', projectSlug(process.cwd()));
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
const mainTranscript = path.join(baseDir, `${sessionId}.jsonl`);
|
|
83
|
+
const subDir = path.join(baseDir, sessionId, 'subagents');
|
|
84
|
+
|
|
85
|
+
if (!fs.existsSync(mainTranscript)) skip(`main transcript not found: ${mainTranscript}`);
|
|
86
|
+
|
|
87
|
+
// ---- jsonl helpers ---------------------------------------------------------
|
|
88
|
+
|
|
89
|
+
function* readLines(file) {
|
|
90
|
+
let raw;
|
|
91
|
+
try {
|
|
92
|
+
raw = fs.readFileSync(file, 'utf8');
|
|
93
|
+
} catch (err) {
|
|
94
|
+
return;
|
|
95
|
+
}
|
|
96
|
+
for (const line of raw.split(/\r?\n/)) {
|
|
97
|
+
if (!line.trim()) continue;
|
|
98
|
+
let obj;
|
|
99
|
+
try {
|
|
100
|
+
obj = JSON.parse(line);
|
|
101
|
+
} catch (err) {
|
|
102
|
+
continue; // tolerate partial / malformed lines
|
|
103
|
+
}
|
|
104
|
+
yield obj;
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
function contentBlocks(rec) {
|
|
109
|
+
const c = rec && rec.message && rec.message.content;
|
|
110
|
+
if (Array.isArray(c)) return c;
|
|
111
|
+
if (typeof c === 'string') return [{ type: 'text', text: c }];
|
|
112
|
+
return [];
|
|
113
|
+
}
|
|
114
|
+
|
|
115
|
+
function blocksText(blocks) {
|
|
116
|
+
return blocks
|
|
117
|
+
.map((b) => (typeof b === 'string' ? b : b && typeof b.text === 'string' ? b.text : ''))
|
|
118
|
+
.join('')
|
|
119
|
+
.trim();
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
// Aggregate the four usage counters + wall-clock span of one transcript file.
|
|
123
|
+
function aggregateFile(file) {
|
|
124
|
+
const acc = {
|
|
125
|
+
fresh_input: 0,
|
|
126
|
+
cache_read: 0,
|
|
127
|
+
cache_creation: 0,
|
|
128
|
+
output: 0,
|
|
129
|
+
turns: 0,
|
|
130
|
+
firstTs: null,
|
|
131
|
+
lastTs: null,
|
|
132
|
+
};
|
|
133
|
+
for (const rec of readLines(file)) {
|
|
134
|
+
const ts = rec.timestamp ? Date.parse(rec.timestamp) : NaN;
|
|
135
|
+
if (!Number.isNaN(ts)) {
|
|
136
|
+
if (acc.firstTs === null || ts < acc.firstTs) acc.firstTs = ts;
|
|
137
|
+
if (acc.lastTs === null || ts > acc.lastTs) acc.lastTs = ts;
|
|
138
|
+
}
|
|
139
|
+
if (rec.type !== 'assistant') continue;
|
|
140
|
+
const u = (rec.message && rec.message.usage) || null;
|
|
141
|
+
if (!u) continue;
|
|
142
|
+
acc.fresh_input += u.input_tokens || 0;
|
|
143
|
+
acc.cache_read += u.cache_read_input_tokens || 0;
|
|
144
|
+
acc.cache_creation += u.cache_creation_input_tokens || 0;
|
|
145
|
+
acc.output += u.output_tokens || 0;
|
|
146
|
+
acc.turns += 1;
|
|
147
|
+
}
|
|
148
|
+
return acc;
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
function costUnits(a) {
|
|
152
|
+
return Math.round(a.output + a.fresh_input + a.cache_creation * 1.25 + a.cache_read * 0.1);
|
|
153
|
+
}
|
|
154
|
+
|
|
155
|
+
function wallMs(a) {
|
|
156
|
+
return a.firstTs !== null && a.lastTs !== null ? a.lastTs - a.firstTs : 0;
|
|
157
|
+
}
|
|
158
|
+
|
|
159
|
+
// ---- role map from main transcript -----------------------------------------
|
|
160
|
+
|
|
161
|
+
// Collect every Agent spawn: { role, promptPrefix } in spawn order.
|
|
162
|
+
const agentSpawns = [];
|
|
163
|
+
for (const rec of readLines(mainTranscript)) {
|
|
164
|
+
for (const blk of contentBlocks(rec)) {
|
|
165
|
+
if (blk && blk.type === 'tool_use' && blk.name === 'Agent' && blk.input) {
|
|
166
|
+
const role = blk.input.subagent_type || 'unknown';
|
|
167
|
+
const prompt = (blk.input.prompt || '').trim();
|
|
168
|
+
agentSpawns.push({ role, prompt });
|
|
169
|
+
}
|
|
170
|
+
}
|
|
171
|
+
}
|
|
172
|
+
|
|
173
|
+
function matchRole(firstPrompt) {
|
|
174
|
+
if (!firstPrompt) return null;
|
|
175
|
+
for (const s of agentSpawns) {
|
|
176
|
+
if (!s.prompt) continue;
|
|
177
|
+
// bidirectional prefix match tolerates truncation on either side
|
|
178
|
+
if (firstPrompt.startsWith(s.prompt.slice(0, 50)) || s.prompt.startsWith(firstPrompt.slice(0, 50))) {
|
|
179
|
+
return s.role;
|
|
180
|
+
}
|
|
181
|
+
}
|
|
182
|
+
return null;
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
// ---- aggregate subagents + main --------------------------------------------
|
|
186
|
+
|
|
187
|
+
const byRole = new Map(); // role → { agg, agents }
|
|
188
|
+
function addToRole(role, agg) {
|
|
189
|
+
if (!byRole.has(role)) {
|
|
190
|
+
byRole.set(role, {
|
|
191
|
+
fresh_input: 0,
|
|
192
|
+
cache_read: 0,
|
|
193
|
+
cache_creation: 0,
|
|
194
|
+
output: 0,
|
|
195
|
+
turns: 0,
|
|
196
|
+
agents: 0,
|
|
197
|
+
wallSumMs: 0,
|
|
198
|
+
});
|
|
199
|
+
}
|
|
200
|
+
const r = byRole.get(role);
|
|
201
|
+
r.fresh_input += agg.fresh_input;
|
|
202
|
+
r.cache_read += agg.cache_read;
|
|
203
|
+
r.cache_creation += agg.cache_creation;
|
|
204
|
+
r.output += agg.output;
|
|
205
|
+
r.turns += agg.turns;
|
|
206
|
+
r.agents += 1;
|
|
207
|
+
r.wallSumMs += wallMs(agg);
|
|
208
|
+
}
|
|
209
|
+
|
|
210
|
+
let subFiles = [];
|
|
211
|
+
if (fs.existsSync(subDir)) {
|
|
212
|
+
try {
|
|
213
|
+
subFiles = fs
|
|
214
|
+
.readdirSync(subDir)
|
|
215
|
+
.filter((f) => f.endsWith('.jsonl'))
|
|
216
|
+
.map((f) => path.join(subDir, f));
|
|
217
|
+
} catch (err) {
|
|
218
|
+
subFiles = [];
|
|
219
|
+
}
|
|
220
|
+
}
|
|
221
|
+
|
|
222
|
+
let unmatchedFiles = 0;
|
|
223
|
+
for (const f of subFiles) {
|
|
224
|
+
const agg = aggregateFile(f);
|
|
225
|
+
if (agg.turns === 0 && wallMs(agg) === 0) continue; // empty/aborted subagent
|
|
226
|
+
// role: first user prompt of the subagent file
|
|
227
|
+
let firstPrompt = '';
|
|
228
|
+
for (const rec of readLines(f)) {
|
|
229
|
+
if (rec.type === 'user') {
|
|
230
|
+
firstPrompt = blocksText(contentBlocks(rec));
|
|
231
|
+
break;
|
|
232
|
+
}
|
|
233
|
+
}
|
|
234
|
+
const role = matchRole(firstPrompt);
|
|
235
|
+
if (!role) unmatchedFiles += 1;
|
|
236
|
+
addToRole(role || 'unknown', agg);
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
// orchestrator (main transcript) as its own row
|
|
240
|
+
const mainAgg = aggregateFile(mainTranscript);
|
|
241
|
+
addToRole('orchestrator', mainAgg);
|
|
242
|
+
|
|
243
|
+
// team-mode nesting caveat: Agent calls in main with no local subagent file
|
|
244
|
+
// (background / nested orchestrators write to a different session dir).
|
|
245
|
+
const localSubCount = subFiles.length;
|
|
246
|
+
const spawnedCount = agentSpawns.length;
|
|
247
|
+
const missingNested = Math.max(0, spawnedCount - localSubCount);
|
|
248
|
+
|
|
249
|
+
// ---- totals ----------------------------------------------------------------
|
|
250
|
+
|
|
251
|
+
const totals = {
|
|
252
|
+
fresh_input: 0,
|
|
253
|
+
cache_read: 0,
|
|
254
|
+
cache_creation: 0,
|
|
255
|
+
output: 0,
|
|
256
|
+
turns: 0,
|
|
257
|
+
agents: 0,
|
|
258
|
+
};
|
|
259
|
+
for (const r of byRole.values()) {
|
|
260
|
+
totals.fresh_input += r.fresh_input;
|
|
261
|
+
totals.cache_read += r.cache_read;
|
|
262
|
+
totals.cache_creation += r.cache_creation;
|
|
263
|
+
totals.output += r.output;
|
|
264
|
+
totals.turns += r.turns;
|
|
265
|
+
totals.agents += r.agents;
|
|
266
|
+
}
|
|
267
|
+
totals.cost_units = costUnits(totals);
|
|
268
|
+
|
|
269
|
+
// run wall-clock = span of the main transcript (first→last timestamp)
|
|
270
|
+
const runWallMs = wallMs(mainAgg);
|
|
271
|
+
|
|
272
|
+
// ---- formatting helpers -----------------------------------------------------
|
|
273
|
+
|
|
274
|
+
function fmtTokens(n) {
|
|
275
|
+
if (n >= 1e6) return `${(n / 1e6).toFixed(1)}M`;
|
|
276
|
+
if (n >= 1e3) return `${(n / 1e3).toFixed(1)}k`;
|
|
277
|
+
return String(n);
|
|
278
|
+
}
|
|
279
|
+
function fmtDuration(ms) {
|
|
280
|
+
if (!ms || ms < 0) return '-';
|
|
281
|
+
const s = Math.round(ms / 1000);
|
|
282
|
+
const m = Math.floor(s / 60);
|
|
283
|
+
const rs = s % 60;
|
|
284
|
+
return m > 0 ? `${m}m${rs.toString().padStart(2, '0')}s` : `${rs}s`;
|
|
285
|
+
}
|
|
286
|
+
function pad(s, w) {
|
|
287
|
+
s = String(s);
|
|
288
|
+
return s.length >= w ? s : s + ' '.repeat(w - s.length);
|
|
289
|
+
}
|
|
290
|
+
function padL(s, w) {
|
|
291
|
+
s = String(s);
|
|
292
|
+
return s.length >= w ? s : ' '.repeat(w - s.length) + s;
|
|
293
|
+
}
|
|
294
|
+
|
|
295
|
+
// rows sorted by cost_units desc
|
|
296
|
+
const roleRows = [...byRole.entries()]
|
|
297
|
+
.map(([role, r]) => ({ role, ...r, cost_units: costUnits(r) }))
|
|
298
|
+
.sort((a, b) => b.cost_units - a.cost_units);
|
|
299
|
+
|
|
300
|
+
// ---- write per-run markdown summary ----------------------------------------
|
|
301
|
+
|
|
302
|
+
const lines = [];
|
|
303
|
+
lines.push(`# Session telemetry — ${runId}`);
|
|
304
|
+
lines.push('');
|
|
305
|
+
lines.push(`- **skill**: ${skill}`);
|
|
306
|
+
lines.push(`- **session**: ${sessionId}`);
|
|
307
|
+
lines.push(`- **run wall-clock** (orchestrator span, includes idle): ${fmtDuration(runWallMs)}`);
|
|
308
|
+
lines.push(`- **agents**: ${totals.agents - 1} subagent(s) + 1 orchestrator`);
|
|
309
|
+
lines.push(`- **total cost units**: ${fmtTokens(totals.cost_units)} _(output + fresh_in + cache_creation×1.25 + cache_read×0.1)_`);
|
|
310
|
+
lines.push('');
|
|
311
|
+
lines.push('## By role');
|
|
312
|
+
lines.push('');
|
|
313
|
+
const H = ['role', 'agents', 'wall(sum)', 'output', 'fresh_in', 'cache_cr', 'cache_rd', 'cost_units', '% cost'];
|
|
314
|
+
const W = [20, 7, 10, 9, 9, 9, 9, 11, 7];
|
|
315
|
+
lines.push(H.map((h, i) => pad(h, W[i])).join(' | '));
|
|
316
|
+
lines.push(W.map((w) => '-'.repeat(w)).join('-|-'));
|
|
317
|
+
for (const r of roleRows) {
|
|
318
|
+
const pct = totals.cost_units ? `${((r.cost_units / totals.cost_units) * 100).toFixed(0)}%` : '-';
|
|
319
|
+
lines.push(
|
|
320
|
+
[
|
|
321
|
+
pad(r.role, W[0]),
|
|
322
|
+
padL(r.agents, W[1]),
|
|
323
|
+
padL(fmtDuration(r.wallSumMs), W[2]),
|
|
324
|
+
padL(fmtTokens(r.output), W[3]),
|
|
325
|
+
padL(fmtTokens(r.fresh_input), W[4]),
|
|
326
|
+
padL(fmtTokens(r.cache_creation), W[5]),
|
|
327
|
+
padL(fmtTokens(r.cache_read), W[6]),
|
|
328
|
+
padL(fmtTokens(r.cost_units), W[7]),
|
|
329
|
+
padL(pct, W[8]),
|
|
330
|
+
].join(' | ')
|
|
331
|
+
);
|
|
332
|
+
}
|
|
333
|
+
lines.push(W.map((w) => '-'.repeat(w)).join('-|-'));
|
|
334
|
+
lines.push(
|
|
335
|
+
[
|
|
336
|
+
pad('TOTAL', W[0]),
|
|
337
|
+
padL(totals.agents, W[1]),
|
|
338
|
+
padL(fmtDuration(runWallMs), W[2]),
|
|
339
|
+
padL(fmtTokens(totals.output), W[3]),
|
|
340
|
+
padL(fmtTokens(totals.fresh_input), W[4]),
|
|
341
|
+
padL(fmtTokens(totals.cache_creation), W[5]),
|
|
342
|
+
padL(fmtTokens(totals.cache_read), W[6]),
|
|
343
|
+
padL(fmtTokens(totals.cost_units), W[7]),
|
|
344
|
+
padL('100%', W[8]),
|
|
345
|
+
].join(' | ')
|
|
346
|
+
);
|
|
347
|
+
lines.push('');
|
|
348
|
+
lines.push('> wall(sum) is the sum of per-agent spans and reflects ACTIVE compute time; the');
|
|
349
|
+
lines.push('> run wall-clock is the orchestrator first→last span and INCLUDES human idle gaps.');
|
|
350
|
+
lines.push('> Subagents may run in parallel, so wall(sum) can exceed the run wall-clock.');
|
|
351
|
+
lines.push('> cache_read tokens are ~10% the cost of fresh input — kept separate on purpose.');
|
|
352
|
+
if (unmatchedFiles > 0) {
|
|
353
|
+
lines.push('>');
|
|
354
|
+
lines.push(`> ⚠ ${unmatchedFiles} subagent file(s) could not be mapped to a role (bucketed as "unknown").`);
|
|
355
|
+
}
|
|
356
|
+
if (missingNested > 0) {
|
|
357
|
+
lines.push('>');
|
|
358
|
+
lines.push(
|
|
359
|
+
`> ⚠ team-mode caveat: ${missingNested} Agent spawn(s) in the main transcript have no local`
|
|
360
|
+
);
|
|
361
|
+
lines.push(
|
|
362
|
+
`> subagent file (likely background / nested orchestrators in separate sessions) — their`
|
|
363
|
+
);
|
|
364
|
+
lines.push(`> tokens are NOT included in these totals. Sequential-mode runs are fully accurate.`);
|
|
365
|
+
}
|
|
366
|
+
lines.push('');
|
|
367
|
+
|
|
368
|
+
// ---- emit ------------------------------------------------------------------
|
|
369
|
+
|
|
370
|
+
const sessionsDir = path.join(outDir, 'sessions');
|
|
371
|
+
try {
|
|
372
|
+
fs.mkdirSync(sessionsDir, { recursive: true });
|
|
373
|
+
fs.writeFileSync(path.join(sessionsDir, `${runId}.md`), lines.join('\n'));
|
|
374
|
+
} catch (err) {
|
|
375
|
+
skip(`could not write summary: ${err.message}`);
|
|
376
|
+
}
|
|
377
|
+
|
|
378
|
+
// human summary to stdout
|
|
379
|
+
console.log(lines.join('\n'));
|
|
380
|
+
|
|
381
|
+
// machine-readable totals for the skill to embed in skill-runs.jsonl
|
|
382
|
+
const jsonOut = {
|
|
383
|
+
run_wall_ms: runWallMs,
|
|
384
|
+
by_role: Object.fromEntries(
|
|
385
|
+
roleRows.map((r) => [
|
|
386
|
+
r.role,
|
|
387
|
+
{
|
|
388
|
+
agents: r.agents,
|
|
389
|
+
output: r.output,
|
|
390
|
+
fresh_input: r.fresh_input,
|
|
391
|
+
cache_creation: r.cache_creation,
|
|
392
|
+
cache_read: r.cache_read,
|
|
393
|
+
cost_units: r.cost_units,
|
|
394
|
+
wall_sum_ms: r.wallSumMs,
|
|
395
|
+
},
|
|
396
|
+
])
|
|
397
|
+
),
|
|
398
|
+
totals: {
|
|
399
|
+
agents: totals.agents,
|
|
400
|
+
output: totals.output,
|
|
401
|
+
fresh_input: totals.fresh_input,
|
|
402
|
+
cache_creation: totals.cache_creation,
|
|
403
|
+
cache_read: totals.cache_read,
|
|
404
|
+
cost_units: totals.cost_units,
|
|
405
|
+
turns: totals.turns,
|
|
406
|
+
},
|
|
407
|
+
unmatched_subagents: unmatchedFiles,
|
|
408
|
+
nested_agents_excluded: missingNested,
|
|
409
|
+
};
|
|
410
|
+
console.log(`TELEMETRY_JSON=${JSON.stringify(jsonOut)}`);
|
|
411
|
+
process.exit(0);
|