baldart 3.28.1 → 3.28.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,51 @@ All notable changes to BALDART will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [3.28.3] - 2026-05-28
|
|
9
|
+
|
|
10
|
+
Release B of the 3-release plan on `/new` fix-application authority. Closes two inline-apply violation patterns observed in real `/new` runs (doc fixes applied by the orchestrator instead of delegated to coder; Codex HIGH security findings written inline) and shuts the structural loophole in `SKILL.md`'s sub-agent failure protocol that was being used to bypass delegation even when no agent had crashed. Telemetry from v3.28.2 will measure the effectiveness of these changes; Release C (conditional threshold rule) waits for that data.
|
|
11
|
+
|
|
12
|
+
### Added — Domain-Override Domains sub-section
|
|
13
|
+
|
|
14
|
+
- **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)**: new sub-section under "Fix Application Log Schema" enumerates tassativamente the domains where inline orchestrator apply is **never** safe, regardless of patch size: `doc` (any `*.md` under references/, prd/, root CHANGELOG, ssot-registry), `security` (Phase 3.7 detector Triggers #2/#3 + SQL RLS policy mutations), `migration` (`supabase/migrations/*.sql` or `${paths.migrations_dir}`). The coder agent's system prompt + project overlay enforces invariants (freshness, tabular formatting, RLS structure) that orchestrator inline edits routinely break. Edge case explicit: mechanical CHANGELOG / ssot-registry append-a-row stays `doc` (uniformity > spawn cost).
|
|
15
|
+
|
|
16
|
+
### Changed — Phase 3 doc delegation reinforced
|
|
17
|
+
|
|
18
|
+
- **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)** Phase 3 step 15: explicit chiosa "Doc fixes are NEVER applied inline by the orchestrator, regardless of size or perceived triviality" added below the existing coder-spawn instruction. The rule already existed (line 1116, pre-v3.28.2: "invoke the coder agent once") but was being bypassed in practice — the chiosa makes the rule non-discretionary.
|
|
19
|
+
|
|
20
|
+
### Changed — Phase 3.7 coder applies Codex patches verbatim
|
|
21
|
+
|
|
22
|
+
- **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)** Phase 3.7 step 4 sub-bullet "spawn coder": coder now receives the report path + list of VERIFIED bugs **plus the patches Codex suggested inline in the report**, and applies them verbatim. The coder does NOT re-do the analysis, does NOT re-grep, does NOT re-read files extensively. Codex has already produced both the diagnosis and the patch; the coder's value-add is the right system prompt (project conventions, naming, testing patterns), not redoing security/correctness review. Saves an estimated 5-10k tokens per Codex-driven coder spawn.
|
|
23
|
+
|
|
24
|
+
### Changed — Sub-agent failure protocol (loophole closed)
|
|
25
|
+
|
|
26
|
+
- **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)** "Sub-agent failure protocol" section: replaced single bullet "attempt the work yourself directly" with a 4-step protocol. (1) log failure with full trace, (2) retry once (transient errors are common), (3) **if domain-override (doc / security / migration) and retry still fails: STOP and AskUserQuestion — never inline-fallback for these domains**, (4) non-override domains may fall back inline but MUST log `applied_by=orchestrator-fallback` for telemetry visibility. Razionale: the pre-v3.28.3 wording was being used as a shortcut to bypass delegation rules even when no agent had crashed; closing that defeats the violation pattern at the structural level.
|
|
27
|
+
|
|
28
|
+
### Rationale
|
|
29
|
+
|
|
30
|
+
Release B addresses 4 of the 17 critiques raised in the adversarial review of the original threshold-rule plan (#4-5 domain override definition, #7 Codex re-analysis ridondante, #10 escape valve aperta). The remaining critiques are either addressed by Release A telemetry, deferred to conditional Release C, or accepted as design trade-offs documented in the plan file. **No new `baldart.config.yml` keys** — domain-override is enforced by content/path matching, not by config flag (KISS).
|
|
31
|
+
|
|
32
|
+
## [3.28.2] - 2026-05-28
|
|
33
|
+
|
|
34
|
+
Telemetry-only release. Adds structured logging for fix-application decisions in `/new` Phases 2.55 / 3 / 3.5 / 3.7. **Zero behavior change** — the orchestrator's decisions remain identical to v3.28.1. The goal is to measure, with real data, whether the per-phase delegation rules need to change in a future release.
|
|
35
|
+
|
|
36
|
+
### Added — `## Fix Application Log` section in tracker
|
|
37
|
+
|
|
38
|
+
- **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)**: new section **"Fix Application Log Schema (telemetry)"** inserted before Phase 2.55. Defines a pipe-separated row format (`<phase> | <domain> | est_lines=<n> | decision=<...> | applied_by=<...> | <key=val>...`) appended to the tracker's `## Fix Application Log` section. Phases 2.55 / 3 / 3.5 / 3.7 each instructed to log one row per finding processed (including skipped). Cost: ~1-2 KB per card, negligible context impact.
|
|
39
|
+
- Per-phase logging instructions added to Phase 2.55 step 4 (simplify reuse/quality/efficiency), Phase 3 step 15 (doc), Phase 3.5 step 23 (qa-sentinel blockers), Phase 3.7 step 4 (codex BLOCKER/HIGH + false-positive-filtered). Each phase classifies finding `domain` consistently with the schema enum.
|
|
40
|
+
|
|
41
|
+
### Added — Aggregator script
|
|
42
|
+
|
|
43
|
+
- **[framework/scripts/analyze-fix-application.js](framework/scripts/analyze-fix-application.js)**: zero-dependency Node script. Reads N tracker files (shell-expanded glob), parses rows in the `## Fix Application Log` section, aggregates by `phase × domain × decision × applied_by`, outputs a tabular summary plus an automatic violation footer flagging inline-apply on doc / security / migration domains. Usage: `node framework/scripts/analyze-fix-application.js docs/references/trackers/*.md`.
|
|
44
|
+
|
|
45
|
+
### Added — Telemetry reading guide
|
|
46
|
+
|
|
47
|
+
- **[framework/docs/FIX-APPLICATION-TELEMETRY.md](framework/docs/FIX-APPLICATION-TELEMETRY.md)**: new doc. Explains why telemetry exists, the row schema (cross-references SKILL.md as SSOT), how to read the analyzer output, violation patterns vs delegation candidates vs healthy patterns, and the trigger conditions for the future Release C decision (threshold rule vs Phase 2.55→3 merge).
|
|
48
|
+
|
|
49
|
+
### Rationale
|
|
50
|
+
|
|
51
|
+
This release is the first of a 3-release sequence (A: telemetry → B: loophole + 2 known violations → C: conditional threshold rule). The decision to instrument *before* changing rules avoids inventing arbitrary thresholds without data — see plan file at `/Users/antoniobaldassarre/.claude/plans/sto-notando-una-cosa-misty-lightning.md` for full reasoning. Schema-change propagation rule: **no new `baldart.config.yml` keys** in this release.
|
|
52
|
+
|
|
8
53
|
## [3.28.1] - 2026-05-28
|
|
9
54
|
|
|
10
55
|
Patch su tre drift segnalati dal curator dopo v3.28.0 install in `mayo`:
|
package/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
3.28.
|
|
1
|
+
3.28.3
|
|
@@ -938,6 +938,45 @@ If at any point in Phases 2 / 2.5 / 2.5b you detect context-window pressure (you
|
|
|
938
938
|
|
|
939
939
|
Auto Mode does NOT override this rule. "Bias toward proceeding" applies to routine decisions ("should I rerun this check?", "should I confirm this safe rename?"), NEVER to scope reduction.
|
|
940
940
|
|
|
941
|
+
### Fix Application Log Schema (telemetry)
|
|
942
|
+
|
|
943
|
+
Every fix-application decision in Phases 2.55 / 3 / 3.5 / 3.7 appends one row to the tracker's `## Fix Application Log` section. Telemetry only — does NOT alter behaviour. The aggregated log feeds `framework/scripts/analyze-fix-application.js`, which surfaces which fixes the orchestrator applied inline vs delegated to coder, per-phase distribution of `est_lines`, and violation patterns (doc/security findings applied inline). Full reading guide: `framework/docs/FIX-APPLICATION-TELEMETRY.md`.
|
|
944
|
+
|
|
945
|
+
**Row format** (one row per finding processed, including skipped):
|
|
946
|
+
|
|
947
|
+
```
|
|
948
|
+
<phase> | <domain> | est_lines=<n> | decision=<inline|coder|coder-batch|skipped> | applied_by=<orchestrator|coder|orchestrator-fallback|-> | <key=val>...
|
|
949
|
+
```
|
|
950
|
+
|
|
951
|
+
| Field | Values |
|
|
952
|
+
|---|---|
|
|
953
|
+
| `phase` | `2.55`, `3`, `3.5`, `3.7` |
|
|
954
|
+
| `domain` | `simplify-reuse`, `simplify-quality`, `simplify-efficiency`, `doc`, `qa-blocker`, `qa-major`, `qa-minor`, `qa-none`, `codex-security`, `codex-correctness`, `codex-other` |
|
|
955
|
+
| `est_lines` | Orchestrator's at-a-glance estimate of the diff size suggested by the finding. Bucketed `1-5` / `6-20` / `21+` is sufficient — exact precision not required. Use `0` if no patch (e.g. skipped). |
|
|
956
|
+
| `decision` | `inline` (orchestrator applied), `coder` (single coder spawn for this finding), `coder-batch` (folded into a batched coder spawn covering ≥2 findings), `skipped` (false positive / not worth addressing / no findings) |
|
|
957
|
+
| `applied_by` | `orchestrator`, `coder`, `orchestrator-fallback` (agent crash with non-domain-override fallback — see Sub-agent failure protocol), `-` (skipped) |
|
|
958
|
+
| Trailing `key=val` | Phase-specific extras: `finding=<1-line summary>`, `severity=<BLOCKER\|HIGH\|MEDIUM\|FALSE-POSITIVE>`, `retry=<n>`, `reason=<...>` |
|
|
959
|
+
|
|
960
|
+
**Write target**: the same tracker file the orchestrator already writes per-card status to (path resolved from `${paths.references_dir}/trackers/` or the project's tracker convention). Section header `## Fix Application Log` is created on first row of the card. Append-only — never rewrite previous rows.
|
|
961
|
+
|
|
962
|
+
**Cost**: ~80-120 bytes per row, ~5-15 rows per card. Negligible context impact, large analytical value.
|
|
963
|
+
|
|
964
|
+
#### Domain-Override Domains (since v3.28.3)
|
|
965
|
+
|
|
966
|
+
Some fix domains are **never** safe for inline orchestrator apply, regardless of size. The coder agent's system prompt + project overlay enforces invariants (doc freshness, tabular formatting, RLS policy structure, migration ordering) that orchestrator inline edits routinely break. The orchestrator MUST delegate every fix in these domains to `coder`, even when the patch is a one-liner.
|
|
967
|
+
|
|
968
|
+
Enumerated tassativamente:
|
|
969
|
+
|
|
970
|
+
| Domain | Match rule |
|
|
971
|
+
|---|---|
|
|
972
|
+
| `doc` | File path matching `*.md` under `${paths.references_dir}`, `${paths.prd_dir}`, project root `CHANGELOG.md`, or any `ssot-registry.md`. |
|
|
973
|
+
| `security` | File path matching the Phase 3.7 detector Triggers #2 (auth/permissions: `src/lib/auth/middleware.ts`, `src/lib/permissions.ts`, anything matching `withAuth`) or #3 (payments: `^src/lib/payments/`, `^src/app/api/v1/billing/`). Also any SQL migration whose content matches `CREATE POLICY|ALTER POLICY|DROP POLICY` (RLS policy mutations). |
|
|
974
|
+
| `migration` | File path matching `supabase/migrations/*.sql` (or `${paths.migrations_dir}/*.sql` if defined in `baldart.config.yml`). |
|
|
975
|
+
|
|
976
|
+
**Edge case explicit** — a mechanical append-a-row update to `CHANGELOG.md` or `ssot-registry.md` is still classified `doc` and still goes through coder. The uniformity of the rule matters more than the cost of the individual spawn. If telemetry from Release A shows that doc-mechanical-append is a large fraction of doc coder spawns, a sub-domain `doc-mechanical-append` may be carved out in a later release.
|
|
977
|
+
|
|
978
|
+
Domains NOT listed here remain governed by the per-phase rules of the corresponding phase (e.g. `simplify-*` follows Phase 2.55 inline rule).
|
|
979
|
+
|
|
941
980
|
### Phase 2.55 — Simplify (code cleanup before review)
|
|
942
981
|
|
|
943
982
|
After completeness is verified, clean up the implementation before it reaches reviewers and E2E tests. This phase launches three parallel agents on the card's diff, then applies fixes directly.
|
|
@@ -954,7 +993,9 @@ After completeness is verified, clean up the implementation before it reaches re
|
|
|
954
993
|
|
|
955
994
|
4. Aggregate findings from all three agents. For each finding:
|
|
956
995
|
- **Valid** → fix directly (no coder agent spawn — apply edits inline).
|
|
957
|
-
- **False positive / not worth addressing** → skip silently, do not
|
|
996
|
+
- **False positive / not worth addressing** → skip silently, do not surface in user output.
|
|
997
|
+
|
|
998
|
+
**Telemetry (Fix Application Log)** — for EVERY finding (valid OR skipped) append one row to the tracker's `## Fix Application Log` section per the schema above. Use `domain=simplify-{reuse|quality|efficiency}` matching the originating agent. Valid: `decision=inline | applied_by=orchestrator | est_lines=<n> | finding=<1-line>`. Skipped: `decision=skipped | applied_by=- | est_lines=0 | reason=<false-positive|not-worth-addressing>`.
|
|
958
999
|
|
|
959
1000
|
5. After all fixes, run `npm run lint` and `npx tsc --noEmit` to confirm nothing broke.
|
|
960
1001
|
If either fails, fix the regression (up to 2 retries).
|
|
@@ -1114,6 +1155,10 @@ skill's Phase 1 falls back to deriving Gherkin scenarios from
|
|
|
1114
1155
|
Doc-reviewer collects findings WITHOUT making changes.
|
|
1115
1156
|
14. **Obsidian Corpus Dispatch**: Parse section H from doc-reviewer findings. If `Trigger: YES`, dispatch the `obsidian-sync` agent (`.claude/agents/obsidian-sync.md`) with the listed paths after all fixes are applied (step 16). If `Trigger: NO`, skip. This is non-blocking -- do not wait for obsidian-sync to complete before proceeding.
|
|
1116
1157
|
15. If doc findings exist, invoke the **coder** agent once to apply **ALL doc fixes in one pass**.
|
|
1158
|
+
|
|
1159
|
+
**Doc fixes are NEVER applied inline by the orchestrator**, regardless of size or perceived triviality (since v3.28.3). `doc` is a domain-override domain — see "Domain-Override Domains" sub-section. A one-line `CHANGELOG.md` append and a 50-line `data-model.md` migration row update both go through coder. The freshness invariants and tabular formatting that the coder's system prompt enforces are routinely broken by orchestrator inline edits; the cost of a coder spawn is the cost of correctness.
|
|
1160
|
+
|
|
1161
|
+
**Telemetry** — after the coder returns, append one row per doc finding to `## Fix Application Log`: `3 | doc | est_lines=<n> | decision=coder-batch | applied_by=coder | finding=<1-line>`. If 0 findings, append one row: `3 | doc | est_lines=0 | decision=skipped | applied_by=- | reason=no-findings`.
|
|
1117
1162
|
16. Run `npm run lint` and `npx tsc --noEmit` to verify nothing broke. If any check fails, apply the self-healing retry loop (up to 3 times, no user prompt).
|
|
1118
1163
|
17. **Update tracker**: phase = "3-doc-review DONE", log doc findings count, fixes applied.
|
|
1119
1164
|
If doc-reviewer found a recurring gap, append 1-line to `## Lessons Learned`:
|
|
@@ -1154,6 +1199,7 @@ skill's Phase 1 falls back to deriving Gherkin scenarios from
|
|
|
1154
1199
|
- After coder fixes, re-invoke `qa-sentinel` in the same mode to re-validate. Repeat up to **2 times**.
|
|
1155
1200
|
- If still FAIL after 2 retries: log in `## Issues & Flags` and **ask the user** whether to proceed or stop.
|
|
1156
1201
|
- The commit in Phase 4 MUST NOT happen until QA verdict is PASS (or user explicitly overrides).
|
|
1202
|
+
- **Telemetry** — after each coder spawn, append one row per blocker to `## Fix Application Log`: `3.5 | qa-blocker | est_lines=<n> | decision=coder | applied_by=coder | retry=<0|1|2>`. After PASS without any spawn (initial verdict PASS), append one row: `3.5 | qa-none | est_lines=0 | decision=skipped | applied_by=- | reason=qa-pass`.
|
|
1157
1203
|
24. **Update tracker**: phase = "3.5-qa DONE", log: profile used, verdict (PASS/FAIL/SKIP), confidence %, findings count (blockers/majors/minors), findings file path.
|
|
1158
1204
|
|
|
1159
1205
|
### Phase 3.7 — Pre-Merge Codex Review Gate (MANDATORY — UNCONDITIONAL)
|
|
@@ -1246,8 +1292,9 @@ For EVERY card (no conditional skip):
|
|
|
1246
1292
|
|
|
1247
1293
|
4. **Apply fix sub-loop** (mirror of Phase 3.5 retry pattern):
|
|
1248
1294
|
- If 0 BLOCKER and 0 HIGH → log `verdict: PASS — proceeding to Phase 4` in tracker. Done.
|
|
1249
|
-
- If 1+ BLOCKER OR 1+ HIGH → spawn `coder` agent with the report path + list of VERIFIED bugs to
|
|
1295
|
+
- If 1+ BLOCKER OR 1+ HIGH → spawn `coder` agent with the report path + list of VERIFIED bugs **+ the patch(es) suggested by Codex inline in the report**. The coder applies the suggested patches verbatim — **does NOT re-do the analysis, does NOT re-grep, does NOT re-read files extensively** (since v3.28.3). Codex has already produced both the diagnosis and the patch; the coder is here to apply edits with the right system prompt (project conventions, naming, testing patterns), not to redo Codex's work. This saves 5-10k tokens of redundant security/correctness review per spawn. After coder fixes, re-invoke `/codexreview <CARD-ID>` to re-validate. Repeat **max 2 times**.
|
|
1250
1296
|
- If still BLOCKER/HIGH after 2 retries → log in `## Issues & Flags` and **ask the user** whether to proceed, escalate, or stop. The Phase 4 commit MUST NOT happen until High-Risk Gate verdict is PASS or user explicitly overrides.
|
|
1297
|
+
- **Telemetry** — for EVERY codex finding processed (verified BLOCKER, verified HIGH, or false-positive-filtered), append one row to `## Fix Application Log`: `3.7 | codex-<security|correctness|other> | est_lines=<n> | decision=<coder|skipped> | applied_by=<coder|-> | severity=<BLOCKER|HIGH|FALSE-POSITIVE> | retry=<n>`. Classify domain: `security` for findings touching RLS / auth / permissions / payments; `correctness` for logic / data integrity / race conditions; `other` for everything else.
|
|
1251
1298
|
|
|
1252
1299
|
5. **Update tracker**: phase = `3.7-highrisk DONE`, log final verdict, retry count, list of fixed findings, and the report path.
|
|
1253
1300
|
|
|
@@ -1285,9 +1332,16 @@ The detector (Step A) is bash + grep — guaranteed to run, no LLM skip. The dow
|
|
|
1285
1332
|
e. Stage BOTH the updated YAML AND ssot-registry.md, then commit (or as an immediate follow-up commit if Phase 4 commit already happened).
|
|
1286
1333
|
28. **Update tracker**: move card to `## Completed Cards` with commit hash, summary, flags, **and `card_status: DONE (verified)`**.
|
|
1287
1334
|
|
|
1288
|
-
### Sub-agent failure protocol
|
|
1289
|
-
|
|
1290
|
-
-
|
|
1335
|
+
### Sub-agent failure protocol (since v3.28.3)
|
|
1336
|
+
|
|
1337
|
+
If any sub-agent **crashes or errors** during any phase, follow this 4-step protocol. The pre-v3.28.3 wording ("attempt the work yourself directly") was being used as a shortcut to bypass delegation rules even when no agent had crashed — that escape valve is closed.
|
|
1338
|
+
|
|
1339
|
+
1. **Log the failure** in the tracker with the full error trace (under `## Issues & Flags`, prefix `[AGENT-CRASH]`).
|
|
1340
|
+
2. **Retry the same agent once**. Transient errors (network, rate-limit, race) are common; a single retry resolves most.
|
|
1341
|
+
3. **If retry fails AND the fix is in a domain-override domain** (`doc`, `security`, `migration` — see "Domain-Override Domains" sub-section): **STOP the pipeline**, log in `## Issues & Flags`, and invoke `AskUserQuestion` with options (a) skip the finding, (b) hand off to user, (c) escalate to a different agent type. **NEVER** fall through to inline orchestrator apply for these domains. The whole point of delegation here is that the orchestrator does not have the right system prompt — bypassing on crash defeats the rule.
|
|
1342
|
+
4. **If retry fails AND the fix is non-domain-override**: orchestrator MAY apply inline, but MUST log a Fix Application Log row with `applied_by=orchestrator-fallback` + `reason=<agent>-crash-retry-failed` so the telemetry surfaces frequency. If this happens often for the same agent, that agent's reliability is the problem, not the protocol.
|
|
1343
|
+
|
|
1344
|
+
Never block the pipeline indefinitely — recover and continue per the rules above. AskUserQuestion in step 3 is bounded by normal user response time.
|
|
1291
1345
|
|
|
1292
1346
|
### Phase 5 — Context Clean & Continue
|
|
1293
1347
|
29. Archive the card from Active Code Context in `${paths.references_dir}/project-status.md`.
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# Fix Application Telemetry
|
|
2
|
+
|
|
3
|
+
This doc describes the telemetry layer added to `/new` (Phases 2.55 / 3 / 3.5 / 3.7) that records, for every check finding, who applied the fix (orchestrator inline vs `coder` subagent) and how big the diff was. The layer is **observation only** — it does not change `/new`'s behaviour. It exists so we can decide, with real data, whether the per-phase delegation rules need to change.
|
|
4
|
+
|
|
5
|
+
## Why this exists
|
|
6
|
+
|
|
7
|
+
Empirical observation showed that fixes flagged by the simplify / doc-reviewer / qa-sentinel / codex agents often get applied **directly by the orchestrator** instead of being delegated to the `coder` agent. The cost is twofold: (1) the orchestrator does not inherit the coder's system prompt + project overlay (testing conventions, naming, project rules), and (2) inlining 5+ agent outputs saturates the orchestrator's context window mid-card.
|
|
8
|
+
|
|
9
|
+
Before changing the rules — which would mean inventing thresholds, domain overrides, and escape valves — we measure the current state. The schema below is the contract.
|
|
10
|
+
|
|
11
|
+
## Schema (SSOT)
|
|
12
|
+
|
|
13
|
+
Authoritative definition lives in `framework/.claude/skills/new/SKILL.md` → section **"Fix Application Log Schema (telemetry)"**. The row format:
|
|
14
|
+
|
|
15
|
+
```
|
|
16
|
+
<phase> | <domain> | est_lines=<n> | decision=<inline|coder|coder-batch|skipped> | applied_by=<orchestrator|coder|orchestrator-fallback|-> | <key=val>...
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Each row is appended to the tracker's `## Fix Application Log` section. One row per finding, including findings the orchestrator skipped as false-positive. The trailing `key=val` pairs are phase-specific extras (`finding=<1-line>`, `severity=<...>`, `retry=<n>`, `reason=<...>`).
|
|
20
|
+
|
|
21
|
+
## How to read the data
|
|
22
|
+
|
|
23
|
+
Run the aggregator:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
node framework/scripts/analyze-fix-application.js docs/references/trackers/*.md
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
(Substitute the actual tracker directory if your project uses a different `${paths.references_dir}`.)
|
|
30
|
+
|
|
31
|
+
Output is a tabular summary grouped by `phase × domain × decision × applied_by`, plus a violation-pattern footer.
|
|
32
|
+
|
|
33
|
+
### Patterns to look for
|
|
34
|
+
|
|
35
|
+
**Violation patterns** — these should be zero. If they're non-zero, the orchestrator is bypassing a delegation rule:
|
|
36
|
+
|
|
37
|
+
| Pattern | What it means |
|
|
38
|
+
|---|---|
|
|
39
|
+
| `3 \| doc \| ... \| applied_by=orchestrator` | Phase 3 doc-review fix applied inline (violates `SKILL.md:1116` "invoke the coder agent once"). |
|
|
40
|
+
| `3.7 \| codex-security \| ... \| applied_by=orchestrator` | Codex HIGH security finding fixed inline (violates `SKILL.md:1249` "spawn `coder` agent"). |
|
|
41
|
+
| `* \| * \| ... \| applied_by=orchestrator-fallback` | Sub-agent failed and orchestrator fell back to inline. Expected to be rare; investigate if frequent. |
|
|
42
|
+
|
|
43
|
+
The analyzer script flags these in its footer automatically.
|
|
44
|
+
|
|
45
|
+
**Delegation candidates** — patterns suggesting the rule could be tightened:
|
|
46
|
+
|
|
47
|
+
| Pattern | What it suggests |
|
|
48
|
+
|---|---|
|
|
49
|
+
| `2.55 \| simplify-* \| est_lines≥6 \| decision=inline \| applied_by=orchestrator` (high count) | Large simplify refactors applied inline. Candidate for threshold-based delegation. |
|
|
50
|
+
| `2.55 \| simplify-* \| ... \| applied_by=orchestrator` total lines > 50/card avg | Cumulative inline diff bloating orchestrator context. Candidate for `coder-batch`. |
|
|
51
|
+
| `3.5 \| qa-blocker \| retry=2` (many rows) | Coder not converging on QA fixes in 2 retries. Candidate for revised qa-sentinel feedback loop. |
|
|
52
|
+
|
|
53
|
+
**Healthy patterns**:
|
|
54
|
+
|
|
55
|
+
| Pattern | What it means |
|
|
56
|
+
|---|---|
|
|
57
|
+
| `2.55 \| simplify-* \| est_lines∈[1-5] \| decision=inline \| applied_by=orchestrator` | Small inline fixes — efficient, low cost. Expected majority of Phase 2.55. |
|
|
58
|
+
| `3 \| doc \| ... \| decision=coder-batch \| applied_by=coder` | Doc fixes batched into one coder spawn. Expected for every card with doc impact. |
|
|
59
|
+
| `3.7 \| * \| decision=skipped \| applied_by=- \| severity=FALSE-POSITIVE` | Codex false positives correctly filtered. |
|
|
60
|
+
|
|
61
|
+
## When to act on the data
|
|
62
|
+
|
|
63
|
+
After roughly **10 cards** of telemetry, run the analyzer and look at:
|
|
64
|
+
|
|
65
|
+
1. **Violation count**. If non-zero, the rules are being bypassed — Release B (loophole fix + Phase 3 reinforcement) should already have made these zero.
|
|
66
|
+
2. **Phase 2.55 inline distribution**. If `est_lines≥6 | applied_by=orchestrator` exceeds 30% of Phase 2.55 rows, Release C (threshold rule) becomes worth considering.
|
|
67
|
+
3. **Mean inline diff per card**. Compute `(sum of est_lines for orchestrator rows) / (number of cards)`. > 50 indicates context bloat.
|
|
68
|
+
|
|
69
|
+
The full decision tree for Release C lives in the plan file (`/Users/antoniobaldassarre/.claude/plans/sto-notando-una-cosa-misty-lightning.md` → "Release C — Threshold Rule (CONDIZIONALE)").
|
|
70
|
+
|
|
71
|
+
## Cost
|
|
72
|
+
|
|
73
|
+
Per-card overhead: 5-15 rows × ~80-120 bytes = ~1-2 KB added to the tracker. Orchestrator context impact: negligible (tracker is appended to disk, not held in context after each phase). Analysis cost: a single Node script invocation. The aggregator has zero dependencies and runs in well under a second on hundreds of trackers.
|
|
74
|
+
|
|
75
|
+
## Pointers
|
|
76
|
+
|
|
77
|
+
- Schema definition (canonical): `framework/.claude/skills/new/SKILL.md` → "Fix Application Log Schema (telemetry)"
|
|
78
|
+
- Per-phase logging instructions: SKILL.md Phase 2.55 step 4, Phase 3 step 15, Phase 3.5 step 23, Phase 3.7 step 4
|
|
79
|
+
- Aggregator script: `framework/scripts/analyze-fix-application.js`
|
|
80
|
+
- Decision plan: `/Users/antoniobaldassarre/.claude/plans/sto-notando-una-cosa-misty-lightning.md`
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Read N tracker files passed as args (shell-expanded glob), extract rows from
|
|
3
|
+
// each file's "## Fix Application Log" section, aggregate by
|
|
4
|
+
// phase × domain × decision × applied_by, print a summary table.
|
|
5
|
+
//
|
|
6
|
+
// Usage:
|
|
7
|
+
// node framework/scripts/analyze-fix-application.js docs/references/trackers/*.md
|
|
8
|
+
// node framework/scripts/analyze-fix-application.js path/to/tracker1.md path/to/tracker2.md
|
|
9
|
+
//
|
|
10
|
+
// Schema (must match framework/.claude/skills/new/SKILL.md "Fix Application Log Schema"):
|
|
11
|
+
// <phase> | <domain> | est_lines=<n> | decision=<...> | applied_by=<...> | <key=val>...
|
|
12
|
+
|
|
13
|
+
'use strict';
|
|
14
|
+
const fs = require('fs');
|
|
15
|
+
const path = require('path');
|
|
16
|
+
|
|
17
|
+
const args = process.argv.slice(2).filter((a) => !a.startsWith('-'));
|
|
18
|
+
if (args.length === 0) {
|
|
19
|
+
console.error('Usage: analyze-fix-application.js <tracker-file>...');
|
|
20
|
+
console.error('Pass shell-expanded glob, e.g. docs/references/trackers/*.md');
|
|
21
|
+
process.exit(2);
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
const SECTION_RE = /^##\s+Fix Application Log\s*$/i;
|
|
25
|
+
const ROW_RE = /^([\w.]+)\s*\|\s*([\w-]+)\s*\|\s*est_lines=(\d+)\s*\|\s*decision=([\w-]+)\s*\|\s*applied_by=([\w-]+)(?:\s*\|\s*(.*))?$/;
|
|
26
|
+
|
|
27
|
+
const buckets = new Map(); // key = "phase|domain|decision|applied_by" → { count, est_sum, est_n }
|
|
28
|
+
let totalRows = 0;
|
|
29
|
+
let filesParsed = 0;
|
|
30
|
+
let filesSkippedNoSection = 0;
|
|
31
|
+
|
|
32
|
+
for (const arg of args) {
|
|
33
|
+
let content;
|
|
34
|
+
try {
|
|
35
|
+
content = fs.readFileSync(arg, 'utf8');
|
|
36
|
+
} catch (err) {
|
|
37
|
+
console.error(`skip ${arg}: ${err.message}`);
|
|
38
|
+
continue;
|
|
39
|
+
}
|
|
40
|
+
filesParsed += 1;
|
|
41
|
+
|
|
42
|
+
const lines = content.split(/\r?\n/);
|
|
43
|
+
let inSection = false;
|
|
44
|
+
let sawSection = false;
|
|
45
|
+
for (const line of lines) {
|
|
46
|
+
if (SECTION_RE.test(line)) {
|
|
47
|
+
inSection = true;
|
|
48
|
+
sawSection = true;
|
|
49
|
+
continue;
|
|
50
|
+
}
|
|
51
|
+
if (inSection && /^##\s+\S/.test(line)) {
|
|
52
|
+
// next H2 section starts — stop
|
|
53
|
+
inSection = false;
|
|
54
|
+
continue;
|
|
55
|
+
}
|
|
56
|
+
if (!inSection) continue;
|
|
57
|
+
const m = line.match(ROW_RE);
|
|
58
|
+
if (!m) continue;
|
|
59
|
+
const [, phase, domain, estLinesStr, decision, appliedBy] = m;
|
|
60
|
+
const key = `${phase}|${domain}|${decision}|${appliedBy}`;
|
|
61
|
+
const est = parseInt(estLinesStr, 10);
|
|
62
|
+
const bucket = buckets.get(key) || { count: 0, estSum: 0, estN: 0 };
|
|
63
|
+
bucket.count += 1;
|
|
64
|
+
if (est > 0) {
|
|
65
|
+
bucket.estSum += est;
|
|
66
|
+
bucket.estN += 1;
|
|
67
|
+
}
|
|
68
|
+
buckets.set(key, bucket);
|
|
69
|
+
totalRows += 1;
|
|
70
|
+
}
|
|
71
|
+
if (!sawSection) filesSkippedNoSection += 1;
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
const rows = [];
|
|
75
|
+
for (const [key, b] of buckets) {
|
|
76
|
+
const [phase, domain, decision, appliedBy] = key.split('|');
|
|
77
|
+
const avgLines = b.estN > 0 ? (b.estSum / b.estN).toFixed(1) : '-';
|
|
78
|
+
const pct = totalRows > 0 ? ((b.count / totalRows) * 100).toFixed(1) + '%' : '-';
|
|
79
|
+
rows.push({ phase, domain, decision, appliedBy, count: b.count, avgLines, pct });
|
|
80
|
+
}
|
|
81
|
+
rows.sort((a, b) => a.phase.localeCompare(b.phase) || b.count - a.count);
|
|
82
|
+
|
|
83
|
+
const headers = ['Phase', 'Domain', 'Decision', 'Applied-by', 'Count', 'Avg-Lines', 'Pct-Total'];
|
|
84
|
+
const widths = headers.map((h, i) => {
|
|
85
|
+
const colKey = ['phase', 'domain', 'decision', 'appliedBy', 'count', 'avgLines', 'pct'][i];
|
|
86
|
+
return Math.max(h.length, ...rows.map((r) => String(r[colKey]).length));
|
|
87
|
+
});
|
|
88
|
+
|
|
89
|
+
function fmtRow(cells) {
|
|
90
|
+
return cells.map((c, i) => String(c).padEnd(widths[i])).join(' ');
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
console.log(fmtRow(headers));
|
|
94
|
+
console.log(widths.map((w) => '-'.repeat(w)).join(' '));
|
|
95
|
+
for (const r of rows) {
|
|
96
|
+
console.log(fmtRow([r.phase, r.domain, r.decision, r.appliedBy, r.count, r.avgLines, r.pct]));
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
console.log('');
|
|
100
|
+
console.log(`Files parsed: ${filesParsed} | Files w/o log section: ${filesSkippedNoSection} | Rows: ${totalRows}`);
|
|
101
|
+
|
|
102
|
+
// Flag candidate violations: inline apply on doc/security/migration domains
|
|
103
|
+
const violationRows = rows.filter(
|
|
104
|
+
(r) =>
|
|
105
|
+
r.appliedBy === 'orchestrator' &&
|
|
106
|
+
(r.domain === 'doc' || r.domain === 'codex-security' || r.domain.startsWith('migration'))
|
|
107
|
+
);
|
|
108
|
+
if (violationRows.length > 0) {
|
|
109
|
+
console.log('');
|
|
110
|
+
console.log('⚠ Candidate violations (inline apply on domain-override domains):');
|
|
111
|
+
for (const r of violationRows) {
|
|
112
|
+
console.log(` ${r.phase} | ${r.domain} | applied_by=${r.appliedBy} | count=${r.count}`);
|
|
113
|
+
}
|
|
114
|
+
}
|