baldart 4.98.3 → 5.0.0

package/CHANGELOG.md

@@ -5,6 +5,92 @@ All notable changes to BALDART will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [5.0.0] - 2026-07-03
+
+**The agent-spine major.** Measured driver (1,934 subagent runs / 191 sessions on the
+`mayo` consumer, one month, deduplicated by API-call id): 5.0B context tokens, of which
+coder 53% and the whole review chain ~10%; the fixed per-spawn preamble (~45k, agent
+definition 6–10k of it) is re-paid on EVERY API call; Read results are 66–75% of
+tool_result bytes with the same hot files re-read at every chain link; the top-10
+monster runs (200–440 calls) alone are 14% of total spend. 5.0 restructures the core
+agents and the `/new` pipeline around those numbers — no gate is removed anywhere.
+
+- **MAJOR — agents install as generated files when slot-bearing.** Core agent
+  definitions are restructured to 3 layers: core inline (I/O contracts + 1-line binding
+  rules) + shared protocol modules read on demand + `{{#flag}}` install-time slots
+  resolved per-consumer from EXISTING `baldart.config.yml` keys (a consumer ships ONE
+  `stack.database` variant, not four). A slot-bearing agent is GENERATED per-consumer
+  (`.baldart/generated/agents/` + symlink, marker gains trailing `config_sha`,
+  byte-stable rewrites); slotless agents stay plain symlinks; H2 headings stay stable
+  (no-op body when a flag is off) so existing overlays keep matching. First
+  `baldart update` after 5.0 flips the affected agents symlink→generated in one
+  auto-committed reconcile (allowlist already covers it). Machinery:
+  `src/utils/agent-slots.js` (new) + `symlinks.js` slot detection (fill → overlay-merge
+  → marker → transpile) + doctor `generatedAgentsStale` probe / `regenerate-agents`
+  action + `configure` re-merge after config writes. NO new config key (schema-change
+  propagation rule: N/A). Codex parity: adapter-generated — the TOML transpile consumes
+  the config-effective markdown, so Codex agents get the same resolved variant.
+- **7 core agents restructured** (`coder`, `code-reviewer`, `doc-reviewer`,
+  `plan-auditor`, `qa-sentinel`, `codebase-architect`, `ui-expert`) — source cuts
+  30–49% with ZERO normative loss (two independent adversarial verification passes:
+  no LOST, no WEAKENED rules). All carry `version:` frontmatter (own SemVer line,
+  history in the new `framework/.claude/agents/CHANGELOG.md` — NOT installed, excluded
+  from every agent enumerator incl. the discovery hooks) and ≤2-line descriptions (the
+  old multi-example descriptions were paid by the orchestrator every session). Codex
+  parity: adapter-generated (transpiler unchanged, consumes the effective markdown).
+- **3 new protocol modules** (`agents/agent-operating-protocol.md`,
+  `agents/review-protocol.md`, `agents/doc-audit-protocol.md`) — the deduplicated SSOT
+  of the shared operating blocks (injection guard / retrieval / memory / tool budget),
+  the reviewer verification engine (Challenge+Actionability / Simulation diff+plan /
+  CoVe / absolute risk scoring / specialist-spawn discipline incl. the new
+  `specialist_dispatch` orchestrated-mode suppression), and doc-reviewer's audit-mode
+  procedures (read ONLY without card context). `SECTION=` dispatch — agents read only
+  the matching section. Codex parity: portable (markdown modules).
+- **`/new` pipeline wave (skill `new` 2.0.0, `new2` 1.4.0)** — six additive mechanisms,
+  each with pre-5.0 graceful degradation and `check-new-parity.mjs` tripwire rules
+  (16 total): review bundle (`build-review-bundle.mjs` — deterministic per-card
+  handoff + binding anti-re-read phrase in every reviewer briefing, Hot-File Map
+  honored downstream); doc-invariants PASS-fast (`doc-invariants.mjs` — the invariant
+  table as executable rules; zero blocking rows + no doc signals + non-deep → per-card
+  doc-reviewer spawn skipped, Final F.3 backstop unchanged; measured headroom: 84/127
+  per-card runs returned PASS); cap-and-handoff (soft 80 / hard 100 tool calls,
+  checkpoint + fresh continuation ×2 — attacks the monster-run tail); explicit i18n
+  fill pass (step 7c, `i18n-translator` BY NAME — closes the measured general-purpose
+  leakage); fix-pass model routing A/B (fix-pass coder → `model: sonnet` best-effort,
+  overlay kill-switch `## [OVERRIDE] Fix-pass model`, `model=` in Fix Application Log
+  for recurrence telemetry; initial builds and specialists untouched). Codex parity:
+  scripts portable (.mjs), prose additive; the new2 workflow mirrors are Claude-only
+  with the /new prose as the inline fallback.
+- **`finding-mine` routine (monthly) + `skill-improver` 2.0.0 `MODE: miner`** — the
+  anticipatory loop: 30 days of pooled findings / QA reports / batch trackers /
+  reviewer memories distilled into CLASSIFIED proposals (author-time coder rules,
+  never-demote rows, deterministic-gate candidates, card-schema proposals —
+  proposal-only, upstream candidates consolidated). Routine count 8 → 9. Codex parity:
+  adapter-generated (3 routine backends; agent transpiled).
+- **`scripts/agent-telemetry.mjs` (maintainer tool)** — fleet-level per-family
+  telemetry (calls p50/p90/max, ctx, read-share, verdicts, rework ratio, cap-hit rate,
+  fix-pass models) with `--baseline save|compare`; the pre-5.0 baseline is saved.
+  Dedup by `message.id` keeping the LAST record (the JSONL double-count trap). Codex
+  parity: N/A (maintainer-only, Claude transcripts).
+- **Post-release measurement plan** (2 weeks on a live consumer, vs the saved
+  baseline): coder ctx/run −30%+, Read-share <66%, p100 calls down, doc-reviewer
+  PASS-fast rate ≈ measured headroom, fix-pass recurrence sonnet ≤ opus, reviewer
+  verdict distributions unchanged.
+
+MAJOR — changes the on-disk layout of installed agents (symlink → generated file for
+slot-bearing agents). No new `baldart.config.yml` keys anywhere in the release.
+
+## [4.99.0] - 2026-07-03
+
+**Consumer-lessons wave: four recurring failure patterns surfaced by mayo's nightly/weekly routine reports (20260628-skill-improve + 20260701-doc-graph-align) upstreamed into the base framework.** The consumer had fixed each locally via overlays, but three of the four root causes live in the base payload — every future consumer would have re-discovered them. The overarching meta-defect (#2) is that the lesson pipeline itself had no upstream channel: overlay fixes died in the consumer.
+
+1. **Epic-closure gate in `/new` (skill `new` 1.4.0 → 1.5.0).** `commit.md` step 28c said only "add/update the entry for this card's feature area" — nothing about the card being the LAST open child of a parent epic, so epics stayed `IN_PROGRESS` after closure (3 confirmed occurrences across two audit windows, caught only by the nightly). New deterministic sub-step **28.c-bis**: when the card YAML declares `parent:`, count non-DONE siblings; zero → flip the parent's ssot-registry entry (and its own YAML if present) to DONE in the SAME commit. Idempotent, non-triggers enumerated. Mirrored in `new2.js`'s mechanical commit prompt (step 4b + ROLE BOUNDARY widened to the parent-epic YAML status).
+2. **`skill-improve` routine + `skill-improver` agent modernized (pre-overlay-era → current model).** The routine prompt (since_version 2.1.0) instructed editing `.claude/skills/**` / `.claude/agents/**` directly — symlinked framework files that the `framework-edit-gate` (v3.3.0+) DENIES in consumers — and read only `*-code-review/doc-review/qa` reports (absent in real consumers; the agent had to improvise onto the nightly reports). Now: write surface = **`.baldart/overlays/`** (skills / agents / root primitives, with section markers); sources = **ALL routine report types** (`doc-graph-align`, `wiki-review`, `i18n-align`, `design-sync`, `ds-drift`, `full-sweep`, + prior skill-improve for context); graceful degradation on missing metrics/report types; and a new mandatory **`## Upstream Candidates`** report section — when a pattern's root cause is in the BASE skill/agent, the overlay fix is applied locally AND the base defect is flagged for the maintainer to carry via `baldart push` (closing the loop this very release exercises).
+3. **`API_BODY_CONSTRAINT_DRIFT` — contract-value body check in `doc-reviewer`.** A review/security fix that changes a validation constraint (`.max()`/`.min()`/length/regex/enum/default on a route handler, server action, or API schema) landed with the API reference *body* still stating the old value (real case: DoS cap 500 → 100); frontmatter-freshness checks don't catch body-level factual drift. New invariant-table row: verify the documented value EQUALS the code value; the doc-reviewer is the canonical writer and fixes the body directly. Covered per-card (Phase 3) and at the batch-Final doc pass (fixes applied after the per-card doc pass are caught there).
+4. **`update_triggers:` — declarative synthesis freshness (opt-in).** Coverage-map syntheses (query catalogs, schema coverage maps, endpoint inventories) silently rot when new code lands (3 missing entries across 3 cards in one week, all caught only by the nightly; the consumer hand-wrote a per-project reviewer rule). New OPTIONAL frontmatter field on wiki synthesis pages — path globs + `content:` added-line patterns that OBLIGATE updating the page when a diff matches, else `WIKI_SYNTH_STALE`. SSOT in `agents/llm-wiki-methodology.md § Update triggers`; consumed by three layers: `doc-reviewer` invariant table (per-task, fixes directly), `wiki-curator` (nightly, updates or flags), `doc-graph-align` routine (nightly commit-range cross-check, propose-don't-author). No retroactive obligation — pages without the field are exempt.
+
+**MINOR** — adds capability to existing skills/agents/routines (no new agent/skill/command/routine, so README counts unchanged; no install-layout change). **Codex parity: portable** — all changes are markdown skill/agent/protocol/routine bodies shared by both runtimes (`skill-improver`, `doc-reviewer`, `wiki-curator` auto-transpile to `.codex/agents/*.toml` on install/update; `new2.js` is the Claude-only accelerator whose inline `/new` path carries the same gate). **Not a `baldart.config.yml` key** (`update_triggers` is wiki-page frontmatter, not config) → the schema-change propagation rule does NOT apply.
+
 ## [4.98.3] - 2026-07-02
 
 **Bugfix: `baldart update` re-flags `multi_tenant_theming` as a "new config key" on every run.** A user reported the schema-drift detector nagging `⚠ New config keys in this version: multi_tenant_theming` at every `baldart update`, even after configuring it. Root cause: `multi_tenant_theming` was the **only** `features.*` key whose autodetected default was hardcoded `null` (`configure.js`), because there is no on-disk signal for per-tenant theming. `mergePreserving` fills it to `false` from the template, but the **non-interactive** configure branch strips every feature whose *detected* value is `null` (the "always-ask, never-assume" contract) — so a `--non-interactive` / `update --auto` / CI / auto-flow configure wrote a config **without** the key, and `update`'s drift detector (`update.js`) then re-flagged it as new on every subsequent run. The interactive path persisted it, so it appeared to "stick sometimes then forget" — a single auto flow in between re-dropped it.

package/README.md

@@ -328,6 +328,7 @@ drift-detection loops actually run. Ship-bundled routines:
 | `ds-drift` | weekly Mon 03:00 UTC | doc-reviewer | Design-system drift check (optional) |
 | `full-sweep` | weekly Sun 03:00 UTC | doc-reviewer | Full SSOT audit (optional) |
 | `i18n-align` | weekly Mon 04:00 UTC | i18n-translator | Translates target languages, lints, commits direct to trunk (optional, needs `has_i18n`) |
+| `finding-mine` | monthly 1st 03:00 UTC | skill-improver | Deep 30-day mining of pooled findings/QA/trackers/reviewer memories into classified improvement proposals (`MODE: miner`, v5.0.0) |
 
 Three backend adapters are bundled:
 

package/VERSION

@@ -1 +1 @@
-4.98.3
+5.0.0

package/framework/.claude/agents/CHANGELOG.md

@@ -0,0 +1,82 @@
+# Changelog — agent definitions
+
+Per-agent version history (the `version:` frontmatter each agent carries since
+framework v5.0.0). This file is NOT installed into consumers (excluded from the
+per-item agent merge, the doctor probes, and the discovery hooks — like
+`REGISTRY.md` it is framework metadata living beside the agents).
+
+Formato: [Keep a Changelog](https://keepachangelog.com/) · per-agent SemVer.
+
+## [coder v2.0.0] · [code-reviewer v2.0.0] · [doc-reviewer v2.0.0] · [plan-auditor v2.0.0] · [qa-sentinel v2.0.0] · [codebase-architect v2.0.0] · [ui-expert v2.0.0] · [skill-improver v2.0.0] — 2026-07-03 (framework v5.0.0)
+
+**The 3-layer restructuring** (core inline + shared modules + install-time slots).
+The pre-5.0 monoliths are implicitly `1.x`. Every normative rule survives — either
+inline as a 1-line binding rule, in a shared module the core cites
+(`agents/agent-operating-protocol.md`, `agents/review-protocol.md`,
+`agents/doc-audit-protocol.md`), or in a pre-existing SSOT — verified by two
+independent adversarial passes (zero LOST, zero WEAKENED).
+
+Common changes (all 8):
+- `version:` frontmatter added; `description` trimmed to ≤2 lines (inline examples
+  removed — they were paid by the ORCHESTRATOR's context every session).
+- Shared operating blocks (injection guard, doc-retrieval, persistent-memory
+  hygiene, tool budget) deduplicated into `agent-operating-protocol.md`;
+  reviewer verification passes (Challenge/Actionability, Simulation, CoVe, risk
+  scoring, specialist-spawn discipline) into `review-protocol.md`. Cores keep
+  1-line binding versions + `SECTION=` cites.
+- Conditional content became `{{#flag}}` install-time slots resolved from
+  EXISTING `baldart.config.yml` keys (features.*, stack.database → `db_*`,
+  `fw_nextjs`, `deploy_vercel`, `e2e_playwright`). H2 headings stay STABLE with a
+  one-line no-op body when a flag is off (overlay `[OVERRIDE]/[APPEND]` targets
+  keep matching).
+- I/O contracts stayed VERBATIM inline: completion report, verdict lines, pooled
+  findings YAML, TARGET TAG table, QA report format, 10-section FULL output,
+  Return Contracts, QUICK mode.
+
+Per-agent highlights:
+- **coder**: 4 DB variants → one `{{#db_*}}` slot; Reference-Aliasing lore → 1-line
+  + `coding-standards.md` cite; NEW `## Tool-Call Budget & Checkpoint` (cap-and-
+  handoff, `[CAP-HANDOFF]` marker); completion report ALSO persisted to
+  `/tmp/completion-<CARD-ID>.md`; deterministic DTCG token lookup (never grep the
+  CSS); Linked Skills conditional on stack.
+- **code-reviewer**: DS compliance compressed under `{{#has_design_system}}`
+  (protocol module is the SSOT); NEW review-bundle-first scope rule; NEW
+  `specialist_dispatch` orchestrated-mode suppression (records
+  `dispatch_deferred:<agent>`).
+- **doc-reviewer**: split by mode — per-card core stays; audit-mode procedures
+  (drift validators, DocAgent topological, epistemic metadata, SCIP, schema/
+  registry drift, coverage gauges) moved to `doc-audit-protocol.md`, read ONLY
+  without card context; NEW deterministic-first invariant check (consumes
+  `/tmp/docinv-<CARD-ID>.json` when present).
+- **plan-auditor**: 4-example description → 2 lines; persistence checks →
+  `{{#db_*}}`; QUICK mode, checklist A–H, ADR delta, high-risk triggers, TARGET
+  TAG table, YAML schema, 10-section output all inline; NEW dispatch suppression.
+- **qa-sentinel**: 3-example description → 2 lines; i18n gate / ds-gate /
+  toolchain resolution under their flags; review-bundle `files_changed`+`gate_logs`
+  as authoritative inputs.
+- **codebase-architect**: 4-example description → 2 lines; LSP/graph escalations
+  under `{{#has_lsp_layer}}`/`{{#has_code_graph}}`; investigation engine, terse
+  contract (incl. Hot-File Map) and architect-specific memory hygiene (TTL,
+  archived/, cleanup) kept verbatim-compressed.
+- **ui-expert**: i18n under `{{#has_i18n}}`; Design Reference (code-first) and
+  build-with-eyes kept intact; expertise/delegations/red-flags compressed with no
+  category dropped; deterministic token lookup.
+- **skill-improver**: NEW `## MODE: miner` (monthly deep pass → classified
+  proposals; consumed by the `finding-mine` routine) + COMPACT return contract.
+
+**Heading map (old → new) for consumer overlays** — overlay sections targeting a
+heading that moved/merged are appended as trailing sections with a warning (never
+lost); re-point them to:
+
+| Old H2 (pre-5.0) | Now |
+|---|---|
+| `## Prompt Injection Guard` (code-reviewer, plan-auditor) | `## Operating Protocol` (1-line) + module `SECTION=injection-guard` |
+| `## Memory Retrieval Step` / `## MEMORY RETRIEVAL STEP` | `## Operating Protocol` + module `SECTION=memory` |
+| `## Retrieval Protocol Consumption` (all) | `## Operating Protocol` / `## Retrieval Protocol` + module `SECTION=retrieval` |
+| `## Search discipline` (all) | `## Operating Protocol` (cite `code-search-protocol.md`) |
+| `## Challenge Pass` / `## Diff Simulation Pass` / `## Chain-of-Verification Pass` / `## Quantified Risk Scoring` | `## Verification Passes` + `review-protocol.md` sections |
+| `## Specialist Auto-Spawn` | same name; suppression clause added (matrix unchanged) |
+| `## Reference-Aliasing Mutation Hazards` (coder) | 1-line in `## Code Standards & Author-Time Simplicity Discipline` + `coding-standards.md` |
+| `## Database Index Invariant` variants (coder) / `### Persistence-Specific` (plan-auditor) | same heading, body is now the single `{{#db_*}}` variant |
+| doc-reviewer `## Audit Process`, `## Drift Validator Suite`, `## Dependency-Aware Generation Protocol`, `## Epistemic Metadata Protocol`, `## SCIP Code Reference Protocol`, `## Schema Field-Level Drift Detection`, `## Canonical API registries`, `## Endpoint Count Reconciliation`, `## Coverage Gauges` | `agents/doc-audit-protocol.md` (agent keeps `## Audit Mode` dispatch) |
+| `# Persistent Agent Memory` boilerplate tails | compressed under `# Persistent Agent Memory` + module `SECTION=memory` |

package/framework/.claude/agents/REGISTRY.md

@@ -6,7 +6,9 @@ Quick-reference for all custom agents. Use this to route tasks to the right spec
 
 > **Codex-native (since v4.80.0)**: every agent here is portable to Codex. The same `<name>.md` is **transpiled** to `.codex/agents/<name>.toml` on install/update (`name`/`description` → same, body → `developer_instructions`, `effort` → `model_reasoning_effort`, `model` omitted). Authoring stays single-source: drop a `.md`, it auto-installs for both tools. Keep `name`+`description` frontmatter, put behaviour in the body, and don't rely on an agent spawning a further agent (Codex `agents.max_depth: 1`). Design: [`framework/docs/CODEX-AGENTS.md`](../../docs/CODEX-AGENTS.md).
 
-> **Machine-readable source of truth (since v3.20.0)**: the filesystem directory `framework/.claude/agents/` is authoritative for the list of BALDART agents. The `agent-discovery-gate.js` PreToolUse hook, the `agent-discovery-info.sh` SessionStart hook, and `baldart doctor` enumerate the directory directly (every `<name>.md` except this file is an agent — no manifest JSON is shipped). Adding a new agent means dropping `<name>.md` here; no other touchpoint.
+> **Machine-readable source of truth (since v3.20.0)**: the filesystem directory `framework/.claude/agents/` is authoritative for the list of BALDART agents. The `agent-discovery-gate.js` PreToolUse hook, the `agent-discovery-info.sh` SessionStart hook, and `baldart doctor` enumerate the directory directly (every `<name>.md` except this file and `CHANGELOG.md` is an agent — no manifest JSON is shipped). Adding a new agent means dropping `<name>.md` here; no other touchpoint.
+
+> **3-layer structure + versioning + slots (since v5.0.0)**: the core agents follow a 3-layer authoring contract. (1) **Core inline** — role boundary, gating, I/O contracts (verdict lines, pooled findings YAML, completion reports) and every always-applicable rule as a 1-line BINDING statement. (2) **Modules on demand** — shared procedures live in `agents/agent-operating-protocol.md` (injection guard / retrieval / memory / tool budget) and `agents/review-protocol.md` (Challenge / Simulation / CoVe / risk scoring / specialist-spawn discipline), cited as `SECTION=<name>`; **reading the cited section when the procedure applies is normative**, and degrade is safe (the 1-line rule still binds; contracts never live in modules). (3) **Install-time slots** — `{{#flag}}` conditionals resolved per-consumer from existing `baldart.config.yml` keys (see `src/utils/agent-slots.js`); a slot-bearing agent installs as a GENERATED file (marker + `config_sha`), H2 headings stay stable with a no-op body when the flag is off (overlays keep matching). Every agent carries a `version:` frontmatter (its own SemVer line); bump it + add an entry to `framework/.claude/agents/CHANGELOG.md` (NOT installed) in the same edit. Keep frontmatter `description` ≤2 lines — it is paid by the orchestrator's context every session.
 
 > **Return contract (since v4.59.0)**: every orchestrator-facing subagent bounds its **final message** so a long body doesn't flood the spawning context — COMPACT return (headline/verdict + `path:line` findings + disk pointer), long form persisted to disk. Agents already structured this way are the exemplars (`qa-sentinel` Return Protocol, `visual-fidelity-verifier` JSON, the verdict+pooled-YAML reviewers — `code-reviewer` / `security-reviewer` / `api-perf-cost-auditor`); free-form analysts carry an explicit `## Return Contract` block (`codebase-architect`, `doc-reviewer`, `plan-auditor`, `senior-researcher`). When adding a new analysis/audit/research agent, paste the block from `framework/templates/agent-return-contract.snippet.md`. Protocol: `framework/agents/return-contract-protocol.md`.