@tyroneross/build-loop 0.34.0 → 0.36.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/plugins/marketplace.json +1 -1
- package/.claude-plugin/marketplace.json +3 -3
- package/.claude-plugin/plugin.json +2 -2
- package/.codex-plugin/plugin.json +1 -1
- package/AGENTS.md +71 -18
- package/README.md +208 -299
- package/agents/advisor.md +2 -0
- package/agents/alignment-checker.md +2 -0
- package/agents/api-assessor.md +3 -1
- package/agents/architecture-scout.md +2 -0
- package/agents/assessment-orchestrator.md +2 -0
- package/agents/build-orchestrator.md +13 -8
- package/agents/database-assessor.md +107 -3
- package/agents/design-contract-specialist.md +2 -0
- package/agents/fact-checker.md +2 -0
- package/agents/fix-critique.md +26 -5
- package/agents/frontend-assessor.md +3 -1
- package/agents/implementer.md +2 -0
- package/agents/independent-auditor.md +2 -0
- package/agents/mock-scanner.md +2 -0
- package/agents/optimize-runner.md +2 -0
- package/agents/overfitting-reviewer.md +2 -0
- package/agents/performance-assessor.md +3 -1
- package/agents/plan-critic.md +5 -2
- package/agents/promotion-reviewer.md +2 -0
- package/agents/recurring-pattern-detector.md +4 -1
- package/agents/retrospective-synthesizer.md +14 -0
- package/agents/root-cause-investigator.md +39 -2
- package/agents/scope-auditor.md +2 -0
- package/agents/security-reviewer.md +3 -0
- package/agents/self-improvement-architect.md +2 -0
- package/agents/synthesis-critic.md +2 -0
- package/agents/transcript-pattern-miner.md +2 -0
- package/agents/ui-validator.md +2 -0
- package/assets/build-loop-plugin-icon.png +0 -0
- package/bin/build-loop-install.js +234 -0
- package/commands/run.md +1 -1
- package/hooks/git/pre-commit +40 -0
- package/hooks/git/pre-push +35 -10
- package/hooks/hooks.json +42 -5
- package/hooks/pre-edit-architecture.sh +2 -2
- package/hooks/route-guard.sh +15 -0
- package/hooks/route_guard.py +221 -0
- package/hooks/session-start-extensions.sh +10 -0
- package/hooks/session-start-git-hooks.sh +6 -1
- package/hooks/session-start-rally-point.sh +23 -1
- package/hooks/test_session_start_rally_version.sh +123 -0
- package/package.json +7 -2
- package/plugin-artifacts/codex/.codex-plugin/plugin.json +1 -1
- package/plugin-artifacts/codex/AGENTS.md +77 -16
- package/plugin-artifacts/codex/README.md +208 -299
- package/plugin-artifacts/codex/assets/build-loop-plugin-icon.png +0 -0
- package/plugin-artifacts/codex/references/agent-role-taxonomy.md +34 -0
- package/plugin-artifacts/codex/references/backlog-system.md +285 -0
- package/plugin-artifacts/codex/references/capability-routing.md +49 -0
- package/plugin-artifacts/codex/references/coordination-file-template.md +2 -2
- package/plugin-artifacts/codex/references/coordination-rules.md +275 -6
- package/plugin-artifacts/codex/references/memory-systems.md +25 -2
- package/plugin-artifacts/codex/references/memory.md +16 -0
- package/plugin-artifacts/codex/references/model-tier-mapping.md +45 -6
- package/plugin-artifacts/codex/references/phase-1-assess.md +19 -1
- package/plugin-artifacts/codex/references/phase-2-plan.md +10 -1
- package/plugin-artifacts/codex/references/phase-3-execute.md +5 -3
- package/plugin-artifacts/codex/references/phase-4-review.md +22 -1
- package/plugin-artifacts/codex/references/phase-5-iterate.md +4 -3
- package/plugin-artifacts/codex/references/phase-6-learn.md +3 -1
- package/plugin-artifacts/codex/references/verify-dispatch.md +85 -0
- package/plugin-artifacts/codex/skills/build-loop/SKILL.md +19 -10
- package/plugin-artifacts/codex/skills/build-loop/fallbacks.md +1 -1
- package/plugin-artifacts/codex/skills/build-loop/references/capability-routing.md +49 -0
- package/plugin-artifacts/codex/skills/build-loop/references/experiment-results-template.md +101 -0
- package/plugin-artifacts/codex/skills/build-loop/references/memory.md +16 -0
- package/plugin-artifacts/codex/skills/build-loop/references/output-style.md +98 -5
- package/plugin-artifacts/codex/skills/build-loop/references/phase-1-assess.md +19 -1
- package/plugin-artifacts/codex/skills/build-loop/references/phase-2-plan.md +10 -1
- package/plugin-artifacts/codex/skills/build-loop/references/phase-3-execute.md +5 -3
- package/plugin-artifacts/codex/skills/build-loop/references/phase-4-review.md +22 -1
- package/plugin-artifacts/codex/skills/build-loop/references/phase-5-iterate.md +4 -3
- package/plugin-artifacts/codex/skills/build-loop/references/phase-6-learn.md +3 -1
- package/plugin-artifacts/codex/skills/build-loop/references/verify-dispatch.md +85 -0
- package/plugin-artifacts/codex/skills/build-loop/templates/codex-worker-prompt.md +4 -0
- package/scripts/build_codex_plugin_artifact.py +7 -0
- package/scripts/embed_backend.py +17 -13
- package/scripts/lessons_index/query.py +1 -1
- package/scripts/memory_graph/__init__.py +2 -2
- package/scripts/sync_navgator_lessons.py +4 -4
- package/skills/api-registry-bridge/SKILL.md +1 -1
- package/skills/auto-decision-capture/SKILL.md +4 -3
- package/skills/auto-finding-capture/SKILL.md +119 -0
- package/skills/build-loop/SKILL.md +19 -10
- package/skills/build-loop/fallbacks.md +1 -1
- package/skills/build-loop/references/capability-routing.md +49 -0
- package/skills/build-loop/references/experiment-results-template.md +101 -0
- package/skills/build-loop/references/memory.md +14 -0
- package/skills/build-loop/references/output-style.md +98 -5
- package/skills/build-loop/references/phase-1-assess.md +19 -1
- package/skills/build-loop/references/phase-2-plan.md +10 -1
- package/skills/build-loop/references/phase-3-execute.md +5 -3
- package/skills/build-loop/references/phase-4-review.md +20 -1
- package/skills/build-loop/references/phase-5-iterate.md +4 -3
- package/skills/build-loop/references/phase-6-learn.md +2 -0
- package/skills/build-loop/references/verify-dispatch.md +85 -0
- package/skills/build-loop/templates/codex-worker-prompt.md +4 -0
- package/skills/debug-loop/SKILL.md +4 -2
- package/skills/debugging-memory/SKILL.md +24 -12
- package/skills/{debugging/assess/SKILL.md → debugging-memory/references/assess.md} +6 -13
- package/skills/{debugging/memory/SKILL.md → debugging-memory/references/search.md} +10 -17
- package/skills/{debugging/store/SKILL.md → debugging-memory/references/store.md} +5 -13
- package/skills/debugging-memory/references/subagent-integration.md +3 -3
- package/skills/defenseclaw-bridge/SKILL.md +1 -1
- package/skills/{loop-builder → focused-loop-builder}/SKILL.md +6 -6
- package/skills/ibr-bridge/SKILL.md +1 -1
- package/skills/knowledge/SKILL.md +13 -2
- package/skills/{knowledge-review/SKILL.md → knowledge/references/review-mode.md} +8 -20
- package/skills/logging-tracer/SKILL.md +20 -4
- package/skills/model-bakeoff/SKILL.md +44 -0
- package/skills/model-tiering/SKILL.md +55 -20
- package/skills/optimize/SKILL.md +2 -2
- package/skills/prd-bridge/SKILL.md +1 -1
- package/skills/recursive-retrospective/SKILL.md +46 -0
- package/skills/recursive-retrospective/references/01-retrospective.md +136 -0
- package/skills/recursive-retrospective/references/02-learning-capture.md +72 -0
- package/skills/recursive-retrospective/references/03-judge.md +45 -0
- package/skills/research/SKILL.md +1 -1
- package/skills/root-cause-analysis/SKILL.md +42 -0
- package/skills/security-methodology/SKILL.md +10 -1
- package/skills/security-scan/SKILL.md +55 -0
- package/skills/spec-writing/SKILL.md +42 -1
- package/skills/sync-skills/SKILL.md +8 -7
- package/commands/assess.md +0 -41
- package/commands/compose-handoff.md +0 -113
- package/commands/debug.md +0 -20
- package/commands/debugger-detail.md +0 -34
- package/commands/debugger-scan.md +0 -15
- package/commands/debugger-status.md +0 -20
- package/commands/debugger.md +0 -32
- package/commands/optimize-run.md +0 -31
- package/commands/promote-experiment.md +0 -129
- package/commands/rally-point.md +0 -301
- package/commands/research-run.md +0 -22
- package/commands/review-knowledge.md +0 -16
- package/commands/self-review.md +0 -51
- package/commands/setup-memory.md +0 -32
- package/commands/start-prd.md +0 -57
- package/commands/test.md +0 -34
- package/commands/verify-plan.md +0 -39
- package/skills/debugging/debug-loop/SKILL.md +0 -231
- package/skills/logging-tracer-bridge/SKILL.md +0 -70
- /package/skills/{loop-builder → focused-loop-builder}/presets/active-project-evidence.yaml +0 -0
- /package/skills/{loop-builder → focused-loop-builder}/presets/generic-artifact-loop.yaml +0 -0
- /package/skills/{loop-builder → focused-loop-builder}/presets/presentation-audit.yaml +0 -0
- /package/skills/{loop-builder → focused-loop-builder}/presets/research-synthesis.yaml +0 -0
- /package/skills/{loop-builder → focused-loop-builder}/presets/source-ingestion-raw-data-audit.yaml +0 -0
- /package/skills/{loop-builder → focused-loop-builder}/references/spec-format.md +0 -0
- /package/skills/{loop-builder → focused-loop-builder}/scripts/loop_builder.py +0 -0
|
@@ -55,7 +55,7 @@ This project uses build-loop native debugging memory.
|
|
|
55
55
|
|
|
56
56
|
**Before investigating any bug:**
|
|
57
57
|
\`\`\`
|
|
58
|
-
Skill("build-loop:debugging-memory
|
|
58
|
+
Skill("build-loop:debugging-memory") with input { op: "search", symptom: "<symptom description>" }
|
|
59
59
|
\`\`\`
|
|
60
60
|
|
|
61
61
|
**Interpret results:**
|
|
@@ -88,10 +88,10 @@ When subagents cannot directly access debugging memory (no Bash tool, sandboxed,
|
|
|
88
88
|
|
|
89
89
|
**Step 1: Search before spawning**
|
|
90
90
|
|
|
91
|
-
Use `build-loop:debugging-memory
|
|
91
|
+
Use `build-loop:debugging-memory` `{op:"search"}` with the symptom description. If the subagent cannot invoke skills, the parent reads `.build-loop/issues/` and passes compact matches:
|
|
92
92
|
|
|
93
93
|
```
|
|
94
|
-
Skill("build-loop:debugging-memory
|
|
94
|
+
Skill("build-loop:debugging-memory") with input { op: "search", symptom: "user login failing with 401" }
|
|
95
95
|
```
|
|
96
96
|
|
|
97
97
|
**Step 2: Format context for subagent**
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: build-loop:defenseclaw-bridge
|
|
3
|
-
description: Use when the user is working on the defenseclaw project and build-loop's Phase 1 detects defenseclaw-specific files (CLAUDE.md indicates the bridge target). Bridges build-loop's orchestrator to defenseclaw's eval/test conventions; otherwise no-op.
|
|
3
|
+
description: Use when the user is working on the defenseclaw project and build-loop's Phase 1 detects defenseclaw-specific files (CLAUDE.md indicates the bridge target). Bridges build-loop's orchestrator to defenseclaw's eval/test conventions; otherwise no-op. NOT a general eval/test bridge (defenseclaw project only); for API/PRD/UI use `api-registry-bridge` / `prd-bridge` / `ibr-bridge`.
|
|
4
4
|
version: 0.1.0
|
|
5
5
|
user-invocable: false
|
|
6
6
|
---
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
name: loop-builder
|
|
2
|
+
name: focused-loop-builder
|
|
3
3
|
description: Use when the user asks to "create a custom build loop", "build a loop spec", "make a focused loop", "generate a workflow loop", "adapt a framework into a loop", or asks whether a workflow should use skill chaining. Generates declarative focused-loop specs, presets, validators, and skill-chain plans.
|
|
4
4
|
version: 0.1.0
|
|
5
5
|
user-invocable: false
|
|
@@ -7,7 +7,7 @@ user-invocable: false
|
|
|
7
7
|
|
|
8
8
|
<!-- SPDX-FileCopyrightText: 2025-2026 Tyrone Ross, Jr <46267523+tyroneross@users.noreply.github.com> | SPDX-License-Identifier: Apache-2.0 -->
|
|
9
9
|
|
|
10
|
-
# loop-builder
|
|
10
|
+
# focused-loop-builder
|
|
11
11
|
|
|
12
12
|
Create declarative focused-loop specs that reuse build-loop's phase discipline outside pure coding work. The skill turns a preset or framework into a small loop pack: `loop.yaml`, `rubric.md`, report template, validator stub, and explicit skill-chain guidance.
|
|
13
13
|
|
|
@@ -37,7 +37,7 @@ Keep the runner generic and the loop domain-specific. Do not create a new orches
|
|
|
37
37
|
4. **Generate the loop pack.**
|
|
38
38
|
- Run:
|
|
39
39
|
```bash
|
|
40
|
-
python3 skills/loop-builder/scripts/loop_builder.py create <loop-id> --preset <preset-name>
|
|
40
|
+
python3 skills/focused-loop-builder/scripts/loop_builder.py create <loop-id> --preset <preset-name>
|
|
41
41
|
```
|
|
42
42
|
- Default output is `.build-loop/loops/<loop-id>/`.
|
|
43
43
|
- Use `--output <dir>` for a WorkWiki, ObsidianVault, or non-code project workspace.
|
|
@@ -76,19 +76,19 @@ Avoid chaining when:
|
|
|
76
76
|
List presets:
|
|
77
77
|
|
|
78
78
|
```bash
|
|
79
|
-
python3 skills/loop-builder/scripts/loop_builder.py list
|
|
79
|
+
python3 skills/focused-loop-builder/scripts/loop_builder.py list
|
|
80
80
|
```
|
|
81
81
|
|
|
82
82
|
Create a loop:
|
|
83
83
|
|
|
84
84
|
```bash
|
|
85
|
-
python3 skills/loop-builder/scripts/loop_builder.py create active-project-evidence --preset active-project-evidence
|
|
85
|
+
python3 skills/focused-loop-builder/scripts/loop_builder.py create active-project-evidence --preset active-project-evidence
|
|
86
86
|
```
|
|
87
87
|
|
|
88
88
|
Inspect a preset:
|
|
89
89
|
|
|
90
90
|
```bash
|
|
91
|
-
python3 skills/loop-builder/scripts/loop_builder.py inspect presentation-audit
|
|
91
|
+
python3 skills/focused-loop-builder/scripts/loop_builder.py inspect presentation-audit
|
|
92
92
|
```
|
|
93
93
|
|
|
94
94
|
## Additional Resources
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: build-loop:ibr-bridge
|
|
3
|
-
description: Routing bridge to the IBR plugin for UI visual verification. Build-loop prefers IBR `scan` / `scan_macos` when the IBR plugin is installed; otherwise falls back to build-loop's own `native-ax-driver` / `ui-validator`. Never falls back to nm/strings.
|
|
3
|
+
description: Routing bridge to the IBR plugin for UI visual verification. Build-loop prefers IBR `scan` / `scan_macos` when the IBR plugin is installed; otherwise falls back to build-loop's own `native-ax-driver` / `ui-validator`. Never falls back to nm/strings. NOT for API registration (use `api-registry-bridge`) or PRD grounding (use `prd-bridge`).
|
|
4
4
|
version: 0.3.0
|
|
5
5
|
user-invocable: false
|
|
6
6
|
---
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: knowledge
|
|
3
|
-
description: Canonical build-loop-memory framework. Use when the user asks to "record a decision", "log an ADR", "write an MADR", "capture this choice", "regenerate the decisions index", "validate knowledge", "migrate feedback to decisions", or "recall <topic>". Active durable writes go to `~/dev/git-folder/build-loop-memory`; legacy `.episodic/` paths are migration/archive inputs only.
|
|
3
|
+
description: Canonical build-loop-memory framework. Use when the user asks to "record a decision", "log an ADR", "write an MADR", "capture this choice", "regenerate the decisions index", "validate knowledge", "migrate feedback to decisions", or "recall <topic>". ALSO the read-only review surface (review mode): "review my decisions", "show review queue", "check decision rot", "list open conflicts", "find stale procedures", or `/knowledge:review`. Active durable writes go to `~/dev/git-folder/build-loop-memory`; legacy `.episodic/` paths are migration/archive inputs only.
|
|
4
4
|
user-invocable: true
|
|
5
5
|
when_to_use: |
|
|
6
6
|
- User wants to record a substantive choice with rationale
|
|
@@ -114,12 +114,23 @@ postgresql://tyroneross@localhost:5432/agent_memory). Per-project schema:
|
|
|
114
114
|
this repo uses `build_loop_memory`. The schema name is configurable via
|
|
115
115
|
the `--schema` flag on each DB-aware script.
|
|
116
116
|
|
|
117
|
+
## Review mode (read-only)
|
|
118
|
+
|
|
119
|
+
The read-only review surface — `/knowledge:review` and asks like "review my
|
|
120
|
+
decisions", "show review queue", "check decision rot", "list open conflicts",
|
|
121
|
+
"find stale procedures" — lists the four sections of decisions/procedures
|
|
122
|
+
awaiting human attention (review queue, decision rot, open conflicts, stale
|
|
123
|
+
procedures) with a suggested action per item. It NEVER auto-resolves; humans
|
|
124
|
+
take the action. Full surface, invocation flags, and the consolidation
|
|
125
|
+
cross-reference: **`references/review-mode.md`**. (Namespace note: this surface
|
|
126
|
+
reviews the legacy `.episodic/` paths; active durable writes target
|
|
127
|
+
build-loop-memory canonical.)
|
|
128
|
+
|
|
117
129
|
## What's NOT in this skill
|
|
118
130
|
|
|
119
131
|
- Auto-capture from conversation — Phase 3 (`auto-decision-capture`
|
|
120
132
|
skill, Stop hook with `scan_transcript_for_decisions.py`)
|
|
121
133
|
- Memory consolidation — Phase 4 (`consolidate_memory.py`)
|
|
122
|
-
- `/knowledge:review` slash command — Phase 4
|
|
123
134
|
- `derived/libraries.json` and `derived/CHANGELOG.md` generators —
|
|
124
135
|
Phase 1.5 / 4
|
|
125
136
|
|
|
@@ -1,24 +1,7 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: knowledge-review
|
|
3
|
-
description: Repo-local episodic memory review surface. Use when the user asks to "review my decisions", "show review queue", "check decision rot", "list open conflicts", "find stale procedures", or runs `/knowledge:review`. Read-only — never auto-resolves.
|
|
4
|
-
user-invocable: false
|
|
5
|
-
when_to_use: |
|
|
6
|
-
- User runs `/knowledge:review` or asks to surface review-needing items
|
|
7
|
-
- User wants to see decisions older than the staleness threshold
|
|
8
|
-
- User wants to see `_review/` queue items awaiting promotion
|
|
9
|
-
- User wants to see open `fact_conflicts` rows
|
|
10
|
-
- User wants to see procedures whose `depends_on` symbols are missing from the codebase
|
|
11
|
-
namespace: .episodic/decisions/_review/, .episodic/decisions/, .procedural/, agent_memory.<schema>.fact_conflicts
|
|
12
|
-
companion_scripts:
|
|
13
|
-
- scripts/knowledge_review.py — aggregates all four sections into a markdown report
|
|
14
|
-
- scripts/detect_decision_rot.py — drives the rot section
|
|
15
|
-
- scripts/procedural_governance.py — drives the stale-procedures section (validate-symbols mode)
|
|
16
|
-
- scripts/consolidate_memory.py — referenced as the next-step action when surface items accumulate
|
|
17
|
-
---
|
|
18
|
-
|
|
19
1
|
<!-- SPDX-FileCopyrightText: 2025-2026 Tyrone Ross, Jr <46267523+tyroneross@users.noreply.github.com> | SPDX-License-Identifier: Apache-2.0 -->
|
|
2
|
+
<!-- PROVENANCE: folded from skills/knowledge-review/SKILL.md (v0.1.0, user-invocable:false) on 2026-07-02 (pool-consolidation Inc 3). Review-intent selection triggers preserved in skills/knowledge/SKILL.md frontmatter description (W4); this reference holds the review-surface body. Namespace note: this surface reviews the LEGACY .episodic/ paths (the knowledge skill's active writes target build-loop-memory canonical); .episodic is migration/archive input. -->
|
|
20
3
|
|
|
21
|
-
#
|
|
4
|
+
# Knowledge review mode — surface review-needing items
|
|
22
5
|
|
|
23
6
|
`/knowledge:review` is the read-only review surface for the four-memory-types
|
|
24
7
|
framework (Working / Episodic / Semantic / Procedural). It does NOT modify
|
|
@@ -28,6 +11,11 @@ The full design lives at
|
|
|
28
11
|
`~/dev/research/topics/repo-episodic-memory-framework/repo-episodic-memory-framework.md`
|
|
29
12
|
(see §11–§14).
|
|
30
13
|
|
|
14
|
+
Companion scripts: `scripts/knowledge_review.py` (aggregates all four sections into a
|
|
15
|
+
markdown report), `scripts/detect_decision_rot.py` (rot section),
|
|
16
|
+
`scripts/procedural_governance.py` (stale-procedures section, validate-symbols mode),
|
|
17
|
+
`scripts/consolidate_memory.py` (next-step action when surface items accumulate).
|
|
18
|
+
|
|
31
19
|
## What it surfaces
|
|
32
20
|
|
|
33
21
|
```
|
|
@@ -101,7 +89,7 @@ records the action in `.semantic/_candidates_history.jsonl`.
|
|
|
101
89
|
|
|
102
90
|
## Read-only contract
|
|
103
91
|
|
|
104
|
-
This
|
|
92
|
+
This surface never:
|
|
105
93
|
- promotes items from `_review/` (user does the `mv`)
|
|
106
94
|
- mutates `last_validated` (user edits the file)
|
|
107
95
|
- resolves conflicts (user updates the rows)
|
|
@@ -157,15 +157,31 @@ After the instrumentation lands:
|
|
|
157
157
|
3. If tests still fail silently → instrumentation did not solve the visibility problem; escalate to user
|
|
158
158
|
4. **Always revert** at session end unless the user explicitly approved keep-in-diff via the prompt above. The orchestrator (or caller) verifies no `build-loop:trace/` stash entries remain and no unguarded trace calls landed.
|
|
159
159
|
|
|
160
|
-
## Extended capability —
|
|
160
|
+
## Extended capability — Coding Debugger escalation
|
|
161
161
|
|
|
162
|
-
|
|
162
|
+
When the bundled tier selection / codegen / placement isn't enough — the project requires a tracer backend or placement intelligence that lives only in the standalone Coding Debugger plugin, or you need cross-build log correlation — this skill escalates directly (folded in from the former `logging-tracer-bridge` skill, 2026-07, pool-consolidation Inc 4). This is a secondary, outbound-only hop: the orchestrator owns when-to-fire (Phase 1 Assess observability scan, Phase 5 Iterate `evidence_gap` trigger) and routes to this skill; this skill owns tier selection / codegen / ephemeral policy; the escalation below is optional.
|
|
163
|
+
|
|
164
|
+
**Pre-flight (always run first):**
|
|
163
165
|
|
|
164
166
|
```
|
|
165
|
-
|
|
167
|
+
if (!state.availablePlugins.codingDebugger) {
|
|
168
|
+
return { delegated: false, reason: "standalone Coding Debugger plugin not installed" }
|
|
169
|
+
}
|
|
166
170
|
```
|
|
167
171
|
|
|
168
|
-
|
|
172
|
+
If the standalone plugin is not installed, continue with bundled-only Tier 1/2/3 capability — never hard-fail (graceful degradation).
|
|
173
|
+
|
|
174
|
+
**Delegations available** (pass through the caller-supplied `{ symptom, target_files, tier_hint }`; fold the enriched result back into this skill's own codegen):
|
|
175
|
+
|
|
176
|
+
| Capability needed | Standalone Skill / MCP call |
|
|
177
|
+
|---|---|
|
|
178
|
+
| Extended tracer backends not in bundle | `Skill("coding-debugger:logging-tracer")` with `tier: <upstream-only>` |
|
|
179
|
+
| Cross-build log correlation (correlation IDs across multiple build-loop runs) | standalone-only MCP tools |
|
|
180
|
+
| Advanced placement intelligence (function-call-graph-aware insertion) | standalone-only assessor skills |
|
|
181
|
+
|
|
182
|
+
**Does NOT:** reimplement tier selection / stack detection / ephemeral mechanisms / placement (those stay in this skill); replace the orchestrator's when-to-fire policy; introduce new logging dependencies without explicit user approval; mutate `.build-loop/issues/`; hard-fail when standalone is absent.
|
|
183
|
+
|
|
184
|
+
**State:** optional escalations log to `.build-loop/state.json.observability.escalations[]` — `{ "ts": "ISO", "calledBy": "logging-tracer", "reason": "tier_3_otel_required", "delegated": true|false }`.
|
|
169
185
|
|
|
170
186
|
## Log Analysis Guidance
|
|
171
187
|
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: model-bakeoff
|
|
3
|
+
description: Use to run a controlled multi-model bake-off — have N models (e.g. Opus 4.8, Sonnet 5.0, GPT-5.5) each independently diagnose→plan→execute the SAME bounded change in isolated git worktrees, then deterministically re-score their committed code on a fresh server, merge the best-of (grafting distinct wins from the others), and repeat per change. Triggers on "bake-off", "compare models on this task", "which model is best at", "run the same change across models and score them". NOT for choosing a model tier/segment for one agent — use `model-tiering`.
|
|
4
|
+
user-invocable: false
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Model Bake-off Harness
|
|
8
|
+
|
|
9
|
+
Run a fair, evidence-based competition where several models each solve the *same* change end-to-end, then merge the best result. One orchestrator (this session) coordinates; contestants are single agents (measure the model, not a multi-agent loop). Repeat per change, accumulating merges on one experiment branch.
|
|
10
|
+
|
|
11
|
+
## Roster & dispatch (verified handles)
|
|
12
|
+
- Opus 4.8 → `Agent(model: "opus")`; Sonnet 5.0 → `Agent(model: "sonnet")` (`sonnet` = latest, NOT 4.x — older Sonnets have no clean subagent handle).
|
|
13
|
+
- GPT-5.5 → Codex MCP `mcp__codex__codex` with `model: "gpt-5.5"`, `config: {model_reasoning_effort: "xhigh", sandbox_workspace_write:{network_access:true}}`, `approval-policy: "never"`, `sandbox: "workspace-write"`. (Check `~/.codex/config.toml` for the exact model id; `-codex` suffixes fail on ChatGPT-account Codex.)
|
|
14
|
+
- Independent judge: prefer a NON-contestant model (e.g. Fable). If unavailable, the orchestrator scores subjective dims with over-cited evidence + a stated caveat, and leans on deterministic dims.
|
|
15
|
+
|
|
16
|
+
## Per-change protocol
|
|
17
|
+
1. **Baseline:** branch the experiment off clean `origin/main` (not a dirty/active branch). Confirm no concurrent session collides.
|
|
18
|
+
2. **Scaffold** one worktree per contestant off the experiment branch HEAD: `git worktree add -b bakeoff/<Cn>-<model> <path> <branch>`; `npm ci` (or lockfile-equiv) per worktree; copy `.env.local`.
|
|
19
|
+
3. **Brief** (IDENTICAL for all): give the *symptom* + acceptance criteria + a fair equal entrypoint pointer — WITHHOLD the diagnosis (that's what's scored). Add repo guardrails (see below).
|
|
20
|
+
4. **Dispatch** all contestants in parallel (Agent arms `run_in_background: true`; Codex arm blocks the turn — fine, the others run concurrently).
|
|
21
|
+
5. **Commit stranded Codex work:** Codex's sandbox usually cannot write an external worktree's `.git` (`index.lock: Operation not permitted`). The orchestrator commits it: `git -C <worktree> add -A && git commit`. (RESULT.md is often gitignored → `git add -f`.)
|
|
22
|
+
6. **Score deterministically** (don't trust self-reports — re-run each contestant's committed code):
|
|
23
|
+
- Objective dims computed in code: typecheck, build, test, betterer, + the change-specific success criterion run on a fresh server. Weight these highest.
|
|
24
|
+
- Subjective dims (diagnosis depth, plan, code quality, intent fidelity) by the judge reading diffs + captured outputs.
|
|
25
|
+
7. **Scorecard** (rubric below) → **merge** best base onto the experiment branch, **grafting** distinct wins from the others (each graft: fixes a real gap the winner has, is isolable from the loser's *harmful* parts, verified by re-running). Document graft rationale.
|
|
26
|
+
8. **Re-verify the merged result**, regenerate coverage, commit. Then next change.
|
|
27
|
+
|
|
28
|
+
## Rubric (max 50; tune weights per task)
|
|
29
|
+
success-criteria attainment ×3 (objective) · build/typecheck/test/betterer ×2 (objective) · diagnosis accuracy ×2 · plan quality ×1 · code quality+scope ×1 · intent fidelity ×1. Objective dims dominate; the LLM judge is confined to subjective dims.
|
|
30
|
+
|
|
31
|
+
## Hard-won lessons (do these or the scoring is wrong)
|
|
32
|
+
- **UI-faithful inputs.** Score with inputs the real UI can actually send. (A driver that sent an out-of-range `timeHorizonDays` the UI caps at 90 unfairly zeroed 2 of 3 contestants whose validators — correctly — rejected it.)
|
|
33
|
+
- **Dynamic free-port allocation; NEVER blanket-kill by port.** External processes steal fixed ports; `pkill`-by-port killed *peer contestants'* live dev servers mid-run twice. Allocate a guaranteed-free port per server, check before bind, kill only your own PIDs.
|
|
34
|
+
- **Multi-sample runtime scoring.** LLM output is non-deterministic (temp>0). A single sample misled once (a model's terse run read as failure). Take ≥3 samples for the pass/fail criterion; report pass-rate.
|
|
35
|
+
- **Write-path changes contaminate a shared DB.** For UI/render/summary fixes that self-heal or persist, contestants' writes to a shared DB poison each other's before/after and self-heal the very rows you test. Prefer **function-level tests** (feed the exact bad input through each contestant's exported cleaner — no DB writes). For write-heavy/auth stages, DO NOT share a live DB (per-contestant schema/branch, or serialize); require a UNIQUE test-user id per contestant and id-scoped (not suffix-scoped) cleanup.
|
|
36
|
+
- **Verify against the REAL path, not the self-reported one.** A contestant's "5/5 pass" exercised a code path real traffic doesn't; the real-UI re-run showed 0/3. Re-run the actual user flow.
|
|
37
|
+
- **Betterer/coverage baseline.** Fresh worktrees need `npm run test:coverage` before betterer (coverage-summary.json). On merge, keep the repo-level baseline; `betterer --update` only to *include new tested code* — NEVER bake in a *lowered* baseline caused by skipped/failing tests (that weakens the guardrail for everyone).
|
|
38
|
+
- **No schema migrations against a shared DB.** Forbid `prisma migrate`/`db push`; use existing columns / JSON blobs.
|
|
39
|
+
|
|
40
|
+
## Repo guardrails to put in every brief
|
|
41
|
+
Work ONLY in your worktree; no edits outside it; no deploy/push; no `--no-verify`; no DDL against the shared DB; unique per-model test-user id + id-scoped cleanup; verify by RUNNING (name the exact verification mechanism: curl the endpoint, CDP virtual authenticator for WebAuthn, function-level test for cleaners, screenshots for UI).
|
|
42
|
+
|
|
43
|
+
## Output
|
|
44
|
+
Per stage: a `SCORECARD.md` (rubric table + verdict + merge/graft rationale). At the end: a consolidated `RESULTS.md` (cross-stage scoreboard, per-model performance pattern, where multi-model merge beat any single model, scoring-integrity caveats).
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: model-tiering
|
|
3
|
-
description: Use when choosing a model tier for a subagent, deciding
|
|
3
|
+
description: Use when choosing a model tier or segment for a subagent, deciding a role descriptor (segment + tier) in frontmatter, or escalating mid-flow. Covers the two-axis taxonomy (work-role segment × 7-rung capability ladder) — Opus/Sonnet/Haiku are Anthropic-default mappings; selection is provider-portable and data-driven. NOT for benchmarking models head-to-head on a task — use `model-bakeoff`.
|
|
4
4
|
user-invocable: false
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -13,13 +13,22 @@ Governs model selection across all build-loop phases. Build-loop is **multi-mode
|
|
|
13
13
|
Use `references/agent-role-taxonomy.md` for responsibility boundaries. This
|
|
14
14
|
skill answers "which tier should run the role?", not "who owns the work?".
|
|
15
15
|
|
|
16
|
-
##
|
|
16
|
+
## Two-axis taxonomy (the source of truth)
|
|
17
|
+
|
|
18
|
+
Selection runs on **two orthogonal axes**, encoded as data in `references/model-taxonomy.json` (loader: `scripts/model_taxonomy.py`):
|
|
19
|
+
|
|
20
|
+
- **SEGMENT** — the work role / primary output: Generative Reasoning, Agentic Execution, Representation/Retrieval, Realtime Interaction, Perception/Input Understanding, Generative Media, Governance/Evaluation. Segment is the *primary product role* — a reasoning model that accepts image/audio input is Generative Reasoning with a `multimodal-input` tag, not Perception.
|
|
21
|
+
- **CAPABILITY-TIER** — a 7-rung ladder: T0 experimental/restricted frontier · T1 ultra-frontier · T2 frontier · T3 balanced workhorse · T4 efficient near-frontier · T5 utility/nano/edge · T-S specialist infrastructure (off the ladder).
|
|
22
|
+
|
|
23
|
+
Agents declare a `(segment, tier)` ROLE; the resolver (`scripts/model_resolver.py resolve_role`) walks the per-cell ordered preferred list and returns the highest-ranked AVAILABLE + host-reachable model (ties → release recency). New models are classified once (`scripts/classify_model_tier.py`, host-LLM, both axes) — no agent edits. **The 4-tier table below is the legacy alias view** — `frontier/thinking/code/pattern` fold to `T1/T2/T3/T4` (Generative Reasoning segment) and remain accepted everywhere for back-compat. Dormant segments (Realtime/Perception/Media) are data + reference only — no resolver wiring yet.
|
|
24
|
+
|
|
25
|
+
## Tier abstraction (legacy 4-token alias view — Generative Reasoning T1–T4)
|
|
17
26
|
|
|
18
27
|
| Tier | Anthropic default | Role | Equivalents (advisory — verify benchmarks before swapping) |
|
|
19
28
|
|---|---|---|---|
|
|
20
29
|
| **Frontier** | Fable 5 | **Phase 2 Plan synthesis (frame goal, draft spec/ADRs, F-criteria, MECE partition) via the Advisor dispatch ladder when stakes-gated** — `advisor` agent / peer host / already-Fable session; honestly-labeled inline-Opus fallback otherwise (`references/advisor-dispatch-ladder.md`). (Advisor v1 = Phase 2 only; Phase 1 Assess synthesis runs inline as today until v2.) AND verification judgment (plan-critic, scope-auditor, independent-auditor, fix-critique, fact-checker, security-reviewer, overfitting-reviewer, promotion-reviewer) | GPT-5.5 Thinking (or whichever tier scores above the prior Thinking-tier ceiling), future Claude tier above Opus; any model that benchmarks above the Thinking-tier contract on SWE-bench Verified AND ARC-AGI / GPQA Diamond |
|
|
21
30
|
| **Thinking** | Opus 4.8 | Coordination — build-orchestrator, assessment-orchestrator — and the escalation target for execution (ambiguous spec, 2 consecutive failures, cross-file surprise) and audit/learnings synthesis when Frontier is unavailable | GPT-5 Thinking, Gemini 2.5 Pro; any model >= Opus 4.6 on SWE-bench Verified + Frontier-class on ARC-AGI / MMLU-Pro |
|
|
22
|
-
| **Code** | Sonnet
|
|
31
|
+
| **Code** | Sonnet 5 | Application — apply rule to bounded input, scoped implementation, mechanical refactor, bounded domain assessment | GPT-5 Codex, qwen2.5-coder-32B (local); any model with SWE-bench Verified within ~5pt of the Code-tier default (last published Anthropic Sonnet figure: 4.6 ~79.6%; Sonnet 5 reaches prior Opus-tier coding/agentic quality per claude-api T1) |
|
|
23
32
|
| **Pattern** | Haiku 4.5 | Recognition — regex/syntactic match, classification into known buckets, log scan, deterministic checklist | Haiku 4.6, GPT-5 Mini, llama3.2-3b (local); any small/fast model that handles structured pattern matching |
|
|
24
33
|
|
|
25
34
|
**Rule of substitution:** tier A's swap target must score within tolerance of the default on the benchmark relevant to its role. For Code tier that's SWE-bench Verified ≥75% AND tool-use accuracy ≥85%; for Thinking tier that's SWE-bench ≥78% AND ARC-AGI / GPQA Diamond competitive; for Frontier tier that's clearing the Thinking-tier contract AND scoring above the prior-generation Thinking-tier ceiling on at least one of SWE-bench Verified / ARC-AGI / GPQA Diamond; for Pattern tier no benchmark — just "fast and cheap, doesn't hallucinate on bounded structured tasks."
|
|
@@ -30,12 +39,31 @@ skill answers "which tier should run the role?", not "who owns the work?".
|
|
|
30
39
|
|
|
31
40
|
Build-loop's agent frontmatter uses Anthropic model aliases (`fable`, `opus`, `sonnet`, `haiku`) because Claude Code is the primary host. To run on a different provider:
|
|
32
41
|
|
|
33
|
-
1. **
|
|
42
|
+
1. **Edit the INDEX, not each agent.** `model:` frontmatter is index-DERIVED (generated by `scripts/sync_agent_model_defaults.py`), so do not hand-edit it. To swap providers, reorder the preferred list / change the default for the `(segment, tier)` cell in `references/model-taxonomy.json` (or classify a new model once via `scripts/classify_model_tier.py`), then run `python3 scripts/sync_agent_model_defaults.py --apply` to regenerate every agent's `model:`. The role's `(segment, tier)` is the durable key; the tier determines the substitution target.
|
|
34
43
|
2. **Runtime override:** `.build-loop/config.json.modelOverrides` accepts `{ frontier: "<id>", thinking: "<id>", code: "<id>", pattern: "<id>" }`. The orchestrator resolves this through `scripts/model_overrides.py` before dispatching subagents (see `references/model-tier-mapping.md` for full schema). Configs without `frontier` resolve frontier → `fable` by default.
|
|
35
44
|
3. **Per-dispatch override:** any orchestrator dispatch may pass `model: <id>` in the subagent prompt to force that call.
|
|
36
45
|
|
|
37
46
|
The role-and-task table below uses tier names. The Anthropic-default mapping in the right column is illustrative; substitute your equivalents at swap time.
|
|
38
47
|
|
|
48
|
+
## Chat-triggered index maintenance (host-LLM-driven)
|
|
49
|
+
|
|
50
|
+
The model index (`references/model-taxonomy.json`) is **user-editable and chat-maintainable**. When the user expresses model intent in conversation, recognize it and act on the index directly — this is host-LLM-driven per the repo's "host coding agent is the LLM" rule: you recognize the intent and run deterministic scripts; there is NO vendor API call and NO hard hook. (A `UserPromptSubmit` hook that pre-detects these phrasings is an OPTIONAL future hardening, not required — the LLM recognizing intent is the mechanism.)
|
|
51
|
+
|
|
52
|
+
**Trigger phrasings (illustrative, not exhaustive):** "check the model(s)", "is there a newer model", "what's the current frontier model", "change the `<tier>`/`<segment>` model", "use `<model>` for `<role>`", "what model is `<agent/tier>` using", "swap `<model>` in", "reorder the preferred list".
|
|
53
|
+
|
|
54
|
+
**On a CHECK / NEWER intent** ("check the models", "is there a newer model", "what is X using"):
|
|
55
|
+
1. Read the index — `python3 scripts/model_taxonomy.py --segment <seg> --tier <tier>` for one cell, or `--json` for the summary.
|
|
56
|
+
2. Report the CURRENTLY RECOMMENDED model vs what is AVAILABLE: run `python3 scripts/resolve_agent_model.py <agent> --json` (or `model_resolver.py --segment <s> --tier <t> --json`) and read back `model` + `resolution_path`.
|
|
57
|
+
3. If the user names a model the index does not know, OFFER to classify it via the existing host-LLM flow: `python3 scripts/classify_model_tier.py lookup <id>` returns a WebSearch query + parse rubric; you run the search, decide segment + tier, then `record <id> --tier <tier> --segment <seg> --provider <vendor> [--provenance verified]`. No vendor API call — you (the host LLM) do the interpretation.
|
|
58
|
+
|
|
59
|
+
**On a CHANGE / USE intent** ("use gpt-5.5 for frontier", "change the code model to X", "make sonnet first"):
|
|
60
|
+
1. Edit the index `references/model-taxonomy.json` with the smallest change: reorder the `preferred[<segment>][<tier>]` list (capability-rank order — the first available wins), change a cell's default, or add an already-classified model id. A documented jsonpatch-style single-field edit is enough — no helper script needed (KISS). If the model is not yet classified, classify it first (step above).
|
|
61
|
+
2. Regenerate the derived defaults: `python3 scripts/sync_agent_model_defaults.py --apply`. This rewrites every affected agent's `model:` from the new index state (only harness-valid tokens are written; a cross-provider recommendation keeps the existing token and is reported).
|
|
62
|
+
3. Confirm: `python3 scripts/sync_agent_model_defaults.py --check` returns 0 drift; report the changed agents back.
|
|
63
|
+
|
|
64
|
+
Dispatch always resolves the role LIVE through `resolve_agent_model.py`, so an index edit takes effect on the next dispatch even before a sync — `sync` only keeps the on-disk `model:` fallback honest.
|
|
65
|
+
|
|
66
|
+
|
|
39
67
|
## When to use this skill
|
|
40
68
|
|
|
41
69
|
- Choosing `model:` field in an agent frontmatter
|
|
@@ -48,10 +76,12 @@ The role-and-task table below uses tier names. The Anthropic-default mapping in
|
|
|
48
76
|
|
|
49
77
|
| Claim | Source | Certainty |
|
|
50
78
|
|-------|--------|-----------|
|
|
51
|
-
| Sonnet
|
|
52
|
-
|
|
|
53
|
-
| Sonnet 4.6
|
|
54
|
-
|
|
|
79
|
+
| Sonnet 5 reaches prior Opus-tier quality on coding/agentic work (Code-tier default since 2026-06) | claude-api skill cache (T1 — Anthropic) | ✅ T1 qualitative; no single SWE-bench figure published yet |
|
|
80
|
+
| Sonnet 5 uses a new tokenizer — ~30% more tokens for the same text vs Sonnet 4.6 | claude-api skill cache (T1 — Anthropic) | ✅ T1 — re-baseline token budgets, do not reuse 4.6 counts |
|
|
81
|
+
| (prior-gen baseline) Sonnet 4.6: 79.6% SWE-bench Verified | Anthropic announcement + SWE-bench leaderboard | ⚠️ T2, single-source; historical calibration |
|
|
82
|
+
| (prior-gen baseline) Opus 4.6: 80.8% SWE-bench Verified (1.2pt gap — smallest in Claude history) | Same | ⚠️ T2, single-source; historical |
|
|
83
|
+
| (prior-gen baseline) Sonnet 4.6 uses 70% fewer tokens than 4.5 on complex file ops with +38% accuracy | Anthropic Sonnet 4.6 announcement | ⚠️ T2, single-source; superseded by Sonnet 5's new tokenizer |
|
|
84
|
+
| Pricing: Sonnet 5 $3/$15 per MTok ($2/$10 introductory through 2026-08-31) | claude-api skill cache 2026-06-24 (T1 — Anthropic) | ⚠️ verify before billing |
|
|
55
85
|
| Pricing: Opus 4.8 $5/$25 per MTok input/output | Anthropic pricing page | ⚠️ verify before billing |
|
|
56
86
|
| Pricing: Fable 5 $10/$50 per MTok input/output (1M context, capability tier above Opus 4.8) | claude-api skill cache 2026-05-26 (T1 — Anthropic) | ✅ T1 source, advisory until re-confirmed at next billing audit |
|
|
57
87
|
|
|
@@ -142,7 +172,7 @@ If the ambiguity surfaces a **planning** problem (the original plan no longer fi
|
|
|
142
172
|
- **Self-refine with external verification** (tests, lint, type-check). External oracle is non-negotiable — without it, self-refine is circular.
|
|
143
173
|
- **Adversarial critic loop** (writer agent + read-only reviewer agent). Separation is what makes it work. Same model reviewing its own output doesn't catch errors.
|
|
144
174
|
- **Best-of-N sampling with self-certainty voting** on HARD chunks only (flagged by plan or first-pass failure). N=3. Cost = ~3x Sonnet, still under 1.5x single-pass Opus.
|
|
145
|
-
- **Test-time compute** (effort=
|
|
175
|
+
- **Test-time compute** (effort=high default, effort=xhigh on retry — Sonnet 5 is the first Sonnet with `xhigh`; coding/agentic work wants high/xhigh per claude-api T1). Easier problems benefit from revisions; harder problems need parallel sampling — not just more thinking on one path.
|
|
146
176
|
- **Plan-then-execute split** (Opus plans once, Sonnet executes many). Established pattern. Amortizes Opus cost across N subagent calls.
|
|
147
177
|
|
|
148
178
|
## Techniques to avoid
|
|
@@ -156,15 +186,19 @@ If the ambiguity surfaces a **planning** problem (the original plan no longer fi
|
|
|
156
186
|
|
|
157
187
|
> ⚠️ **Advisory only.** The numbers below are directional heuristics based on single-source token-profile estimates and public pricing as of the skill's last update. They are **not** verified against real usage telemetry and should not be used as hard routing logic. Treat them as "this tier costs roughly this much more than that tier," not as commitments. Pricing, token profiles, and model output lengths all drift over time. Before using these ratios in any cost-minimization decision, pull actual usage data from the last 30 days of builds and re-derive the numbers for your workload.
|
|
158
188
|
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
|
162
|
-
|
|
163
|
-
| Sonnet 4.
|
|
164
|
-
| Sonnet
|
|
165
|
-
|
|
|
189
|
+
**Price ratios only** (T1, claude-api — per-MTok, in/out): Fable 5 $10/$50 · Opus 4.8 $5/$25 · Sonnet 5 $3/$15 ($2/$10 intro through 2026-08-31) · Haiku 4.5 $1/$5.
|
|
190
|
+
|
|
191
|
+
| Pair (same task) | Price ratio | Basis |
|
|
192
|
+
|---|---|---|
|
|
193
|
+
| Sonnet 5 vs Opus 4.8 | **~0.6x** | $3/$15 ÷ $5/$25 = 0.6 in & out. Sonnet 5 and Opus 4.8 share the **same** new tokenizer (T1), so token counts are comparable — the ratio is price-driven, not token-driven |
|
|
194
|
+
| Sonnet 5 vs Fable 5 | **~0.3x** | $3/$15 ÷ $10/$50 |
|
|
195
|
+
| Haiku 4.5 vs Sonnet 5 | **~0.33x** | $1/$5 ÷ $3/$15 |
|
|
196
|
+
|
|
197
|
+
> ⚠️ **These are price ratios, not measured task costs.** Real per-task cost also depends on token volume and effort — running Sonnet 5 at `high`/`xhigh` (the new default + escalation rung) raises its output-token count, narrowing the gap to Opus. A *measured* multiplier cannot yet be stated: the cost ledger (`~/.bookmark/cost-ledger.jsonl`) is currently dominated by local-model rows with no Anthropic build telemetry. **Re-derivation trigger:** once the ledger carries real Sonnet-5 + Opus build rows, pull the last 30 days and re-derive measured task-cost multipliers per workload.
|
|
198
|
+
>
|
|
199
|
+
> The retired "~0.3x single-pass Sonnet" figure was a Sonnet-**4.6** token-profile artifact ("70% fewer tokens than 4.5") and does **not** carry to Sonnet 5 — whose new tokenizer emits ~30% *more* tokens per unit text than 4.6. That ~30% delta matters only when comparing Sonnet 5 to the old 4.6 numbers, **not** to Opus 4.8 (shared tokenizer).
|
|
166
200
|
|
|
167
|
-
❓ Best-of-N + critic vs single-pass Opus on SWE-bench has not been directly benchmarked.
|
|
201
|
+
❓ Best-of-N + critic vs single-pass Opus on SWE-bench has not been directly benchmarked on Sonnet 5.
|
|
168
202
|
|
|
169
203
|
**How to convert these into routing decisions**: don't. Use the numbers to sanity-check a tier choice after the fact ("was this worth the 5x?"), not to justify forcing a model swap. When real telemetry disagrees with this table, trust telemetry and file an issue to update the table.
|
|
170
204
|
|
|
@@ -172,7 +206,7 @@ If the ambiguity surfaces a **planning** problem (the original plan no longer fi
|
|
|
172
206
|
|
|
173
207
|
**Fable plans (when stakes-gated) and verifies. Opus coordinates. Sonnet executes. Haiku recognizes.**
|
|
174
208
|
|
|
175
|
-
Phase 2 Plan synthesis reaches **Fable** through the **Advisor dispatch ladder** when stakes-gating trips (`synthesisDensity > 5`, `riskSurfaceChange`, `stakes >= medium`, or `dispatch_tier: frontier`): the orchestrator dispatches the `advisor` agent (Rung 1), routes to a peer host (Rung 2), or — if its own session is already Fable — synthesizes inline at Frontier (Rung 0). When no trigger fires or no dispatch path is reachable, the orchestrator synthesizes the plan **inline on its own model (Opus)** and labels it honestly (Rung 3 = today's behavior; the floor equals current state). So "Fable plans" is the *guarantee on high-stakes plans*, with an honestly-labeled inline fallback otherwise — not unconditional. Full protocol: `references/advisor-dispatch-ladder.md`. The Advisor frames the goal, drafts the spec/ADRs, sets F-criteria, and MECE-partitions the work. The orchestrator (**Opus**, `build-orchestrator`, `assessment-orchestrator`) coordinates: it routes dispatches, runs deterministic gates, manages parallel fan-out, walks the Advisor ladder, and handles the escalation ladder. Phase 3 implementer subagents run on **Sonnet** at effort=
|
|
209
|
+
Phase 2 Plan synthesis reaches **Fable** through the **Advisor dispatch ladder** when stakes-gating trips (`synthesisDensity > 5`, `riskSurfaceChange`, `stakes >= medium`, or `dispatch_tier: frontier`): the orchestrator dispatches the `advisor` agent (Rung 1), routes to a peer host (Rung 2), or — if its own session is already Fable — synthesizes inline at Frontier (Rung 0). When no trigger fires or no dispatch path is reachable, the orchestrator synthesizes the plan **inline on its own model (Opus)** and labels it honestly (Rung 3 = today's behavior; the floor equals current state). So "Fable plans" is the *guarantee on high-stakes plans*, with an honestly-labeled inline fallback otherwise — not unconditional. Full protocol: `references/advisor-dispatch-ladder.md`. The Advisor frames the goal, drafts the spec/ADRs, sets F-criteria, and MECE-partitions the work. The orchestrator (**Opus**, `build-orchestrator`, `assessment-orchestrator`) coordinates: it routes dispatches, runs deterministic gates, manages parallel fan-out, walks the Advisor ladder, and handles the escalation ladder. Phase 3 implementer subagents run on **Sonnet** at effort=high (default workhorse; xhigh on hard/code-heavy chunks) → external verification gate (tests/lint/types) → adversarial **Fable** verification surface (`plan-critic`, `scope-auditor`, `independent-auditor`, `fix-critique`, `fact-checker`, `security-reviewer`, `overfitting-reviewer`, `promotion-reviewer`). On a first execution-problem chunk failure, retry the same Sonnet 5 implementer at **effort=xhigh** (intermediate rung — near-Opus coding at lower cost) before paying for Opus; a strong-checkpoint finding or a 2nd consecutive failure after the xhigh retry escalates to **Opus** for judgment; if the failure traces back to a planning miss, route back to Fable to re-plan. See `agents/build-orchestrator.md §Escalation Triggers`. The **tier mapping** is the policy; the cost numbers above are advisory context, not the basis for overrides.
|
|
176
210
|
|
|
177
211
|
Haiku is only used for Phase 7B mock scanning and recurring-pattern detection across `runs[]`. Never for reasoning tasks.
|
|
178
212
|
|
|
@@ -184,11 +218,12 @@ Not every agent should hard-pin its model. Use this rule:
|
|
|
184
218
|
- **Inherit** (`model: inherit`) when user intent should flow through. The user's main-session choice is itself a cost/speed preference; respect it. Pair with a "recommended: X" note in this skill rather than forcing via frontmatter. Example: `root-cause-investigator` — recommended Opus on causal-tree work, but inherit honors whatever tier the user picked upstream.
|
|
185
219
|
- **Override mechanism**: users can override any pin by passing `model:` when spawning the agent or by editing the frontmatter. Pins are defaults, not locks. The deliberate exceptions documented above (`alignment-checker`, `synthesis-critic` on Sonnet despite being verification-shaped) are exactly this kind of cost-vs-judgment pin and can be lifted if telemetry says so.
|
|
186
220
|
|
|
187
|
-
Forward-compat note: pinned family aliases (`fable`, `sonnet`, `opus`) auto-track latest versions in their tier (e.g., Sonnet 4.6 →
|
|
221
|
+
Forward-compat note: pinned family aliases (`fable`, `sonnet`, `opus`) auto-track latest versions in their tier (e.g., Sonnet 4.6 → 5, Opus 4.7 → 4.8, Fable 5 → 6). `inherit` additionally picks up brand-new tiers (e.g., a future Flash-class model) without frontmatter edits.
|
|
188
222
|
|
|
189
223
|
## Limitations of this guidance
|
|
190
224
|
|
|
191
|
-
- ⚠️ Sonnet 4.6 token-efficiency claim is single-source (Anthropic announcement). Treat as directionally correct, not proven.
|
|
225
|
+
- ⚠️ Sonnet 4.6 token-efficiency claim is single-source (Anthropic announcement). Treat as directionally correct, not proven; superseded for the active default by Sonnet 5's new tokenizer (~30% more tokens/text, claude-api T1).
|
|
226
|
+
- ❓ No single published SWE-bench Verified figure for Sonnet 5 yet; the Code-tier capability claim rests on the T1 qualitative "reaches prior Opus-tier coding/agentic quality." Re-confirm with a benchmark when available.
|
|
192
227
|
- ❓ Best-of-N + critic hasn't been tested against single-pass Opus on SWE-bench specifically.
|
|
193
228
|
- ⚠️ Escalation triggers are heuristics, not proven thresholds. Revise after observing 5+ real builds and logging outcomes to `.build-loop/memory/`.
|
|
194
229
|
|
package/skills/optimize/SKILL.md
CHANGED
|
@@ -40,7 +40,7 @@ Highest-leverage phase. Wrong metric = Goodhart's Law. Wrong factors = wasted ru
|
|
|
40
40
|
|---|---|---|
|
|
41
41
|
| **A. Power-user explicit** | User supplied factors via CLI flag, `.build-loop/optimize/factors.json`, or inline ("optimize batch_size, retries, workers for throughput") | Skip suggestion; use the user's factors directly |
|
|
42
42
|
| **B. Vague optimization** *(default)* | "run optimization", "make my app faster", "improve performance", "speed up", "reduce <metric>" without naming factors | Run factor-identification scan; propose candidates; **AskUserQuestion to confirm before running** |
|
|
43
|
-
| **C. Single-variable explicit** | "simplify this file", "reduce build time", scoped `/build-loop:
|
|
43
|
+
| **C. Single-variable explicit** | "simplify this file", "reduce build time", a scoped optimize request via `/build-loop:run <known-target>` | Skip DOE; run autoresearch (existing behavior, Phase 2 LOOP unchanged) |
|
|
44
44
|
|
|
45
45
|
### Step 1.2 — Branch A or B: factor identification
|
|
46
46
|
|
|
@@ -186,7 +186,7 @@ Dispatch the `optimize-runner` agent. It executes:
|
|
|
186
186
|
|
|
187
187
|
Phase 4.7 (AUTO-OPTIMIZE): after Phase 4 Execute completes and commits, check for optimization targets. Run sequentially (not parallel with Phase 4).
|
|
188
188
|
|
|
189
|
-
|
|
189
|
+
Reached via `/build-loop:run [target]` + optimize language (no separate command).
|
|
190
190
|
|
|
191
191
|
## State Files
|
|
192
192
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: build-loop:prd-bridge
|
|
3
|
-
description: Use when Phase 1 Assess runs, the user mentions a PRD, or asks to "ground the build in product strategy". Surfaces always-true principles and Navigation Map from the project PRD so subsequent phases reason from strategic frame; recommends prd-builder if none exists.
|
|
3
|
+
description: Use when Phase 1 Assess runs, the user mentions a PRD, or asks to "ground the build in product strategy". Surfaces always-true principles and Navigation Map from the project PRD so subsequent phases reason from strategic frame; recommends prd-builder if none exists. NOT for authoring a PRD from scratch (use `prd-builder`), API registration (use `api-registry-bridge`), or UI verification (use `ibr-bridge`).
|
|
4
4
|
version: 0.1.0
|
|
5
5
|
user-invocable: false
|
|
6
6
|
---
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: recursive-retrospective
|
|
3
|
+
description: Run a recursive-learning retrospective on an app/agent/plugin/build-loop project — analyze build history, behavior, and current state to extract reusable learning objects and encode them into memory, agents, skills, plugins, evals, preflights, and approval gates. Use after a substantial build, when reviewing a project's trajectory, when deciding preserve/refine/redirect/reset, or when converting a session into durable system improvement. Discovery-first (observe behavior → cluster patterns → encode learning); RCA is one module, not the whole job.
|
|
4
|
+
user-invocable: false
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Recursive Learning Retrospective
|
|
8
|
+
|
|
9
|
+
A three-stage pipeline that turns a project's history into durable, encoded system improvement. Discovery-first: observe revealed behavior, cluster patterns, then encode learning — categories are seed scaffolds, never closed taxonomies (preserve emergent findings).
|
|
10
|
+
|
|
11
|
+
> **v2 (2026-06-18) — test-validated.** This revision was hardened against a live run: the full pipeline was executed on a real project (ross-labs-astro) and scored by an independent judge at 4.27/5, "Accept with revisions." The judge's penalties are fixed here (see Changelog). The v1 structure (peer draft `feat/recursive-retrospective@1c24d4a`) is preserved; only the validated fixes are layered on.
|
|
12
|
+
|
|
13
|
+
## When to use
|
|
14
|
+
- After a substantial build/session, to extract what the system should learn.
|
|
15
|
+
- When deciding whether to preserve / refine / redirect / reset a mid-build project.
|
|
16
|
+
- To convert a retrospective into concrete learning objects (memory, agent rules, skills, plugin/app behavior, evals, preflights, approval gates).
|
|
17
|
+
- NOT only for failures — RCA is one diagnostic module used when there are failures, regressions, steering loops, brittle decisions, or outcome gaps.
|
|
18
|
+
|
|
19
|
+
## The pipeline
|
|
20
|
+
1. **Run** — `references/01-retrospective.md` (Prompt 1): the recursive-learning retrospective. Produces maturity classification, spec→current→desired comparison, behavior/workflow discovery, steering mining, learning objects, a selective RCA module, preflight/hard-gate analysis, a counterfactual simulation, a prioritized (ordinal) roadmap, layered recommendations, a from-scratch option, emergent patterns, and an executive summary.
|
|
21
|
+
2. **Capture** — `references/02-learning-capture.md` (Prompt 2): a PACKAGER/DEDUPER of Prompt 1's §7 + §12. It does not re-extract from scratch; it dedupes the learning objects Prompt 1 already produced, adds only net-new ones it missed (flagged), and emits the copy-ready encoding package.
|
|
22
|
+
3. **Judge** — `references/03-judge.md` (Prompt 3): an INDEPENDENT evaluator scores the retrospective on 11 criteria and returns accept / accept-with-revisions / reject. It MUST verify headline claims against source before scoring evidence grounding.
|
|
23
|
+
|
|
24
|
+
## Operating rules (carried through all three stages)
|
|
25
|
+
- **Discovery flexibility:** start from evidence before classification; add/rename/split/merge categories when evidence requires; use Other/Emergent; preserve unexpected findings.
|
|
26
|
+
- **Density governor:** reward density, not completeness. Collapse a low-signal section to a single line; omit a conditional section entirely when it yields no non-obvious finding. A shorter retrospective that drops empty scaffolding scores HIGHER than an exhaustive one. Do not pad a section to look complete.
|
|
27
|
+
- **Cross-reference, don't restate:** assign each finding a stable id (LO-1, RCA-1, …) once, then reference the id in later sections instead of repeating the finding.
|
|
28
|
+
- **Evidence discipline:** separate explicit statements from revealed behavior; mark unknowns `UNKNOWN — evidence not available` and unverified claims `TAG:INFERRED`.
|
|
29
|
+
- **Memory discipline:** do not encode one-off comments as durable memory unless explicitly framed durable or recurring.
|
|
30
|
+
- **Appropriate autonomy + approval authority:** the dispatching (main) agent holds approval for this pipeline's own outputs — SAFE, reversible encodes (memory / note / skill-source / agent-instruction) auto-persist under that authority and are never bounced to the human. Reserve human gates for the hard-gate taxonomy only: security, privacy, cost, live/production deploy, irreversible/destructive actions, or promoting an experimental artifact into a globally runtime-active skill/agent.
|
|
31
|
+
- **No false precision:** rank by ordinal priority (P0/P1/P2) with qualitative justification; do not emit multiplied point-scores that imply a calibration the inputs cannot support.
|
|
32
|
+
|
|
33
|
+
## Model tiering
|
|
34
|
+
- Stage 1 (retrospective = assessment) and Stage 3 (judge = verification verdict) are Frontier-tier work (Fable) per the model org. Stage 3 MUST run in a context independent of Stage 1's author (external evaluator). Frontier-unavailable → Thinking-tier (Opus) fallback, never Code tier.
|
|
35
|
+
- Stage 2 (capture) can run at the executor tier.
|
|
36
|
+
|
|
37
|
+
## Output homes
|
|
38
|
+
- Retrospective + learning-object package → the project's retrospective lane and (for cross-project learning) build-loop-memory via the canonical `memory_writer.py`.
|
|
39
|
+
- Learning objects with encoding target = memory / project-note / skill-source / agent-instruction are SAFE + reversible: the **dispatching (main) agent approves them and they persist automatically** — do not bounce a medium-confidence memory note to the human. Human approval is reserved for the hard-gate taxonomy only (security / privacy / cost / live-deploy / irreversible-destructive / promoting an experimental artifact into a globally runtime-active skill or agent).
|
|
40
|
+
|
|
41
|
+
## Changelog (v1 → v2; each fix traces to the live judge run)
|
|
42
|
+
- **Density governor + conditional sections** — judge flagged §11 (counterfactual) and the near-1:1 §5 behavior→LO mapping as forced-completeness padding. §11 and parts of §5/§15 are now conditional; the density rule is explicit in the operating rules and acceptance criteria.
|
|
43
|
+
- **Ordinal priority replaces the multiply formula** — judge flagged `(F×I×R×C)/Difficulty` point-scores (250, 200, 160) as false precision. §12 now ranks P0/P1/P2 with F/I/R/C/Difficulty kept as qualitative justification columns only.
|
|
44
|
+
- **Cross-reference rule** — judge flagged §5/§6/§7/§15 reformatting the same findings 3–4×. Findings now carry ids and are referenced, not restated.
|
|
45
|
+
- **Prompt 2 reframed as packager/deduper** — Prompt 1 §7 already emits a full learning-object table; v1 Prompt 2 re-extracted from scratch (double work, divergence risk). v2 Prompt 2 ingests §7+§12 and dedupes.
|
|
46
|
+
- **Prompt 3 mandates verification** — in the live test the judge only verified facts because it was told to. v2 §1 (evidence grounding) requires independent tool-verification of ≥3 headline claims when source is available, and caps the grounding score at 3 if verification was possible but skipped.
|