forgedock 1.0.16 → 1.0.18

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (190) hide show
  1. package/README.md +54 -3
  2. package/bin/cross-publish.mjs +344 -0
  3. package/bin/forgedock.mjs +2205 -541
  4. package/bin/github-app-manifest.json.license +2 -0
  5. package/bin/hooks/session-start.mjs +4 -5
  6. package/bin/registry.mjs +20 -0
  7. package/bin/tui.mjs +172 -96
  8. package/commands/adopt.md +433 -0
  9. package/commands/analytics.md +2 -0
  10. package/commands/audit-agents.md +2 -0
  11. package/commands/audit.md +2 -0
  12. package/commands/autopilot.md +77 -0
  13. package/commands/changelog.md +354 -0
  14. package/commands/cleanup.md +2 -0
  15. package/commands/compat-audit.md +330 -0
  16. package/commands/deploy-info.md +6 -5
  17. package/commands/design-bench.md +176 -0
  18. package/commands/design-render-critique-loop.md +103 -0
  19. package/commands/design.md +259 -0
  20. package/commands/diagnose.md +483 -0
  21. package/commands/explain.md +349 -0
  22. package/commands/forge-stats.md +2 -0
  23. package/commands/forgedock-init.md +2 -0
  24. package/commands/geo-audit.md +2 -0
  25. package/commands/incident-response.md +2 -0
  26. package/commands/issue.md +9 -3
  27. package/commands/milestone.md +2 -0
  28. package/commands/optimize.md +411 -0
  29. package/commands/orchestrate.md +239 -0
  30. package/commands/pipeline-health.md +60 -1
  31. package/commands/qa-sweep.md +39 -2
  32. package/commands/quality-gate.md +67 -0
  33. package/commands/replay.md +263 -0
  34. package/commands/resume.md +264 -0
  35. package/commands/review-pr-agents.md +5 -1
  36. package/commands/review-pr-staging.md +160 -2
  37. package/commands/review-pr.md +85 -3
  38. package/commands/rollback.md +2 -0
  39. package/commands/scope.md +280 -0
  40. package/commands/security-audit.md +2 -0
  41. package/commands/status.md +262 -0
  42. package/commands/sync-ecosystem.md +2 -0
  43. package/commands/test-gate.md +929 -0
  44. package/commands/validate.md +2 -0
  45. package/commands/work-on/build/architect.md +128 -18
  46. package/commands/work-on/build/context.md +130 -17
  47. package/commands/work-on/build/implement.md +7 -3
  48. package/commands/work-on/build/validate.md +2 -0
  49. package/commands/work-on/build.md +32 -7
  50. package/commands/work-on/close.md +85 -0
  51. package/commands/work-on/decompose.md +2 -0
  52. package/commands/work-on/investigate.md +30 -3
  53. package/commands/work-on/review.md +13 -1
  54. package/commands/work-on-monolithic.md +2 -0
  55. package/commands/work-on.md +676 -38
  56. package/docs/BRAND.md +51 -0
  57. package/docs/CI.md +177 -0
  58. package/docs/CONFIG.md +94 -0
  59. package/docs/FORGE-PROTOCOL.md +324 -0
  60. package/docs/articles/b-vs-c-baseline.md +210 -0
  61. package/docs/articles/full-abc-benchmark.md +205 -0
  62. package/docs/articles/per-repo-adaptive-scripts-token-savings.md +174 -0
  63. package/docs/articles/selective-spec-loading-token-savings.md +140 -0
  64. package/docs/claude-breakpoints.json +47 -0
  65. package/docs/design/abc-benchmark.md +195 -0
  66. package/docs/design/design-architect-rationale.md +123 -0
  67. package/docs/design/design-memory.md +61 -0
  68. package/docs/design/design-spec-schema.md +248 -0
  69. package/docs/design/design-system-lint.md +150 -0
  70. package/docs/design/divergent-generation.md +106 -0
  71. package/docs/design/effects-appropriateness.md +133 -0
  72. package/docs/design/fixtures/briefs/README.md +47 -0
  73. package/docs/design/fixtures/briefs/cadence.md +39 -0
  74. package/docs/design/fixtures/briefs/plume.md +39 -0
  75. package/docs/design/fixtures/briefs/slipstream.md +39 -0
  76. package/docs/design/fixtures/briefs/tender.md +39 -0
  77. package/docs/design/fixtures/briefs/voltage.md +39 -0
  78. package/docs/design/fixtures/runs/b-vs-c-baseline/cadence/arm-b/run-1.html +364 -0
  79. package/docs/design/fixtures/runs/b-vs-c-baseline/cadence/arm-b/run-2.html +420 -0
  80. package/docs/design/fixtures/runs/b-vs-c-baseline/cadence/arm-b/run-3.html +506 -0
  81. package/docs/design/fixtures/runs/b-vs-c-baseline/plume/arm-b/run-1.html +611 -0
  82. package/docs/design/fixtures/runs/b-vs-c-baseline/plume/arm-b/run-2.html +594 -0
  83. package/docs/design/fixtures/runs/b-vs-c-baseline/plume/arm-b/run-3.html +817 -0
  84. package/docs/design/fixtures/runs/b-vs-c-baseline/runs.json +175 -0
  85. package/docs/design/fixtures/runs/b-vs-c-baseline/scorecard.json +138 -0
  86. package/docs/design/fixtures/runs/b-vs-c-baseline/slipstream/arm-b/run-1.html +373 -0
  87. package/docs/design/fixtures/runs/b-vs-c-baseline/slipstream/arm-b/run-2.html +398 -0
  88. package/docs/design/fixtures/runs/b-vs-c-baseline/slipstream/arm-b/run-3.html +595 -0
  89. package/docs/design/fixtures/runs/b-vs-c-baseline/tender/arm-b/run-1.html +531 -0
  90. package/docs/design/fixtures/runs/b-vs-c-baseline/tender/arm-b/run-2.html +565 -0
  91. package/docs/design/fixtures/runs/b-vs-c-baseline/tender/arm-b/run-3.html +674 -0
  92. package/docs/design/fixtures/runs/b-vs-c-baseline/voltage/arm-b/run-1.html +388 -0
  93. package/docs/design/fixtures/runs/b-vs-c-baseline/voltage/arm-b/run-2.html +463 -0
  94. package/docs/design/fixtures/runs/b-vs-c-baseline/voltage/arm-b/run-3.html +568 -0
  95. package/docs/design/fixtures/runs/full-abc/cadence/arm-a/run-1.html +562 -0
  96. package/docs/design/fixtures/runs/full-abc/cadence/arm-a/run-2.html +618 -0
  97. package/docs/design/fixtures/runs/full-abc/cadence/arm-a/run-3.html +618 -0
  98. package/docs/design/fixtures/runs/full-abc/cadence/arm-b/run-1.html +364 -0
  99. package/docs/design/fixtures/runs/full-abc/cadence/arm-b/run-2.html +420 -0
  100. package/docs/design/fixtures/runs/full-abc/cadence/arm-b/run-3.html +506 -0
  101. package/docs/design/fixtures/runs/full-abc/plume/arm-a/run-1.html +527 -0
  102. package/docs/design/fixtures/runs/full-abc/plume/arm-a/run-2.html +445 -0
  103. package/docs/design/fixtures/runs/full-abc/plume/arm-a/run-3.html +483 -0
  104. package/docs/design/fixtures/runs/full-abc/plume/arm-b/run-1.html +611 -0
  105. package/docs/design/fixtures/runs/full-abc/plume/arm-b/run-2.html +594 -0
  106. package/docs/design/fixtures/runs/full-abc/plume/arm-b/run-3.html +817 -0
  107. package/docs/design/fixtures/runs/full-abc/runs.json +190 -0
  108. package/docs/design/fixtures/runs/full-abc/scorecard.json +144 -0
  109. package/docs/design/fixtures/runs/full-abc/slipstream/arm-a/run-1.html +496 -0
  110. package/docs/design/fixtures/runs/full-abc/slipstream/arm-a/run-2.html +447 -0
  111. package/docs/design/fixtures/runs/full-abc/slipstream/arm-a/run-3.html +564 -0
  112. package/docs/design/fixtures/runs/full-abc/slipstream/arm-b/run-1.html +373 -0
  113. package/docs/design/fixtures/runs/full-abc/slipstream/arm-b/run-2.html +398 -0
  114. package/docs/design/fixtures/runs/full-abc/slipstream/arm-b/run-3.html +595 -0
  115. package/docs/design/fixtures/runs/full-abc/tender/arm-a/run-1.html +622 -0
  116. package/docs/design/fixtures/runs/full-abc/tender/arm-a/run-2.html +475 -0
  117. package/docs/design/fixtures/runs/full-abc/tender/arm-a/run-3.html +660 -0
  118. package/docs/design/fixtures/runs/full-abc/tender/arm-b/run-1.html +531 -0
  119. package/docs/design/fixtures/runs/full-abc/tender/arm-b/run-2.html +565 -0
  120. package/docs/design/fixtures/runs/full-abc/tender/arm-b/run-3.html +674 -0
  121. package/docs/design/fixtures/runs/full-abc/voltage/arm-a/run-1.html +525 -0
  122. package/docs/design/fixtures/runs/full-abc/voltage/arm-a/run-2.html +446 -0
  123. package/docs/design/fixtures/runs/full-abc/voltage/arm-a/run-3.html +463 -0
  124. package/docs/design/fixtures/runs/full-abc/voltage/arm-b/run-1.html +388 -0
  125. package/docs/design/fixtures/runs/full-abc/voltage/arm-b/run-2.html +463 -0
  126. package/docs/design/fixtures/runs/full-abc/voltage/arm-b/run-3.html +568 -0
  127. package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-a/run-1.html +1068 -0
  128. package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-a/run-2.html +1423 -0
  129. package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-a/run-3.html +1245 -0
  130. package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-b/run-1.html +305 -0
  131. package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-b/run-2.html +327 -0
  132. package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-b/run-3.html +428 -0
  133. package/docs/design/fixtures/runs/full-abc-opus/plume/arm-a/run-1.html +1270 -0
  134. package/docs/design/fixtures/runs/full-abc-opus/plume/arm-a/run-2.html +1172 -0
  135. package/docs/design/fixtures/runs/full-abc-opus/plume/arm-a/run-3.html +1532 -0
  136. package/docs/design/fixtures/runs/full-abc-opus/plume/arm-b/run-1.html +315 -0
  137. package/docs/design/fixtures/runs/full-abc-opus/plume/arm-b/run-2.html +342 -0
  138. package/docs/design/fixtures/runs/full-abc-opus/plume/arm-b/run-3.html +353 -0
  139. package/docs/design/fixtures/runs/full-abc-opus/runs.json +190 -0
  140. package/docs/design/fixtures/runs/full-abc-opus/scorecard.json +144 -0
  141. package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-a/run-1.html +345 -0
  142. package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-a/run-2.html +396 -0
  143. package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-a/run-3.html +502 -0
  144. package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-b/run-1.html +408 -0
  145. package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-b/run-2.html +446 -0
  146. package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-b/run-3.html +496 -0
  147. package/docs/design/fixtures/runs/full-abc-opus/tender/arm-a/run-1.html +813 -0
  148. package/docs/design/fixtures/runs/full-abc-opus/tender/arm-a/run-2.html +711 -0
  149. package/docs/design/fixtures/runs/full-abc-opus/tender/arm-a/run-3.html +721 -0
  150. package/docs/design/fixtures/runs/full-abc-opus/tender/arm-b/run-1.html +280 -0
  151. package/docs/design/fixtures/runs/full-abc-opus/tender/arm-b/run-2.html +291 -0
  152. package/docs/design/fixtures/runs/full-abc-opus/tender/arm-b/run-3.html +366 -0
  153. package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-a/run-1.html +1010 -0
  154. package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-a/run-2.html +836 -0
  155. package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-a/run-3.html +930 -0
  156. package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-b/run-1.html +407 -0
  157. package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-b/run-2.html +396 -0
  158. package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-b/run-3.html +529 -0
  159. package/docs/design/foundations/brutalist.css +294 -0
  160. package/docs/design/foundations/editorial.css +300 -0
  161. package/docs/design/foundations/minimal-luxury.css +295 -0
  162. package/docs/design/foundations/technical.css +293 -0
  163. package/docs/design/foundations/warm-photo.css +299 -0
  164. package/docs/design/reference-corpus.md +1726 -0
  165. package/docs/design/render-critique-loop.md +57 -0
  166. package/docs/site/getting-started.md +21 -0
  167. package/docs/spec-graph-schema.md +397 -0
  168. package/package.json +3 -3
  169. package/scripts/bench-scorecard.mjs +239 -0
  170. package/scripts/build-spec-graph.mjs +509 -0
  171. package/scripts/classify-lane.sh +138 -0
  172. package/scripts/design-system-lint.mjs +993 -0
  173. package/scripts/graph-query.sh +486 -0
  174. package/scripts/transition-label.sh +169 -0
  175. package/scripts/validate-pr-target.sh +30 -0
  176. package/scripts/validate-spec-graph.sh +423 -0
  177. package/templates/devdocs/README.md +72 -0
  178. package/templates/devdocs/agent/using-forgedock.md +3 -0
  179. package/templates/devdocs/agent/using-github.md +3 -0
  180. package/templates/devdocs/index.yaml +162 -0
  181. package/templates/devdocs/project/architecture.md +3 -0
  182. package/templates/devdocs/project/conventions.md +3 -0
  183. package/templates/devdocs/project/custom-instructions.md +3 -0
  184. package/templates/devdocs/project/glossary.md +3 -0
  185. package/templates/devdocs/project/stack.md +3 -0
  186. package/templates/devdocs/templates/commit-convention.md +3 -0
  187. package/templates/devdocs/templates/issue.md +3 -0
  188. package/templates/devdocs/templates/pull-request.md +3 -0
  189. package/templates/devdocs/templates/review-finding.md +3 -0
  190. package/templates/workflows/forgedock-review.yml +96 -0
@@ -0,0 +1,140 @@
1
+ ---
2
+ title: "Selective Spec Loading: Measured Token Savings From a Self-Map of Command Specs"
3
+ description: "ForgeDock symlinks ~27 command specs (~1.1 MB) into every Claude Code session. We built a queryable graph of the specs and load only the ones a task actually reaches — and measured how many tokens that saves."
4
+ tags: ["ai", "performance", "tokens", "productivity", "opensource"]
5
+ cover_image:
6
+ canonical_url: https://github.com/RapierCraftStudios/ForgeDock
7
+ published: false
8
+ ---
9
+
10
+ <!-- Generated by ForgeDock pipeline — Issue #868 -->
11
+
12
+ AI coding agents pay for context by the token. ForgeDock's pipeline is driven by
13
+ markdown command specs — `work-on`, `review-pr`, `quality-gate`, and ~40 more —
14
+ that are symlinked into `~/.claude/commands/` and therefore loaded into the model's
15
+ context at session start, whether or not a given run ever touches them. This
16
+ article measures that cost and quantifies how a **self-map of the specs** lets a
17
+ run load only the specs it actually reaches.
18
+
19
+ ---
20
+
21
+ ## The Problem: The Whole Corpus Loads Even Though One Command Runs
22
+
23
+ A single ForgeDock run invokes exactly one entry command — usually `work-on`.
24
+ That command reaches a small, well-defined set of downstream specs: its own
25
+ sub-phases (`work-on/build/*`, `work-on/review`, `work-on/close`) and the review
26
+ orchestrator it delegates to (`review-pr`). It never reads `pipeline-health.md`,
27
+ `geo-audit.md`, `audit.md`, `incident-response.md`, or the two dozen other
28
+ specs that have nothing to do with the issue at hand.
29
+
30
+ But all of them are present in context. Measured on the current spec set:
31
+
32
+ | Metric | Value |
33
+ |--------|-------|
34
+ | Command spec files | 44 |
35
+ | Total bytes | 1,104,004 |
36
+ | Approx. tokens (1 token ≈ 4 chars) | **~276,000** |
37
+
38
+ That is ~276K tokens of spec text loaded on every session — the overwhelming
39
+ majority of which is dead weight for any single task.
40
+
41
+ ---
42
+
43
+ ## The Mechanism: A Queryable Graph of the Specs
44
+
45
+ ForgeDock already builds a **spec knowledge graph** — a zero-dependency self-map
46
+ of its own command specs (`scripts/build-spec-graph.mjs`, queried by
47
+ `scripts/graph-query.sh`). The graph records which command *contains* which
48
+ sub-phase, which command *requires* which devdoc, which annotation a command
49
+ *reads* or *writes*, and so on.
50
+
51
+ The graph already answered "what is the blast radius if I change node X?" via its
52
+ `impact` query (transitive **reverse** reachability). Selective loading needs the
53
+ mirror image: the **forward** transitive reachable set of a command. We added a
54
+ `load-set` subcommand that does exactly that:
55
+
56
+ ```bash
57
+ $ graph-query.sh load-set work-on
58
+ ["commands/review-pr.md","commands/work-on.md","commands/work-on/build.md",
59
+ "commands/work-on/build/architect.md","commands/work-on/build/context.md",
60
+ "commands/work-on/build/implement.md","commands/work-on/build/validate.md",
61
+ "commands/work-on/close.md","commands/work-on/decompose.md",
62
+ "commands/work-on/investigate.md","commands/work-on/review.md"]
63
+ ```
64
+
65
+ `load-set` seeds at the command node and walks `CONTAINS` (sub-phases) and
66
+ `REQUIRES` (devdocs) edges forward to a fixpoint, then maps the reachable nodes to
67
+ their repo-relative file paths. It is read-only, auto-builds the graph if the
68
+ gitignored JSON is absent, and shares the existing `impact` resolver's cycle-safe
69
+ termination. No graph logic was reimplemented — the feature *consumes* the
70
+ builder and query script that were already there.
71
+
72
+ `work-on` Phase 0 now calls `load-set work-on` and reads **only** the returned
73
+ paths instead of relying on the whole symlinked corpus.
74
+
75
+ ---
76
+
77
+ ## Methodology
78
+
79
+ - Token counts are estimated at the standard rate of **1 token ≈ 4 characters**.
80
+ - **Baseline (load-all)** = the byte sum of every spec under `commands/`.
81
+ - **Selective (load-set)** = `wc -c` of just the files returned by
82
+ `graph-query.sh load-set <command>` for the entry command of a session.
83
+ - Measured across **five representative sessions**, each keyed by its entry command.
84
+
85
+ This mirrors the methodology used in the per-repo adaptive-scripts measurement,
86
+ so the numbers are directly comparable.
87
+
88
+ ---
89
+
90
+ ## Baseline
91
+
92
+ Every session, with the load-all corpus:
93
+
94
+ | | Files | Bytes | Approx. tokens |
95
+ |--|------:|------:|---------------:|
96
+ | **Load-all corpus** | 44 | 1,104,004 | ~276,000 |
97
+
98
+ ---
99
+
100
+ ## Target: Per-Session Selective Load
101
+
102
+ | Session (entry command) | Files | Bytes | Approx. tokens | Tokens saved vs baseline | % saved |
103
+ |-------------------------|------:|------:|---------------:|-------------------------:|--------:|
104
+ | `work-on` (full feature build) | 11 | 296,172 | ~74,000 | ~202,000 | **73%** |
105
+ | `orchestrate` (multi-issue) | 14 | 390,054 | ~98,000 | ~178,000 | **65%** |
106
+ | `review-pr` (PR review) | 1 | 64,395 | ~16,000 | ~260,000 | **94%** |
107
+ | `quality-gate` (pre-commit) | 1 | 47,676 | ~12,000 | ~264,000 | **96%** |
108
+ | `issue` (fast-lane triage) | 1 | 19,945 | ~5,000 | ~271,000 | **98%** |
109
+
110
+ ---
111
+
112
+ ## Savings
113
+
114
+ Across the five representative sessions, selective loading removes between
115
+ **~178,000 and ~271,000 tokens** of spec text from context — a **65%–98%**
116
+ reduction depending on how much of the pipeline the entry command actually
117
+ reaches. The heaviest run (`orchestrate`, which fans out to many sub-phases)
118
+ still drops nearly two-thirds of the corpus; the lightest (`issue`) drops 98%.
119
+
120
+ The win scales with breadth: the more specialized commands ForgeDock grows, the
121
+ larger the dead-weight fraction the load-all corpus carries — and the more a run
122
+ saves by loading only what its graph neighborhood reaches. Because the resolution
123
+ is graph-driven and deterministic (the transitive walk lives in `jq`, not in the
124
+ prompt), adding a new command automatically narrows the set for every *other*
125
+ command, with no manual bookkeeping.
126
+
127
+ ---
128
+
129
+ ## How To Reproduce
130
+
131
+ ```bash
132
+ # Baseline: full corpus
133
+ find commands -name '*.md' -type f -exec cat {} + | wc -c
134
+
135
+ # Selective: minimal set for a given entry command
136
+ graph-query.sh load-set work-on | jq -r '.[]' \
137
+ | while read -r p; do cat "$p"; done | wc -c
138
+ ```
139
+
140
+ Divide bytes by 4 for the approximate token count.
@@ -0,0 +1,47 @@
1
+ {
2
+ "_comment": "claude-breakpoints.json — structured registry of known Claude Code version breakpoints. Each entry documents a version where Claude Code changed behavior that ForgeDock depends on, introduced a feature ForgeDock leverages, or broke a capability ForgeDock uses. Format is append-only and human-editable. Consumers: detectClaudeVersion() (issue #680), /compat-audit (issue #686), forgedock doctor (issue #682).",
3
+ "schema": {
4
+ "version": "Claude Code version string where the breakpoint was introduced (e.g. '2.1.157')",
5
+ "type": "Category of change: 'feature' | 'behavior_change' | 'compatibility' | 'deprecation'",
6
+ "description": "Human-readable description of what changed in Claude Code at this version",
7
+ "forgedock_impact": "How this change affects ForgeDock — what commands, scripts, or behaviors are affected",
8
+ "severity": "Impact level if a user is on this version or older: 'high' | 'medium' | 'low' | 'info'"
9
+ },
10
+ "breakpoints": [
11
+ {
12
+ "version": "2.1.157",
13
+ "type": "behavior_change",
14
+ "description": "EnterWorktree now unlocks worktrees automatically when the agent session finishes",
15
+ "forgedock_impact": "ForgeDock no longer needs --force on worktree cleanup commands. The cleanup phase in work-on.md can use standard git worktree remove without the force flag.",
16
+ "severity": "low"
17
+ },
18
+ {
19
+ "version": "2.1.157",
20
+ "type": "feature",
21
+ "description": "Plugins in .claude/skills are auto-loaded without requiring marketplace installation",
22
+ "forgedock_impact": "Symlink-based install (the original forgedock install method) remains valid but is no longer required for skill discovery. The stub file install pattern introduced in #619 is the preferred approach and works the same way.",
23
+ "severity": "info"
24
+ },
25
+ {
26
+ "version": "2.1.163",
27
+ "type": "compatibility",
28
+ "description": "requiredMinimumVersion and requiredMaximumVersion managed settings added for enterprise deployments",
29
+ "forgedock_impact": "ForgeDock's own version check (forgedock doctor, #682) is the OSS equivalent of this enterprise feature. No conflict — enterprise users with managed settings may have stricter version constraints than ForgeDock requires.",
30
+ "severity": "info"
31
+ },
32
+ {
33
+ "version": "2.1.172",
34
+ "type": "feature",
35
+ "description": "Sub-agents can spawn sub-agents up to 5 levels deep (increased nesting depth)",
36
+ "forgedock_impact": "ForgeDock's orchestrate.md uses multi-agent orchestration. The 5-level nesting limit is compatible with ForgeDock's orchestration patterns. No changes needed.",
37
+ "severity": "info"
38
+ },
39
+ {
40
+ "version": "2.1.154",
41
+ "type": "compatibility",
42
+ "description": "Native /workflows dynamic workflows introduced as a built-in Claude Code feature",
43
+ "forgedock_impact": "Parallel to ForgeDock's orchestrate.md. ForgeDock's approach adds GitHub persistence (FORGE: annotations, label-state machine, compaction resilience) that native /workflows lacks. Both can coexist — ForgeDock is the orchestration layer with state, not a replacement for native workflows.",
44
+ "severity": "info"
45
+ }
46
+ ]
47
+ }
@@ -0,0 +1,195 @@
1
+ # The ABC Benchmark — Methodology
2
+
3
+ > **Status:** Foundation spec for issue #878 (milestone #13, UI Taste Harness).
4
+ > The benchmark is the milestone's **fitness function** — built *before* the harness. It is what tells us
5
+ > whether each lever the harness adds actually moves design quality. Harness development reduces to one
6
+ > question this rig answers: *"did arm A's score against C go up?"*
7
+ >
8
+ > Companion to [design-spec-schema](design-spec-schema.md) (#881), [reference-corpus](reference-corpus.md) (#880),
9
+ > and the [`/design-bench`](../../commands/design-bench.md) command spec.
10
+ > Results are carried as a `FORGE:BENCH_SCORECARD` annotation — registered in
11
+ > [`../FORGE-PROTOCOL.md`](../FORGE-PROTOCOL.md).
12
+
13
+ ## Why this exists first
14
+
15
+ A taste benchmark is not a test we run at the end. It is the reward signal. Without it, every "improvement" to
16
+ the harness is a guess. With it, each lever (archetype sampling, anti-slop negatives, the critique loop) is a
17
+ measurable hypothesis: add the lever, re-run the benchmark, confirm arm A's win-rate against C went up. Build the
18
+ ruler before building the thing it measures.
19
+
20
+ ## The three arms — same input, one variable
21
+
22
+ The **input is a single design-blind product brief** derived from a real reference page. It describes only what the
23
+ product *does* — no "make it like X", no colors, no type, no layout hints. The real product name is **anonymized**
24
+ so the generation model cannot recall the real brand's design from training. (Brand recall would measure memory,
25
+ not taste, and would unfairly inflate arm B — see [Anonymization](#anonymization-is-load-bearing).)
26
+
27
+ | Arm | What it is | Role |
28
+ |-----|------------|------|
29
+ | **A** | **ForgeDock harness** — the brief run through the `/design` pipeline (#888). | The thing under test. |
30
+ | **B** | **Raw frontier model**, one shot, no tools, no iteration, given the same brief. | Control / lower bound. |
31
+ | **C** | **The real reference page** the brief was stripped from. | Gold standard + judge calibration. |
32
+
33
+ **The one variable rule.** Arms A and B MUST use the **same generation model**. The harness (its pipeline,
34
+ structured design language, render→critique loop) is then the *only* difference between A and B — which is exactly
35
+ what makes the A-vs-B delta attributable to the harness. If A and B use different models, you are benchmarking
36
+ models, not the harness. This is non-negotiable.
37
+
38
+ ## C does double duty
39
+
40
+ C is both the gold standard and a **free validity check on the judge**:
41
+
42
+ - As gold standard, win-rate against C is the primary metric for A and B.
43
+ - As calibration, if the judge ever ranks C **below** A or B in the blind pairwise comparison, the judge is
44
+ miscalibrated — its result for that run is suspect and flagged in the scorecard. A real, professionally designed,
45
+ funded-SaaS page losing to a one-shot generation is a judge defect, not a harness triumph.
46
+
47
+ ## Rendering — kill the confounds
48
+
49
+ All three arms are rendered through the **identical** pipeline so the only thing the judge compares is design, not
50
+ rendering artifacts.
51
+
52
+ - **Self-contained HTML** for arms A and B (Tailwind CDN is fine; no build step).
53
+ - **Playwright** (milestone render dependency) renders all three at:
54
+ - **1440px** desktop and **390px** mobile,
55
+ - `networkidle` + fonts loaded before capture,
56
+ - **full-page** screenshot.
57
+ - **C is captured through the same pipeline.** Trim cookie banners and hero video so C is compared on its design,
58
+ not on a consent modal. Document any trim applied to C in the run record for reproducibility.
59
+
60
+ ## Judging — three layers
61
+
62
+ The judge is a **named, independent stage**. See [Anti-Goodhart](#anti-goodhart-judge-independence) — it MUST NOT
63
+ be the harness's own critique loop (#882).
64
+
65
+ ### Layer 1 — Blind randomized pairwise (primary metric)
66
+
67
+ The judge sees **two anonymized screenshots in shuffled order** and answers one question:
68
+
69
+ > *"Which reads more like a professionally designed, well-funded SaaS landing page?"*
70
+
71
+ Run all three pairings: **A-vs-C**, **B-vs-C**, **A-vs-B**. Screenshots carry no arm labels; left/right order is
72
+ randomized per comparison. **Metric = win-rate against C.** The harness improves when A's win-rate against C rises
73
+ (and ideally approaches or exceeds B's gap to C closing).
74
+
75
+ ### Layer 2 — Rubric (1–5 per dimension)
76
+
77
+ A per-arm absolute score across six dimensions:
78
+
79
+ | Dimension | Looks for |
80
+ |-----------|-----------|
81
+ | Hierarchy | clear primary action, deliberate reading order |
82
+ | Typography | confident scale, non-default families, tight headline tracking |
83
+ | Color discipline | restrained palette, one accent, no "AI gradient" tell |
84
+ | Whitespace rhythm | composed negative space, not uniform padding |
85
+ | Originality vs slop | a committed direction, not the hero→3-cards→testimonial→CTA mean |
86
+ | Mobile | the 390px capture holds up, not a squished desktop |
87
+
88
+ ### Layer 3 — Slop detector (binary checklist)
89
+
90
+ A nearly objective pass/fail checklist over the **negatives** corpus (#880): default Inter at default weights,
91
+ slate/indigo gradient, rounded-everything, boilerplate section skeleton, stock 3D blobs, etc. Each negative is a
92
+ yes/no. The slop score is the count of negatives present (lower is better).
93
+
94
+ ## Two non-negotiable methodology rules
95
+
96
+ ### n=1 is a lie
97
+
98
+ Taste output is high-variance. A single generation per arm tells you almost nothing. **Run 3–5 generations per arm
99
+ across 3–5 products** and report **distributions** — win-rate over runs, and the mean and spread of rubric scores.
100
+ The scorecard aggregator (`scripts/bench-scorecard.mjs`) **rejects runs with n<3** to enforce this.
101
+
102
+ ### Anti-Goodhart — judge independence
103
+
104
+ The benchmark judge **MUST be independent from the harness's own critique loop (#882)**. If the harness optimizes
105
+ against the same judge that scores the benchmark, you are training to the test — the number goes up while real
106
+ quality does not. Concretely: the benchmark judge uses its own prompt and is invoked as a distinct stage; it does
107
+ **not** import, share state with, or reuse the #882 critique judge. This independence is asserted in both this doc
108
+ and the [`/design-bench`](../../commands/design-bench.md) command spec.
109
+
110
+ ## Anonymization is load-bearing
111
+
112
+ Each brief is derived from a real reference (arm C) but uses an **anonymized product name** and strips every visual
113
+ cue. The mapping from anonymized brief → real reference lives in
114
+ [`fixtures/briefs/README.md`](fixtures/briefs/README.md) (kept separate from the briefs so the generator never sees
115
+ it). A real brand name in a brief lets the model reproduce that brand's actual page from training — measuring recall,
116
+ not taste, and inflating arm B. Treat any real-brand leak in a brief as a fixture bug.
117
+
118
+ ## Seed inputs (fixtures)
119
+
120
+ Briefs are **fixed and hand-written**, kept as reproducible fixtures under
121
+ [`fixtures/briefs/`](fixtures/briefs/): **Cadence**, **Tender**, **Slipstream**, **Voltage**, **Plume**. Fixed
122
+ inputs are what make the benchmark reproducible across runs and over time — the same five briefs are re-run as the
123
+ harness evolves, so score movement is attributable to the harness, not to changing inputs.
124
+
125
+ ## Scorecard schema
126
+
127
+ The aggregator consumes a **runs file** (the raw per-run judge output) and emits a **scorecard**.
128
+
129
+ ### Input — runs file (`runs.json`)
130
+
131
+ ```jsonc
132
+ {
133
+ "corpus_version": "2026.2", // grammar snapshot the briefs were designed against
134
+ "generation_model": "…", // the SAME model used for arms A and B
135
+ "judge_model": "…", // the INDEPENDENT benchmark judge (not #882)
136
+ "products": [
137
+ {
138
+ "product": "Voltage", // anonymized brief name
139
+ "reference": "modal.com", // real arm-C reference (internal record only)
140
+ "runs": [
141
+ {
142
+ "run": 1,
143
+ "pairwise": { // winner of each blind pairwise comparison
144
+ "A_vs_C": "C", // "A" | "B" | "C" | "tie"
145
+ "B_vs_C": "C",
146
+ "A_vs_B": "A"
147
+ },
148
+ "rubric": { // 1–5 per arm per dimension
149
+ "A": { "hierarchy": 4, "typography": 3, "color": 4, "whitespace": 4, "originality": 3, "mobile": 4 },
150
+ "B": { "hierarchy": 3, "typography": 2, "color": 2, "whitespace": 3, "originality": 2, "mobile": 3 },
151
+ "C": { "hierarchy": 5, "typography": 5, "color": 5, "whitespace": 5, "originality": 5, "mobile": 5 }
152
+ },
153
+ "slop": { // count of negatives present (lower is better)
154
+ "A": 1, "B": 4, "C": 0
155
+ }
156
+ }
157
+ // … runs 2..n (n >= 3)
158
+ ]
159
+ }
160
+ // … 3–5 products
161
+ ]
162
+ }
163
+ ```
164
+
165
+ ### Output — scorecard
166
+
167
+ ```jsonc
168
+ {
169
+ "n_products": 5,
170
+ "n_runs_per_product": 4,
171
+ "win_rate_vs_C": { // primary metric — fraction of runs the arm beat C
172
+ "A": { "mean": 0.15, "by_product": { "Voltage": 0.25, "…": 0.10 } },
173
+ "B": { "mean": 0.05, "by_product": { "…": 0.00 } }
174
+ },
175
+ "a_vs_b_win_rate": 0.70, // fraction of runs A beat B (the harness delta)
176
+ "rubric": { // mean + spread per arm per dimension across all runs
177
+ "A": { "hierarchy": { "mean": 3.8, "stdev": 0.6 }, "…": {} },
178
+ "B": { "…": {} },
179
+ "C": { "…": {} }
180
+ },
181
+ "slop": { // mean negatives present per arm (lower better)
182
+ "A": 1.2, "B": 3.9, "C": 0.1
183
+ },
184
+ "judge_calibration": { // C must win A-vs-C and B-vs-C; flag if not
185
+ "miscalibrated_runs": [], // runs where C lost to A or B
186
+ "ok": true
187
+ }
188
+ }
189
+ ```
190
+
191
+ ## Running the benchmark
192
+
193
+ The full reproducible workflow — reference URL → brief → render arms → judge → scorecard — is driven by the
194
+ [`/design-bench`](../../commands/design-bench.md) command. The deterministic aggregation step is
195
+ `node scripts/bench-scorecard.mjs runs.json`.
@@ -0,0 +1,123 @@
1
+ # Design-Architect Phase & FORGE:DESIGN_RATIONALE — the designer's diary
2
+
3
+ > **Status:** Committed foundation — design-architect phase spec for issue #886 (milestone #13).
4
+ > The reasoning-before-generation phase. Produces the [DESIGN_SPEC](design-spec-schema.md) (#881);
5
+ > queries [design-memory](design-memory.md) (#887); read by generate + the critique loop (#882).
6
+ > Extended: motion tier selection added as diary step 5 (#1043); product mock decision added as diary step 6 (#1045); scroll narrative planning added as diary step 7.5 (#1046).
7
+
8
+ ## Why this exists
9
+
10
+ Everything else in the harness (spec, effects, critique loop) can still **raw-dog** the output: jump straight to a
11
+ page and defend it afterward. A page produced without deliberation is a guess. What separates craft from a guess is
12
+ **thinking before generating** — and capturing that thinking so it's reviewable.
13
+
14
+ ForgeDock already does this for code: `investigate → architect → implement → validate`, where the *architect* phase
15
+ reasons about *how* before any code is written, and writes a FORGE annotation the next stage reads. **A design harness
16
+ without a design-architect phase is just vibes.** So this is not a new paradigm — it is the ForgeDock pattern applied
17
+ to design:
18
+
19
+ ```
20
+ design-investigate (what's the message?) → design-architect (diary + rationale → spec) → generate → critique-loop
21
+ ```
22
+
23
+ The architect phase's output is a `FORGE:DESIGN_RATIONALE` annotation — the diary — which *produces* the
24
+ [DESIGN_SPEC](design-spec-schema.md) instead of the spec appearing from nowhere.
25
+
26
+ ## The diary — structured chain of thought
27
+
28
+ The phase must reason through all ten, in order. This is what a designer's diary actually contains.
29
+
30
+ 1. **Intent & emotional register** — One core message. One feeling to evoke (trust / speed / power / craft / play).
31
+ What should the visitor feel, and what should they do?
32
+ 2. **Audience & objection** — Who lands here? What do they already believe? What is the single objection this page
33
+ must overcome?
34
+ 3. **Communication hierarchy** — What gets said 1st, 2nd, 3rd. This *is* the visual hierarchy — it is derived from
35
+ meaning, not chosen as decoration. (Feeds `layout_grammar.sections` in the spec.)
36
+ 4. **Direction & why** — The chosen archetype/concept *and the reasoning for it* — why it serves this message for this
37
+ audience. **Plus the alternatives considered and explicitly rejected, and why.** (Rejection is where taste shows.)
38
+ 5. **Motion tier & hero technique** — Which motion tier (1–3) from the [corpus hero motion vocabulary](reference-corpus.md#hero-motion-vocabulary--making-the-hero-feel-alive)
39
+ is appropriate for this archetype and product nature? Which specific technique? Commit to both. The default per-archetype
40
+ motion profile in the corpus is the starting point; deviation requires explicit reasoning. Every page requires a
41
+ committed motion posture — "no motion" for `bold-brutalist` or `minimal-luxury` is a valid deliberate choice, but
42
+ it must be stated, not omitted.
43
+ - Feeds `motion.tier` and `motion.hero_technique` in the [DESIGN_SPEC](design-spec-schema.md).
44
+ - If a video placeholder is appropriate (Tier 3): set `motion.video_placeholder: true` — the generator scaffolds
45
+ the HTML and CSS gradient poster.
46
+ - State the `prefers-reduced-motion` fallback explicitly (e.g., "gradient-shift pauses; text remains visible").
47
+ 6. **Product mock decision** — Does this brief imply a product interface that can be shown live in the hero? <!-- Added: forge#1045 -->
48
+ Consult the [interactive product mock vocabulary](reference-corpus.md#interactive-product-mock-vocabulary) to
49
+ identify the matching product type.
50
+ - **Decision signals**: does the brief describe a dashboard, issue board, payment flow, deploy pipeline, code
51
+ API, or inbox? → commit a product mock. Does the product have no single canonical interface, is it image-driven
52
+ (e.g. physical device, marketplace), or is the archetype `minimal-luxury` and restraint is the move? → commit
53
+ `product_mock.type: "none"` with explicit reasoning.
54
+ - If committing a mock: select exactly 2 interactions from the per-type defaults table in the corpus. State
55
+ whether they are CSS-only or require lightweight JS. State the `prefers-reduced-motion` fallback.
56
+ - Feeds `product_mock.type`, `product_mock.interactions`, and `product_mock.css_only` in the
57
+ [DESIGN_SPEC](design-spec-schema.md).
58
+ - Restraint rule: 2 interactions only. The mock should feel alive, not be a demo. Never animate primary copy.
59
+ 7. **The signature move** — The one memorable, non-obvious idea that makes it *this* page and not a template. The "hook."
60
+ 7.5. **Scroll narrative planning** — Commit the full-page below-fold narrative before generation. <!-- Added: forge#1046 -->
61
+ Consult the [below-the-fold narrative vocabulary](reference-corpus.md#below-the-fold-narrative-vocabulary--scroll-driven-full-page-structure)
62
+ and the per-archetype scroll narrative defaults table.
63
+ - **Section sequence**: list the committed section order (e.g. `problem-context → solution-showcase → social-proof → deep-features → final-cta`). Deviate from the archetype default with explicit reasoning. Minimum 4 below-fold sections (5 total including hero); `minimal-luxury` archetype may use 3 below-fold sections with explicit restraint rationale.
64
+ - **Sticky nav decision**: commit `sticky_nav: true` or `false`. True is the default for all archetypes when total sections ≥ 5. False requires explicit rationale. For `minimal-luxury`: omit backdrop-filter blur (glassmorphism contradiction) and use fully-opaque background instead.
65
+ - **Social proof type**: commit one of `logos | metrics | testimonials | none`. Consult the archetype social proof defaults. `none` requires explicit rationale (e.g., pre-launch product with no social proof yet — acceptable).
66
+ - **CTA rhythm**: commit ≥ 2 CTA placements — one in hero, one in final-cta. Note any additional mid-page CTA placement and its rationale.
67
+ - Feeds `scroll_narrative.section_sequence`, `scroll_narrative.sticky_nav`, `scroll_narrative.social_proof_section`, and `scroll_narrative.cta_placements_min` in the [DESIGN_SPEC](design-spec-schema.md).
68
+ 8. **What I'm trying this time** — A technique or learning being deliberately applied. Queries
69
+ [design-memory](design-memory.md) so it builds on, and diverges from, prior work.
70
+ 9. **Non-goals** — What this page deliberately will *not* do. Restraint stated as a decision, not an omission.
71
+
72
+ ## FORGE:DESIGN_RATIONALE — annotation format
73
+
74
+ Posted to the design issue at the architect stage (see the `/design` pipeline, #888):
75
+
76
+ ```markdown
77
+ <!-- FORGE:DESIGN_RATIONALE -->
78
+ ## Design Rationale — {product}
79
+
80
+ **Intent / feeling:** {one message} · {one feeling}
81
+ **Audience / objection:** {who} — must overcome: {objection}
82
+ **Communication hierarchy:** 1) {…} 2) {…} 3) {…}
83
+ **Direction:** {archetype} — because {reasoning}
84
+ - Considered & rejected: {alt A} (because {…}); {alt B} (because {…})
85
+ **Motion:** Tier {1|2|3} — technique: {hero_technique} — because {reasoning}
86
+ - `prefers-reduced-motion` fallback: {what degrades gracefully}
87
+ - Video placeholder: {yes — Tier 3 scaffold generated | no}
88
+ **Product mock:** {type id | none} — because {reasoning}
89
+ - Interactions: {interaction-id-1}, {interaction-id-2} (or "n/a — type: none")
90
+ - CSS-only: {yes | no — lightweight JS for {reason}}
91
+ - `prefers-reduced-motion` fallback: {what degrades gracefully, or "n/a"}
92
+ **Scroll narrative:** {section sequence} — sticky nav: {yes|no} — social proof: {logos|metrics|testimonials|none}
93
+ - CTA placements: {count} ({hero + final-cta} or note any additional mid-page CTA)
94
+ - Deviation from archetype default: {reasoning, or "none — follows archetype default"}
95
+ **Signature move:** {the one non-obvious idea}
96
+ **Trying this time:** {technique/learning} (from memory: avoiding {prior move})
97
+ **Non-goals:** {what this won't do}
98
+
99
+ → Produces DESIGN_SPEC: {link}
100
+ ```
101
+
102
+ ## Why it matters for a non-designer
103
+
104
+ The rationale is **reviewable without judging pixels.** A non-designer can read the *thinking* and catch when the
105
+ reasoning is thin or generic — "this hook is the same as every other page," "the direction has no justification" —
106
+ **before a single pixel renders.** The harness externalizes the designer's thought process the user may not have.
107
+ This is the review checkpoint that makes the whole pipeline trustworthy.
108
+
109
+ ## Acceptance
110
+
111
+ - Every generated page is preceded by a DESIGN_RATIONALE covering all 10 elements, with ≥1 explicitly rejected
112
+ alternative, a committed motion tier + technique, a product mock decision, a scroll narrative plan, and a named signature move.
113
+ - The `**Motion:**` line must be present in every rationale — "no motion" is acceptable only for `bold-brutalist`
114
+ and `minimal-luxury` archetypes, and must be stated with reasoning, not left absent.
115
+ - The `**Product mock:**` line must be present in every rationale — `type: none` is acceptable when the product
116
+ has no single canonical interface or when archetype restraint is the deliberate choice, but must be stated with
117
+ reasoning, not left absent.
118
+ - The `**Scroll narrative:**` line must be present in every rationale — a committed section sequence, sticky nav
119
+ decision, social proof type, and CTA count are all required. `social_proof_section: none` is acceptable for
120
+ pre-launch products with no social proof, but must be stated with reasoning, not left absent.
121
+ - On the seed set, rationales for Voltage / Plume / Slipstream show genuinely different intent, hierarchy,
122
+ motion techniques, product mock decisions, scroll narrative sequences, and signature moves — derived from the
123
+ briefs alone (no design hints in the brief).
@@ -0,0 +1,61 @@
1
+ # Design Memory & Deliberate Divergence
2
+
3
+ > **Status:** DRAFT for review — foundation spec for issue #887 (milestone #13).
4
+ > Queried by the [design-architect phase](design-architect-rationale.md) (#886) for the "different from my previous
5
+ > work / what new thing am I applying" questions. Enforced by the critique loop (#882).
6
+
7
+ ## The problem it solves
8
+
9
+ Mode-collapse is not only a *per-page* phenomenon — it is why *all* AI output looks the same *across* pages. Per-page
10
+ taste (the spec, the critique loop) fixes one page. **Design memory fixes the portfolio.** To answer "how is this
11
+ different from before," the harness must *remember before*.
12
+
13
+ ## The store
14
+
15
+ Local, file-based, on the AGPL CLI side — under `.forgedock/design-memory/` (like per-repo adaptive scripts:
16
+ `.gitignore`d by default, opt-in to commit). One record per shipped design:
17
+
18
+ ```jsonc
19
+ {
20
+ "product": "…",
21
+ "shipped_at": "<timestamp>",
22
+ "corpus_version": "2026.2", // taste shifts; record what was "current" then
23
+ "archetype": "technical-dense",
24
+ "signature_move": "…", // the hook — the thing to NOT repeat next time
25
+ "palette_move": "…", // the distinctive color decision
26
+ "type_pairing": "…",
27
+ "effects_used": ["…"],
28
+ "rationale_summary": "…",
29
+ "learnings": ["…"] // techniques discovered/applied — grows the vocabulary
30
+ }
31
+ ```
32
+
33
+ ## Two mechanisms
34
+
35
+ ### 1. Diverge (anti-sameness)
36
+ The architect phase receives a summary of recent records and is required to **not repeat** the last N designs':
37
+ - signature moves (the hook),
38
+ - archetype (don't ship `technical-dense` three times running for similar briefs),
39
+ - palette/type/effect moves.
40
+
41
+ The [render → vision-critique loop](render-critique-loop.md) (#882) can enforce a **divergence check**: "is this meaningfully distinct from prior outputs,
42
+ or a reskin?" A reskin fails.
43
+
44
+ ### 2. Evolve (incorporate learnings)
45
+ `learnings` accumulate into a rotating technique/pattern library. The architect's "what I'm trying this time"
46
+ (rationale item 6) draws from it, so the harness deliberately applies and rotates new techniques rather than
47
+ converging on a comfortable default. This is the autopilot / self-improvement angle: **the system gets better the
48
+ more it ships.**
49
+
50
+ ## Open-core boundary
51
+
52
+ - Memory is **local, AGPL CLI side**. No cross-repo import.
53
+ - It emits nothing the Platform needs to function; if a user opts in, a summary could feed the Platform's
54
+ observability (per `devdocs/project/architecture.md` boundary rules) — but the CLI never depends on the Platform.
55
+
56
+ ## Acceptance
57
+
58
+ - Architect phase receives prior signature moves and avoids repeating them.
59
+ - Across N consecutive generations for similar briefs, outputs show measurable variance in concept / hook / palette —
60
+ not the same page reskinned.
61
+ - `learnings` demonstrably influence later designs ("trying this time" references a prior learning).