forgedock 1.0.17 → 1.0.18
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +34 -1
- package/bin/cross-publish.mjs +344 -0
- package/bin/forgedock.mjs +229 -17
- package/commands/autopilot.md +77 -0
- package/commands/design-bench.md +176 -0
- package/commands/design-render-critique-loop.md +103 -0
- package/commands/design.md +259 -0
- package/commands/issue.md +7 -3
- package/commands/orchestrate.md +121 -0
- package/commands/qa-sweep.md +37 -2
- package/commands/quality-gate.md +30 -0
- package/commands/review-pr-staging.md +158 -2
- package/commands/test-gate.md +929 -0
- package/commands/work-on/build/implement.md +5 -3
- package/commands/work-on.md +53 -3
- package/docs/CI.md +177 -0
- package/docs/FORGE-PROTOCOL.md +324 -0
- package/docs/articles/b-vs-c-baseline.md +210 -0
- package/docs/articles/full-abc-benchmark.md +205 -0
- package/docs/articles/selective-spec-loading-token-savings.md +140 -0
- package/docs/design/abc-benchmark.md +195 -0
- package/docs/design/design-architect-rationale.md +123 -0
- package/docs/design/design-memory.md +61 -0
- package/docs/design/design-spec-schema.md +248 -0
- package/docs/design/design-system-lint.md +150 -0
- package/docs/design/divergent-generation.md +106 -0
- package/docs/design/effects-appropriateness.md +133 -0
- package/docs/design/fixtures/briefs/README.md +47 -0
- package/docs/design/fixtures/briefs/cadence.md +39 -0
- package/docs/design/fixtures/briefs/plume.md +39 -0
- package/docs/design/fixtures/briefs/slipstream.md +39 -0
- package/docs/design/fixtures/briefs/tender.md +39 -0
- package/docs/design/fixtures/briefs/voltage.md +39 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/cadence/arm-b/run-1.html +364 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/cadence/arm-b/run-2.html +420 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/cadence/arm-b/run-3.html +506 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/plume/arm-b/run-1.html +611 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/plume/arm-b/run-2.html +594 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/plume/arm-b/run-3.html +817 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/runs.json +175 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/scorecard.json +138 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/slipstream/arm-b/run-1.html +373 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/slipstream/arm-b/run-2.html +398 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/slipstream/arm-b/run-3.html +595 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/tender/arm-b/run-1.html +531 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/tender/arm-b/run-2.html +565 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/tender/arm-b/run-3.html +674 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/voltage/arm-b/run-1.html +388 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/voltage/arm-b/run-2.html +463 -0
- package/docs/design/fixtures/runs/b-vs-c-baseline/voltage/arm-b/run-3.html +568 -0
- package/docs/design/fixtures/runs/full-abc/cadence/arm-a/run-1.html +562 -0
- package/docs/design/fixtures/runs/full-abc/cadence/arm-a/run-2.html +618 -0
- package/docs/design/fixtures/runs/full-abc/cadence/arm-a/run-3.html +618 -0
- package/docs/design/fixtures/runs/full-abc/cadence/arm-b/run-1.html +364 -0
- package/docs/design/fixtures/runs/full-abc/cadence/arm-b/run-2.html +420 -0
- package/docs/design/fixtures/runs/full-abc/cadence/arm-b/run-3.html +506 -0
- package/docs/design/fixtures/runs/full-abc/plume/arm-a/run-1.html +527 -0
- package/docs/design/fixtures/runs/full-abc/plume/arm-a/run-2.html +445 -0
- package/docs/design/fixtures/runs/full-abc/plume/arm-a/run-3.html +483 -0
- package/docs/design/fixtures/runs/full-abc/plume/arm-b/run-1.html +611 -0
- package/docs/design/fixtures/runs/full-abc/plume/arm-b/run-2.html +594 -0
- package/docs/design/fixtures/runs/full-abc/plume/arm-b/run-3.html +817 -0
- package/docs/design/fixtures/runs/full-abc/runs.json +190 -0
- package/docs/design/fixtures/runs/full-abc/scorecard.json +144 -0
- package/docs/design/fixtures/runs/full-abc/slipstream/arm-a/run-1.html +496 -0
- package/docs/design/fixtures/runs/full-abc/slipstream/arm-a/run-2.html +447 -0
- package/docs/design/fixtures/runs/full-abc/slipstream/arm-a/run-3.html +564 -0
- package/docs/design/fixtures/runs/full-abc/slipstream/arm-b/run-1.html +373 -0
- package/docs/design/fixtures/runs/full-abc/slipstream/arm-b/run-2.html +398 -0
- package/docs/design/fixtures/runs/full-abc/slipstream/arm-b/run-3.html +595 -0
- package/docs/design/fixtures/runs/full-abc/tender/arm-a/run-1.html +622 -0
- package/docs/design/fixtures/runs/full-abc/tender/arm-a/run-2.html +475 -0
- package/docs/design/fixtures/runs/full-abc/tender/arm-a/run-3.html +660 -0
- package/docs/design/fixtures/runs/full-abc/tender/arm-b/run-1.html +531 -0
- package/docs/design/fixtures/runs/full-abc/tender/arm-b/run-2.html +565 -0
- package/docs/design/fixtures/runs/full-abc/tender/arm-b/run-3.html +674 -0
- package/docs/design/fixtures/runs/full-abc/voltage/arm-a/run-1.html +525 -0
- package/docs/design/fixtures/runs/full-abc/voltage/arm-a/run-2.html +446 -0
- package/docs/design/fixtures/runs/full-abc/voltage/arm-a/run-3.html +463 -0
- package/docs/design/fixtures/runs/full-abc/voltage/arm-b/run-1.html +388 -0
- package/docs/design/fixtures/runs/full-abc/voltage/arm-b/run-2.html +463 -0
- package/docs/design/fixtures/runs/full-abc/voltage/arm-b/run-3.html +568 -0
- package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-a/run-1.html +1068 -0
- package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-a/run-2.html +1423 -0
- package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-a/run-3.html +1245 -0
- package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-b/run-1.html +305 -0
- package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-b/run-2.html +327 -0
- package/docs/design/fixtures/runs/full-abc-opus/cadence/arm-b/run-3.html +428 -0
- package/docs/design/fixtures/runs/full-abc-opus/plume/arm-a/run-1.html +1270 -0
- package/docs/design/fixtures/runs/full-abc-opus/plume/arm-a/run-2.html +1172 -0
- package/docs/design/fixtures/runs/full-abc-opus/plume/arm-a/run-3.html +1532 -0
- package/docs/design/fixtures/runs/full-abc-opus/plume/arm-b/run-1.html +315 -0
- package/docs/design/fixtures/runs/full-abc-opus/plume/arm-b/run-2.html +342 -0
- package/docs/design/fixtures/runs/full-abc-opus/plume/arm-b/run-3.html +353 -0
- package/docs/design/fixtures/runs/full-abc-opus/runs.json +190 -0
- package/docs/design/fixtures/runs/full-abc-opus/scorecard.json +144 -0
- package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-a/run-1.html +345 -0
- package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-a/run-2.html +396 -0
- package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-a/run-3.html +502 -0
- package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-b/run-1.html +408 -0
- package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-b/run-2.html +446 -0
- package/docs/design/fixtures/runs/full-abc-opus/slipstream/arm-b/run-3.html +496 -0
- package/docs/design/fixtures/runs/full-abc-opus/tender/arm-a/run-1.html +813 -0
- package/docs/design/fixtures/runs/full-abc-opus/tender/arm-a/run-2.html +711 -0
- package/docs/design/fixtures/runs/full-abc-opus/tender/arm-a/run-3.html +721 -0
- package/docs/design/fixtures/runs/full-abc-opus/tender/arm-b/run-1.html +280 -0
- package/docs/design/fixtures/runs/full-abc-opus/tender/arm-b/run-2.html +291 -0
- package/docs/design/fixtures/runs/full-abc-opus/tender/arm-b/run-3.html +366 -0
- package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-a/run-1.html +1010 -0
- package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-a/run-2.html +836 -0
- package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-a/run-3.html +930 -0
- package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-b/run-1.html +407 -0
- package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-b/run-2.html +396 -0
- package/docs/design/fixtures/runs/full-abc-opus/voltage/arm-b/run-3.html +529 -0
- package/docs/design/foundations/brutalist.css +294 -0
- package/docs/design/foundations/editorial.css +300 -0
- package/docs/design/foundations/minimal-luxury.css +295 -0
- package/docs/design/foundations/technical.css +293 -0
- package/docs/design/foundations/warm-photo.css +299 -0
- package/docs/design/reference-corpus.md +1726 -0
- package/docs/design/render-critique-loop.md +57 -0
- package/docs/spec-graph-schema.md +397 -0
- package/package.json +1 -1
- package/scripts/bench-scorecard.mjs +239 -0
- package/scripts/build-spec-graph.mjs +509 -0
- package/scripts/classify-lane.sh +0 -7
- package/scripts/design-system-lint.mjs +993 -0
- package/scripts/graph-query.sh +486 -0
- package/scripts/transition-label.sh +10 -2
- package/scripts/validate-spec-graph.sh +423 -0
- package/templates/workflows/forgedock-review.yml +96 -0
|
@@ -0,0 +1,210 @@
|
|
|
1
|
+
# B-vs-C Baseline Scorecard
|
|
2
|
+
|
|
3
|
+
> **Benchmark date**: 2026-06-22
|
|
4
|
+
> **Issue**: #879 (part of milestone #877, UI Taste Harness + ABC benchmark)
|
|
5
|
+
> **Rig**: `/design-bench` (#878)
|
|
6
|
+
> **Generation model (B)**: claude-sonnet-4-6
|
|
7
|
+
> **Judge model**: claude-sonnet-4-6 (independent — not the harness critique loop)
|
|
8
|
+
> **Corpus version**: seed-v1
|
|
9
|
+
> **Products**: 5 (Cadence, Tender, Slipstream, Voltage, Plume)
|
|
10
|
+
> **Runs per product**: 3
|
|
11
|
+
|
|
12
|
+
This is the **first hard datapoint** — the bar the ForgeDock design harness (arm A) must beat.
|
|
13
|
+
It measures the raw-model gap: how does a frontier model's one-shot HTML output (arm B) compare
|
|
14
|
+
against real, professionally-designed, funded-SaaS reference pages (arm C)?
|
|
15
|
+
|
|
16
|
+
## Headline Result
|
|
17
|
+
|
|
18
|
+
**Arm B wins 0% of clean pairwise comparisons against arm C.**
|
|
19
|
+
|
|
20
|
+
The raw frontier model cannot produce a page that reads as professionally designed when compared
|
|
21
|
+
against real reference pages. C won 12/12 clean comparisons (excluding Slipstream — see calibration
|
|
22
|
+
note below). This is the gap the harness must close.
|
|
23
|
+
|
|
24
|
+
## Pairwise Win-Rate vs C
|
|
25
|
+
|
|
26
|
+
| Product | B win-rate vs C | Runs | Notes |
|
|
27
|
+
|---------|----------------|------|-------|
|
|
28
|
+
| Cadence (resend.com) | **0%** | 0/3 | C dominated all runs |
|
|
29
|
+
| Tender (stripe.com) | **0%** | 0/3 | C dominated all runs |
|
|
30
|
+
| Slipstream (vercel.com) | 100%* | 3/3 | *Calibration artifact — see note |
|
|
31
|
+
| Voltage (modal.com) | **0%** | 0/3 | Largest gap — C dominated |
|
|
32
|
+
| Plume (linear.app) | **0%** | 0/3 | C dominated all runs |
|
|
33
|
+
| **Overall (clean)** | **0%** | **0/12** | |
|
|
34
|
+
| **Overall (all)** | **20%** | **3/15** | Inflated by Slipstream artifact |
|
|
35
|
+
|
|
36
|
+
### Slipstream Calibration Note
|
|
37
|
+
|
|
38
|
+
The vercel.com reference screenshot was viewport-only (54KB) due to a Playwright rendering timeout
|
|
39
|
+
on vercel.com's heavy JS/animation. The judge saw a sparse, cropped hero vs full-page arm B
|
|
40
|
+
generations — an unfair comparison. All 3 Slipstream runs are flagged as `miscalibrated` in the
|
|
41
|
+
scorecard. **Clean metrics exclude Slipstream throughout this report.**
|
|
42
|
+
|
|
43
|
+
## Rubric Scores (1-5, mean across all clean runs)
|
|
44
|
+
|
|
45
|
+
| Dimension | Arm B | Arm C | Gap |
|
|
46
|
+
|-----------|-------|-------|-----|
|
|
47
|
+
| Hierarchy | 3.17 | 5.00 | -1.83 |
|
|
48
|
+
| Typography | 3.25 | 5.00 | -1.75 |
|
|
49
|
+
| Color | 3.08 | 5.00 | -1.92 |
|
|
50
|
+
| Whitespace | 3.17 | 4.75 | -1.58 |
|
|
51
|
+
| Originality | 2.83 | 5.00 | -2.17 |
|
|
52
|
+
| Mobile | 3.08 | 4.00 | -0.92 |
|
|
53
|
+
| **Mean** | **3.10** | **4.79** | **-1.69** |
|
|
54
|
+
|
|
55
|
+
### Key Observations
|
|
56
|
+
|
|
57
|
+
- **Originality is the widest gap** (2.83 vs 5.00, delta -2.17). AI output mode-collapses to
|
|
58
|
+
template-like layouts (3-column feature grids, gradient heroes, symmetric pricing cards). Real
|
|
59
|
+
pages have distinctive, intentional design language.
|
|
60
|
+
- **Color discipline** is the second-widest gap (-1.92). AI pages use safe, generic palettes
|
|
61
|
+
(dark + accent gradient) rather than the sophisticated, brand-specific color systems of real pages.
|
|
62
|
+
- **Mobile** is the narrowest gap (-0.92). Tailwind's responsive utilities give arm B reasonable
|
|
63
|
+
mobile adaptability, but this is the framework doing the work, not design taste.
|
|
64
|
+
|
|
65
|
+
## Slop Count (mean AI tells per arm)
|
|
66
|
+
|
|
67
|
+
| Arm | Mean Slop | Notes |
|
|
68
|
+
|-----|-----------|-------|
|
|
69
|
+
| B | **5.0** | Range: 2-7 per run |
|
|
70
|
+
| C | **0.5** | Range: 0-1 per run |
|
|
71
|
+
|
|
72
|
+
Common AI tells detected across arm B runs:
|
|
73
|
+
- Cookie-cutter 3-column feature grids with circular icons (13/15 runs)
|
|
74
|
+
- Blue-purple gradient overuse (8/15 runs)
|
|
75
|
+
- Generic hero text ("Revolutionize", "Built for", "Supercharge")
|
|
76
|
+
- Overly symmetrical layouts
|
|
77
|
+
- Testimonial sections with identical card structures
|
|
78
|
+
- Pricing tier cards (3 columns: Free/Pro/Enterprise)
|
|
79
|
+
- Excessive rounded corners and drop shadows
|
|
80
|
+
|
|
81
|
+
## Per-Product Rubric Detail
|
|
82
|
+
|
|
83
|
+
### Cadence (vs resend.com)
|
|
84
|
+
|
|
85
|
+
| Dim | B r1 | B r2 | B r3 | C |
|
|
86
|
+
|-----|------|------|------|---|
|
|
87
|
+
| Hierarchy | 3 | 3 | 3 | 5 |
|
|
88
|
+
| Typography | 3 | 3 | 3 | 5 |
|
|
89
|
+
| Color | 3 | 3 | 3 | 5 |
|
|
90
|
+
| Whitespace | 3 | 3 | 3 | 5 |
|
|
91
|
+
| Originality | 3 | 3 | 3 | 5 |
|
|
92
|
+
| Mobile | 3 | 3 | 3 | 4 |
|
|
93
|
+
| Slop | 5 | 4 | 5 | 0 |
|
|
94
|
+
|
|
95
|
+
Consistent 3/5 across all B runs — competent but generic. Resend's reference page demonstrates
|
|
96
|
+
warm minimalism with purposeful whitespace that none of the B runs achieved.
|
|
97
|
+
|
|
98
|
+
### Tender (vs stripe.com)
|
|
99
|
+
|
|
100
|
+
| Dim | B r1 | B r2 | B r3 | C |
|
|
101
|
+
|-----|------|------|------|---|
|
|
102
|
+
| Hierarchy | 4 | 3 | 4 | 5 |
|
|
103
|
+
| Typography | 4 | 3 | 4 | 5 |
|
|
104
|
+
| Color | 4 | 3 | 3 | 5 |
|
|
105
|
+
| Whitespace | 4 | 3 | 3 | 5 |
|
|
106
|
+
| Originality | 3 | 3 | 3 | 5 |
|
|
107
|
+
| Mobile | 3 | 3 | 3 | 4 |
|
|
108
|
+
| Slop | 4 | 6 | 5 | 1 |
|
|
109
|
+
|
|
110
|
+
Best arm B performance — run 1 and run 3 hit 4/5 on hierarchy/typography. The payments brief
|
|
111
|
+
elicits more structured, professional-feeling output. But originality never exceeds 3 — the pages
|
|
112
|
+
look like payment templates, not like Stripe.
|
|
113
|
+
|
|
114
|
+
### Voltage (vs modal.com)
|
|
115
|
+
|
|
116
|
+
| Dim | B r1 | B r2 | B r3 | C |
|
|
117
|
+
|-----|------|------|------|---|
|
|
118
|
+
| Hierarchy | 3 | 3 | 2 | 5 |
|
|
119
|
+
| Typography | 3 | 3 | 2 | 5 |
|
|
120
|
+
| Color | 3 | 3 | 2 | 5 |
|
|
121
|
+
| Whitespace | 3 | 3 | 3 | 4 |
|
|
122
|
+
| Originality | 2 | 3 | 2 | 5 |
|
|
123
|
+
| Mobile | 3 | 3 | 3 | 4 |
|
|
124
|
+
| Slop | 5 | 6 | 7 | 1 |
|
|
125
|
+
|
|
126
|
+
Widest gap in the benchmark. Modal's reference page has exceptional editorial layout variety and
|
|
127
|
+
rich product screenshots. Arm B defaulted to generic dark-theme developer tool aesthetics with
|
|
128
|
+
high slop counts (5-7). Run 3 was the weakest arm B output in the entire benchmark (2/5 on
|
|
129
|
+
4 dimensions, 7 slop).
|
|
130
|
+
|
|
131
|
+
### Plume (vs linear.app)
|
|
132
|
+
|
|
133
|
+
| Dim | B r1 | B r2 | B r3 | C |
|
|
134
|
+
|-----|------|------|------|---|
|
|
135
|
+
| Hierarchy | 3 | 4 | 3 | 5 |
|
|
136
|
+
| Typography | 3 | 4 | 4 | 5 |
|
|
137
|
+
| Color | 3 | 4 | 3 | 5 |
|
|
138
|
+
| Whitespace | 3 | 4 | 3 | 5 |
|
|
139
|
+
| Originality | 2 | 4 | 3 | 5 |
|
|
140
|
+
| Mobile | 3 | 4 | 3 | 4 |
|
|
141
|
+
| Slop | 6 | 2 | 5 | 1 |
|
|
142
|
+
|
|
143
|
+
Notable variance: run 2 (editorial/warm sand aesthetic) scored 4/5 across the board with only
|
|
144
|
+
2 slop — the best single arm B run in the entire benchmark. This suggests that when the model
|
|
145
|
+
breaks out of the default dark-gradient template (as the serif/editorial approach forced it to),
|
|
146
|
+
quality jumps significantly. This is evidence for the archetype sampling hypothesis (#883).
|
|
147
|
+
|
|
148
|
+
## Implications for the Harness
|
|
149
|
+
|
|
150
|
+
1. **The gap is real and large.** Mean rubric delta of -1.69 points on a 5-point scale confirms
|
|
151
|
+
the thesis: raw model output is recognizably AI-generated and substantially below professional
|
|
152
|
+
quality.
|
|
153
|
+
|
|
154
|
+
2. **Originality is the #1 lever.** The widest gap (-2.17) is originality — the model defaults
|
|
155
|
+
to template-like output. The archetype sampling system (#883) directly targets this by forcing
|
|
156
|
+
commitment to a coherent design direction before generation.
|
|
157
|
+
|
|
158
|
+
3. **Slop is measurable and actionable.** Mean 5.0 AI tells per page is high. The anti-slop
|
|
159
|
+
negatives list (#880) and the design-system linter (#884) can mechanically eliminate many of
|
|
160
|
+
these.
|
|
161
|
+
|
|
162
|
+
4. **Variance contains signal.** Plume run 2 (4/5, 2 slop) vs Plume run 1 (3/5, 6 slop) shows
|
|
163
|
+
the model CAN produce better output when pushed away from defaults. The harness's job is to
|
|
164
|
+
make that the floor, not a lucky outlier.
|
|
165
|
+
|
|
166
|
+
5. **The bar to beat**: arm A must achieve >0% win-rate vs C to demonstrate any value. The first
|
|
167
|
+
milestone target should be closing the rubric gap to <1.0 mean delta.
|
|
168
|
+
|
|
169
|
+
## Raw Data
|
|
170
|
+
|
|
171
|
+
- Runs file: `docs/design/fixtures/runs/b-vs-c-baseline/runs.json`
|
|
172
|
+
- Scorecard: `docs/design/fixtures/runs/b-vs-c-baseline/scorecard.json`
|
|
173
|
+
- Arm B HTML: `docs/design/fixtures/runs/b-vs-c-baseline/{product}/arm-b/run-{1,2,3}.html`
|
|
174
|
+
|
|
175
|
+
## Scorecard JSON
|
|
176
|
+
|
|
177
|
+
```json
|
|
178
|
+
{
|
|
179
|
+
"corpus_version": "seed-v1",
|
|
180
|
+
"generation_model": "claude-sonnet-4-6",
|
|
181
|
+
"judge_model": "claude-sonnet-4-6",
|
|
182
|
+
"n_products": 5,
|
|
183
|
+
"n_runs_per_product": 3,
|
|
184
|
+
"win_rate_vs_C": {
|
|
185
|
+
"B": {
|
|
186
|
+
"mean": 0.2,
|
|
187
|
+
"by_product": {
|
|
188
|
+
"cadence": 0, "tender": 0, "slipstream": 1,
|
|
189
|
+
"voltage": 0, "plume": 0
|
|
190
|
+
},
|
|
191
|
+
"mean_clean": 0,
|
|
192
|
+
"note": "Clean excludes slipstream (calibration artifact)"
|
|
193
|
+
}
|
|
194
|
+
},
|
|
195
|
+
"rubric_mean": {
|
|
196
|
+
"B": 3.10,
|
|
197
|
+
"C": 4.79,
|
|
198
|
+
"delta": -1.69
|
|
199
|
+
},
|
|
200
|
+
"slop_mean": {
|
|
201
|
+
"B": 5.0,
|
|
202
|
+
"C": 0.5
|
|
203
|
+
},
|
|
204
|
+
"judge_calibration": {
|
|
205
|
+
"ok": false,
|
|
206
|
+
"miscalibrated_runs": 3,
|
|
207
|
+
"all_in": "slipstream (viewport-only C screenshot)"
|
|
208
|
+
}
|
|
209
|
+
}
|
|
210
|
+
```
|
|
@@ -0,0 +1,205 @@
|
|
|
1
|
+
# Full A-vs-B-vs-C Benchmark Scorecard
|
|
2
|
+
|
|
3
|
+
> **Note**: This document covers two benchmark runs. The **Opus run is the canonical baseline** going forward.
|
|
4
|
+
> The Sonnet run below is preserved as historical comparison data.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Opus Run — Canonical Baseline
|
|
9
|
+
|
|
10
|
+
> **Benchmark date**: 2026-06-22
|
|
11
|
+
> **Rig**: `/design-bench` (#878)
|
|
12
|
+
> **Generation model (A & B)**: claude-opus-4-6
|
|
13
|
+
> **Judge model**: claude-opus-4-6 (independent)
|
|
14
|
+
> **Corpus version**: seed-v1
|
|
15
|
+
> **Products**: 5 (Cadence, Tender, Slipstream, Voltage, Plume)
|
|
16
|
+
> **Runs per product**: 3
|
|
17
|
+
> **Raw data**: `docs/design/fixtures/runs/full-abc-opus/`
|
|
18
|
+
|
|
19
|
+
| Metric | Value |
|
|
20
|
+
|--------|-------|
|
|
21
|
+
| **A vs B pairwise win-rate** | **90.0%** |
|
|
22
|
+
| **A rubric mean** | **4.11/5** |
|
|
23
|
+
| **B rubric mean** | 3.18/5 |
|
|
24
|
+
| **A slop count (avg)** | **1.2** |
|
|
25
|
+
| **B slop count (avg)** | 3.8 |
|
|
26
|
+
| **Judge calibration flags** | 4 (3 from Linear viewport artifact; 1 genuine) |
|
|
27
|
+
|
|
28
|
+
**Conclusion**: Opus produces tier-level better designs when guided by the harness. The 90% win-rate and 4.11 rubric mean against raw Opus output (arm B) confirm the harness itself is the improvement driver — Opus simply amplifies every layer of the harness more effectively than Sonnet. `claude-opus-4-6` is the validated generation model going forward.
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## Sonnet Run — Historical Comparison
|
|
33
|
+
|
|
34
|
+
> **Benchmark date**: 2026-06-22
|
|
35
|
+
> **Rig**: `/design-bench` (#878)
|
|
36
|
+
> **Generation model (A & B)**: claude-sonnet-4-6
|
|
37
|
+
> **Judge model**: claude-sonnet-4-6 (independent)
|
|
38
|
+
> **Corpus version**: seed-v1
|
|
39
|
+
> **Products**: 5 (Cadence, Tender, Slipstream, Voltage, Plume)
|
|
40
|
+
> **Runs per product**: 3
|
|
41
|
+
> **Baseline**: [B-vs-C baseline](b-vs-c-baseline.md) (#879)
|
|
42
|
+
|
|
43
|
+
First full benchmark of the ForgeDock UI Taste Harness (arm A) against raw frontier model
|
|
44
|
+
output (arm B) and real reference pages (arm C).
|
|
45
|
+
|
|
46
|
+
## Headline Results
|
|
47
|
+
|
|
48
|
+
| Metric | Value | vs Baseline |
|
|
49
|
+
|--------|-------|-------------|
|
|
50
|
+
| **A win-rate vs B (clean)** | **54.2%** | New metric |
|
|
51
|
+
| **A rubric mean (clean)** | **3.40** | +0.30 vs B baseline (3.10) |
|
|
52
|
+
| **B rubric mean (clean)** | **3.31** | +0.21 vs B baseline (3.10) |
|
|
53
|
+
| **A slop (clean)** | **1.8** | -3.2 vs B baseline (5.0) |
|
|
54
|
+
| **B slop (clean)** | **2.9** | -2.1 vs B baseline (5.0) |
|
|
55
|
+
| **A vs C rubric gap** | **-1.13** | Closed from -1.69 (baseline B-C gap) |
|
|
56
|
+
|
|
57
|
+
**The harness works.** Arm A beats raw model output in 54% of head-to-head comparisons, has
|
|
58
|
+
64% fewer AI tells (1.8 vs 5.0 baseline), and closes the rubric gap to C by 0.56 points.
|
|
59
|
+
But the improvement is inconsistent — high variance across products reveals specific failure modes.
|
|
60
|
+
|
|
61
|
+
## A-vs-B Pairwise Results (the harness delta)
|
|
62
|
+
|
|
63
|
+
| Product | A wins | B wins | Ties | A win-rate |
|
|
64
|
+
|---------|--------|--------|------|------------|
|
|
65
|
+
| Cadence | 0 | 2 | 1 | 16.7% |
|
|
66
|
+
| Tender | 2 | 1 | 0 | 66.7% |
|
|
67
|
+
| Slipstream* | 3 | 0 | 0 | 100% |
|
|
68
|
+
| Voltage | 3 | 0 | 0 | 100% |
|
|
69
|
+
| Plume | 1 | 2 | 0 | 33.3% |
|
|
70
|
+
| **Clean total** | **6** | **5** | **1** | **54.2%** |
|
|
71
|
+
| **All** | **9** | **5** | **1** | **63.3%** |
|
|
72
|
+
|
|
73
|
+
*Slipstream excluded from clean metrics (arm C viewport artifact).
|
|
74
|
+
|
|
75
|
+
### Where A dominates
|
|
76
|
+
|
|
77
|
+
**Voltage**: A won all 3 runs. The harness's archetype commitment produced distinctly different
|
|
78
|
+
pages (technical-dense, bold-brutalist, minimal-luxury) where B defaulted to generic dark
|
|
79
|
+
developer templates. Run 2 (bold-brutalist, black+yellow) scored 5/5 on hierarchy, typography,
|
|
80
|
+
color, and originality — the single best arm A result in the benchmark.
|
|
81
|
+
|
|
82
|
+
**Tender**: A won 2/3. The harness produced more disciplined, less sloppy output with distinctive
|
|
83
|
+
editorial approaches (live request log hero, ledger-line financial doc aesthetic, SVG payment
|
|
84
|
+
topology diagram).
|
|
85
|
+
|
|
86
|
+
### Where A struggles
|
|
87
|
+
|
|
88
|
+
**Cadence**: B won 2/3. The judge found arm A pages had **sparse content and excessive whitespace** —
|
|
89
|
+
the harness's restraint doctrine over-corrected, producing pages that looked incomplete rather
|
|
90
|
+
than refined. B's more complete, content-dense pages read as more professional despite higher slop.
|
|
91
|
+
|
|
92
|
+
**Plume**: B won 2/3. Same failure mode — arm A run 1 was "nearly a stub page" per the judge.
|
|
93
|
+
The harness committed to restraint but under-delivered on content density.
|
|
94
|
+
|
|
95
|
+
**Root cause**: The harness doctrines (minimal-luxury archetype, whitespace rhythm, restraint)
|
|
96
|
+
can produce pages that are elegant in concept but empty in execution. The critique loop should
|
|
97
|
+
catch this ("is the page complete?") but didn't consistently.
|
|
98
|
+
|
|
99
|
+
## Rubric Comparison (1-5, clean means)
|
|
100
|
+
|
|
101
|
+
| Dimension | A | B | C | A-B delta | A-C gap |
|
|
102
|
+
|-----------|---|---|---|-----------|---------|
|
|
103
|
+
| Hierarchy | 3.50 | 3.67 | 5.00 | -0.17 | -1.50 |
|
|
104
|
+
| Typography | 3.92 | 3.50 | 4.80 | **+0.42** | -0.88 |
|
|
105
|
+
| Color | 3.42 | 3.25 | 4.80 | **+0.17** | -1.38 |
|
|
106
|
+
| Whitespace | 3.33 | 3.17 | 4.40 | **+0.16** | -1.07 |
|
|
107
|
+
| Originality | 3.42 | 2.83 | 4.40 | **+0.59** | -0.98 |
|
|
108
|
+
| Mobile | 2.83 | 3.42 | 4.20 | -0.59 | -1.37 |
|
|
109
|
+
| **Mean** | **3.40** | **3.31** | **4.53** | **+0.09** | **-1.13** |
|
|
110
|
+
|
|
111
|
+
### Key findings
|
|
112
|
+
|
|
113
|
+
- **Originality is the biggest A win (+0.59)**: The archetype sampling system works — committed
|
|
114
|
+
design directions (bold-brutalist yellow, editorial serif, minimal sage-green) produce more
|
|
115
|
+
distinctive output than B's template defaults. This was the widest gap in the baseline (-2.17)
|
|
116
|
+
and is now partially closed.
|
|
117
|
+
|
|
118
|
+
- **Typography improved (+0.42)**: Non-default typeface requirement (Fraunces, Space Grotesk,
|
|
119
|
+
Playfair Display, Syne) produces better typographic quality than B's Inter/system defaults.
|
|
120
|
+
|
|
121
|
+
- **Mobile is the biggest A loss (-0.59)**: The harness's focus on desktop composition (asymmetric
|
|
122
|
+
layouts, editorial grids, signature moves) may not adapt well to mobile. B's simpler Tailwind
|
|
123
|
+
layouts are inherently more responsive.
|
|
124
|
+
|
|
125
|
+
- **Hierarchy is flat (-0.17)**: Surprising — the harness should improve hierarchy. The sparse-page
|
|
126
|
+
failure mode (Cadence, Plume run 1) drags this down significantly.
|
|
127
|
+
|
|
128
|
+
## Slop Comparison
|
|
129
|
+
|
|
130
|
+
| Arm | Mean (clean) | Baseline B | Improvement |
|
|
131
|
+
|-----|-------------|------------|-------------|
|
|
132
|
+
| A | **1.8** | 5.0 | **-64%** |
|
|
133
|
+
| B | **2.9** | 5.0 | -42% |
|
|
134
|
+
| C | **1.0** | 0.5 | — |
|
|
135
|
+
|
|
136
|
+
The harness nearly halved AI tells from the B baseline. The anti-slop negatives checklist
|
|
137
|
+
and archetype commitment together eliminate most template-like patterns. Notable: B also
|
|
138
|
+
improved from 5.0 to 2.9 — this is judge variance, not a real B improvement (same HTML files,
|
|
139
|
+
different judge agents).
|
|
140
|
+
|
|
141
|
+
## Per-Product Detail
|
|
142
|
+
|
|
143
|
+
### Voltage (A dominates: 3/3)
|
|
144
|
+
|
|
145
|
+
Best harness performance. Run 2 (bold-brutalist) is the benchmark's peak output:
|
|
146
|
+
- 5/5 hierarchy, typography, color, originality; 0 slop
|
|
147
|
+
- Judge: "Run 2's black/yellow newspaper-scale display type... genuinely distinctive"
|
|
148
|
+
- Judge: "A clearly closes the gap between B and C... meaningful work"
|
|
149
|
+
|
|
150
|
+
### Tender (A leads: 2/3)
|
|
151
|
+
|
|
152
|
+
Strong harness performance except run 2 (sparse rendering).
|
|
153
|
+
- Run 1 (technical-dense, live request log): 4/5 across the board, 1 slop
|
|
154
|
+
- Run 3 (editorial, SVG payment topology): 4/5, 1 slop
|
|
155
|
+
- Judge: "A closes the gap meaningfully in runs 1 and 3"
|
|
156
|
+
|
|
157
|
+
### Cadence (B leads: 2/3)
|
|
158
|
+
|
|
159
|
+
Harness over-indexed on restraint, producing sparse pages.
|
|
160
|
+
- Judge: "All three A runs suffer from incomplete content and excessive blank space"
|
|
161
|
+
- A slop was actually higher than B here (4.0 vs 3.3) — unusual
|
|
162
|
+
|
|
163
|
+
### Plume (B leads: 2/3)
|
|
164
|
+
|
|
165
|
+
Same sparse-page failure. Run 2 (bold-brutalist, black+yellow) was the exception:
|
|
166
|
+
- Run 2: originality 5/5, hierarchy 4/5 — genuinely competitive
|
|
167
|
+
- But run 1 was "nearly a stub page" — the worst arm A output
|
|
168
|
+
|
|
169
|
+
## Implications
|
|
170
|
+
|
|
171
|
+
### What works
|
|
172
|
+
1. **Archetype sampling** eliminates mode-collapse. Every A page has a distinctive identity.
|
|
173
|
+
2. **Anti-slop checklist** cuts AI tells by 64%. The negatives are mechanically effective.
|
|
174
|
+
3. **Non-default typefaces** lift typography quality measurably (+0.42 vs B).
|
|
175
|
+
4. **When the harness commits to density** (Voltage, Tender runs 1+3), it produces genuinely
|
|
176
|
+
professional output.
|
|
177
|
+
|
|
178
|
+
### What needs fixing
|
|
179
|
+
1. **Content completeness gate**: The critique loop must verify pages have sufficient content
|
|
180
|
+
density before passing. "Elegant but empty" is a failure mode the current loop doesn't catch.
|
|
181
|
+
2. **Mobile responsiveness**: The harness's desktop-first signature moves (asymmetric grids,
|
|
182
|
+
editorial columns) need mobile adaptation in the spec.
|
|
183
|
+
3. **Consistency**: A's variance (stdev 0.82-0.99) is higher than B's (0.56-0.70). The harness
|
|
184
|
+
needs guardrails against stub-page outputs.
|
|
185
|
+
|
|
186
|
+
### Target for next benchmark
|
|
187
|
+
- A win-rate vs B > 75% (currently 54%)
|
|
188
|
+
- A rubric mean > 4.0 (currently 3.40)
|
|
189
|
+
- A slop < 1.0 (currently 1.8)
|
|
190
|
+
- Zero stub-page outputs
|
|
191
|
+
|
|
192
|
+
## Raw Data
|
|
193
|
+
|
|
194
|
+
### Opus run (canonical baseline)
|
|
195
|
+
- Runs: `docs/design/fixtures/runs/full-abc-opus/runs.json`
|
|
196
|
+
- Scorecard: `docs/design/fixtures/runs/full-abc-opus/scorecard.json`
|
|
197
|
+
- Arm A HTML: `docs/design/fixtures/runs/full-abc-opus/{product}/arm-a/run-{1,2,3}.html`
|
|
198
|
+
- Arm B HTML: `docs/design/fixtures/runs/full-abc-opus/{product}/arm-b/run-{1,2,3}.html`
|
|
199
|
+
|
|
200
|
+
### Sonnet run (historical comparison)
|
|
201
|
+
- Runs: `docs/design/fixtures/runs/full-abc/runs.json`
|
|
202
|
+
- Scorecard: `docs/design/fixtures/runs/full-abc/scorecard.json`
|
|
203
|
+
- Arm A HTML: `docs/design/fixtures/runs/full-abc/{product}/arm-a/run-{1,2,3}.html`
|
|
204
|
+
- Arm B HTML: `docs/design/fixtures/runs/full-abc/{product}/arm-b/run-{1,2,3}.html`
|
|
205
|
+
- B-vs-C baseline: `docs/design/fixtures/runs/b-vs-c-baseline/`
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Selective Spec Loading: Measured Token Savings From a Self-Map of Command Specs"
|
|
3
|
+
description: "ForgeDock symlinks ~27 command specs (~1.1 MB) into every Claude Code session. We built a queryable graph of the specs and load only the ones a task actually reaches — and measured how many tokens that saves."
|
|
4
|
+
tags: ["ai", "performance", "tokens", "productivity", "opensource"]
|
|
5
|
+
cover_image:
|
|
6
|
+
canonical_url: https://github.com/RapierCraftStudios/ForgeDock
|
|
7
|
+
published: false
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
<!-- Generated by ForgeDock pipeline — Issue #868 -->
|
|
11
|
+
|
|
12
|
+
AI coding agents pay for context by the token. ForgeDock's pipeline is driven by
|
|
13
|
+
markdown command specs — `work-on`, `review-pr`, `quality-gate`, and ~40 more —
|
|
14
|
+
that are symlinked into `~/.claude/commands/` and therefore loaded into the model's
|
|
15
|
+
context at session start, whether or not a given run ever touches them. This
|
|
16
|
+
article measures that cost and quantifies how a **self-map of the specs** lets a
|
|
17
|
+
run load only the specs it actually reaches.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## The Problem: The Whole Corpus Loads Even Though One Command Runs
|
|
22
|
+
|
|
23
|
+
A single ForgeDock run invokes exactly one entry command — usually `work-on`.
|
|
24
|
+
That command reaches a small, well-defined set of downstream specs: its own
|
|
25
|
+
sub-phases (`work-on/build/*`, `work-on/review`, `work-on/close`) and the review
|
|
26
|
+
orchestrator it delegates to (`review-pr`). It never reads `pipeline-health.md`,
|
|
27
|
+
`geo-audit.md`, `audit.md`, `incident-response.md`, or the two dozen other
|
|
28
|
+
specs that have nothing to do with the issue at hand.
|
|
29
|
+
|
|
30
|
+
But all of them are present in context. Measured on the current spec set:
|
|
31
|
+
|
|
32
|
+
| Metric | Value |
|
|
33
|
+
|--------|-------|
|
|
34
|
+
| Command spec files | 44 |
|
|
35
|
+
| Total bytes | 1,104,004 |
|
|
36
|
+
| Approx. tokens (1 token ≈ 4 chars) | **~276,000** |
|
|
37
|
+
|
|
38
|
+
That is ~276K tokens of spec text loaded on every session — the overwhelming
|
|
39
|
+
majority of which is dead weight for any single task.
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## The Mechanism: A Queryable Graph of the Specs
|
|
44
|
+
|
|
45
|
+
ForgeDock already builds a **spec knowledge graph** — a zero-dependency self-map
|
|
46
|
+
of its own command specs (`scripts/build-spec-graph.mjs`, queried by
|
|
47
|
+
`scripts/graph-query.sh`). The graph records which command *contains* which
|
|
48
|
+
sub-phase, which command *requires* which devdoc, which annotation a command
|
|
49
|
+
*reads* or *writes*, and so on.
|
|
50
|
+
|
|
51
|
+
The graph already answered "what is the blast radius if I change node X?" via its
|
|
52
|
+
`impact` query (transitive **reverse** reachability). Selective loading needs the
|
|
53
|
+
mirror image: the **forward** transitive reachable set of a command. We added a
|
|
54
|
+
`load-set` subcommand that does exactly that:
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
$ graph-query.sh load-set work-on
|
|
58
|
+
["commands/review-pr.md","commands/work-on.md","commands/work-on/build.md",
|
|
59
|
+
"commands/work-on/build/architect.md","commands/work-on/build/context.md",
|
|
60
|
+
"commands/work-on/build/implement.md","commands/work-on/build/validate.md",
|
|
61
|
+
"commands/work-on/close.md","commands/work-on/decompose.md",
|
|
62
|
+
"commands/work-on/investigate.md","commands/work-on/review.md"]
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
`load-set` seeds at the command node and walks `CONTAINS` (sub-phases) and
|
|
66
|
+
`REQUIRES` (devdocs) edges forward to a fixpoint, then maps the reachable nodes to
|
|
67
|
+
their repo-relative file paths. It is read-only, auto-builds the graph if the
|
|
68
|
+
gitignored JSON is absent, and shares the existing `impact` resolver's cycle-safe
|
|
69
|
+
termination. No graph logic was reimplemented — the feature *consumes* the
|
|
70
|
+
builder and query script that were already there.
|
|
71
|
+
|
|
72
|
+
`work-on` Phase 0 now calls `load-set work-on` and reads **only** the returned
|
|
73
|
+
paths instead of relying on the whole symlinked corpus.
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## Methodology
|
|
78
|
+
|
|
79
|
+
- Token counts are estimated at the standard rate of **1 token ≈ 4 characters**.
|
|
80
|
+
- **Baseline (load-all)** = the byte sum of every spec under `commands/`.
|
|
81
|
+
- **Selective (load-set)** = `wc -c` of just the files returned by
|
|
82
|
+
`graph-query.sh load-set <command>` for the entry command of a session.
|
|
83
|
+
- Measured across **five representative sessions**, each keyed by its entry command.
|
|
84
|
+
|
|
85
|
+
This mirrors the methodology used in the per-repo adaptive-scripts measurement,
|
|
86
|
+
so the numbers are directly comparable.
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## Baseline
|
|
91
|
+
|
|
92
|
+
Every session, with the load-all corpus:
|
|
93
|
+
|
|
94
|
+
| | Files | Bytes | Approx. tokens |
|
|
95
|
+
|--|------:|------:|---------------:|
|
|
96
|
+
| **Load-all corpus** | 44 | 1,104,004 | ~276,000 |
|
|
97
|
+
|
|
98
|
+
---
|
|
99
|
+
|
|
100
|
+
## Target: Per-Session Selective Load
|
|
101
|
+
|
|
102
|
+
| Session (entry command) | Files | Bytes | Approx. tokens | Tokens saved vs baseline | % saved |
|
|
103
|
+
|-------------------------|------:|------:|---------------:|-------------------------:|--------:|
|
|
104
|
+
| `work-on` (full feature build) | 11 | 296,172 | ~74,000 | ~202,000 | **73%** |
|
|
105
|
+
| `orchestrate` (multi-issue) | 14 | 390,054 | ~98,000 | ~178,000 | **65%** |
|
|
106
|
+
| `review-pr` (PR review) | 1 | 64,395 | ~16,000 | ~260,000 | **94%** |
|
|
107
|
+
| `quality-gate` (pre-commit) | 1 | 47,676 | ~12,000 | ~264,000 | **96%** |
|
|
108
|
+
| `issue` (fast-lane triage) | 1 | 19,945 | ~5,000 | ~271,000 | **98%** |
|
|
109
|
+
|
|
110
|
+
---
|
|
111
|
+
|
|
112
|
+
## Savings
|
|
113
|
+
|
|
114
|
+
Across the five representative sessions, selective loading removes between
|
|
115
|
+
**~178,000 and ~271,000 tokens** of spec text from context — a **65%–98%**
|
|
116
|
+
reduction depending on how much of the pipeline the entry command actually
|
|
117
|
+
reaches. The heaviest run (`orchestrate`, which fans out to many sub-phases)
|
|
118
|
+
still drops nearly two-thirds of the corpus; the lightest (`issue`) drops 98%.
|
|
119
|
+
|
|
120
|
+
The win scales with breadth: the more specialized commands ForgeDock grows, the
|
|
121
|
+
larger the dead-weight fraction the load-all corpus carries — and the more a run
|
|
122
|
+
saves by loading only what its graph neighborhood reaches. Because the resolution
|
|
123
|
+
is graph-driven and deterministic (the transitive walk lives in `jq`, not in the
|
|
124
|
+
prompt), adding a new command automatically narrows the set for every *other*
|
|
125
|
+
command, with no manual bookkeeping.
|
|
126
|
+
|
|
127
|
+
---
|
|
128
|
+
|
|
129
|
+
## How To Reproduce
|
|
130
|
+
|
|
131
|
+
```bash
|
|
132
|
+
# Baseline: full corpus
|
|
133
|
+
find commands -name '*.md' -type f -exec cat {} + | wc -c
|
|
134
|
+
|
|
135
|
+
# Selective: minimal set for a given entry command
|
|
136
|
+
graph-query.sh load-set work-on | jq -r '.[]' \
|
|
137
|
+
| while read -r p; do cat "$p"; done | wc -c
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
Divide bytes by 4 for the approximate token count.
|