@codewizard-dt/bootstrap 2.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (186) hide show
  1. package/.github/workflows/build.yml +88 -0
  2. package/.github/workflows/security.yml +24 -0
  3. package/.gitleaks.toml +15 -0
  4. package/README.md +472 -0
  5. package/bin/cli.js +47 -0
  6. package/lib/README.md +27 -0
  7. package/lib/hooks/README.md +316 -0
  8. package/lib/hooks/env-file-guard.js +45 -0
  9. package/lib/hooks/git-protected-ops-block.js +77 -0
  10. package/lib/hooks/lib/serena-languages.js +158 -0
  11. package/lib/hooks/lib/serena.js +787 -0
  12. package/lib/hooks/mv-absolute-path-block.js +53 -0
  13. package/lib/hooks/serena-bash-grep-block.js +501 -0
  14. package/lib/hooks/serena-edit-guard.js +74 -0
  15. package/lib/hooks/serena-first-glob-guard.js +94 -0
  16. package/lib/hooks/serena-first-guard.js +74 -0
  17. package/lib/hooks/serena-first-read-guard.js +167 -0
  18. package/lib/hooks/serena-pre-delegation.js +117 -0
  19. package/lib/hooks/serena-session-reset.js +45 -0
  20. package/lib/hooks/serena-usage-tracker.js +186 -0
  21. package/lib/hooks/serena-write-guard.js +58 -0
  22. package/lib/prompts/README.md +16 -0
  23. package/lib/prompts/migrate-wiki.md +111 -0
  24. package/lib/prompts/setup-deployment.md +322 -0
  25. package/lib/prompts/setup-design-system.md +162 -0
  26. package/lib/prompts/setup-strict-typechecks.md +139 -0
  27. package/lib/scripts/README.md +49 -0
  28. package/lib/scripts/bootstrap-serena.sh +145 -0
  29. package/lib/scripts/build-mcp-guide.sh +108 -0
  30. package/lib/scripts/install-global.sh +78 -0
  31. package/lib/scripts/install-mcps.sh +131 -0
  32. package/lib/scripts/lib.sh +154 -0
  33. package/lib/scripts/merge-gitignore.sh +65 -0
  34. package/lib/scripts/migrate-project.sh +273 -0
  35. package/lib/scripts/setup-deployment.sh +104 -0
  36. package/lib/scripts/setup-project.sh +55 -0
  37. package/lib/scripts/setup-runner.sh +99 -0
  38. package/lib/scripts/setup-strict-typechecks.sh +29 -0
  39. package/lib/scripts/startup.sh +42 -0
  40. package/lib/scripts/sync-wiki-scaffold.sh +104 -0
  41. package/lib/scripts/templates/CLAUDE-env-safety.md +3 -0
  42. package/lib/scripts/templates/CLAUDE-wiki.md +60 -0
  43. package/lib/scripts/templates/gitignore +136 -0
  44. package/lib/scripts/templates/guides/stubs/00-header.md +15 -0
  45. package/lib/scripts/templates/guides/stubs/brave-search.md +20 -0
  46. package/lib/scripts/templates/guides/stubs/context7.md +26 -0
  47. package/lib/scripts/templates/guides/stubs/playwright.md +26 -0
  48. package/lib/scripts/templates/guides/stubs/serena.md +219 -0
  49. package/lib/scripts/templates/wiki/conventions.md +100 -0
  50. package/lib/scripts/templates/wiki/dashboard.html +1733 -0
  51. package/lib/scripts/templates/wiki/hot.md +21 -0
  52. package/lib/scripts/templates/wiki/index.md +46 -0
  53. package/lib/scripts/templates/wiki/knowledge/concepts/.gitkeep +0 -0
  54. package/lib/scripts/templates/wiki/knowledge/entities/components/.gitkeep +0 -0
  55. package/lib/scripts/templates/wiki/knowledge/entities/organisations/.gitkeep +0 -0
  56. package/lib/scripts/templates/wiki/knowledge/entities/people/.gitkeep +0 -0
  57. package/lib/scripts/templates/wiki/knowledge/entities/tools/.gitkeep +0 -0
  58. package/lib/scripts/templates/wiki/knowledge/sources/.gitkeep +0 -0
  59. package/lib/scripts/templates/wiki/log.md +14 -0
  60. package/lib/scripts/templates/wiki/work/bugs/.gitkeep +0 -0
  61. package/lib/scripts/templates/wiki/work/bugs/archive/index.md +6 -0
  62. package/lib/scripts/templates/wiki/work/bugs/index.md +12 -0
  63. package/lib/scripts/templates/wiki/work/bugs/lifecycle.md +39 -0
  64. package/lib/scripts/templates/wiki/work/decisions/.gitkeep +0 -0
  65. package/lib/scripts/templates/wiki/work/decisions/archive/index.md +6 -0
  66. package/lib/scripts/templates/wiki/work/decisions/index.md +12 -0
  67. package/lib/scripts/templates/wiki/work/decisions/lifecycle.md +42 -0
  68. package/lib/scripts/templates/wiki/work/requirements/.gitkeep +0 -0
  69. package/lib/scripts/templates/wiki/work/requirements/archive/index.md +6 -0
  70. package/lib/scripts/templates/wiki/work/requirements/index.md +12 -0
  71. package/lib/scripts/templates/wiki/work/requirements/lifecycle.md +34 -0
  72. package/lib/scripts/templates/wiki/work/roadmaps/.gitkeep +0 -0
  73. package/lib/scripts/templates/wiki/work/roadmaps/archive/index.md +6 -0
  74. package/lib/scripts/templates/wiki/work/roadmaps/index.md +12 -0
  75. package/lib/scripts/templates/wiki/work/roadmaps/lifecycle.md +32 -0
  76. package/lib/scripts/templates/wiki/work/tasks/.gitkeep +0 -0
  77. package/lib/scripts/templates/wiki/work/tasks/archive/index.md +6 -0
  78. package/lib/scripts/templates/wiki/work/tasks/index.md +12 -0
  79. package/lib/scripts/templates/wiki/work/tasks/lifecycle.md +39 -0
  80. package/lib/scripts/templates/wiki/work/uat/.gitkeep +0 -0
  81. package/lib/scripts/templates/wiki/work/uat/archive/index.md +6 -0
  82. package/lib/scripts/templates/wiki/work/uat/index.md +12 -0
  83. package/lib/scripts/templates/wiki/work/uat/lifecycle.md +48 -0
  84. package/lib/scripts/templates/wiki/work/uat/screenshots/.gitkeep +0 -0
  85. package/lib/scripts/update-project.sh +75 -0
  86. package/lib/scripts/wiki-dashboard-server.js +146 -0
  87. package/lib/skills/README.md +146 -0
  88. package/lib/skills/anti-slop/SKILL.md +183 -0
  89. package/lib/skills/bug-close/SKILL.md +136 -0
  90. package/lib/skills/bug-file/SKILL.md +114 -0
  91. package/lib/skills/bug-triage/SKILL.md +140 -0
  92. package/lib/skills/company-align/SKILL.md +203 -0
  93. package/lib/skills/debug-logs/SKILL.md +129 -0
  94. package/lib/skills/decision-create/SKILL.md +220 -0
  95. package/lib/skills/decision-finalize/SKILL.md +181 -0
  96. package/lib/skills/decision-next/SKILL.md +129 -0
  97. package/lib/skills/decision-walkthrough/SKILL.md +169 -0
  98. package/lib/skills/demo/SKILL.md +183 -0
  99. package/lib/skills/elevator-pitch/SKILL.md +125 -0
  100. package/lib/skills/eval-create/SKILL.md +344 -0
  101. package/lib/skills/eval-gap/SKILL.md +150 -0
  102. package/lib/skills/eval-run/SKILL.md +214 -0
  103. package/lib/skills/extract-feature/SKILL.md +344 -0
  104. package/lib/skills/flashcard/SKILL.md +513 -0
  105. package/lib/skills/frontend-taste/SKILL.md +184 -0
  106. package/lib/skills/gap-assess/SKILL.md +97 -0
  107. package/lib/skills/git-commit/SKILL.md +164 -0
  108. package/lib/skills/lint/SKILL.md +99 -0
  109. package/lib/skills/marp-slideshow/SKILL.md +212 -0
  110. package/lib/skills/mermaid-flowchart/SKILL.md +194 -0
  111. package/lib/skills/now/SKILL.md +36 -0
  112. package/lib/skills/port-feature/SKILL.md +315 -0
  113. package/lib/skills/power-mode/SKILL.md +184 -0
  114. package/lib/skills/primer/SKILL.md +84 -0
  115. package/lib/skills/project-readme/SKILL.md +166 -0
  116. package/lib/skills/req-compile/SKILL.md +131 -0
  117. package/lib/skills/req-create/SKILL.md +124 -0
  118. package/lib/skills/req-extract-decisions/SKILL.md +241 -0
  119. package/lib/skills/req-finalize/SKILL.md +251 -0
  120. package/lib/skills/req-retire/SKILL.md +185 -0
  121. package/lib/skills/req-update/SKILL.md +265 -0
  122. package/lib/skills/research/SKILL.md +250 -0
  123. package/lib/skills/research-company/SKILL.md +345 -0
  124. package/lib/skills/roadmap-add/SKILL.md +238 -0
  125. package/lib/skills/roadmap-create/SKILL.md +223 -0
  126. package/lib/skills/roadmap-next/SKILL.md +132 -0
  127. package/lib/skills/security-audit/SKILL.md +246 -0
  128. package/lib/skills/serena-config/SKILL.md +92 -0
  129. package/lib/skills/simplify/SKILL.md +227 -0
  130. package/lib/skills/tackle/SKILL.md +156 -0
  131. package/lib/skills/task-add/SKILL.md +208 -0
  132. package/lib/skills/task-audit/SKILL.md +294 -0
  133. package/lib/skills/task-trash/SKILL.md +115 -0
  134. package/lib/skills/task-update/SKILL.md +136 -0
  135. package/lib/skills/uat-auto/SKILL.md +69 -0
  136. package/lib/skills/uat-auto-plus/SKILL.md +115 -0
  137. package/lib/skills/uat-generate/SKILL.md +289 -0
  138. package/lib/skills/uat-skip/SKILL.md +169 -0
  139. package/lib/skills/uat-walk/SKILL.md +164 -0
  140. package/lib/skills/uat-walk/UAT-CORE.md +140 -0
  141. package/lib/skills/update-docs/SKILL.md +75 -0
  142. package/lib/skills/wiki-archive/SKILL.md +96 -0
  143. package/lib/skills/wiki-ingest/SKILL.md +139 -0
  144. package/lib/skills/wiki-lint/SKILL.md +131 -0
  145. package/lib/skills/wiki-query/SKILL.md +71 -0
  146. package/lib/skills/wiki-rotate-log/SKILL.md +90 -0
  147. package/lib/skills/wiki-tidy/SKILL.md +107 -0
  148. package/package.json +32 -0
  149. package/raw/.gitkeep +0 -0
  150. package/raw/design-principles.md +65 -0
  151. package/raw/guides/command-anti-patterns.md +79 -0
  152. package/raw/guides/deployment-strategy.md +448 -0
  153. package/raw/guides/evals-framework.md +452 -0
  154. package/raw/guides/mcp-tools.md +335 -0
  155. package/raw/guides/task-spec.md +79 -0
  156. package/raw/guides/type-checking-templates/README.md +112 -0
  157. package/raw/guides/type-checking-templates/eslint/eslint.config.ts +105 -0
  158. package/raw/guides/type-checking-templates/python/mypy.ini +69 -0
  159. package/raw/guides/type-checking-templates/python/pyproject.toml +164 -0
  160. package/raw/guides/type-checking-templates/python/pyrightconfig.json +67 -0
  161. package/raw/guides/type-checking-templates/tsconfig/tsconfig.app.jsonc +45 -0
  162. package/raw/guides/type-checking-templates/tsconfig/tsconfig.lib.jsonc +44 -0
  163. package/raw/guides/type-checking-templates/tsconfig/tsconfig.node-esm.jsonc +40 -0
  164. package/raw/guides/type-checking-templates/tsconfig/tsconfig.node.jsonc +34 -0
  165. package/raw/guides/type-checking-templates/tsconfig/tsconfig.vite-root.json +8 -0
  166. package/raw/harness-notes.md +20 -0
  167. package/raw/house-style/components/Badge.tsx +16 -0
  168. package/raw/house-style/components/Button.tsx +32 -0
  169. package/raw/house-style/components/Card.tsx +26 -0
  170. package/raw/house-style/components/Chip.tsx +57 -0
  171. package/raw/house-style/components/IconButton.tsx +67 -0
  172. package/raw/house-style/components/Modal.tsx +18 -0
  173. package/raw/house-style/components/Stepper.tsx +141 -0
  174. package/raw/house-style/components/icons.tsx +128 -0
  175. package/raw/house-style/preview/src/index.css +46 -0
  176. package/raw/house-style/tokens/tailwind.preset.ts +49 -0
  177. package/raw/house-style/tokens/tokens.json +62 -0
  178. package/raw/llm-wiki-2.md +162 -0
  179. package/raw/llm-wiki.md +75 -0
  180. package/raw/research/dark-mode-theme/index.md +85 -0
  181. package/raw/research/dark-mode-theme/sources.md +49 -0
  182. package/raw/research/serena-mcp-disconnect/index.md +81 -0
  183. package/raw/research/serena-mcp-disconnect/sources.md +87 -0
  184. package/raw/research/wiki-tooling-improvements/index.md +126 -0
  185. package/raw/research/wiki-tooling-improvements/sources.md +73 -0
  186. package/raw/software-factory-case-study.pdf +0 -0
@@ -0,0 +1,452 @@
1
+ # Evals That Actually Work: A 5-Stage Framework for Production AI Quality
2
+
3
+ > A generic, project-agnostic guide to building an evaluation suite for any production AI system. Distills the "Evals That Actually Work" framework into the test types, artifacts, cadences, and anti-patterns a team needs to go from *"we tested it manually"* to *"is this better?" is a question we answer with data*.
4
+
5
+ This guide is **not specific to any one codebase**. Drop it into any project that ships LLM-backed features (agents, RAG systems, classifiers, summarizers, judges) and use it as a blueprint for what to build and in what order.
6
+
7
+ ---
8
+
9
+ ## 1. Why evals
10
+
11
+ Without evals you:
12
+
13
+ - Guess whether changes helped
14
+ - Find regressions in production
15
+ - Debate quality subjectively
16
+ - Ship and hope
17
+
18
+ With evals you:
19
+
20
+ - Measure whether changes helped
21
+ - Catch regressions before shipping
22
+ - Compare quality with data
23
+ - Ship and know
24
+
25
+ If your answer to *"how do you know your AI is good?"* is *"we tested it manually"* or *"it looked fine in review,"* you don't have an eval strategy — you have vibes.
26
+
27
+ ---
28
+
29
+ ## 2. The framework: five stages, each builds on the last
30
+
31
+ | # | Stage | Purpose | When it runs |
32
+ |---|-------|---------|--------------|
33
+ | 1 | **Golden Sets** | Baseline correctness | Every commit |
34
+ | 2 | **Labeled Scenarios** | Coverage mapping | Every release |
35
+ | 3 | **Replay Harnesses** | Reproducibility + ML metrics | Weekly / nightly |
36
+ | 4 | **Rubrics** | Multi-dimensional quality scoring | Before shipping |
37
+ | 5 | **Experiments** | Data-driven decisions on changes | On any change |
38
+
39
+ **Start at Stage 1. Add stages as your system matures.** Do not try to build all five at once. A team with 15 good golden cases is shipping better software than a team with a half-finished rubric harness and no regression suite.
40
+
41
+ These stages map cleanly to test layers:
42
+
43
+ - **Unit-grade** → Stage 1 (deterministic, fast, zero LLM cost)
44
+ - **Integration-grade** → Stages 2–3 (recorded fixtures, no live model calls)
45
+ - **E2E-grade** → Stages 4–5 (live system, real cost, release gates)
46
+
47
+ ---
48
+
49
+ ## 3. Stage 1 — Golden Sets
50
+
51
+ **Goal:** A small, curated set of canonical cases that define what "correct" looks like. If these fail, something is fundamentally broken.
52
+
53
+ ### Shape
54
+
55
+ - **Size:** 10–20 cases. Quality > quantity.
56
+ - **Format:** YAML or JSON, one record per case. Lives in a top-level `evals/golden/` directory, *separate from the code under test*, so the eval substrate stays portable.
57
+ - **Cost:** Zero LLM calls. Pure code evals — assertions over the system's recorded output.
58
+ - **Cadence:** Every commit, in CI.
59
+
60
+ ### Schema
61
+
62
+ ```yaml
63
+ - id: "gs-001"
64
+ query: "What is our remote work policy?"
65
+ expected_tools:
66
+ - vector_search
67
+ expected_sources:
68
+ - remote_work_policy.md
69
+ must_contain:
70
+ - "remote"
71
+ - "core hours"
72
+ must_not_contain:
73
+ - "I don't know"
74
+ - "no information"
75
+ ```
76
+
77
+ ### The four check types
78
+
79
+ | Check | What it catches |
80
+ |---|---|
81
+ | **Tool selection** | Agent used the wrong tool |
82
+ | **Source citation** | Agent cited the wrong document |
83
+ | **Content validation** (`must_contain`) | Response is missing key facts |
84
+ | **Negative validation** (`must_not_contain`) | Agent hallucinated or gave up |
85
+
86
+ All four are **deterministic, binary, no LLM needed**. They run as plain `assert` statements over recorded system output.
87
+
88
+ ```python
89
+ # Tool selection
90
+ assert "vector_search" in actual_tools
91
+
92
+ # Source citation
93
+ assert "refund_policy.md" in response_text
94
+
95
+ # Content validation
96
+ assert "30-day" in response_text
97
+ assert "$500" in response_text
98
+
99
+ # Negative validation
100
+ assert "I don't know" not in response_text
101
+ ```
102
+
103
+ ### Four rules for golden sets that stay useful
104
+
105
+ | Rule | Detail |
106
+ |---|---|
107
+ | **Start small** | 10–20 quality cases beats 100 sloppy ones |
108
+ | **Run on every commit** | These *are* your regression tests |
109
+ | **Add from production bugs** | Every confirmed prod bug becomes a new case |
110
+ | **Never** | Change expected output just to make tests pass |
111
+
112
+ ### Artifacts
113
+
114
+ Every run emits per-case verdict files (JSON) and a summary (Markdown). Diffs vs. the baseline surface regressions immediately on PR review.
115
+
116
+ ---
117
+
118
+ ## 4. Stage 2 — Labeled Scenarios
119
+
120
+ **Goal:** Answer *"does it work for **all types**?"* not just *"does it work?"* Golden sets prove correctness on the canonical happy path. Labeled scenarios prove **coverage** across the full input space.
121
+
122
+ ### Shape
123
+
124
+ - **Size:** 30–100+ cases.
125
+ - **Format:** Same as golden sets, plus **tags** for category, subcategory, and difficulty.
126
+ - **Cadence:** Every release (not every commit — too slow / too expensive).
127
+ - **All must pass?** No. You watch pass-rate *trends* per cell, not a binary gate.
128
+
129
+ ### Schema
130
+
131
+ ```yaml
132
+ - id: "sc-m-001"
133
+ query: "What's our refund policy and how many refunds last quarter?"
134
+ expected_tools: ["vector_search", "sql_query"]
135
+ category: multi_tool
136
+ subcategory: vector_and_sql
137
+ difficulty: straightforward
138
+ ```
139
+
140
+ The tags don't change *how the test runs* — they change *what the results tell you.*
141
+
142
+ ### Coverage matrix
143
+
144
+ Aggregate pass-rate by `category × difficulty` (or any two tag axes). Empty cells show you where to write tests next.
145
+
146
+ ```
147
+ | vector | sql | jira | slack | multi |
148
+ ------------------|--------|------|------|-------|-------|
149
+ straightforward | 3/3 | 2/3 | 2/2 | 2/2 | 1/1 |
150
+ ambiguous | 1/1 | 1/2 | 0/1 | 1/1 | 0/1 |
151
+ edge_case | 1/1 | 1/1 | 1/1 | -- | -- |
152
+ ```
153
+
154
+ The empty cells are your test-writing backlog.
155
+
156
+ ### Difficulty bands (suggested)
157
+
158
+ - **straightforward** — single-tool, unambiguous query, golden-path inputs
159
+ - **ambiguous** — multiple plausible interpretations, requires disambiguation
160
+ - **edge_case** — adversarial inputs, rare paths, known-hard examples from production
161
+
162
+ ### Golden Sets vs. Labeled Scenarios
163
+
164
+ | | Golden Sets | Labeled Scenarios |
165
+ |---|---|---|
166
+ | **Question** | "Does it work?" | "Does it work for all types?" |
167
+ | **Size** | 10–20 | 30–100+ |
168
+ | **All must pass?** | Yes | No |
169
+ | **When to run** | Every commit | Every release |
170
+
171
+ ---
172
+
173
+ ## 5. Stage 3 — Replay Harnesses
174
+
175
+ **Goal:** **Record once. Score anytime.** Capture a real session against the live system to a JSON fixture; evaluate that frozen snapshot whenever you want — immediately, next week, or after a human has annotated ground truth.
176
+
177
+ ### Shape
178
+
179
+ ```python
180
+ # Record once (costs tokens)
181
+ session = record_session(
182
+ query="What's our refund policy and how many refunds in Q4?",
183
+ session_id="refund-001",
184
+ )
185
+ # → saves to fixtures/refund-001.json
186
+
187
+ # Replay forever (costs nothing)
188
+ replayed = replay_session("refund-001")
189
+ scores = evaluate_session(replayed)
190
+ ```
191
+
192
+ **Record real production examples.** Real queries make the best test cases — synthetic queries don't surface the weird ways users actually phrase things.
193
+
194
+ ### ML-grade metrics this unlocks
195
+
196
+ | Metric | What it measures |
197
+ |---|---|
198
+ | **Precision** | How many retrieved docs are relevant? |
199
+ | **Recall** | How many relevant docs were retrieved? |
200
+ | **Groundedness** | Is the response grounded in sources? |
201
+ | **Faithfulness** | Does it stay true to sources (no hallucination)? |
202
+ | **Tool accuracy** | Did it use the correct tools? |
203
+
204
+ Precision, recall, and tool accuracy are deterministic. **Groundedness and faithfulness require an LLM judge** — which is where Stage 4 (rubrics) earns its keep.
205
+
206
+ ### Why this is integration-grade
207
+
208
+ Replays let your full pipeline (retrieval → tool calls → generation → judging) run end-to-end against frozen inputs. You exercise the system's wiring without paying live LLM cost per CI run. **Add a cost-per-replay assertion** to catch token-usage regressions automatically.
209
+
210
+ ### Fixture hygiene
211
+
212
+ - Store fixtures under version control alongside the eval suite.
213
+ - Re-record fixtures on a schedule (e.g., monthly) so the test set tracks real system behavior, not a 2024-vintage snapshot.
214
+ - When the system's I/O schema changes, fixtures must be re-recorded — flag this in CI.
215
+
216
+ ---
217
+
218
+ ## 6. Stage 4 — Rubrics & LLM-as-Judge
219
+
220
+ **Goal:** Move from binary (pass/fail) to multi-dimensional *quality* scoring. Use an LLM judge — but **only after calibrating it against humans**.
221
+
222
+ ### When to reach for an LLM judge
223
+
224
+ After you have exhausted code evals. **Use binary checks where you can; reserve LLM judges for what can't be checked programmatically** (groundedness, faithfulness, coherence, helpfulness).
225
+
226
+ ### Anchor every claim against its source
227
+
228
+ The judge's core operation is **claim → source verification**, not free-form rating:
229
+
230
+ ```
231
+ claim: "We offer 30-day refunds"
232
+ source: refund_policy.md → "...30-day refund window..."
233
+ result: ✅ grounded
234
+
235
+ claim: "We offer 60-day refunds"
236
+ source: refund_policy.md → "...30-day refund window..."
237
+ result: ❌ not grounded (hallucination)
238
+ ```
239
+
240
+ The judge makes a binary call **per claim**, then aggregates into a groundedness score.
241
+
242
+ ### Rubrics: weighted dimensions
243
+
244
+ A rubric is a set of weighted, named quality dimensions. A reasonable starting shape:
245
+
246
+ | Dimension | Weight | Question |
247
+ |---|---|---|
248
+ | Relevance | 30% | Does it address the question? |
249
+ | Accuracy | 40% | Are the facts correct? |
250
+ | Completeness | 20% | Does it fully answer? |
251
+ | Clarity | 10% | Is it easy to understand? |
252
+
253
+ Weighted average → single quality score. Track *trends* across releases. A 5% drop in accuracy is a red flag even if overall pass-rate holds.
254
+
255
+ Domain-specific rubrics swap these dimensions for what matters in your domain (e.g., a medical-advice judge might use *safety / accuracy / scope-adherence / clarity*).
256
+
257
+ ### Every score needs an explicit anchor
258
+
259
+ ```yaml
260
+ accuracy:
261
+ weight: 0.4
262
+ scores:
263
+ 5: "All facts correct and verifiable from cited sources"
264
+ 3: "Mostly correct with one minor inaccuracy"
265
+ 1: "Contains significant errors or misleading information"
266
+ 0: "Completely incorrect or fabricated"
267
+ ```
268
+
269
+ **No anchor = no consistency.** A judge that interprets "3" differently each run is useless. If you can't write down what a 5 looks like in concrete terms, neither the judge nor a human can score it.
270
+
271
+ ### Score thresholds → action policy
272
+
273
+ | Score | Quality | Action |
274
+ |---|---|---|
275
+ | 4.5–5.0 | Excellent | Ship it |
276
+ | 3.5–4.4 | Good | Minor tweaks |
277
+ | 2.5–3.4 | Acceptable | Review and improve |
278
+ | 1.5–2.4 | Poor | Significant work needed |
279
+ | 0–1.4 | Critical | Stop. Fix now. |
280
+
281
+ Wire these thresholds into your release gate or orchestrator so the action is automatic, not a meeting.
282
+
283
+ ### Calibration: validate the judge before you trust it
284
+
285
+ ```python
286
+ # Step 1: Score 20 examples by hand
287
+ human_scores = [4, 3, 5, 2, 5, 4, 3, ...]
288
+
289
+ # Step 2: Run the LLM judge on the same examples
290
+ llm_scores = [4, 4, 5, 3, 5, 3, 3, ...]
291
+
292
+ # Step 3: Check correlation
293
+ correlation(human_scores, llm_scores)
294
+ # If < 0.8, your rubric is broken — fix it before trusting the judge
295
+ ```
296
+
297
+ **A judge with a bad rubric produces confident, wrong scores.** Calibration must happen *before* you let the judge gate releases.
298
+
299
+ ---
300
+
301
+ ## 7. Stage 5 — Experiments
302
+
303
+ **Goal:** Replace intuition with evidence. Run two variants (prompts, models, retrieval configs, agent topologies) against the same test set; report deltas in pass-rate, rubric score, latency, and cost.
304
+
305
+ ### Shape
306
+
307
+ ```
308
+ | Variant | Pass % | Rubric | Latency | Cost |
309
+ |-------------|--------|--------|---------|--------|
310
+ | baseline | 87% | 4.1/5 | 1.2s | $0.003 |
311
+ | gpt-4o | 93% | 4.5/5 | 2.1s | $0.015 |
312
+ | new_prompt | 91% | 4.3/5 | 1.3s | $0.003 |
313
+ ```
314
+
315
+ → `new_prompt` gets 91% of `gpt-4o` quality at 20% of the cost. That's a data-driven decision.
316
+
317
+ ### Four rules
318
+
319
+ | Rule | Why |
320
+ |---|---|
321
+ | **One change per experiment** | Isolate variables to understand impact |
322
+ | **Same test set every time** | Apples to apples |
323
+ | **Track cost** | 6% quality gain at 5× cost may not be worth it |
324
+ | **Version your prompts** | Store them as files; commit to git |
325
+
326
+ ### Statistical significance
327
+
328
+ For small test sets the variance per run can swamp the effect you're trying to measure. Either (a) run each variant N times and report mean ± stddev, or (b) grow the test set until the delta is statistically meaningful. Don't ship a 1% rubric bump from a 30-case run as a "win."
329
+
330
+ ### Release gate
331
+
332
+ Combine experiment outputs into a gate: **rubric pass-rate above threshold AND cost-per-run below ceiling**. Both must hold. This is where Stages 1, 4, and 5 meet — and where evals start paying for themselves.
333
+
334
+ ---
335
+
336
+ ## 8. The three eval anti-patterns
337
+
338
+ ### 1. The Likert trap
339
+
340
+ > *"Rate the quality of this response: 1 (poor) to 5 (excellent)"*
341
+
342
+ **What happens:** Human A gives "3" for adequate answers. Human B gives "4". Aggregated scores are noise, not signal.
343
+ **Fix:** Define every point on the scale with a concrete anchor. If you can't write the anchor, you haven't done the work.
344
+
345
+ ### 2. Vague criteria
346
+
347
+ > *"The response demonstrates strategic thinking"*
348
+
349
+ **Problems:** "Strategic thinking" is undefined. Two evaluators will score it differently. An LLM judge will hallucinate a definition. You can't tell what improvement looks like.
350
+ **Fix:** Make it falsifiable: *"Response identifies ≥2 explicit trade-offs with concrete examples."*
351
+
352
+ ### 3. Ambiguous ranges
353
+
354
+ > *"The response length should be between 300 and 500 tokens"*
355
+
356
+ **Problem:** A 450-token response that misses the question passes. A 250-token response that perfectly answers it fails. This criterion measures **form, not quality**.
357
+ **Fix:** Describe what good looks like: *"Answers all parts of the question; does not include unrequested information."*
358
+
359
+ ### The underlying rule
360
+
361
+ Length, scale labels, and vague terms are **proxies for quality, not quality itself.**
362
+
363
+ > **A criterion is done when you can write a concrete example that unambiguously passes and one that unambiguously fails.** If you can't do that, keep writing.
364
+
365
+ ---
366
+
367
+ ## 9. Good practices
368
+
369
+ ### Use binary checks where you can
370
+
371
+ ```python
372
+ # Specific. Deterministic. Fast.
373
+ assert "vector_search" in actual_tools
374
+ assert "refund_policy.md" in response
375
+ assert "I don't know" not in response
376
+ assert "$500 annual stipend" in response
377
+ ```
378
+
379
+ Binary checks have **zero calibration cost, zero API cost, and produce the same result every run.** Reserve LLM judges for what can't be checked programmatically.
380
+
381
+ ### Write rubric anchors *before* you run any evals
382
+
383
+ | Step | Why |
384
+ |---|---|
385
+ | 1. Write score anchors (0, 3, 5 minimum) | Forces you to define quality |
386
+ | 2. Score 20 examples by hand | Creates ground truth |
387
+ | 3. Run LLM judge on same examples | Measures calibration |
388
+ | 4. Adjust until correlation ≥ 0.8 | Validates the judge |
389
+ | 5. Then run at scale | Now you can trust it |
390
+
391
+ ### Match the eval to the moment
392
+
393
+ ```
394
+ Every commit → Golden sets (5 min, catch regressions)
395
+ Every release → Labeled scenarios (coverage check)
396
+ Weekly → Replay harnesses (deep quality analysis)
397
+ Before shipping → Rubric evals (multi-dimensional scoring)
398
+ On any change → Experiment (validate the hypothesis)
399
+ ```
400
+
401
+ The earlier you catch a regression, the cheaper it is to fix.
402
+
403
+ ### Cost tracking is a first-class eval
404
+
405
+ Every run should emit per-call token usage and dollar cost. Without cost as a tracked metric, "we improved quality" can secretly mean "we 5×'d the bill." Cost projections at multiple scales (100 / 1K / 10K / 100K runs) make the budget conversation honest.
406
+
407
+ ---
408
+
409
+ ## 10. Build your eval stack in six steps
410
+
411
+ 1. **Write 10–15 golden cases today.** Tools, sources, content checks, negative validation.
412
+ 2. **Label them by query type and difficulty.** Category, subcategory, complexity.
413
+ 3. **Record real production sessions as fixtures.** Real queries make the best test cases.
414
+ 4. **Define rubrics with explicit anchors.** Every score point needs a concrete description.
415
+ 5. **Calibrate your LLM judge against human scores.** Target correlation ≥ 0.8 before running at scale.
416
+ 6. **A/B test every meaningful change.** One variable at a time.
417
+
418
+ ---
419
+
420
+ ## 11. Test-layer mapping (for repo planning)
421
+
422
+ If you're laying out a test suite around this framework, a natural mapping:
423
+
424
+ | Layer | Lives in | Maps to | Cost per CI run |
425
+ |---|---|---|---|
426
+ | **unit** | `tests/unit/` | Stage 1 golden sets, schema validators, deterministic asserts | $0 |
427
+ | **integration** | `tests/integration/` | Stage 2–3 replay-based pipeline tests | $0 (fixtures) |
428
+ | **e2e** | `tests/e2e/` | Stage 5 single-case live smoke test | small, gated |
429
+ | **golden** | `evals/golden/` | Stage 1 corpus + runner | $0 |
430
+ | **scenarios** | `evals/scenarios/` | Stage 2 corpus + coverage report | $0 |
431
+ | **replays** | `evals/replays/` | Stage 3 fixtures + ML metrics | $0 to replay, $$ to record |
432
+ | **rubrics** | `evals/rubrics/` | Stage 4 YAML rubric + calibration set | $ per release |
433
+
434
+ Keep `evals/` separate from your application package. The eval substrate must be portable, independently versionable, and clearly distinct from the system under test.
435
+
436
+ ---
437
+
438
+ ## 12. The goal
439
+
440
+ **Before:** *"I think the new prompt is better, it felt more accurate."*
441
+
442
+ **After:** *"The new prompt scored 4.3/5 vs 4.1/5 on accuracy, passed 91% of golden cases vs 87%, with no change in latency or cost."*
443
+
444
+ That's the difference between guessing and shipping with confidence.
445
+
446
+ ---
447
+
448
+ ## 13. Start at Stage 1
449
+
450
+ 10–15 golden cases, run after every commit. Add stages as your system matures. Do not skip ahead.
451
+
452
+ The team that ships Stage 1 next week beats the team that's still designing Stage 4 next quarter.