@codewizard-dt/bootstrap 2.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (186) hide show
  1. package/.github/workflows/build.yml +88 -0
  2. package/.github/workflows/security.yml +24 -0
  3. package/.gitleaks.toml +15 -0
  4. package/README.md +472 -0
  5. package/bin/cli.js +47 -0
  6. package/lib/README.md +27 -0
  7. package/lib/hooks/README.md +316 -0
  8. package/lib/hooks/env-file-guard.js +45 -0
  9. package/lib/hooks/git-protected-ops-block.js +77 -0
  10. package/lib/hooks/lib/serena-languages.js +158 -0
  11. package/lib/hooks/lib/serena.js +787 -0
  12. package/lib/hooks/mv-absolute-path-block.js +53 -0
  13. package/lib/hooks/serena-bash-grep-block.js +501 -0
  14. package/lib/hooks/serena-edit-guard.js +74 -0
  15. package/lib/hooks/serena-first-glob-guard.js +94 -0
  16. package/lib/hooks/serena-first-guard.js +74 -0
  17. package/lib/hooks/serena-first-read-guard.js +167 -0
  18. package/lib/hooks/serena-pre-delegation.js +117 -0
  19. package/lib/hooks/serena-session-reset.js +45 -0
  20. package/lib/hooks/serena-usage-tracker.js +186 -0
  21. package/lib/hooks/serena-write-guard.js +58 -0
  22. package/lib/prompts/README.md +16 -0
  23. package/lib/prompts/migrate-wiki.md +111 -0
  24. package/lib/prompts/setup-deployment.md +322 -0
  25. package/lib/prompts/setup-design-system.md +162 -0
  26. package/lib/prompts/setup-strict-typechecks.md +139 -0
  27. package/lib/scripts/README.md +49 -0
  28. package/lib/scripts/bootstrap-serena.sh +145 -0
  29. package/lib/scripts/build-mcp-guide.sh +108 -0
  30. package/lib/scripts/install-global.sh +78 -0
  31. package/lib/scripts/install-mcps.sh +131 -0
  32. package/lib/scripts/lib.sh +154 -0
  33. package/lib/scripts/merge-gitignore.sh +65 -0
  34. package/lib/scripts/migrate-project.sh +273 -0
  35. package/lib/scripts/setup-deployment.sh +104 -0
  36. package/lib/scripts/setup-project.sh +55 -0
  37. package/lib/scripts/setup-runner.sh +99 -0
  38. package/lib/scripts/setup-strict-typechecks.sh +29 -0
  39. package/lib/scripts/startup.sh +42 -0
  40. package/lib/scripts/sync-wiki-scaffold.sh +104 -0
  41. package/lib/scripts/templates/CLAUDE-env-safety.md +3 -0
  42. package/lib/scripts/templates/CLAUDE-wiki.md +60 -0
  43. package/lib/scripts/templates/gitignore +136 -0
  44. package/lib/scripts/templates/guides/stubs/00-header.md +15 -0
  45. package/lib/scripts/templates/guides/stubs/brave-search.md +20 -0
  46. package/lib/scripts/templates/guides/stubs/context7.md +26 -0
  47. package/lib/scripts/templates/guides/stubs/playwright.md +26 -0
  48. package/lib/scripts/templates/guides/stubs/serena.md +219 -0
  49. package/lib/scripts/templates/wiki/conventions.md +100 -0
  50. package/lib/scripts/templates/wiki/dashboard.html +1733 -0
  51. package/lib/scripts/templates/wiki/hot.md +21 -0
  52. package/lib/scripts/templates/wiki/index.md +46 -0
  53. package/lib/scripts/templates/wiki/knowledge/concepts/.gitkeep +0 -0
  54. package/lib/scripts/templates/wiki/knowledge/entities/components/.gitkeep +0 -0
  55. package/lib/scripts/templates/wiki/knowledge/entities/organisations/.gitkeep +0 -0
  56. package/lib/scripts/templates/wiki/knowledge/entities/people/.gitkeep +0 -0
  57. package/lib/scripts/templates/wiki/knowledge/entities/tools/.gitkeep +0 -0
  58. package/lib/scripts/templates/wiki/knowledge/sources/.gitkeep +0 -0
  59. package/lib/scripts/templates/wiki/log.md +14 -0
  60. package/lib/scripts/templates/wiki/work/bugs/.gitkeep +0 -0
  61. package/lib/scripts/templates/wiki/work/bugs/archive/index.md +6 -0
  62. package/lib/scripts/templates/wiki/work/bugs/index.md +12 -0
  63. package/lib/scripts/templates/wiki/work/bugs/lifecycle.md +39 -0
  64. package/lib/scripts/templates/wiki/work/decisions/.gitkeep +0 -0
  65. package/lib/scripts/templates/wiki/work/decisions/archive/index.md +6 -0
  66. package/lib/scripts/templates/wiki/work/decisions/index.md +12 -0
  67. package/lib/scripts/templates/wiki/work/decisions/lifecycle.md +42 -0
  68. package/lib/scripts/templates/wiki/work/requirements/.gitkeep +0 -0
  69. package/lib/scripts/templates/wiki/work/requirements/archive/index.md +6 -0
  70. package/lib/scripts/templates/wiki/work/requirements/index.md +12 -0
  71. package/lib/scripts/templates/wiki/work/requirements/lifecycle.md +34 -0
  72. package/lib/scripts/templates/wiki/work/roadmaps/.gitkeep +0 -0
  73. package/lib/scripts/templates/wiki/work/roadmaps/archive/index.md +6 -0
  74. package/lib/scripts/templates/wiki/work/roadmaps/index.md +12 -0
  75. package/lib/scripts/templates/wiki/work/roadmaps/lifecycle.md +32 -0
  76. package/lib/scripts/templates/wiki/work/tasks/.gitkeep +0 -0
  77. package/lib/scripts/templates/wiki/work/tasks/archive/index.md +6 -0
  78. package/lib/scripts/templates/wiki/work/tasks/index.md +12 -0
  79. package/lib/scripts/templates/wiki/work/tasks/lifecycle.md +39 -0
  80. package/lib/scripts/templates/wiki/work/uat/.gitkeep +0 -0
  81. package/lib/scripts/templates/wiki/work/uat/archive/index.md +6 -0
  82. package/lib/scripts/templates/wiki/work/uat/index.md +12 -0
  83. package/lib/scripts/templates/wiki/work/uat/lifecycle.md +48 -0
  84. package/lib/scripts/templates/wiki/work/uat/screenshots/.gitkeep +0 -0
  85. package/lib/scripts/update-project.sh +75 -0
  86. package/lib/scripts/wiki-dashboard-server.js +146 -0
  87. package/lib/skills/README.md +146 -0
  88. package/lib/skills/anti-slop/SKILL.md +183 -0
  89. package/lib/skills/bug-close/SKILL.md +136 -0
  90. package/lib/skills/bug-file/SKILL.md +114 -0
  91. package/lib/skills/bug-triage/SKILL.md +140 -0
  92. package/lib/skills/company-align/SKILL.md +203 -0
  93. package/lib/skills/debug-logs/SKILL.md +129 -0
  94. package/lib/skills/decision-create/SKILL.md +220 -0
  95. package/lib/skills/decision-finalize/SKILL.md +181 -0
  96. package/lib/skills/decision-next/SKILL.md +129 -0
  97. package/lib/skills/decision-walkthrough/SKILL.md +169 -0
  98. package/lib/skills/demo/SKILL.md +183 -0
  99. package/lib/skills/elevator-pitch/SKILL.md +125 -0
  100. package/lib/skills/eval-create/SKILL.md +344 -0
  101. package/lib/skills/eval-gap/SKILL.md +150 -0
  102. package/lib/skills/eval-run/SKILL.md +214 -0
  103. package/lib/skills/extract-feature/SKILL.md +344 -0
  104. package/lib/skills/flashcard/SKILL.md +513 -0
  105. package/lib/skills/frontend-taste/SKILL.md +184 -0
  106. package/lib/skills/gap-assess/SKILL.md +97 -0
  107. package/lib/skills/git-commit/SKILL.md +164 -0
  108. package/lib/skills/lint/SKILL.md +99 -0
  109. package/lib/skills/marp-slideshow/SKILL.md +212 -0
  110. package/lib/skills/mermaid-flowchart/SKILL.md +194 -0
  111. package/lib/skills/now/SKILL.md +36 -0
  112. package/lib/skills/port-feature/SKILL.md +315 -0
  113. package/lib/skills/power-mode/SKILL.md +184 -0
  114. package/lib/skills/primer/SKILL.md +84 -0
  115. package/lib/skills/project-readme/SKILL.md +166 -0
  116. package/lib/skills/req-compile/SKILL.md +131 -0
  117. package/lib/skills/req-create/SKILL.md +124 -0
  118. package/lib/skills/req-extract-decisions/SKILL.md +241 -0
  119. package/lib/skills/req-finalize/SKILL.md +251 -0
  120. package/lib/skills/req-retire/SKILL.md +185 -0
  121. package/lib/skills/req-update/SKILL.md +265 -0
  122. package/lib/skills/research/SKILL.md +250 -0
  123. package/lib/skills/research-company/SKILL.md +345 -0
  124. package/lib/skills/roadmap-add/SKILL.md +238 -0
  125. package/lib/skills/roadmap-create/SKILL.md +223 -0
  126. package/lib/skills/roadmap-next/SKILL.md +132 -0
  127. package/lib/skills/security-audit/SKILL.md +246 -0
  128. package/lib/skills/serena-config/SKILL.md +92 -0
  129. package/lib/skills/simplify/SKILL.md +227 -0
  130. package/lib/skills/tackle/SKILL.md +156 -0
  131. package/lib/skills/task-add/SKILL.md +208 -0
  132. package/lib/skills/task-audit/SKILL.md +294 -0
  133. package/lib/skills/task-trash/SKILL.md +115 -0
  134. package/lib/skills/task-update/SKILL.md +136 -0
  135. package/lib/skills/uat-auto/SKILL.md +69 -0
  136. package/lib/skills/uat-auto-plus/SKILL.md +115 -0
  137. package/lib/skills/uat-generate/SKILL.md +289 -0
  138. package/lib/skills/uat-skip/SKILL.md +169 -0
  139. package/lib/skills/uat-walk/SKILL.md +164 -0
  140. package/lib/skills/uat-walk/UAT-CORE.md +140 -0
  141. package/lib/skills/update-docs/SKILL.md +75 -0
  142. package/lib/skills/wiki-archive/SKILL.md +96 -0
  143. package/lib/skills/wiki-ingest/SKILL.md +139 -0
  144. package/lib/skills/wiki-lint/SKILL.md +131 -0
  145. package/lib/skills/wiki-query/SKILL.md +71 -0
  146. package/lib/skills/wiki-rotate-log/SKILL.md +90 -0
  147. package/lib/skills/wiki-tidy/SKILL.md +107 -0
  148. package/package.json +32 -0
  149. package/raw/.gitkeep +0 -0
  150. package/raw/design-principles.md +65 -0
  151. package/raw/guides/command-anti-patterns.md +79 -0
  152. package/raw/guides/deployment-strategy.md +448 -0
  153. package/raw/guides/evals-framework.md +452 -0
  154. package/raw/guides/mcp-tools.md +335 -0
  155. package/raw/guides/task-spec.md +79 -0
  156. package/raw/guides/type-checking-templates/README.md +112 -0
  157. package/raw/guides/type-checking-templates/eslint/eslint.config.ts +105 -0
  158. package/raw/guides/type-checking-templates/python/mypy.ini +69 -0
  159. package/raw/guides/type-checking-templates/python/pyproject.toml +164 -0
  160. package/raw/guides/type-checking-templates/python/pyrightconfig.json +67 -0
  161. package/raw/guides/type-checking-templates/tsconfig/tsconfig.app.jsonc +45 -0
  162. package/raw/guides/type-checking-templates/tsconfig/tsconfig.lib.jsonc +44 -0
  163. package/raw/guides/type-checking-templates/tsconfig/tsconfig.node-esm.jsonc +40 -0
  164. package/raw/guides/type-checking-templates/tsconfig/tsconfig.node.jsonc +34 -0
  165. package/raw/guides/type-checking-templates/tsconfig/tsconfig.vite-root.json +8 -0
  166. package/raw/harness-notes.md +20 -0
  167. package/raw/house-style/components/Badge.tsx +16 -0
  168. package/raw/house-style/components/Button.tsx +32 -0
  169. package/raw/house-style/components/Card.tsx +26 -0
  170. package/raw/house-style/components/Chip.tsx +57 -0
  171. package/raw/house-style/components/IconButton.tsx +67 -0
  172. package/raw/house-style/components/Modal.tsx +18 -0
  173. package/raw/house-style/components/Stepper.tsx +141 -0
  174. package/raw/house-style/components/icons.tsx +128 -0
  175. package/raw/house-style/preview/src/index.css +46 -0
  176. package/raw/house-style/tokens/tailwind.preset.ts +49 -0
  177. package/raw/house-style/tokens/tokens.json +62 -0
  178. package/raw/llm-wiki-2.md +162 -0
  179. package/raw/llm-wiki.md +75 -0
  180. package/raw/research/dark-mode-theme/index.md +85 -0
  181. package/raw/research/dark-mode-theme/sources.md +49 -0
  182. package/raw/research/serena-mcp-disconnect/index.md +81 -0
  183. package/raw/research/serena-mcp-disconnect/sources.md +87 -0
  184. package/raw/research/wiki-tooling-improvements/index.md +126 -0
  185. package/raw/research/wiki-tooling-improvements/sources.md +73 -0
  186. package/raw/software-factory-case-study.pdf +0 -0
@@ -0,0 +1,344 @@
1
+ ---
2
+ name: eval-create
3
+ description: Assess the project's eval coverage against the 5-stage framework, identify gaps, and create new evals one at a time with user understanding and approval at each step
4
+ category: executing
5
+ model: claude-sonnet-4-6
6
+ effort: high
7
+ argument-hint: "[stage: golden|scenarios|replay|rubric] [description of the eval to create — e.g. 'golden case for the refund policy query using vector_search']"
8
+ disable-model-invocation: false
9
+ user-invocable: true
10
+ ---
11
+ **Prereqs:** obey `.docs/guides/mcp-tools.md`.
12
+
13
+ $ARGUMENTS
14
+
15
+ ---
16
+
17
+ # Eval Create
18
+
19
+ Create new eval cases for this project by assessing coverage against the 5-stage evals framework and filling the highest-priority gaps.
20
+
21
+ **Core rule**: Every eval you propose must be explained and explicitly approved by the user before it is written. In bootstrap mode (no golden sets yet) this is strictly enforced — one eval at a time, no batching. After golden sets exist the same spirit applies but you may present a small batch (2–3) after the user has warmed up to the process.
22
+
23
+ ---
24
+
25
+ ## Step 0: Parse arguments and pick execution path
26
+
27
+ Parse `$ARGUMENTS` immediately. The argument string can take several forms:
28
+
29
+ | Form | Example | Execution path |
30
+ |------|---------|---------------|
31
+ | Empty | *(nothing)* | Full discovery flow — Steps 1 → 2 → 3 → mode branch |
32
+ | Stage keyword only | `golden` / `scenarios` / `rubric` | Skip system discovery; go straight to gap-fill for that stage |
33
+ | Specific eval description | `"golden case for the refund policy query"` | **Direct mode** — skip discovery entirely, jump to Step 4-Direct |
34
+ | Stage + description | `"golden refund policy query using vector_search"` | **Direct mode** scoped to that stage |
35
+
36
+ **Direct mode** (any non-empty argument that is not a bare stage keyword):
37
+
38
+ 1. Acknowledge the request: "Creating a golden set eval for: `<extracted description>`"
39
+ 2. Read `.docs/guides/evals-framework.md` (always required)
40
+ 3. Survey `evals/golden/` to find the next ID and check bootstrap vs gap-fill state
41
+ 4. Go directly to **Step 4-Direct** — skip Steps 1–3
42
+
43
+ **Stage keyword only**: skip Step 2 (system discovery Q&A) and go straight to Step 3 survey, then the gap-fill branch for that stage.
44
+
45
+ **Empty arguments**: full flow starting at Step 1.
46
+
47
+ ---
48
+
49
+ ## Step 1: Read the framework
50
+
51
+ Read `.docs/guides/evals-framework.md` in full. This is authoritative for stage definitions, schemas, check types, and anti-patterns. Do not proceed until you have read it.
52
+
53
+ ---
54
+
55
+ ## Step 2: Understand the system under test
56
+
57
+ Before proposing any evals you need to understand what you are evaluating.
58
+
59
+ Use `AskUserQuestion` to gather:
60
+
61
+ 1. **What does this system do?** — one-sentence description of the AI feature or agent being evaluated (e.g. "RAG chatbot that answers HR policy questions", "code review agent", "SQL query assistant")
62
+ 2. **What tools or capabilities does it have?** — list of tool names (e.g. `vector_search`, `sql_query`, `jira_lookup`) or capabilities (e.g. multi-turn conversation, citation, tool-calling)
63
+ 3. **What are the 2–3 most important things it must always do correctly?** — these become the seed of the golden set
64
+
65
+ If `$ARGUMENTS` names a specific stage to target (e.g. `golden`, `scenarios`), skip questions that are irrelevant to that stage.
66
+
67
+ ---
68
+
69
+ ## Step 3: Survey existing evals
70
+
71
+ Use Serena to inspect the eval directory structure:
72
+
73
+ ```
74
+ mcp__serena__list_dir("evals", recursive=true)
75
+ ```
76
+
77
+ If the `evals/` directory does not exist, note that as **bootstrap mode** and proceed to Step 4A.
78
+
79
+ For each sub-directory that exists, count the eval files:
80
+
81
+ | Directory | Stage | Files found |
82
+ |-----------|-------|-------------|
83
+ | `evals/golden/` | Stage 1 — Golden Sets | N |
84
+ | `evals/scenarios/` | Stage 2 — Labeled Scenarios | N |
85
+ | `evals/replays/` | Stage 3 — Replay Fixtures | N |
86
+ | `evals/rubrics/` | Stage 4 — Rubrics | N |
87
+
88
+ Read existing golden set files (if any) to understand what is already covered — note the IDs, query types, and check patterns already present.
89
+
90
+ **Determine mode**:
91
+
92
+ - **Bootstrap mode**: `evals/golden/` does not exist OR contains zero YAML files → go to Step 4A
93
+ - **Gap-fill mode**: golden sets exist → go to Step 4B
94
+
95
+ ---
96
+
97
+ ## Step 4A: Bootstrap mode — first golden sets (STRICTLY one at a time)
98
+
99
+ The user has no golden sets yet. The goal is to build a small, high-quality foundation of 10–15 cases that become the permanent regression suite.
100
+
101
+ ### Explain before starting
102
+
103
+ Present this framing to the user before proposing anything:
104
+
105
+ > **Why we start with golden sets**
106
+ >
107
+ > Golden sets are the foundation. They are the 10–15 cases that define what "correct" looks like for your system. They cost nothing to run (no LLM calls — pure assertions), run on every commit, and catch regressions immediately.
108
+ >
109
+ > We will create them one at a time. For each one I will show you:
110
+ > - The query
111
+ > - What tools or sources the system should use
112
+ > - What the response must contain
113
+ > - What the response must never contain
114
+ >
115
+ > You confirm or edit each one before it is written. You can stop after any eval.
116
+
117
+ ### Propose the first eval
118
+
119
+ Based on the system description from Step 2, propose the single highest-value golden case:
120
+
121
+ - Choose the **canonical happy path** — the most common, most important query type
122
+ - Show the full YAML for one eval:
123
+
124
+ ```yaml
125
+ - id: "gs-001"
126
+ description: "<one sentence — what scenario this tests>"
127
+ query: "<realistic query the system would receive>"
128
+ tested_files:
129
+ - <path/to/file_under_test.ext>
130
+ expected_tools:
131
+ - <tool_name>
132
+ expected_sources:
133
+ - <source_document.ext>
134
+ must_contain:
135
+ - "<key phrase that must appear in the response>"
136
+ must_not_contain:
137
+ - "I don't know"
138
+ - "I cannot find"
139
+ ```
140
+
141
+ - Explain **why** each field was chosen:
142
+ - Why this query? (It's the golden path / most common scenario)
143
+ - Why these `tested_files`? (Name the specific implementation file(s) whose logic this eval exercises — e.g. the retrieval module, the prompt template, the tool handler)
144
+ - Why these tools? (This query type always needs X)
145
+ - Why these must_contain terms? (These are the facts the response must include)
146
+ - Why these must_not_contain terms? (These indicate failure modes)
147
+ - Explain **what regression this catches**: "If someone changes the retrieval logic and this query stops hitting the right document, this eval will fail in CI."
148
+
149
+ Use `AskUserQuestion` with options:
150
+ - **Approve — write it** (write the file, then loop to the next eval)
151
+ - **Edit it** (show fields as editable — re-ask with their corrections applied)
152
+ - **Skip this one** (propose a different eval instead)
153
+ - **Stop here** (end the session)
154
+
155
+ **Do not write the file until the user selects Approve.**
156
+
157
+ ### Loop: next eval
158
+
159
+ After writing an approved eval, immediately propose the next one. For the next eval, cover a **different scenario type**:
160
+
161
+ Suggested progression (adapt to the actual system):
162
+ 1. Canonical happy path (single-tool, unambiguous)
163
+ 2. Second most common query type
164
+ 3. A negative validation case (query that should NOT hallucinate a specific wrong answer)
165
+ 4. A multi-tool or multi-source query (if the system supports it)
166
+ 5. An edge case (ambiguous query, empty result set, adversarial phrasing)
167
+
168
+ After each approval, ask:
169
+ > Written as `evals/golden/gs-NNN.yaml`. Continue with the next eval, or stop here?
170
+
171
+ Continue until the user stops or you reach 15 evals.
172
+
173
+ ### Strict enforcement rules for bootstrap mode
174
+
175
+ - **Never propose two evals at once.** One proposal per round, full stop.
176
+ - **Never write a file without explicit "Approve" from the user.**
177
+ - **Never skip the explanation.** Every eval must have: why this query, why these checks, what regression it catches.
178
+ - **Never invent facts about the system.** If you are unsure what tool a query should use, ask.
179
+
180
+ ---
181
+
182
+ ## Step 4B: Gap-fill mode — assess coverage and propose additions
183
+
184
+ Golden sets exist. Now assess gaps and propose the highest-priority evals to fill them.
185
+
186
+ ### Coverage audit
187
+
188
+ 1. Read all existing golden set YAML files. Build a coverage summary:
189
+
190
+ ```
191
+ Golden sets: N cases
192
+ - Tool coverage: [list of tools referenced]
193
+ - Query types covered: [inferred from queries]
194
+ - Missing tools: [any tool from Step 2 not yet covered]
195
+ - Missing scenarios: [obvious gaps]
196
+ ```
197
+
198
+ 2. If `evals/scenarios/` exists, read those files and build a coverage matrix by `category × difficulty`. Show empty cells as gaps.
199
+
200
+ 3. Present the audit to the user as a table. Highlight the top 3 gaps.
201
+
202
+ ### Propose evals for the highest-priority gap
203
+
204
+ For the top gap, propose 1–3 evals (scale up only once the user has approved the first one in this session):
205
+
206
+ - For the very first proposal in this session: **one eval only** (same as bootstrap mode — earn trust before batching)
207
+ - After first approval in this session: may present up to 3 at a time, clearly separated
208
+
209
+ Each proposal must include:
210
+ - Full YAML (following the schema for the target stage)
211
+ - One-sentence explanation of what gap it fills
212
+ - What regression it catches
213
+
214
+ Use `AskUserQuestion` with options:
215
+ - **Approve all** (write all proposed evals)
216
+ - **Approve some** (user selects which)
217
+ - **Edit** (re-propose with corrections)
218
+ - **Propose different gap** (skip this gap, show the next one)
219
+ - **Stop**
220
+
221
+ ### Stage progression check
222
+
223
+ After covering gaps within the current stage, check whether the project is ready to advance:
224
+
225
+ | Current state | Suggest next |
226
+ |--------------|-------------|
227
+ | < 10 golden cases | Add more golden cases |
228
+ | ≥ 10 golden cases, no scenarios | Introduce Stage 2: first labeled scenario |
229
+ | Scenarios exist, no replays | Introduce Stage 3: first replay fixture |
230
+ | Replays exist, no rubrics | Introduce Stage 4: first rubric definition |
231
+
232
+ When suggesting a new stage, explain what it adds and why now is the right time.
233
+
234
+ ---
235
+
236
+ ## Step 4-Direct: Direct mode — create from description
237
+
238
+ This step runs only when `$ARGUMENTS` contains a specific eval description (non-empty, non-bare-keyword). It replaces the full discovery and gap-analysis flow with a focused single-eval creation loop.
239
+
240
+ ### Parse the description
241
+
242
+ Extract from `$ARGUMENTS`:
243
+ - **Stage** (default: `golden` if not specified)
244
+ - **Query or scenario described** — what the user said the eval should cover
245
+ - **Any hints** about tools, sources, expected content (pick these out of the description if present)
246
+
247
+ Fill in any gaps you can infer from context; ask only for what is genuinely ambiguous.
248
+
249
+ ### Check bootstrap state
250
+
251
+ Scan `evals/golden/` (or the relevant stage directory) to determine the next ID and whether bootstrap mode applies. If bootstrap mode is active (no golden sets yet), the strict one-at-a-time rule applies — even in direct mode.
252
+
253
+ ### Propose the eval immediately
254
+
255
+ Present the proposed YAML derived from the description — no preamble, no multi-question discovery. Show the user what you built from their description:
256
+
257
+ ```yaml
258
+ - id: "gs-NNN"
259
+ description: "<one sentence — what this tests>"
260
+ query: "<query derived from the description>"
261
+ tested_files:
262
+ - <path/to/file_under_test.ext>
263
+ expected_tools:
264
+ - <inferred or explicit>
265
+ expected_sources:
266
+ - <inferred or explicit>
267
+ must_contain:
268
+ - "<key terms>"
269
+ must_not_contain:
270
+ - "I don't know"
271
+ ```
272
+
273
+ Then briefly note:
274
+ - Any fields you had to infer (so the user can correct them), including `tested_files` if you guessed at the path
275
+ - What regression this catches
276
+
277
+ Use `AskUserQuestion`:
278
+ - **Approve — write it**
279
+ - **Edit** (show correctable fields)
280
+ - **Stop**
281
+
282
+ After writing, ask if the user wants to continue with the next eval (gap-fill mode) or stop.
283
+
284
+ ---
285
+
286
+ ## Step 5: Write eval files
287
+
288
+ After approval, write each eval to the correct location:
289
+
290
+ | Stage | Directory | File format |
291
+ |-------|-----------|-------------|
292
+ | Stage 1 — Golden | `evals/golden/` | `gs-NNN.yaml` (one case per file) or append to `golden.yaml` — match existing convention |
293
+ | Stage 2 — Scenarios | `evals/scenarios/` | `sc-NNN.yaml` |
294
+ | Stage 3 — Replays | `evals/replays/fixtures/` | `<session-id>.json` (record actual; create placeholder if live recording not possible) |
295
+ | Stage 4 — Rubrics | `evals/rubrics/` | `<dimension>.yaml` |
296
+
297
+ Create parent directories if they do not exist (use `Bash mkdir -p`).
298
+
299
+ For golden and scenario YAML files, prefer **one case per file** unless the project already uses a different convention. File names match the eval ID: `gs-001.yaml`, `sc-047.yaml`.
300
+
301
+ ---
302
+
303
+ ## Step 6: Update the eval index
304
+
305
+ If `evals/README.md` exists, update it to record the new eval(s). If it does not exist, create it with:
306
+
307
+ ```markdown
308
+ # Evals
309
+
310
+ ## Stage 1 — Golden Sets (`evals/golden/`)
311
+
312
+ | ID | Query summary | Tools | Added |
313
+ |----|--------------|-------|-------|
314
+ | gs-001 | <summary> | vector_search | YYYY-MM-DD |
315
+ ```
316
+
317
+ Add a row per new eval written in this session.
318
+
319
+ ---
320
+
321
+ ## Step 7: Report
322
+
323
+ Print a summary:
324
+
325
+ | Field | Value |
326
+ |-------|-------|
327
+ | Mode | bootstrap / gap-fill |
328
+ | Evals written | N (list IDs) |
329
+ | Stage covered | Stage N — <name> |
330
+ | Total golden cases now | N |
331
+ | Next recommended action | e.g. "Add 5 more golden cases", "Run `/eval-gap` for coverage report", "Run `/eval-run` to verify runner finds new cases" |
332
+
333
+ ---
334
+
335
+ ## CRITICAL rules
336
+
337
+ 1. **Never write an eval without explicit user approval.** Propose → explain → approve → write. In that order, every time.
338
+ 2. **In bootstrap mode: strictly one eval at a time.** No exceptions. No batching. No "here are three options" followed by writing all three.
339
+ 3. **Never invent system behavior.** If you don't know what tool a query uses, ask.
340
+ 4. **No must_contain terms that are implementation artifacts** (log lines, internal IDs, debug output). Only observable response content.
341
+ 5. **No vague must_not_contain terms** like "bad" or "wrong". Specific failure-mode strings only (e.g. `"I don't know"`, `"no information available"`, `"hallucinated_term"`).
342
+ 6. **Every eval needs a regression story.** If you cannot articulate "this eval fails when X breaks", the eval is not ready.
343
+ 7. **Never skip Step 1** (reading the framework). The quality of evals degrades without it.
344
+ 8. **Every eval must include at least one `tested_files` entry.** Use Serena (`find_file`, `get_symbols_overview`) to locate the actual implementation file(s) the eval exercises before proposing. A path you cannot verify in the repo is not a valid entry — ask the user rather than guessing.
@@ -0,0 +1,150 @@
1
+ ---
2
+ name: eval-gap
3
+ description: Analyse the project's eval suite against the 5-stage framework and produce a prioritised gap report — what's missing, what's thin, and what to build next
4
+ category: researching
5
+ model: claude-sonnet-4-6
6
+ effort: low
7
+ argument-hint: [optional: stage — golden | scenarios | replay | rubric | all]
8
+ disable-model-invocation: false
9
+ user-invocable: true
10
+ ---
11
+ **Prereqs:** obey `.docs/guides/mcp-tools.md`.
12
+
13
+ $ARGUMENTS
14
+
15
+ ---
16
+
17
+ # Eval Gap
18
+
19
+ Read-only audit of the project's eval suite. Produces a prioritised gap report and recommends the next `/eval-create` invocations. Does not write any eval files.
20
+
21
+ **Argument**: optional stage filter (`golden`, `scenarios`, `replay`, `rubric`, or `all`). Defaults to `all`.
22
+
23
+ ---
24
+
25
+ ## Step 1: Read the framework
26
+
27
+ Read `.docs/guides/evals-framework.md`. This defines the expected shape of each stage and is required context for meaningful gap analysis.
28
+
29
+ ---
30
+
31
+ ## Step 2: Inventory existing evals
32
+
33
+ Use Serena to scan the eval directory:
34
+
35
+ ```
36
+ mcp__serena__list_dir("evals", recursive=true)
37
+ ```
38
+
39
+ For each directory, read the files and extract metadata:
40
+
41
+ **Golden sets** (`evals/golden/`):
42
+ - Count of cases
43
+ - Tools referenced across all cases
44
+ - Query categories (inferred from queries)
45
+ - Check types used (tool_selection / source_citation / must_contain / must_not_contain)
46
+
47
+ **Labeled scenarios** (`evals/scenarios/`):
48
+ - Count of cases
49
+ - Categories and subcategories present
50
+ - Difficulty bands present
51
+ - Coverage matrix: `category × difficulty` — count per cell
52
+
53
+ **Replay fixtures** (`evals/replays/fixtures/`):
54
+ - Count of fixtures
55
+ - Whether they have ground-truth annotations
56
+
57
+ **Rubrics** (`evals/rubrics/`):
58
+ - Whether any rubric YAML files exist
59
+ - Dimensions defined
60
+ - Whether calibration sets are present
61
+
62
+ If `evals/` does not exist, skip to Step 4.
63
+
64
+ ---
65
+
66
+ ## Step 3: Assess against the framework
67
+
68
+ For each stage, score the current state:
69
+
70
+ | Stage | Minimum viable | Current | Gap severity |
71
+ |-------|---------------|---------|-------------|
72
+ | 1 — Golden Sets | 10–15 quality cases, all 4 check types used | N cases | 🔴 / 🟡 / 🟢 |
73
+ | 2 — Labeled Scenarios | 30+ cases, all tool categories covered, 3 difficulty bands | N cases | 🔴 / 🟡 / 🟢 |
74
+ | 3 — Replay Fixtures | At least 1 fixture per major query type | N fixtures | 🔴 / 🟡 / 🟢 |
75
+ | 4 — Rubrics | At least 1 rubric with explicit score anchors | defined / missing | 🔴 / 🟡 / 🟢 |
76
+
77
+ Gap severity:
78
+ - 🔴 Critical: stage not started or severely under-resourced
79
+ - 🟡 Partial: stage exists but has clear gaps
80
+ - 🟢 Healthy: meets minimum viable bar
81
+
82
+ ---
83
+
84
+ ## Step 4: Produce the gap report
85
+
86
+ Print a structured report covering all stages (or the filtered stage if `$ARGUMENTS` specifies one).
87
+
88
+ ### Section: Stage 1 — Golden Sets
89
+
90
+ - Current count vs target (10–15)
91
+ - Check types in use vs missing
92
+ - Tool coverage: which tools have no golden case
93
+ - Query types covered vs obvious missing types
94
+ - Anti-patterns detected (e.g., vague `must_not_contain` terms, missing `expected_tools`)
95
+ - **Recommended**: next 1–3 golden cases to add (describe what they should cover — do not write YAML)
96
+
97
+ ### Section: Stage 2 — Labeled Scenarios
98
+
99
+ - Coverage matrix showing `category × difficulty` pass-rate or case count
100
+ - Empty cells in the matrix (highest priority gaps)
101
+ - Difficulty band distribution (is it all `straightforward`? needs more `edge_case`)
102
+ - **Recommended**: which cells to fill next
103
+
104
+ ### Section: Stage 3 — Replay Fixtures
105
+
106
+ - Coverage vs golden set (does every golden query type have a fixture?)
107
+ - Whether ground-truth annotations exist
108
+ - Fixture age (if date metadata available — stale fixtures should be re-recorded)
109
+ - **Recommended**: which query types need fixtures
110
+
111
+ ### Section: Stage 4 — Rubrics
112
+
113
+ - Whether any rubric exists
114
+ - If yes: which dimensions are defined, whether anchors are explicit, whether calibration exists
115
+ - If no: readiness check — do golden sets and scenarios exist? If yes, rubrics are the next investment
116
+ - **Recommended**: which dimensions to define first
117
+
118
+ ### Section: Stage readiness
119
+
120
+ Which stage is the team actually on, and what is the single highest-leverage next action?
121
+
122
+ ```
123
+ Current stage: Stage N — <name>
124
+ Next action: /eval-create <recommended arg>
125
+ Reason: <one sentence>
126
+ ```
127
+
128
+ ---
129
+
130
+ ## Step 5: Anti-pattern check
131
+
132
+ Scan existing evals for common anti-patterns from the framework:
133
+
134
+ | Anti-pattern | What to check | How to detect |
135
+ |-------------|--------------|--------------|
136
+ | Likert trap | Rubric scores without explicit anchors | Read rubric YAML — any score without a description |
137
+ | Vague criteria | must_contain/must_not_contain terms that are too generic | Terms like "good", "correct", "yes", "no" |
138
+ | Form over quality | must_contain checks on response length or formatting | Numeric length checks |
139
+ | Missing negative validation | Golden cases without must_not_contain | Cases where must_not_contain is empty or absent |
140
+ | No tool selection checks | Cases without expected_tools | Cases missing the field |
141
+
142
+ Report which anti-patterns are present and which cases exhibit them.
143
+
144
+ ---
145
+
146
+ ## CRITICAL rules
147
+
148
+ 1. **Read-only.** This skill never writes files.
149
+ 2. **Only recommend — never create.** Describe what evals should cover; never produce YAML. Direct the user to `/eval-create` for that.
150
+ 3. **No rubric judgements without reading the rubric.** Do not claim a rubric is broken without reading its YAML.