@codewizard-dt/bootstrap 2.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/workflows/build.yml +88 -0
- package/.github/workflows/security.yml +24 -0
- package/.gitleaks.toml +15 -0
- package/README.md +472 -0
- package/bin/cli.js +47 -0
- package/lib/README.md +27 -0
- package/lib/hooks/README.md +316 -0
- package/lib/hooks/env-file-guard.js +45 -0
- package/lib/hooks/git-protected-ops-block.js +77 -0
- package/lib/hooks/lib/serena-languages.js +158 -0
- package/lib/hooks/lib/serena.js +787 -0
- package/lib/hooks/mv-absolute-path-block.js +53 -0
- package/lib/hooks/serena-bash-grep-block.js +501 -0
- package/lib/hooks/serena-edit-guard.js +74 -0
- package/lib/hooks/serena-first-glob-guard.js +94 -0
- package/lib/hooks/serena-first-guard.js +74 -0
- package/lib/hooks/serena-first-read-guard.js +167 -0
- package/lib/hooks/serena-pre-delegation.js +117 -0
- package/lib/hooks/serena-session-reset.js +45 -0
- package/lib/hooks/serena-usage-tracker.js +186 -0
- package/lib/hooks/serena-write-guard.js +58 -0
- package/lib/prompts/README.md +16 -0
- package/lib/prompts/migrate-wiki.md +111 -0
- package/lib/prompts/setup-deployment.md +322 -0
- package/lib/prompts/setup-design-system.md +162 -0
- package/lib/prompts/setup-strict-typechecks.md +139 -0
- package/lib/scripts/README.md +49 -0
- package/lib/scripts/bootstrap-serena.sh +145 -0
- package/lib/scripts/build-mcp-guide.sh +108 -0
- package/lib/scripts/install-global.sh +78 -0
- package/lib/scripts/install-mcps.sh +131 -0
- package/lib/scripts/lib.sh +154 -0
- package/lib/scripts/merge-gitignore.sh +65 -0
- package/lib/scripts/migrate-project.sh +273 -0
- package/lib/scripts/setup-deployment.sh +104 -0
- package/lib/scripts/setup-project.sh +55 -0
- package/lib/scripts/setup-runner.sh +99 -0
- package/lib/scripts/setup-strict-typechecks.sh +29 -0
- package/lib/scripts/startup.sh +42 -0
- package/lib/scripts/sync-wiki-scaffold.sh +104 -0
- package/lib/scripts/templates/CLAUDE-env-safety.md +3 -0
- package/lib/scripts/templates/CLAUDE-wiki.md +60 -0
- package/lib/scripts/templates/gitignore +136 -0
- package/lib/scripts/templates/guides/stubs/00-header.md +15 -0
- package/lib/scripts/templates/guides/stubs/brave-search.md +20 -0
- package/lib/scripts/templates/guides/stubs/context7.md +26 -0
- package/lib/scripts/templates/guides/stubs/playwright.md +26 -0
- package/lib/scripts/templates/guides/stubs/serena.md +219 -0
- package/lib/scripts/templates/wiki/conventions.md +100 -0
- package/lib/scripts/templates/wiki/dashboard.html +1733 -0
- package/lib/scripts/templates/wiki/hot.md +21 -0
- package/lib/scripts/templates/wiki/index.md +46 -0
- package/lib/scripts/templates/wiki/knowledge/concepts/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/knowledge/entities/components/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/knowledge/entities/organisations/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/knowledge/entities/people/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/knowledge/entities/tools/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/knowledge/sources/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/log.md +14 -0
- package/lib/scripts/templates/wiki/work/bugs/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/work/bugs/archive/index.md +6 -0
- package/lib/scripts/templates/wiki/work/bugs/index.md +12 -0
- package/lib/scripts/templates/wiki/work/bugs/lifecycle.md +39 -0
- package/lib/scripts/templates/wiki/work/decisions/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/work/decisions/archive/index.md +6 -0
- package/lib/scripts/templates/wiki/work/decisions/index.md +12 -0
- package/lib/scripts/templates/wiki/work/decisions/lifecycle.md +42 -0
- package/lib/scripts/templates/wiki/work/requirements/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/work/requirements/archive/index.md +6 -0
- package/lib/scripts/templates/wiki/work/requirements/index.md +12 -0
- package/lib/scripts/templates/wiki/work/requirements/lifecycle.md +34 -0
- package/lib/scripts/templates/wiki/work/roadmaps/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/work/roadmaps/archive/index.md +6 -0
- package/lib/scripts/templates/wiki/work/roadmaps/index.md +12 -0
- package/lib/scripts/templates/wiki/work/roadmaps/lifecycle.md +32 -0
- package/lib/scripts/templates/wiki/work/tasks/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/work/tasks/archive/index.md +6 -0
- package/lib/scripts/templates/wiki/work/tasks/index.md +12 -0
- package/lib/scripts/templates/wiki/work/tasks/lifecycle.md +39 -0
- package/lib/scripts/templates/wiki/work/uat/.gitkeep +0 -0
- package/lib/scripts/templates/wiki/work/uat/archive/index.md +6 -0
- package/lib/scripts/templates/wiki/work/uat/index.md +12 -0
- package/lib/scripts/templates/wiki/work/uat/lifecycle.md +48 -0
- package/lib/scripts/templates/wiki/work/uat/screenshots/.gitkeep +0 -0
- package/lib/scripts/update-project.sh +75 -0
- package/lib/scripts/wiki-dashboard-server.js +146 -0
- package/lib/skills/README.md +146 -0
- package/lib/skills/anti-slop/SKILL.md +183 -0
- package/lib/skills/bug-close/SKILL.md +136 -0
- package/lib/skills/bug-file/SKILL.md +114 -0
- package/lib/skills/bug-triage/SKILL.md +140 -0
- package/lib/skills/company-align/SKILL.md +203 -0
- package/lib/skills/debug-logs/SKILL.md +129 -0
- package/lib/skills/decision-create/SKILL.md +220 -0
- package/lib/skills/decision-finalize/SKILL.md +181 -0
- package/lib/skills/decision-next/SKILL.md +129 -0
- package/lib/skills/decision-walkthrough/SKILL.md +169 -0
- package/lib/skills/demo/SKILL.md +183 -0
- package/lib/skills/elevator-pitch/SKILL.md +125 -0
- package/lib/skills/eval-create/SKILL.md +344 -0
- package/lib/skills/eval-gap/SKILL.md +150 -0
- package/lib/skills/eval-run/SKILL.md +214 -0
- package/lib/skills/extract-feature/SKILL.md +344 -0
- package/lib/skills/flashcard/SKILL.md +513 -0
- package/lib/skills/frontend-taste/SKILL.md +184 -0
- package/lib/skills/gap-assess/SKILL.md +97 -0
- package/lib/skills/git-commit/SKILL.md +164 -0
- package/lib/skills/lint/SKILL.md +99 -0
- package/lib/skills/marp-slideshow/SKILL.md +212 -0
- package/lib/skills/mermaid-flowchart/SKILL.md +194 -0
- package/lib/skills/now/SKILL.md +36 -0
- package/lib/skills/port-feature/SKILL.md +315 -0
- package/lib/skills/power-mode/SKILL.md +184 -0
- package/lib/skills/primer/SKILL.md +84 -0
- package/lib/skills/project-readme/SKILL.md +166 -0
- package/lib/skills/req-compile/SKILL.md +131 -0
- package/lib/skills/req-create/SKILL.md +124 -0
- package/lib/skills/req-extract-decisions/SKILL.md +241 -0
- package/lib/skills/req-finalize/SKILL.md +251 -0
- package/lib/skills/req-retire/SKILL.md +185 -0
- package/lib/skills/req-update/SKILL.md +265 -0
- package/lib/skills/research/SKILL.md +250 -0
- package/lib/skills/research-company/SKILL.md +345 -0
- package/lib/skills/roadmap-add/SKILL.md +238 -0
- package/lib/skills/roadmap-create/SKILL.md +223 -0
- package/lib/skills/roadmap-next/SKILL.md +132 -0
- package/lib/skills/security-audit/SKILL.md +246 -0
- package/lib/skills/serena-config/SKILL.md +92 -0
- package/lib/skills/simplify/SKILL.md +227 -0
- package/lib/skills/tackle/SKILL.md +156 -0
- package/lib/skills/task-add/SKILL.md +208 -0
- package/lib/skills/task-audit/SKILL.md +294 -0
- package/lib/skills/task-trash/SKILL.md +115 -0
- package/lib/skills/task-update/SKILL.md +136 -0
- package/lib/skills/uat-auto/SKILL.md +69 -0
- package/lib/skills/uat-auto-plus/SKILL.md +115 -0
- package/lib/skills/uat-generate/SKILL.md +289 -0
- package/lib/skills/uat-skip/SKILL.md +169 -0
- package/lib/skills/uat-walk/SKILL.md +164 -0
- package/lib/skills/uat-walk/UAT-CORE.md +140 -0
- package/lib/skills/update-docs/SKILL.md +75 -0
- package/lib/skills/wiki-archive/SKILL.md +96 -0
- package/lib/skills/wiki-ingest/SKILL.md +139 -0
- package/lib/skills/wiki-lint/SKILL.md +131 -0
- package/lib/skills/wiki-query/SKILL.md +71 -0
- package/lib/skills/wiki-rotate-log/SKILL.md +90 -0
- package/lib/skills/wiki-tidy/SKILL.md +107 -0
- package/package.json +32 -0
- package/raw/.gitkeep +0 -0
- package/raw/design-principles.md +65 -0
- package/raw/guides/command-anti-patterns.md +79 -0
- package/raw/guides/deployment-strategy.md +448 -0
- package/raw/guides/evals-framework.md +452 -0
- package/raw/guides/mcp-tools.md +335 -0
- package/raw/guides/task-spec.md +79 -0
- package/raw/guides/type-checking-templates/README.md +112 -0
- package/raw/guides/type-checking-templates/eslint/eslint.config.ts +105 -0
- package/raw/guides/type-checking-templates/python/mypy.ini +69 -0
- package/raw/guides/type-checking-templates/python/pyproject.toml +164 -0
- package/raw/guides/type-checking-templates/python/pyrightconfig.json +67 -0
- package/raw/guides/type-checking-templates/tsconfig/tsconfig.app.jsonc +45 -0
- package/raw/guides/type-checking-templates/tsconfig/tsconfig.lib.jsonc +44 -0
- package/raw/guides/type-checking-templates/tsconfig/tsconfig.node-esm.jsonc +40 -0
- package/raw/guides/type-checking-templates/tsconfig/tsconfig.node.jsonc +34 -0
- package/raw/guides/type-checking-templates/tsconfig/tsconfig.vite-root.json +8 -0
- package/raw/harness-notes.md +20 -0
- package/raw/house-style/components/Badge.tsx +16 -0
- package/raw/house-style/components/Button.tsx +32 -0
- package/raw/house-style/components/Card.tsx +26 -0
- package/raw/house-style/components/Chip.tsx +57 -0
- package/raw/house-style/components/IconButton.tsx +67 -0
- package/raw/house-style/components/Modal.tsx +18 -0
- package/raw/house-style/components/Stepper.tsx +141 -0
- package/raw/house-style/components/icons.tsx +128 -0
- package/raw/house-style/preview/src/index.css +46 -0
- package/raw/house-style/tokens/tailwind.preset.ts +49 -0
- package/raw/house-style/tokens/tokens.json +62 -0
- package/raw/llm-wiki-2.md +162 -0
- package/raw/llm-wiki.md +75 -0
- package/raw/research/dark-mode-theme/index.md +85 -0
- package/raw/research/dark-mode-theme/sources.md +49 -0
- package/raw/research/serena-mcp-disconnect/index.md +81 -0
- package/raw/research/serena-mcp-disconnect/sources.md +87 -0
- package/raw/research/wiki-tooling-improvements/index.md +126 -0
- package/raw/research/wiki-tooling-improvements/sources.md +73 -0
- package/raw/software-factory-case-study.pdf +0 -0
|
@@ -0,0 +1,452 @@
|
|
|
1
|
+
# Evals That Actually Work: A 5-Stage Framework for Production AI Quality
|
|
2
|
+
|
|
3
|
+
> A generic, project-agnostic guide to building an evaluation suite for any production AI system. Distills the "Evals That Actually Work" framework into the test types, artifacts, cadences, and anti-patterns a team needs to go from *"we tested it manually"* to *"is this better?" is a question we answer with data*.
|
|
4
|
+
|
|
5
|
+
This guide is **not specific to any one codebase**. Drop it into any project that ships LLM-backed features (agents, RAG systems, classifiers, summarizers, judges) and use it as a blueprint for what to build and in what order.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## 1. Why evals
|
|
10
|
+
|
|
11
|
+
Without evals you:
|
|
12
|
+
|
|
13
|
+
- Guess whether changes helped
|
|
14
|
+
- Find regressions in production
|
|
15
|
+
- Debate quality subjectively
|
|
16
|
+
- Ship and hope
|
|
17
|
+
|
|
18
|
+
With evals you:
|
|
19
|
+
|
|
20
|
+
- Measure whether changes helped
|
|
21
|
+
- Catch regressions before shipping
|
|
22
|
+
- Compare quality with data
|
|
23
|
+
- Ship and know
|
|
24
|
+
|
|
25
|
+
If your answer to *"how do you know your AI is good?"* is *"we tested it manually"* or *"it looked fine in review,"* you don't have an eval strategy — you have vibes.
|
|
26
|
+
|
|
27
|
+
---
|
|
28
|
+
|
|
29
|
+
## 2. The framework: five stages, each builds on the last
|
|
30
|
+
|
|
31
|
+
| # | Stage | Purpose | When it runs |
|
|
32
|
+
|---|-------|---------|--------------|
|
|
33
|
+
| 1 | **Golden Sets** | Baseline correctness | Every commit |
|
|
34
|
+
| 2 | **Labeled Scenarios** | Coverage mapping | Every release |
|
|
35
|
+
| 3 | **Replay Harnesses** | Reproducibility + ML metrics | Weekly / nightly |
|
|
36
|
+
| 4 | **Rubrics** | Multi-dimensional quality scoring | Before shipping |
|
|
37
|
+
| 5 | **Experiments** | Data-driven decisions on changes | On any change |
|
|
38
|
+
|
|
39
|
+
**Start at Stage 1. Add stages as your system matures.** Do not try to build all five at once. A team with 15 good golden cases is shipping better software than a team with a half-finished rubric harness and no regression suite.
|
|
40
|
+
|
|
41
|
+
These stages map cleanly to test layers:
|
|
42
|
+
|
|
43
|
+
- **Unit-grade** → Stage 1 (deterministic, fast, zero LLM cost)
|
|
44
|
+
- **Integration-grade** → Stages 2–3 (recorded fixtures, no live model calls)
|
|
45
|
+
- **E2E-grade** → Stages 4–5 (live system, real cost, release gates)
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## 3. Stage 1 — Golden Sets
|
|
50
|
+
|
|
51
|
+
**Goal:** A small, curated set of canonical cases that define what "correct" looks like. If these fail, something is fundamentally broken.
|
|
52
|
+
|
|
53
|
+
### Shape
|
|
54
|
+
|
|
55
|
+
- **Size:** 10–20 cases. Quality > quantity.
|
|
56
|
+
- **Format:** YAML or JSON, one record per case. Lives in a top-level `evals/golden/` directory, *separate from the code under test*, so the eval substrate stays portable.
|
|
57
|
+
- **Cost:** Zero LLM calls. Pure code evals — assertions over the system's recorded output.
|
|
58
|
+
- **Cadence:** Every commit, in CI.
|
|
59
|
+
|
|
60
|
+
### Schema
|
|
61
|
+
|
|
62
|
+
```yaml
|
|
63
|
+
- id: "gs-001"
|
|
64
|
+
query: "What is our remote work policy?"
|
|
65
|
+
expected_tools:
|
|
66
|
+
- vector_search
|
|
67
|
+
expected_sources:
|
|
68
|
+
- remote_work_policy.md
|
|
69
|
+
must_contain:
|
|
70
|
+
- "remote"
|
|
71
|
+
- "core hours"
|
|
72
|
+
must_not_contain:
|
|
73
|
+
- "I don't know"
|
|
74
|
+
- "no information"
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
### The four check types
|
|
78
|
+
|
|
79
|
+
| Check | What it catches |
|
|
80
|
+
|---|---|
|
|
81
|
+
| **Tool selection** | Agent used the wrong tool |
|
|
82
|
+
| **Source citation** | Agent cited the wrong document |
|
|
83
|
+
| **Content validation** (`must_contain`) | Response is missing key facts |
|
|
84
|
+
| **Negative validation** (`must_not_contain`) | Agent hallucinated or gave up |
|
|
85
|
+
|
|
86
|
+
All four are **deterministic, binary, no LLM needed**. They run as plain `assert` statements over recorded system output.
|
|
87
|
+
|
|
88
|
+
```python
|
|
89
|
+
# Tool selection
|
|
90
|
+
assert "vector_search" in actual_tools
|
|
91
|
+
|
|
92
|
+
# Source citation
|
|
93
|
+
assert "refund_policy.md" in response_text
|
|
94
|
+
|
|
95
|
+
# Content validation
|
|
96
|
+
assert "30-day" in response_text
|
|
97
|
+
assert "$500" in response_text
|
|
98
|
+
|
|
99
|
+
# Negative validation
|
|
100
|
+
assert "I don't know" not in response_text
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
### Four rules for golden sets that stay useful
|
|
104
|
+
|
|
105
|
+
| Rule | Detail |
|
|
106
|
+
|---|---|
|
|
107
|
+
| **Start small** | 10–20 quality cases beats 100 sloppy ones |
|
|
108
|
+
| **Run on every commit** | These *are* your regression tests |
|
|
109
|
+
| **Add from production bugs** | Every confirmed prod bug becomes a new case |
|
|
110
|
+
| **Never** | Change expected output just to make tests pass |
|
|
111
|
+
|
|
112
|
+
### Artifacts
|
|
113
|
+
|
|
114
|
+
Every run emits per-case verdict files (JSON) and a summary (Markdown). Diffs vs. the baseline surface regressions immediately on PR review.
|
|
115
|
+
|
|
116
|
+
---
|
|
117
|
+
|
|
118
|
+
## 4. Stage 2 — Labeled Scenarios
|
|
119
|
+
|
|
120
|
+
**Goal:** Answer *"does it work for **all types**?"* not just *"does it work?"* Golden sets prove correctness on the canonical happy path. Labeled scenarios prove **coverage** across the full input space.
|
|
121
|
+
|
|
122
|
+
### Shape
|
|
123
|
+
|
|
124
|
+
- **Size:** 30–100+ cases.
|
|
125
|
+
- **Format:** Same as golden sets, plus **tags** for category, subcategory, and difficulty.
|
|
126
|
+
- **Cadence:** Every release (not every commit — too slow / too expensive).
|
|
127
|
+
- **All must pass?** No. You watch pass-rate *trends* per cell, not a binary gate.
|
|
128
|
+
|
|
129
|
+
### Schema
|
|
130
|
+
|
|
131
|
+
```yaml
|
|
132
|
+
- id: "sc-m-001"
|
|
133
|
+
query: "What's our refund policy and how many refunds last quarter?"
|
|
134
|
+
expected_tools: ["vector_search", "sql_query"]
|
|
135
|
+
category: multi_tool
|
|
136
|
+
subcategory: vector_and_sql
|
|
137
|
+
difficulty: straightforward
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
The tags don't change *how the test runs* — they change *what the results tell you.*
|
|
141
|
+
|
|
142
|
+
### Coverage matrix
|
|
143
|
+
|
|
144
|
+
Aggregate pass-rate by `category × difficulty` (or any two tag axes). Empty cells show you where to write tests next.
|
|
145
|
+
|
|
146
|
+
```
|
|
147
|
+
| vector | sql | jira | slack | multi |
|
|
148
|
+
------------------|--------|------|------|-------|-------|
|
|
149
|
+
straightforward | 3/3 | 2/3 | 2/2 | 2/2 | 1/1 |
|
|
150
|
+
ambiguous | 1/1 | 1/2 | 0/1 | 1/1 | 0/1 |
|
|
151
|
+
edge_case | 1/1 | 1/1 | 1/1 | -- | -- |
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
The empty cells are your test-writing backlog.
|
|
155
|
+
|
|
156
|
+
### Difficulty bands (suggested)
|
|
157
|
+
|
|
158
|
+
- **straightforward** — single-tool, unambiguous query, golden-path inputs
|
|
159
|
+
- **ambiguous** — multiple plausible interpretations, requires disambiguation
|
|
160
|
+
- **edge_case** — adversarial inputs, rare paths, known-hard examples from production
|
|
161
|
+
|
|
162
|
+
### Golden Sets vs. Labeled Scenarios
|
|
163
|
+
|
|
164
|
+
| | Golden Sets | Labeled Scenarios |
|
|
165
|
+
|---|---|---|
|
|
166
|
+
| **Question** | "Does it work?" | "Does it work for all types?" |
|
|
167
|
+
| **Size** | 10–20 | 30–100+ |
|
|
168
|
+
| **All must pass?** | Yes | No |
|
|
169
|
+
| **When to run** | Every commit | Every release |
|
|
170
|
+
|
|
171
|
+
---
|
|
172
|
+
|
|
173
|
+
## 5. Stage 3 — Replay Harnesses
|
|
174
|
+
|
|
175
|
+
**Goal:** **Record once. Score anytime.** Capture a real session against the live system to a JSON fixture; evaluate that frozen snapshot whenever you want — immediately, next week, or after a human has annotated ground truth.
|
|
176
|
+
|
|
177
|
+
### Shape
|
|
178
|
+
|
|
179
|
+
```python
|
|
180
|
+
# Record once (costs tokens)
|
|
181
|
+
session = record_session(
|
|
182
|
+
query="What's our refund policy and how many refunds in Q4?",
|
|
183
|
+
session_id="refund-001",
|
|
184
|
+
)
|
|
185
|
+
# → saves to fixtures/refund-001.json
|
|
186
|
+
|
|
187
|
+
# Replay forever (costs nothing)
|
|
188
|
+
replayed = replay_session("refund-001")
|
|
189
|
+
scores = evaluate_session(replayed)
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
**Record real production examples.** Real queries make the best test cases — synthetic queries don't surface the weird ways users actually phrase things.
|
|
193
|
+
|
|
194
|
+
### ML-grade metrics this unlocks
|
|
195
|
+
|
|
196
|
+
| Metric | What it measures |
|
|
197
|
+
|---|---|
|
|
198
|
+
| **Precision** | How many retrieved docs are relevant? |
|
|
199
|
+
| **Recall** | How many relevant docs were retrieved? |
|
|
200
|
+
| **Groundedness** | Is the response grounded in sources? |
|
|
201
|
+
| **Faithfulness** | Does it stay true to sources (no hallucination)? |
|
|
202
|
+
| **Tool accuracy** | Did it use the correct tools? |
|
|
203
|
+
|
|
204
|
+
Precision, recall, and tool accuracy are deterministic. **Groundedness and faithfulness require an LLM judge** — which is where Stage 4 (rubrics) earns its keep.
|
|
205
|
+
|
|
206
|
+
### Why this is integration-grade
|
|
207
|
+
|
|
208
|
+
Replays let your full pipeline (retrieval → tool calls → generation → judging) run end-to-end against frozen inputs. You exercise the system's wiring without paying live LLM cost per CI run. **Add a cost-per-replay assertion** to catch token-usage regressions automatically.
|
|
209
|
+
|
|
210
|
+
### Fixture hygiene
|
|
211
|
+
|
|
212
|
+
- Store fixtures under version control alongside the eval suite.
|
|
213
|
+
- Re-record fixtures on a schedule (e.g., monthly) so the test set tracks real system behavior, not a 2024-vintage snapshot.
|
|
214
|
+
- When the system's I/O schema changes, fixtures must be re-recorded — flag this in CI.
|
|
215
|
+
|
|
216
|
+
---
|
|
217
|
+
|
|
218
|
+
## 6. Stage 4 — Rubrics & LLM-as-Judge
|
|
219
|
+
|
|
220
|
+
**Goal:** Move from binary (pass/fail) to multi-dimensional *quality* scoring. Use an LLM judge — but **only after calibrating it against humans**.
|
|
221
|
+
|
|
222
|
+
### When to reach for an LLM judge
|
|
223
|
+
|
|
224
|
+
After you have exhausted code evals. **Use binary checks where you can; reserve LLM judges for what can't be checked programmatically** (groundedness, faithfulness, coherence, helpfulness).
|
|
225
|
+
|
|
226
|
+
### Anchor every claim against its source
|
|
227
|
+
|
|
228
|
+
The judge's core operation is **claim → source verification**, not free-form rating:
|
|
229
|
+
|
|
230
|
+
```
|
|
231
|
+
claim: "We offer 30-day refunds"
|
|
232
|
+
source: refund_policy.md → "...30-day refund window..."
|
|
233
|
+
result: ✅ grounded
|
|
234
|
+
|
|
235
|
+
claim: "We offer 60-day refunds"
|
|
236
|
+
source: refund_policy.md → "...30-day refund window..."
|
|
237
|
+
result: ❌ not grounded (hallucination)
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
The judge makes a binary call **per claim**, then aggregates into a groundedness score.
|
|
241
|
+
|
|
242
|
+
### Rubrics: weighted dimensions
|
|
243
|
+
|
|
244
|
+
A rubric is a set of weighted, named quality dimensions. A reasonable starting shape:
|
|
245
|
+
|
|
246
|
+
| Dimension | Weight | Question |
|
|
247
|
+
|---|---|---|
|
|
248
|
+
| Relevance | 30% | Does it address the question? |
|
|
249
|
+
| Accuracy | 40% | Are the facts correct? |
|
|
250
|
+
| Completeness | 20% | Does it fully answer? |
|
|
251
|
+
| Clarity | 10% | Is it easy to understand? |
|
|
252
|
+
|
|
253
|
+
Weighted average → single quality score. Track *trends* across releases. A 5% drop in accuracy is a red flag even if overall pass-rate holds.
|
|
254
|
+
|
|
255
|
+
Domain-specific rubrics swap these dimensions for what matters in your domain (e.g., a medical-advice judge might use *safety / accuracy / scope-adherence / clarity*).
|
|
256
|
+
|
|
257
|
+
### Every score needs an explicit anchor
|
|
258
|
+
|
|
259
|
+
```yaml
|
|
260
|
+
accuracy:
|
|
261
|
+
weight: 0.4
|
|
262
|
+
scores:
|
|
263
|
+
5: "All facts correct and verifiable from cited sources"
|
|
264
|
+
3: "Mostly correct with one minor inaccuracy"
|
|
265
|
+
1: "Contains significant errors or misleading information"
|
|
266
|
+
0: "Completely incorrect or fabricated"
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
**No anchor = no consistency.** A judge that interprets "3" differently each run is useless. If you can't write down what a 5 looks like in concrete terms, neither the judge nor a human can score it.
|
|
270
|
+
|
|
271
|
+
### Score thresholds → action policy
|
|
272
|
+
|
|
273
|
+
| Score | Quality | Action |
|
|
274
|
+
|---|---|---|
|
|
275
|
+
| 4.5–5.0 | Excellent | Ship it |
|
|
276
|
+
| 3.5–4.4 | Good | Minor tweaks |
|
|
277
|
+
| 2.5–3.4 | Acceptable | Review and improve |
|
|
278
|
+
| 1.5–2.4 | Poor | Significant work needed |
|
|
279
|
+
| 0–1.4 | Critical | Stop. Fix now. |
|
|
280
|
+
|
|
281
|
+
Wire these thresholds into your release gate or orchestrator so the action is automatic, not a meeting.
|
|
282
|
+
|
|
283
|
+
### Calibration: validate the judge before you trust it
|
|
284
|
+
|
|
285
|
+
```python
|
|
286
|
+
# Step 1: Score 20 examples by hand
|
|
287
|
+
human_scores = [4, 3, 5, 2, 5, 4, 3, ...]
|
|
288
|
+
|
|
289
|
+
# Step 2: Run the LLM judge on the same examples
|
|
290
|
+
llm_scores = [4, 4, 5, 3, 5, 3, 3, ...]
|
|
291
|
+
|
|
292
|
+
# Step 3: Check correlation
|
|
293
|
+
correlation(human_scores, llm_scores)
|
|
294
|
+
# If < 0.8, your rubric is broken — fix it before trusting the judge
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
**A judge with a bad rubric produces confident, wrong scores.** Calibration must happen *before* you let the judge gate releases.
|
|
298
|
+
|
|
299
|
+
---
|
|
300
|
+
|
|
301
|
+
## 7. Stage 5 — Experiments
|
|
302
|
+
|
|
303
|
+
**Goal:** Replace intuition with evidence. Run two variants (prompts, models, retrieval configs, agent topologies) against the same test set; report deltas in pass-rate, rubric score, latency, and cost.
|
|
304
|
+
|
|
305
|
+
### Shape
|
|
306
|
+
|
|
307
|
+
```
|
|
308
|
+
| Variant | Pass % | Rubric | Latency | Cost |
|
|
309
|
+
|-------------|--------|--------|---------|--------|
|
|
310
|
+
| baseline | 87% | 4.1/5 | 1.2s | $0.003 |
|
|
311
|
+
| gpt-4o | 93% | 4.5/5 | 2.1s | $0.015 |
|
|
312
|
+
| new_prompt | 91% | 4.3/5 | 1.3s | $0.003 |
|
|
313
|
+
```
|
|
314
|
+
|
|
315
|
+
→ `new_prompt` gets 91% of `gpt-4o` quality at 20% of the cost. That's a data-driven decision.
|
|
316
|
+
|
|
317
|
+
### Four rules
|
|
318
|
+
|
|
319
|
+
| Rule | Why |
|
|
320
|
+
|---|---|
|
|
321
|
+
| **One change per experiment** | Isolate variables to understand impact |
|
|
322
|
+
| **Same test set every time** | Apples to apples |
|
|
323
|
+
| **Track cost** | 6% quality gain at 5× cost may not be worth it |
|
|
324
|
+
| **Version your prompts** | Store them as files; commit to git |
|
|
325
|
+
|
|
326
|
+
### Statistical significance
|
|
327
|
+
|
|
328
|
+
For small test sets the variance per run can swamp the effect you're trying to measure. Either (a) run each variant N times and report mean ± stddev, or (b) grow the test set until the delta is statistically meaningful. Don't ship a 1% rubric bump from a 30-case run as a "win."
|
|
329
|
+
|
|
330
|
+
### Release gate
|
|
331
|
+
|
|
332
|
+
Combine experiment outputs into a gate: **rubric pass-rate above threshold AND cost-per-run below ceiling**. Both must hold. This is where Stages 1, 4, and 5 meet — and where evals start paying for themselves.
|
|
333
|
+
|
|
334
|
+
---
|
|
335
|
+
|
|
336
|
+
## 8. The three eval anti-patterns
|
|
337
|
+
|
|
338
|
+
### 1. The Likert trap
|
|
339
|
+
|
|
340
|
+
> *"Rate the quality of this response: 1 (poor) to 5 (excellent)"*
|
|
341
|
+
|
|
342
|
+
**What happens:** Human A gives "3" for adequate answers. Human B gives "4". Aggregated scores are noise, not signal.
|
|
343
|
+
**Fix:** Define every point on the scale with a concrete anchor. If you can't write the anchor, you haven't done the work.
|
|
344
|
+
|
|
345
|
+
### 2. Vague criteria
|
|
346
|
+
|
|
347
|
+
> *"The response demonstrates strategic thinking"*
|
|
348
|
+
|
|
349
|
+
**Problems:** "Strategic thinking" is undefined. Two evaluators will score it differently. An LLM judge will hallucinate a definition. You can't tell what improvement looks like.
|
|
350
|
+
**Fix:** Make it falsifiable: *"Response identifies ≥2 explicit trade-offs with concrete examples."*
|
|
351
|
+
|
|
352
|
+
### 3. Ambiguous ranges
|
|
353
|
+
|
|
354
|
+
> *"The response length should be between 300 and 500 tokens"*
|
|
355
|
+
|
|
356
|
+
**Problem:** A 450-token response that misses the question passes. A 250-token response that perfectly answers it fails. This criterion measures **form, not quality**.
|
|
357
|
+
**Fix:** Describe what good looks like: *"Answers all parts of the question; does not include unrequested information."*
|
|
358
|
+
|
|
359
|
+
### The underlying rule
|
|
360
|
+
|
|
361
|
+
Length, scale labels, and vague terms are **proxies for quality, not quality itself.**
|
|
362
|
+
|
|
363
|
+
> **A criterion is done when you can write a concrete example that unambiguously passes and one that unambiguously fails.** If you can't do that, keep writing.
|
|
364
|
+
|
|
365
|
+
---
|
|
366
|
+
|
|
367
|
+
## 9. Good practices
|
|
368
|
+
|
|
369
|
+
### Use binary checks where you can
|
|
370
|
+
|
|
371
|
+
```python
|
|
372
|
+
# Specific. Deterministic. Fast.
|
|
373
|
+
assert "vector_search" in actual_tools
|
|
374
|
+
assert "refund_policy.md" in response
|
|
375
|
+
assert "I don't know" not in response
|
|
376
|
+
assert "$500 annual stipend" in response
|
|
377
|
+
```
|
|
378
|
+
|
|
379
|
+
Binary checks have **zero calibration cost, zero API cost, and produce the same result every run.** Reserve LLM judges for what can't be checked programmatically.
|
|
380
|
+
|
|
381
|
+
### Write rubric anchors *before* you run any evals
|
|
382
|
+
|
|
383
|
+
| Step | Why |
|
|
384
|
+
|---|---|
|
|
385
|
+
| 1. Write score anchors (0, 3, 5 minimum) | Forces you to define quality |
|
|
386
|
+
| 2. Score 20 examples by hand | Creates ground truth |
|
|
387
|
+
| 3. Run LLM judge on same examples | Measures calibration |
|
|
388
|
+
| 4. Adjust until correlation ≥ 0.8 | Validates the judge |
|
|
389
|
+
| 5. Then run at scale | Now you can trust it |
|
|
390
|
+
|
|
391
|
+
### Match the eval to the moment
|
|
392
|
+
|
|
393
|
+
```
|
|
394
|
+
Every commit → Golden sets (5 min, catch regressions)
|
|
395
|
+
Every release → Labeled scenarios (coverage check)
|
|
396
|
+
Weekly → Replay harnesses (deep quality analysis)
|
|
397
|
+
Before shipping → Rubric evals (multi-dimensional scoring)
|
|
398
|
+
On any change → Experiment (validate the hypothesis)
|
|
399
|
+
```
|
|
400
|
+
|
|
401
|
+
The earlier you catch a regression, the cheaper it is to fix.
|
|
402
|
+
|
|
403
|
+
### Cost tracking is a first-class eval
|
|
404
|
+
|
|
405
|
+
Every run should emit per-call token usage and dollar cost. Without cost as a tracked metric, "we improved quality" can secretly mean "we 5×'d the bill." Cost projections at multiple scales (100 / 1K / 10K / 100K runs) make the budget conversation honest.
|
|
406
|
+
|
|
407
|
+
---
|
|
408
|
+
|
|
409
|
+
## 10. Build your eval stack in six steps
|
|
410
|
+
|
|
411
|
+
1. **Write 10–15 golden cases today.** Tools, sources, content checks, negative validation.
|
|
412
|
+
2. **Label them by query type and difficulty.** Category, subcategory, complexity.
|
|
413
|
+
3. **Record real production sessions as fixtures.** Real queries make the best test cases.
|
|
414
|
+
4. **Define rubrics with explicit anchors.** Every score point needs a concrete description.
|
|
415
|
+
5. **Calibrate your LLM judge against human scores.** Target correlation ≥ 0.8 before running at scale.
|
|
416
|
+
6. **A/B test every meaningful change.** One variable at a time.
|
|
417
|
+
|
|
418
|
+
---
|
|
419
|
+
|
|
420
|
+
## 11. Test-layer mapping (for repo planning)
|
|
421
|
+
|
|
422
|
+
If you're laying out a test suite around this framework, a natural mapping:
|
|
423
|
+
|
|
424
|
+
| Layer | Lives in | Maps to | Cost per CI run |
|
|
425
|
+
|---|---|---|---|
|
|
426
|
+
| **unit** | `tests/unit/` | Stage 1 golden sets, schema validators, deterministic asserts | $0 |
|
|
427
|
+
| **integration** | `tests/integration/` | Stage 2–3 replay-based pipeline tests | $0 (fixtures) |
|
|
428
|
+
| **e2e** | `tests/e2e/` | Stage 5 single-case live smoke test | small, gated |
|
|
429
|
+
| **golden** | `evals/golden/` | Stage 1 corpus + runner | $0 |
|
|
430
|
+
| **scenarios** | `evals/scenarios/` | Stage 2 corpus + coverage report | $0 |
|
|
431
|
+
| **replays** | `evals/replays/` | Stage 3 fixtures + ML metrics | $0 to replay, $$ to record |
|
|
432
|
+
| **rubrics** | `evals/rubrics/` | Stage 4 YAML rubric + calibration set | $ per release |
|
|
433
|
+
|
|
434
|
+
Keep `evals/` separate from your application package. The eval substrate must be portable, independently versionable, and clearly distinct from the system under test.
|
|
435
|
+
|
|
436
|
+
---
|
|
437
|
+
|
|
438
|
+
## 12. The goal
|
|
439
|
+
|
|
440
|
+
**Before:** *"I think the new prompt is better, it felt more accurate."*
|
|
441
|
+
|
|
442
|
+
**After:** *"The new prompt scored 4.3/5 vs 4.1/5 on accuracy, passed 91% of golden cases vs 87%, with no change in latency or cost."*
|
|
443
|
+
|
|
444
|
+
That's the difference between guessing and shipping with confidence.
|
|
445
|
+
|
|
446
|
+
---
|
|
447
|
+
|
|
448
|
+
## 13. Start at Stage 1
|
|
449
|
+
|
|
450
|
+
10–15 golden cases, run after every commit. Add stages as your system matures. Do not skip ahead.
|
|
451
|
+
|
|
452
|
+
The team that ships Stage 1 next week beats the team that's still designing Stage 4 next quarter.
|