@bompus/house-rules 0.0.0-stage → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CODE_OF_CONDUCT.md +83 -0
- package/CONTRIBUTING.md +54 -0
- package/LICENSE +21 -0
- package/README.md +378 -2
- package/THIRD_PARTY_NOTICES.md +188 -0
- package/compose.mjs +125 -0
- package/composition.mjs +361 -0
- package/config-view.mjs +209 -0
- package/config.mjs +381 -0
- package/docs/assets/banner-dark.png +0 -0
- package/docs/assets/banner-light.png +0 -0
- package/docs/assets/overview.png +0 -0
- package/docs/configuration.md +269 -0
- package/docs/skill-names.md +50 -0
- package/examples/person/house-rules.json +15 -0
- package/examples/person/rules/10-tooling.md +8 -0
- package/examples/person/rules/20-reporting.md +9 -0
- package/examples/project/AGENTS.md +18 -0
- package/package.json +42 -4
- package/rule-references.mjs +91 -0
- package/rules/core.md +340 -0
- package/rules/modifiers/backlog-outline.md +25 -0
- package/rules/modifiers/backlog-visual.md +16 -0
- package/rules/modifiers/coded-offers.md +70 -0
- package/rules/modifiers/cross-session-relay.md +23 -0
- package/rules/modifiers/effort-estimates.md +41 -0
- package/rules/modifiers/land-when-done.md +12 -0
- package/rules/modifiers/low-quota-handoff.md +20 -0
- package/rules/modifiers/model-default-questions.md +33 -0
- package/rules/modifiers/multi-agent.md +38 -0
- package/rules/modifiers/no-attribution.md +16 -0
- package/rules/modifiers/periodic-updates.md +25 -0
- package/rules/modifiers/plan-files.md +82 -0
- package/rules/modifiers/question-cards.md +48 -0
- package/rules/modifiers/release-batching.md +15 -0
- package/rules/modifiers/scratch-on-disk.md +15 -0
- package/rules/modifiers/shared-host-load.md +19 -0
- package/rules/modifiers/solo-operator.md +13 -0
- package/rules/modifiers/squash-landing.md +22 -0
- package/rules/modifiers/upstream-contributions.md +45 -0
- package/rules/modifiers/web-image-formats.md +35 -0
- package/rules/references/backlog-view.md +154 -0
- package/rules/references/github-actions.md +45 -0
- package/rules/references/landing.md +39 -0
- package/rules/references/multi-agent.md +51 -0
- package/rules/references/release-batching.md +66 -0
- package/rules/references/squash-landing.md +50 -0
- package/rules/references/thread-visuals.md +45 -0
- package/setup.mjs +445 -0
- package/skills/hr-agent-guidance-audit/LICENSE +21 -0
- package/skills/hr-agent-guidance-audit/SKILL.md +91 -0
- package/skills/hr-agent-guidance-audit/agents/openai.yaml +2 -0
- package/skills/hr-agent-guidance-audit/references/prune-workflow.md +72 -0
- package/skills/hr-agent-guidance-audit/references/records-and-lessons.md +60 -0
- package/skills/hr-agent-guidance-audit/references/retirement-verdicts.md +49 -0
- package/skills/hr-agent-guidance-refresh/LICENSE +21 -0
- package/skills/hr-agent-guidance-refresh/SKILL.md +36 -0
- package/skills/hr-agent-guidance-refresh/agents/openai.yaml +2 -0
- package/skills/hr-agent-guidance-refresh/references/verification.md +47 -0
- package/skills/hr-api-exposure-check/SKILL.md +159 -0
- package/skills/hr-api-exposure-check/agents/openai.yaml +3 -0
- package/skills/hr-audit-choices/SKILL.md +67 -0
- package/skills/hr-audit-choices/agents/openai.yaml +2 -0
- package/skills/hr-benchmarking/NOTICE.md +36 -0
- package/skills/hr-benchmarking/SKILL.md +279 -0
- package/skills/hr-benchmarking/agents/openai.yaml +2 -0
- package/skills/hr-benchmarking/references/javascript.md +130 -0
- package/skills/hr-benchmarking/references/tools.md +165 -0
- package/skills/hr-benchmarking/scripts/process-attribution.ps1 +20 -0
- package/skills/hr-benchmarking/scripts/process-attribution.py +107 -0
- package/skills/hr-better-accessibility/LICENSE +22 -0
- package/skills/hr-better-accessibility/NOTICE.md +46 -0
- package/skills/hr-better-accessibility/SKILL.md +119 -0
- package/skills/hr-better-accessibility/agents/openai.yaml +3 -0
- package/skills/hr-better-accessibility/focus-and-keyboard.md +130 -0
- package/skills/hr-better-accessibility/forms.md +82 -0
- package/skills/hr-better-accessibility/hit-areas.md +98 -0
- package/skills/hr-better-accessibility/motion-and-zoom.md +81 -0
- package/skills/hr-better-accessibility/screen-readers.md +102 -0
- package/skills/hr-better-accessibility/semantics-and-aria.md +84 -0
- package/skills/hr-better-colors/LICENSE +21 -0
- package/skills/hr-better-colors/NOTICE.md +37 -0
- package/skills/hr-better-colors/SKILL.md +114 -0
- package/skills/hr-better-colors/agents/openai.yaml +3 -0
- package/skills/hr-better-colors/color-formats.md +90 -0
- package/skills/hr-better-colors/color-usage.md +140 -0
- package/skills/hr-better-colors/contrast.md +80 -0
- package/skills/hr-better-colors/palette-generation.md +104 -0
- package/skills/hr-better-colors/palette-structure.md +76 -0
- package/skills/hr-better-colors/token-naming.md +97 -0
- package/skills/hr-better-layout/LICENSE +22 -0
- package/skills/hr-better-layout/NOTICE.md +48 -0
- package/skills/hr-better-layout/SKILL.md +92 -0
- package/skills/hr-better-layout/agents/openai.yaml +3 -0
- package/skills/hr-better-layout/grouping-and-alignment.md +144 -0
- package/skills/hr-better-layout/spacing-and-adaptivity.md +215 -0
- package/skills/hr-better-typography/LICENSE +22 -0
- package/skills/hr-better-typography/NOTICE.md +46 -0
- package/skills/hr-better-typography/SKILL.md +172 -0
- package/skills/hr-better-typography/agents/openai.yaml +3 -0
- package/skills/hr-better-typography/choosing-fonts.md +69 -0
- package/skills/hr-better-typography/css-cheat-sheet.md +65 -0
- package/skills/hr-better-typography/details-and-accessibility.md +119 -0
- package/skills/hr-better-typography/spacing-and-sizing.md +131 -0
- package/skills/hr-better-typography/variable-fonts-and-opentype.md +105 -0
- package/skills/hr-better-typography/wrapping-and-punctuation.md +57 -0
- package/skills/hr-better-writing/LICENSE +22 -0
- package/skills/hr-better-writing/NOTICE.md +46 -0
- package/skills/hr-better-writing/SKILL.md +122 -0
- package/skills/hr-better-writing/agents/openai.yaml +3 -0
- package/skills/hr-code-review/LICENSE +21 -0
- package/skills/hr-code-review/NOTICE.md +26 -0
- package/skills/hr-code-review/SKILL.md +163 -0
- package/skills/hr-code-review/agents/openai.yaml +3 -0
- package/skills/hr-code-review/references/acp-client.md +43 -0
- package/skills/hr-code-review/references/evidence.md +57 -0
- package/skills/hr-code-review/references/impact.md +103 -0
- package/skills/hr-code-review/references/panel-adjudication.md +58 -0
- package/skills/hr-code-review/references/panel-plan.md +61 -0
- package/skills/hr-code-review/references/panel-recommendations.json +105 -0
- package/skills/hr-code-review/references/panel-recommendations.md +46 -0
- package/skills/hr-code-review/references/panel-setup.md +54 -0
- package/skills/hr-code-review/references/reviewer-role.md +31 -0
- package/skills/hr-code-review/scripts/acp-panel-client.mjs +299 -0
- package/skills/hr-code-review/scripts/check-receipts.mjs +263 -0
- package/skills/hr-code-review/scripts/panel-plan.mjs +69 -0
- package/skills/hr-code-review/scripts/panel-preferences.mjs +220 -0
- package/skills/hr-design-exploration/SKILL.md +105 -0
- package/skills/hr-design-exploration/agents/openai.yaml +4 -0
- package/skills/hr-diagnosing-bugs/NOTICE.md +45 -0
- package/skills/hr-diagnosing-bugs/SKILL.md +108 -0
- package/skills/hr-diagnosing-bugs/agents/openai.yaml +3 -0
- package/skills/hr-diagnosing-bugs/references/reporting.md +97 -0
- package/skills/hr-diagnosing-bugs/scripts/check-pr-offer.mjs +47 -0
- package/skills/hr-explain-code/NOTICE.md +37 -0
- package/skills/hr-explain-code/SKILL.md +110 -0
- package/skills/hr-explain-code/agents/openai.yaml +5 -0
- package/skills/hr-explain-code/references/helper-brief.md +39 -0
- package/skills/hr-extract-shared-steps/SKILL.md +134 -0
- package/skills/hr-extract-shared-steps/agents/openai.yaml +3 -0
- package/skills/hr-forgotten-work-audit/SKILL.md +130 -0
- package/skills/hr-forgotten-work-audit/references/brief-template.md +56 -0
- package/skills/hr-forgotten-work-audit/references/result-format.md +46 -0
- package/skills/hr-handoff/LICENSE +21 -0
- package/skills/hr-handoff/NOTICE.md +29 -0
- package/skills/hr-handoff/SKILL.md +114 -0
- package/skills/hr-handoff/agents/openai.yaml +5 -0
- package/skills/hr-handoff/assets/ledger.example.json +10 -0
- package/skills/hr-handoff/assets/receipt.example.json +76 -0
- package/skills/hr-handoff/assets/source.example.json +20 -0
- package/skills/hr-handoff/references/reconciliation.md +124 -0
- package/skills/hr-handoff/references/successor-thread.md +76 -0
- package/skills/hr-handoff/scripts/check-reconciliation.mjs +324 -0
- package/skills/hr-house-rules-setup/SKILL.md +114 -0
- package/skills/hr-house-rules-setup/agents/openai.yaml +5 -0
- package/skills/hr-issue-tracker-setup/SKILL.md +49 -0
- package/skills/hr-issue-tracker-setup/agents/openai.yaml +5 -0
- package/skills/hr-lean-plan/SKILL.md +146 -0
- package/skills/hr-lean-plan/agents/openai.yaml +5 -0
- package/skills/hr-maintainability-review/SKILL.md +186 -0
- package/skills/hr-maintainability-review/agents/openai.yaml +5 -0
- package/skills/hr-manual-qa/NOTICE.md +36 -0
- package/skills/hr-manual-qa/SKILL.md +37 -0
- package/skills/hr-manual-qa/agents/openai.yaml +3 -0
- package/skills/hr-navigation-retro/LICENSE +21 -0
- package/skills/hr-navigation-retro/NOTICE.md +5 -0
- package/skills/hr-navigation-retro/SKILL.md +86 -0
- package/skills/hr-navigation-retro/agents/openai.yaml +4 -0
- package/skills/hr-navigation-retro/references/inputs.md +67 -0
- package/skills/hr-navigation-retro/scripts/retro.py +552 -0
- package/skills/hr-navigation-retro/scripts/transcripts.py +205 -0
- package/skills/hr-ordering-tests/SKILL.md +51 -0
- package/skills/hr-ordering-tests/agents/openai.yaml +2 -0
- package/skills/hr-plain-prose/SKILL.md +264 -0
- package/skills/hr-plain-prose/agents/openai.yaml +3 -0
- package/skills/hr-plain-prose/scripts/check.mjs +331 -0
- package/skills/hr-pr-followup/LICENSE +21 -0
- package/skills/hr-pr-followup/NOTICE.md +9 -0
- package/skills/hr-pr-followup/SKILL.md +74 -0
- package/skills/hr-pr-followup/agents/openai.yaml +2 -0
- package/skills/hr-pr-followup/references/review-helpers.md +111 -0
- package/skills/hr-pr-followup/scripts/check-landing.py +204 -0
- package/skills/hr-pr-followup/scripts/check-pr-review.ts +510 -0
- package/skills/hr-pr-followup/scripts/merge-pr.sh +100 -0
- package/skills/hr-pr-followup/scripts/resolve-pr-thread.ts +219 -0
- package/skills/hr-pr-followup/scripts/verify-landed.sh +184 -0
- package/skills/hr-progress-report/SKILL.md +49 -0
- package/skills/hr-progress-report/agents/openai.yaml +5 -0
- package/skills/hr-read-reddit/SKILL.md +44 -0
- package/skills/hr-read-reddit/agents/openai.yaml +2 -0
- package/skills/hr-read-reddit/scripts/reddit.ts +252 -0
- package/skills/hr-read-x-links/SKILL.md +42 -0
- package/skills/hr-read-x-links/agents/openai.yaml +2 -0
- package/skills/hr-read-x-links/scripts/x-post.ts +264 -0
- package/skills/hr-research-shared-links/SKILL.md +103 -0
- package/skills/hr-split-to-prs/LICENSE +21 -0
- package/skills/hr-split-to-prs/NOTICE.md +9 -0
- package/skills/hr-split-to-prs/SKILL.md +61 -0
- package/skills/hr-split-to-prs/agents/openai.yaml +2 -0
- package/skills/hr-sqlite/SKILL.md +174 -0
- package/skills/hr-sqlite/agents/openai.yaml +2 -0
- package/skills/hr-sqlite/references/evidence.md +94 -0
- package/skills/hr-sqlite/references/tool-decisions.md +39 -0
- package/skills/hr-sqlite/scripts/inspect-sqlite.py +245 -0
- package/skills/hr-stock-ui-audit/SKILL.md +132 -0
- package/skills/hr-stock-ui-audit/agents/openai.yaml +3 -0
- package/skills/hr-stock-ui-audit/scripts/scan.mjs +619 -0
- package/skills/hr-tdd/LICENSE +21 -0
- package/skills/hr-tdd/NOTICE.md +10 -0
- package/skills/hr-tdd/SKILL.md +40 -0
- package/skills/hr-tdd/agents/openai.yaml +3 -0
- package/skills/hr-tdd/mocking.md +39 -0
- package/skills/hr-tdd/tests.md +77 -0
- package/skills/hr-test-audit/LICENSE +21 -0
- package/skills/hr-test-audit/NOTICE.md +67 -0
- package/skills/hr-test-audit/SKILL.md +146 -0
- package/skills/hr-test-audit/agents/openai.yaml +3 -0
- package/skills/hr-test-audit/scripts/floor-guard.ts +137 -0
- package/skills/hr-usage-report/SKILL.md +90 -0
- package/skills/hr-usage-report/agents/openai.yaml +2 -0
- package/skills/hr-usage-report/assets/example-report.json +125 -0
- package/skills/hr-usage-report/assets/report-template.html +377 -0
- package/skills/hr-usage-report/scripts/render-report.ts +86 -0
- package/skills/hr-usage-report/scripts/usage-by-model.ts +499 -0
- package/skills/hr-what-next/SKILL.md +151 -0
- package/skills/hr-what-next/agents/openai.yaml +4 -0
- package/skills/hr-what-next/references/reporting.md +59 -0
- package/skills/hr-what-next/scripts/check-open-work.mjs +711 -0
- package/skills/hr-writing-for-agents/NOTICE.md +46 -0
- package/skills/hr-writing-for-agents/SKILL-MECHANICS.md +83 -0
- package/skills/hr-writing-for-agents/SKILL.md +108 -0
- package/skills/hr-writing-for-agents/agents/openai.yaml +3 -0
- package/skills/hr-writing-pr/NOTICE.md +48 -0
- package/skills/hr-writing-pr/SKILL.md +119 -0
- package/skills/hr-writing-pr/agents/openai.yaml +2 -0
- package/skills/hr-writing-pr/scripts/benefit-chart.py +200 -0
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# When to mock
|
|
2
|
+
|
|
3
|
+
Mock at **system boundaries** only:
|
|
4
|
+
|
|
5
|
+
- External APIs (payment, email, etc.)
|
|
6
|
+
- Databases (sometimes; prefer a test DB)
|
|
7
|
+
- Time/randomness
|
|
8
|
+
- File system (sometimes)
|
|
9
|
+
|
|
10
|
+
Don't mock:
|
|
11
|
+
|
|
12
|
+
- Your own classes/modules
|
|
13
|
+
- Internal collaborators
|
|
14
|
+
- Anything you control
|
|
15
|
+
|
|
16
|
+
## Designing for mockability
|
|
17
|
+
|
|
18
|
+
At system boundaries, design interfaces that are easy to mock:
|
|
19
|
+
|
|
20
|
+
**1. Use dependency injection**
|
|
21
|
+
|
|
22
|
+
Pass external dependencies in rather than creating them internally:
|
|
23
|
+
|
|
24
|
+
```typescript
|
|
25
|
+
// Easy to mock
|
|
26
|
+
function processPayment(order, paymentClient) {
|
|
27
|
+
return paymentClient.charge(order.total);
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
// Hard to mock
|
|
31
|
+
function processPayment(order) {
|
|
32
|
+
const client = new StripeClient(process.env.STRIPE_KEY);
|
|
33
|
+
return client.charge(order.total);
|
|
34
|
+
}
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
**2. Reuse production interfaces**
|
|
38
|
+
|
|
39
|
+
Prefer existing operation-specific APIs when they serve production callers. Otherwise mock the transport boundary; request routing in a fake is acceptable. Do not add wrappers solely to simplify mocks. Assert observable results rather than calls to repo-owned helpers.
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Good and bad tests
|
|
2
|
+
|
|
3
|
+
## Good tests
|
|
4
|
+
|
|
5
|
+
**Integration-style**: Test through real interfaces, not mocks of internal parts.
|
|
6
|
+
|
|
7
|
+
```typescript
|
|
8
|
+
// GOOD: Tests observable behavior
|
|
9
|
+
test("user can checkout with valid cart", async () => {
|
|
10
|
+
const cart = createCart();
|
|
11
|
+
cart.add(product);
|
|
12
|
+
const result = await checkout(cart, paymentMethod);
|
|
13
|
+
expect(result.status).toBe("confirmed");
|
|
14
|
+
});
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Characteristics:
|
|
18
|
+
|
|
19
|
+
- Tests behavior users/callers care about
|
|
20
|
+
- Uses public API only
|
|
21
|
+
- Survives internal refactors
|
|
22
|
+
- Describes WHAT, not HOW
|
|
23
|
+
- Assertions establish one observable behavior; use several when the contract needs them
|
|
24
|
+
|
|
25
|
+
## Bad tests
|
|
26
|
+
|
|
27
|
+
**Implementation-detail tests**: Coupled to internal structure.
|
|
28
|
+
|
|
29
|
+
```typescript
|
|
30
|
+
// BAD: Tests implementation details
|
|
31
|
+
test("checkout calls paymentService.process", async () => {
|
|
32
|
+
const mockPayment = { process: jest.fn() };
|
|
33
|
+
await checkout(cart, mockPayment);
|
|
34
|
+
expect(mockPayment.process).toHaveBeenCalledWith(cart.total);
|
|
35
|
+
});
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Red flags:
|
|
39
|
+
|
|
40
|
+
- Mocking internal collaborators
|
|
41
|
+
- Testing private methods
|
|
42
|
+
- Asserting internal collaborator calls/counts/order instead of observable behavior. External-boundary count or order contracts are valid assertions.
|
|
43
|
+
- Test breaks when refactoring without behavior change
|
|
44
|
+
- Test name describes HOW not WHAT
|
|
45
|
+
- Bypassing the tested public interface to inspect private implementation state. Observable storage, wire, and DOM outcomes remain valid; prefer fake state over call logs when state expresses the contract.
|
|
46
|
+
|
|
47
|
+
```typescript
|
|
48
|
+
// BAD: Checks private storage when storage is not the public contract
|
|
49
|
+
test("createUser saves to database", async () => {
|
|
50
|
+
await createUser({ name: "Alice" });
|
|
51
|
+
const row = await db.query("SELECT * FROM users WHERE name = ?", ["Alice"]);
|
|
52
|
+
expect(row).toBeDefined();
|
|
53
|
+
});
|
|
54
|
+
|
|
55
|
+
// GOOD: Verifies through interface
|
|
56
|
+
test("createUser makes user retrievable", async () => {
|
|
57
|
+
const user = await createUser({ name: "Alice" });
|
|
58
|
+
const retrieved = await getUser(user.id);
|
|
59
|
+
expect(retrieved.name).toBe("Alice");
|
|
60
|
+
});
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
**Tautological tests**: Expected value restates the implementation, so the test passes by construction.
|
|
64
|
+
|
|
65
|
+
```typescript
|
|
66
|
+
// BAD: Expected value is recomputed the way the code computes it
|
|
67
|
+
test("calculateTotal sums line items", () => {
|
|
68
|
+
const items = [{ price: 10 }, { price: 5 }];
|
|
69
|
+
const expected = items.reduce((sum, i) => sum + i.price, 0);
|
|
70
|
+
expect(calculateTotal(items)).toBe(expected);
|
|
71
|
+
});
|
|
72
|
+
|
|
73
|
+
// GOOD: Expected value is an independent, known literal
|
|
74
|
+
test("calculateTotal sums line items", () => {
|
|
75
|
+
expect(calculateTotal([{ price: 10 }, { price: 5 }])).toBe(15);
|
|
76
|
+
});
|
|
77
|
+
```
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Aaron Queen
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# Source notices
|
|
2
|
+
|
|
3
|
+
This notice accompanies the hr-test-audit bundle when copied on its own.
|
|
4
|
+
Each source below is published under the MIT License. The licence text at the
|
|
5
|
+
end applies to each of them, with the copyright line shown for that source.
|
|
6
|
+
|
|
7
|
+
## obra/superpowers
|
|
8
|
+
|
|
9
|
+
Source: https://github.com/obra/superpowers (compared at revision `8ca22db`)
|
|
10
|
+
|
|
11
|
+
- `SKILL.md`: cites `writing-good-tests.md`.
|
|
12
|
+
|
|
13
|
+
Copyright (c) 2025 Jesse Vincent
|
|
14
|
+
|
|
15
|
+
## addyosmani/agent-skills
|
|
16
|
+
|
|
17
|
+
Source: https://github.com/addyosmani/agent-skills (compared at revision `9d0c60d`)
|
|
18
|
+
|
|
19
|
+
- `scripts/floor-guard.ts`: adapted from constraint-driven-development.
|
|
20
|
+
|
|
21
|
+
Copyright (c) 2025 Addy Osmani
|
|
22
|
+
|
|
23
|
+
## openclaw/openclaw
|
|
24
|
+
|
|
25
|
+
Source: https://github.com/openclaw/openclaw (linked at revision `65f1e4d`, compared at `1fda703`)
|
|
26
|
+
|
|
27
|
+
- `SKILL.md`: the authoring gate, junk patterns, retention bar and evidence
|
|
28
|
+
fields are adapted from `.agents/skills/test-audit/`.
|
|
29
|
+
|
|
30
|
+
Copyright (c) 2026 OpenClaw Foundation
|
|
31
|
+
|
|
32
|
+
## no-human-ai/no_human
|
|
33
|
+
|
|
34
|
+
Source: https://github.com/no-human-ai/no_human (compared at revision `9b50bf6`)
|
|
35
|
+
|
|
36
|
+
- `SKILL.md`: the rule that weakening a test counts as removing it is adapted
|
|
37
|
+
from its tamper guard.
|
|
38
|
+
|
|
39
|
+
Copyright (c) 2026 Eyal Golan
|
|
40
|
+
|
|
41
|
+
## omkamal/pypict-claude-skill
|
|
42
|
+
|
|
43
|
+
Source: https://github.com/omkamal/pypict-claude-skill (compared at revision `fbda212`)
|
|
44
|
+
|
|
45
|
+
- `SKILL.md`: the pairwise-coverage guidance is adapted from this skill.
|
|
46
|
+
|
|
47
|
+
Copyright (c) 2025 pypict-claude-skill contributors
|
|
48
|
+
|
|
49
|
+
## MIT License text
|
|
50
|
+
|
|
51
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
52
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
53
|
+
in the Software without restriction, including without limitation the rights
|
|
54
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
55
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
56
|
+
furnished to do so, subject to the following conditions:
|
|
57
|
+
|
|
58
|
+
The above copyright notice and this permission notice shall be included in all
|
|
59
|
+
copies or substantial portions of the Software.
|
|
60
|
+
|
|
61
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
62
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
63
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
64
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
65
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
66
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
67
|
+
SOFTWARE.
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: hr-test-audit
|
|
3
|
+
description: Gate new or changed tests on the value they add, and sweep an existing suite for low-value, implementation-coupled or duplicate tests to delete. Use when adding or reviewing tests, or when asked to prune, audit or clean up a test suite.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Test audit
|
|
7
|
+
|
|
8
|
+
Two modes share one value bar: a test earns its upkeep by protecting observable
|
|
9
|
+
behavior, a credible regression, or an independent contract. **Authoring mode**
|
|
10
|
+
gates each new or changed test before it lands. **Audit mode** finds existing
|
|
11
|
+
tests that fail the bar and removes them in small, evidenced batches. For the
|
|
12
|
+
red-green loop itself, use a TDD skill where one is installed; this skill decides which tests are worth
|
|
13
|
+
keeping.
|
|
14
|
+
|
|
15
|
+
## Authoring gate
|
|
16
|
+
|
|
17
|
+
Before adding a test, answer all four. A missing answer means don't add it yet.
|
|
18
|
+
|
|
19
|
+
1. What observable behavior, invariant or contract does it protect?
|
|
20
|
+
2. What credible regression makes it fail? Name the production change that
|
|
21
|
+
turns it red: a wrong constant or argument, a wrong branch, a missing side
|
|
22
|
+
effect, an empty or default return, or a missing check for empty, nil,
|
|
23
|
+
unauthorized or malformed input. If none would, it is a change detector or
|
|
24
|
+
cannot fail.
|
|
25
|
+
3. Why doesn't existing coverage already catch that? Each contract has one
|
|
26
|
+
owning test at its strongest boundary. Another layer needs its own risk the
|
|
27
|
+
owner cannot reach. Prefer a new case in an existing table or fixture over a
|
|
28
|
+
near-duplicate test.
|
|
29
|
+
4. Does it need a production seam (export, flag, wrapper, injection hook) that
|
|
30
|
+
no production caller uses? Then test at the real boundary instead.
|
|
31
|
+
|
|
32
|
+
When a unit must be tested in isolation, write down the ways it can fail before
|
|
33
|
+
writing the code, and test those failures. A test written after the code tends
|
|
34
|
+
to restate the code.
|
|
35
|
+
|
|
36
|
+
A bug regression test must fail on the pre-fix code for the intended reason,
|
|
37
|
+
then pass after the fix. One that never demonstrably failed proves the mock, not
|
|
38
|
+
the fix. One regression at the owning boundary covers the bug; don't replay it
|
|
39
|
+
at every layer it crosses. For a sequence `hr-ordering-tests` found, that
|
|
40
|
+
regression is the named test beside its enumeration.
|
|
41
|
+
|
|
42
|
+
Changing an existing test so it checks less counts as removing it: deleting or
|
|
43
|
+
skipping it, loosening an assertion, or rewriting an expected value to match new
|
|
44
|
+
output. It needs the audit-mode evidence from step 2, not only these answers.
|
|
45
|
+
To catch these moves in a diff, run
|
|
46
|
+
`bun <this skill's directory>/scripts/floor-guard.ts [--base <ref>]` from
|
|
47
|
+
the repository. It reports new suppression comments, added skips or
|
|
48
|
+
`.only`, deleted test files and assertions removed from surviving tests, and
|
|
49
|
+
exits 0 clean, 1 with findings, 2 when it could not run. Each finding still
|
|
50
|
+
needs a reason; a 2 is never a pass.
|
|
51
|
+
|
|
52
|
+
For retry and concurrency failures, exercise the implementation at the boundary
|
|
53
|
+
where order matters. Cover the credible lost-response, interrupted-write or
|
|
54
|
+
restart sequence with controlled barriers and assert durable state or externally
|
|
55
|
+
visible behavior. Do not replace that coverage with a separate model of the code.
|
|
56
|
+
Code that merges events from more than one actor also needs an enumeration test
|
|
57
|
+
from `hr-ordering-tests`, and a change that adds a kind of event adds it to that
|
|
58
|
+
test's event list.
|
|
59
|
+
|
|
60
|
+
When behavior depends on three or more independent settings, cover every pair
|
|
61
|
+
of their values rather than every combination: list each setting's value
|
|
62
|
+
classes and edge values, drop combinations the code rejects, and build the
|
|
63
|
+
smallest table where each pair appears at least once (Microsoft PICT generates
|
|
64
|
+
one). Most interaction bugs need only two settings to line up. Add an explicit
|
|
65
|
+
case for each combination of three or more settings already known to interact,
|
|
66
|
+
such as one behind a past bug or a documented constraint.
|
|
67
|
+
|
|
68
|
+
Then check the test against the junk patterns below. A match fails the gate
|
|
69
|
+
unless the retention bar names the contract it independently guards.
|
|
70
|
+
|
|
71
|
+
## Junk patterns
|
|
72
|
+
|
|
73
|
+
Authoring rejects new tests that match; audits hunt for existing ones that do.
|
|
74
|
+
|
|
75
|
+
- no assertion, or an assertion that cannot fail
|
|
76
|
+
- expected values produced by the code under test, or copied from it
|
|
77
|
+
(fixtures, inventories, export lists, snapshots taken from current output)
|
|
78
|
+
- exact source, import or string greps standing in for behavior
|
|
79
|
+
- private helpers or call shapes tested again where a real boundary already
|
|
80
|
+
covers them
|
|
81
|
+
- the same contract asserted several times, or one shared helper re-tested in
|
|
82
|
+
every caller
|
|
83
|
+
- tests that exist only to keep test-only exports, globals or wrappers alive,
|
|
84
|
+
and production code whose only callers are tests
|
|
85
|
+
- mocks that implement the asserted behavior, or one mock standing in for
|
|
86
|
+
different APIs
|
|
87
|
+
- negative tests that pass for an unrelated reason (a different guard rejects
|
|
88
|
+
the input, or the production path never reaches the check)
|
|
89
|
+
- names or fixtures that promise more than the test exercises
|
|
90
|
+
- tests that break under a behavior-preserving refactor
|
|
91
|
+
|
|
92
|
+
## Retention bar
|
|
93
|
+
|
|
94
|
+
Keep a test that independently enforces a public API, protocol, config,
|
|
95
|
+
migration, storage, security, platform, default or release contract. Also keep
|
|
96
|
+
call-order tests when order is observable, regressions with a credible failure
|
|
97
|
+
mode, and a source check when it is the cheapest guard that fails on a
|
|
98
|
+
user-facing change and survives a rename. Slow or static is not a reason to
|
|
99
|
+
delete. A kept test that fails on the unchanged baseline may be a real bug:
|
|
100
|
+
reproduce and fix it rather than deleting the test.
|
|
101
|
+
|
|
102
|
+
## Audit mode
|
|
103
|
+
|
|
104
|
+
1. **Discover, read-only.** Read the repository's testing guidance first. For a
|
|
105
|
+
large suite, split discovery by area and run the slices in parallel if the
|
|
106
|
+
host allows. Prefer a few high-confidence candidates over a long speculative
|
|
107
|
+
list. Before judging one, read the whole test, the production code it covers,
|
|
108
|
+
that code's callers, and any overlapping tests.
|
|
109
|
+
2. **Record evidence per candidate** before editing: test name and location;
|
|
110
|
+
the failure it can actually detect; non-test callers of what it covers; the
|
|
111
|
+
stronger test that remains, or why none is needed; why it exists (history);
|
|
112
|
+
what production or test-support code its removal frees; the focused
|
|
113
|
+
validation command. A missing field means the candidate is not ready.
|
|
114
|
+
3. **Edit one coherent batch.** Remove the tests along with the test-only
|
|
115
|
+
exports, wrappers and dead paths they kept alive; don't leave aliases. Move a
|
|
116
|
+
retained regression to its owning boundary. Don't add replacement tests that
|
|
117
|
+
restate the implementation, and don't count uncertain deletions as progress.
|
|
118
|
+
4. **Validate.** Run the owning and sibling tests, then the repository's
|
|
119
|
+
required checks. Where a removed test grepped source or a plan, run the
|
|
120
|
+
script or dry run that owns the real contract instead.
|
|
121
|
+
5. **Report**: categories removed, kept false positives and why, checks run,
|
|
122
|
+
and production versus test lines changed (`git diff --numstat`). Land one
|
|
123
|
+
batch at a time through the repository's normal flow, then rerun discovery on
|
|
124
|
+
the updated base for the next batch.
|
|
125
|
+
|
|
126
|
+
## Sources
|
|
127
|
+
|
|
128
|
+
Original synthesis, adapting:
|
|
129
|
+
|
|
130
|
+
- OpenClaw's
|
|
131
|
+
[`test-audit`](https://github.com/openclaw/openclaw/blob/65f1e4d2279eecd6e40d55d82027d5c97b4b272b/.agents/skills/test-audit/SKILL.md):
|
|
132
|
+
the authoring gate, junk patterns, retention bar and evidence fields.
|
|
133
|
+
- [Ansh Nanda via Ray Fernando](https://x.com/RayFernando1337/status/2102927565778522610):
|
|
134
|
+
the failure-list-first rule. Their "E2E tests only" rule is not adopted.
|
|
135
|
+
- no_human's [tamper guard](https://github.com/no-human-ai/no_human): weakening
|
|
136
|
+
a test counts as removing it.
|
|
137
|
+
- addyosmani/agent-skills'
|
|
138
|
+
[floor guard](https://github.com/addyosmani/agent-skills/blob/main/skills/constraint-driven-development/references/floor-guard.md)
|
|
139
|
+
(MIT): the diff checker.
|
|
140
|
+
- obra/superpowers `writing-good-tests.md`: the mutation check behind
|
|
141
|
+
authoring-gate question 2.
|
|
142
|
+
- omkamal's [pypict skill](https://github.com/omkamal/pypict-claude-skill):
|
|
143
|
+
pairwise coverage.
|
|
144
|
+
|
|
145
|
+
For recorded proof that a UI change works, see michaelshimeles'
|
|
146
|
+
[`evidence-driven-testing`](https://github.com/michaelshimeles/skills/tree/main/evidence-driven-testing).
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
#!/usr/bin/env bun
|
|
2
|
+
// Flag changes that make tests or checkers easier to pass: new suppression
|
|
3
|
+
// comments, added skips or `.only`, deleted test files, and assertions removed
|
|
4
|
+
// from test files that still exist. Diff-scoped: merge base to working tree,
|
|
5
|
+
// untracked files included. Tightening is silent; only loosening is reported.
|
|
6
|
+
// bun floor-guard.ts [--base <ref>] (default: origin/HEAD, then origin/main)
|
|
7
|
+
// Exit 0 clean, 1 findings, 2 the guard could not run. Never read 2 as clean.
|
|
8
|
+
// Adapted from addyosmani/agent-skills constraint-driven-development
|
|
9
|
+
// references/floor-guard.md (MIT).
|
|
10
|
+
import { spawnSync } from "node:child_process";
|
|
11
|
+
import { parseArgs } from "node:util";
|
|
12
|
+
|
|
13
|
+
export type Finding = { rule: string; file: string; text: string };
|
|
14
|
+
|
|
15
|
+
const SUPPRESSIONS =
|
|
16
|
+
/@ts-ignore|@ts-nocheck|@ts-expect-error|eslint-disable|oxlint-disable|biome-ignore|#\s*noqa|#\s*type:\s*ignore|istanbul ignore|c8 ignore|nosemgrep|gitleaks:allow|Stryker disable/;
|
|
17
|
+
const SKIPS =
|
|
18
|
+
/\.(skip|todo|only)\s*\(|\b(xit|xdescribe|xtest)\s*\(|@pytest\.mark\.(skip|xfail)|\bt\.Skip\(/;
|
|
19
|
+
const ASSERTION = /\b(expect|assert\w*|should)\b/;
|
|
20
|
+
|
|
21
|
+
export function isTestFile(path: string): boolean {
|
|
22
|
+
return /\.(test|spec)\.[^/]+$|_test\.[^/]+$|(^|\/)test_[^/]+$|(^|\/)(tests?|__tests__)\//.test(
|
|
23
|
+
path,
|
|
24
|
+
);
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
// Reads a unified diff (`--unified=0` is enough) and returns the loosening moves.
|
|
28
|
+
export function scanDiff(diff: string): Finding[] {
|
|
29
|
+
const added: { file: string; text: string }[] = [];
|
|
30
|
+
const removed: { file: string; text: string }[] = [];
|
|
31
|
+
const deleted = new Set<string>();
|
|
32
|
+
const strip = (p: string) => p.replace(/^[ab]\//, "");
|
|
33
|
+
let file = "";
|
|
34
|
+
let oldFile = "";
|
|
35
|
+
// File headers appear only between `diff --git` and the first `@@`; inside a hunk,
|
|
36
|
+
// an added line such as `++ x` is content, not a header.
|
|
37
|
+
let inHunk = false;
|
|
38
|
+
for (const line of diff.split("\n")) {
|
|
39
|
+
if (line.startsWith("diff ")) {
|
|
40
|
+
inHunk = false;
|
|
41
|
+
} else if (line.startsWith("@@")) {
|
|
42
|
+
inHunk = true;
|
|
43
|
+
} else if (!inHunk && line.startsWith("--- ")) {
|
|
44
|
+
oldFile = strip(line.slice(4));
|
|
45
|
+
} else if (!inHunk && line.startsWith("+++ ")) {
|
|
46
|
+
const newFile = strip(line.slice(4));
|
|
47
|
+
file = newFile === "/dev/null" ? oldFile : newFile;
|
|
48
|
+
if (newFile === "/dev/null") {
|
|
49
|
+
deleted.add(oldFile);
|
|
50
|
+
}
|
|
51
|
+
} else if (line.startsWith("+")) {
|
|
52
|
+
added.push({ file, text: line.slice(1) });
|
|
53
|
+
} else if (line.startsWith("-")) {
|
|
54
|
+
removed.push({ file, text: line.slice(1) });
|
|
55
|
+
}
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
const findings: Finding[] = [];
|
|
59
|
+
const flag = (rule: string, f: string, text: string) =>
|
|
60
|
+
findings.push({ rule, file: f, text: text.trim().slice(0, 120) });
|
|
61
|
+
for (const { file: f, text } of added) {
|
|
62
|
+
if (SUPPRESSIONS.test(text)) {
|
|
63
|
+
flag("silenced-checker", f, text);
|
|
64
|
+
}
|
|
65
|
+
if (isTestFile(f) && SKIPS.test(text)) {
|
|
66
|
+
flag("test-skipped", f, text);
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
for (const f of deleted) {
|
|
70
|
+
if (isTestFile(f)) {
|
|
71
|
+
flag("test-deleted", f, "file deleted");
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
// An assertion line that reappears in the same file was moved, not removed.
|
|
75
|
+
const kept = new Set(added.map((a) => `${a.file}\0${a.text.trim()}`));
|
|
76
|
+
for (const { file: f, text } of removed) {
|
|
77
|
+
if (!isTestFile(f) || deleted.has(f) || !ASSERTION.test(text)) {
|
|
78
|
+
continue;
|
|
79
|
+
}
|
|
80
|
+
if (!kept.has(`${f}\0${text.trim()}`)) {
|
|
81
|
+
flag("assertion-removed", f, text);
|
|
82
|
+
}
|
|
83
|
+
}
|
|
84
|
+
return findings;
|
|
85
|
+
}
|
|
86
|
+
|
|
87
|
+
function git(args: string[], okStatuses = [0]): string | null {
|
|
88
|
+
const r = spawnSync("git", args, { encoding: "utf8" });
|
|
89
|
+
return r.status !== null && okStatuses.includes(r.status) ? r.stdout : null;
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
function main(): number {
|
|
93
|
+
const bail = (msg: string) => (console.error(`floor-guard: ${msg}`), 2);
|
|
94
|
+
let base: string | undefined;
|
|
95
|
+
try {
|
|
96
|
+
base = parseArgs({ options: { base: { type: "string" } } }).values.base;
|
|
97
|
+
} catch (error) {
|
|
98
|
+
return bail(error instanceof Error ? error.message : String(error));
|
|
99
|
+
}
|
|
100
|
+
const bases = base ? [base] : ["origin/HEAD", "origin/main"];
|
|
101
|
+
let mergeBase: string | undefined;
|
|
102
|
+
for (const b of bases) {
|
|
103
|
+
mergeBase = git(["merge-base", b, "HEAD"])?.trim();
|
|
104
|
+
if (mergeBase) {
|
|
105
|
+
break;
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
if (!mergeBase) {
|
|
109
|
+
return bail(`no merge base against ${bases.join(" or ")}`);
|
|
110
|
+
}
|
|
111
|
+
const tracked = git(["diff", "--unified=0", mergeBase, "--"]);
|
|
112
|
+
const untracked = git(["ls-files", "--others", "--exclude-standard"]);
|
|
113
|
+
if (tracked === null || untracked === null) {
|
|
114
|
+
return bail("could not read the diff");
|
|
115
|
+
}
|
|
116
|
+
let diff = tracked;
|
|
117
|
+
for (const f of untracked.split("\n").filter(Boolean)) {
|
|
118
|
+
// --no-index exits 1 when the sides differ, which is the normal case here.
|
|
119
|
+
const d = git(["diff", "--no-index", "--unified=0", "/dev/null", f], [0, 1]);
|
|
120
|
+
if (d === null) {
|
|
121
|
+
return bail(`could not diff untracked file ${f}`);
|
|
122
|
+
}
|
|
123
|
+
diff += `\n${d}`;
|
|
124
|
+
}
|
|
125
|
+
const findings = scanDiff(diff);
|
|
126
|
+
for (const f of findings) {
|
|
127
|
+
console.log(`${f.rule}\t${f.file}\t${f.text}`);
|
|
128
|
+
}
|
|
129
|
+
if (findings.length === 0) {
|
|
130
|
+
console.log("floor-guard: clean");
|
|
131
|
+
}
|
|
132
|
+
return findings.length ? 1 : 0;
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
if (import.meta.main) {
|
|
136
|
+
process.exit(main());
|
|
137
|
+
}
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: hr-usage-report
|
|
3
|
+
description: Measure where Claude Code, Codex and OpenCode usage went by provider, model and role, and chart model comparisons as HTML. Use for quota or cost questions and to show a model or role recommendation visually.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Usage report
|
|
7
|
+
|
|
8
|
+
Answer quota and model-role questions from measured usage and published
|
|
9
|
+
evidence, then show the result as charts and tables. Run the scripts from this
|
|
10
|
+
skill's directory with Bun.
|
|
11
|
+
|
|
12
|
+
## 1. Measure where usage went
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
bun scripts/usage-by-model.ts --days 7 [--providers claude,codex,opencode] [--claude-projects <dir,dir>] [--prices <api.json>] [--json]
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
It reads Claude Code transcripts (`projects` under `~/.claude` and under every
|
|
19
|
+
other `~/.claude_*` or `~/.claude-*` configuration directory, so sessions run
|
|
20
|
+
with a separate `CLAUDE_CONFIG_DIR` count; `--claude-projects` replaces that
|
|
21
|
+
list), Codex rollouts
|
|
22
|
+
(`~/.codex/sessions`) and the OpenCode database, all read-only and local. It
|
|
23
|
+
prices each request at [models.dev](https://models.dev) list rates, including
|
|
24
|
+
long-context tiers, and splits usage by provider, model and role: main session,
|
|
25
|
+
subagent, advisor. Without `--prices` it downloads `https://models.dev/api.json`;
|
|
26
|
+
that request carries no local data. Use `--since <ISO>` for a fixed start.
|
|
27
|
+
|
|
28
|
+
Read the output beside the provider's live quota readings:
|
|
29
|
+
|
|
30
|
+
- **Shares, not dollars.** List price stands in for subscription quota; no
|
|
31
|
+
provider publishes the conversion. Say so in the report.
|
|
32
|
+
- **Cache reads.** When cache reads dominate a row's cost, context length, not
|
|
33
|
+
the model's output, drives usage. Compare the main-request context median
|
|
34
|
+
with the session-start median: the gap is conversation growth; the start is
|
|
35
|
+
fixed overhead (system prompt, rules, tool schemas).
|
|
36
|
+
- **Advisor rows.** Each advisor call re-reads the whole transcript uncached at
|
|
37
|
+
the advisor's rates; a few calls can outweigh many main turns.
|
|
38
|
+
- **Cross-provider work.** The same table shows what the author's sessions and
|
|
39
|
+
another vendor's reviewer each cost.
|
|
40
|
+
- **Unpriced models.** The summary lists models models.dev has no price for;
|
|
41
|
+
report them rather than dropping them.
|
|
42
|
+
|
|
43
|
+
## 2. Pick evidence by question
|
|
44
|
+
|
|
45
|
+
| Question | Evidence | Read |
|
|
46
|
+
| ------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------- |
|
|
47
|
+
| Quality per dollar for implementation | Agentic coding boards that publish effort, harness and cost per task | Score against cost at every effort; cost per solved task |
|
|
48
|
+
| Review and bug finding | Bug-finding boards with run counts per row | The effort curve; treat single runs as noisy |
|
|
49
|
+
| Raw speed | Output tokens per second | Generation speed only |
|
|
50
|
+
| Task speed | Boards that time whole tasks | Time to finish; a model that generates faster can still take longer |
|
|
51
|
+
| Fit for your own work | A known-answer trial on your own tasks (`hr-benchmarking`) | Correctness and model time per task shape |
|
|
52
|
+
|
|
53
|
+
Compare models at the same effort and harness, or say they differ. A vendor
|
|
54
|
+
table that sets one model's top effort against another's lower effort is not a
|
|
55
|
+
matched comparison. Keep token speed and task time in separate charts; never
|
|
56
|
+
infer one from the other.
|
|
57
|
+
|
|
58
|
+
## 3. Render the report
|
|
59
|
+
|
|
60
|
+
Write a data file, then:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
bun scripts/render-report.ts <data.json> <out.html>
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
The page is self-contained, follows the host's theme variables when it is
|
|
67
|
+
embedded, and falls back to a dark surface. `assets/example-report.json` uses
|
|
68
|
+
every section type.
|
|
69
|
+
|
|
70
|
+
The data file declares `series` once, so each model keeps one color across
|
|
71
|
+
every chart in the report: up to three colored series with slots 0 to 2, and at
|
|
72
|
+
most one `"reference": true` series drawn dashed and neutral. The renderer
|
|
73
|
+
rejects a fourth colored series because no fourth hue passes the color-vision
|
|
74
|
+
check on the dark surface against the other three. Compare more models in a
|
|
75
|
+
second report file.
|
|
76
|
+
|
|
77
|
+
Section types:
|
|
78
|
+
|
|
79
|
+
- `curve`: points with `x`, `y`, `label`, `tip` per series; `xLog`, domains,
|
|
80
|
+
ticks and axis affixes. Use for score against cost across efforts.
|
|
81
|
+
- `ladder`: `categories` (usually efforts) and `values` per series, `null` for a
|
|
82
|
+
missing row, optional `notes` such as run counts.
|
|
83
|
+
- `bars`: `rows` of `label`, `key`, `value`, optional `display` and `tip`.
|
|
84
|
+
- `pair`: two sections side by side, such as token speed beside task time.
|
|
85
|
+
- `table`: `columns` and `rows`; a cell may be `{ "text", "verdict": "yes" | "maybe" | "no" }`;
|
|
86
|
+
`numeric` lists right-aligned column indexes.
|
|
87
|
+
|
|
88
|
+
End with one verdict table per model or role: role, key evidence with numbers,
|
|
89
|
+
verdict. Name the evidence date in the title, and apply any model or effort
|
|
90
|
+
limits the user has set before recommending a configuration.
|