@ericrisco/rsc 1.0.10 → 1.0.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -13
- package/manifest.json +133 -2
- package/package.json +3 -3
- package/scripts/consult.js +75 -42
- package/scripts/lib/default-skill-floor.js +11 -0
- package/scripts/lib/domains.js +3 -3
- package/scripts/lib/recommend.js +1 -0
- package/scripts/lib/text-rank.js +112 -0
- package/scripts/lib/ui.js +3 -3
- package/scripts/routing-audit.js +163 -0
- package/scripts/rsc.js +11 -10
- package/skills/author-skill/references/rsc-conventions.md +1 -1
- package/skills/bro/SKILL.md +68 -0
- package/skills/bro/evals/README.md +22 -0
- package/skills/bro/evals/cases.yaml +58 -0
- package/skills/decision-challenge/SKILL.md +113 -0
- package/skills/decision-challenge/evals/README.md +3 -0
- package/skills/decision-challenge/evals/cases.yaml +42 -0
- package/skills/deprecation/SKILL.md +123 -0
- package/skills/deprecation/evals/README.md +3 -0
- package/skills/deprecation/evals/cases.yaml +42 -0
- package/skills/idea-refinement/SKILL.md +121 -0
- package/skills/idea-refinement/evals/README.md +3 -0
- package/skills/idea-refinement/evals/cases.yaml +42 -0
- package/skills/observability/SKILL.md +6 -0
- package/skills/observability/evals/cases.yaml +2 -0
- package/skills/performance/SKILL.md +13 -0
- package/skills/performance/evals/cases.yaml +2 -0
- package/skills/simplify-code/SKILL.md +101 -0
- package/skills/simplify-code/evals/README.md +3 -0
- package/skills/simplify-code/evals/cases.yaml +42 -0
- package/skills/source-grounded-development/SKILL.md +108 -0
- package/skills/source-grounded-development/evals/README.md +3 -0
- package/skills/source-grounded-development/evals/cases.yaml +42 -0
- package/skills/specify/SKILL.md +31 -0
- package/skills/suggest/evals/cases.yaml +1 -1
- package/skills/testing-go/SKILL.md +17 -0
- package/skills/testing-py/SKILL.md +18 -0
- package/skills/testing-web/SKILL.md +20 -0
- package/skills/verify/SKILL.md +37 -0
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: source-grounded-development
|
|
3
|
+
description: "Use when implementation or review depends on current version-specific framework, SDK, protocol or platform behaviour and memory is unsafe. Detects exact local versions, verifies primary sources, tests applicability and marks unknowns. NOT a standalone cited memo (`research-ops`), NOT diagnosis of a reproduced failure (`debug`)."
|
|
4
|
+
tags: [official-docs, primary-sources, version-detection, implementation, verification]
|
|
5
|
+
recommends: [research-ops, decision-records, debug, technical-writing]
|
|
6
|
+
profiles: [core, full]
|
|
7
|
+
origin: risco
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# source-grounded-development — make current facts part of the change
|
|
11
|
+
|
|
12
|
+
Use this when correctness depends on what a framework, SDK, API, standard or hosted platform **does now for the exact version in this repository**. The deliverable is still code, a review or an implementation decision; sources are the evidence behind it, not a separate essay.
|
|
13
|
+
|
|
14
|
+
This is narrower than `../research-ops/SKILL.md`: research-ops answers an open question with a cited memo. Here, the question exists because a concrete change cannot safely proceed from memory.
|
|
15
|
+
|
|
16
|
+
## The loop
|
|
17
|
+
|
|
18
|
+
```text
|
|
19
|
+
DETECT → QUESTION → SOURCE → APPLY → PROVE → REPORT
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
### 1. DETECT the actual version and environment
|
|
23
|
+
|
|
24
|
+
Read the repository before the web:
|
|
25
|
+
|
|
26
|
+
- lockfiles and manifests, not only a README;
|
|
27
|
+
- runtime and compiler configuration;
|
|
28
|
+
- generated client/server versions;
|
|
29
|
+
- deployment target and enabled compatibility flags;
|
|
30
|
+
- existing wrapper code that may override documented defaults.
|
|
31
|
+
|
|
32
|
+
Record an evidence line such as: `Next 16.2.1 from pnpm-lock.yaml; Node 24 from .nvmrc; App Router; deployed to Vercel.` If versions conflict, stop treating the docs as applicable until the conflict is resolved.
|
|
33
|
+
|
|
34
|
+
### 2. QUESTION only the claims that can change the implementation
|
|
35
|
+
|
|
36
|
+
Build a small claim ledger. Each row is one decision-sensitive statement:
|
|
37
|
+
|
|
38
|
+
```text
|
|
39
|
+
claim | exact component/version | evidence needed | source | applicability | status
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Examples: whether a callback is invoked on retry, the default cookie policy, an API’s idempotency semantics, a deprecated option’s replacement, or a browser feature’s support boundary. Do not cite obvious local code or generic programming facts just to make the list longer.
|
|
43
|
+
|
|
44
|
+
### 3. SOURCE by authority and freshness
|
|
45
|
+
|
|
46
|
+
Prefer, in order:
|
|
47
|
+
|
|
48
|
+
1. the repository and installed type definitions for local truth;
|
|
49
|
+
2. official versioned documentation and migration/release notes;
|
|
50
|
+
3. specifications, RFCs or provider API references;
|
|
51
|
+
4. upstream source or tests when documentation is ambiguous;
|
|
52
|
+
5. reputable secondary material only as a lead, never as the final authority for a disputed behaviour.
|
|
53
|
+
|
|
54
|
+
Use the current page that matches the detected version. A search snippet, an undated blog, a forum answer or remembered API shape is not proof. When two primary sources disagree, show the disagreement and test the behaviour; do not average them into false certainty.
|
|
55
|
+
|
|
56
|
+
### 4. APPLY the source to this repository
|
|
57
|
+
|
|
58
|
+
For every useful source, state why it applies here:
|
|
59
|
+
|
|
60
|
+
- same major/minor version or a documented compatibility range;
|
|
61
|
+
- same runtime, router, transport, deployment mode or feature flag;
|
|
62
|
+
- no local wrapper/configuration that changes the default;
|
|
63
|
+
- publication or update date appropriate to a fast-moving claim.
|
|
64
|
+
|
|
65
|
+
Then make the smallest implementation decision that the evidence supports. Do not turn examples from docs into architecture by cargo cult. Examples demonstrate an API; the repository still determines ownership, boundaries and error policy.
|
|
66
|
+
|
|
67
|
+
### 5. PROVE behaviour locally
|
|
68
|
+
|
|
69
|
+
Documentation tells you intended behaviour. A local test proves the integration:
|
|
70
|
+
|
|
71
|
+
- add or run a focused contract/regression test;
|
|
72
|
+
- compile against installed types;
|
|
73
|
+
- exercise a minimal reproduction when defaults or runtime order matter;
|
|
74
|
+
- run the stack’s normal verification gate after the focused proof.
|
|
75
|
+
|
|
76
|
+
If local evidence contradicts the source, preserve the reproduction and investigate version/config drift. Do not silently choose whichever result is more convenient.
|
|
77
|
+
|
|
78
|
+
### 6. REPORT without polluting the code
|
|
79
|
+
|
|
80
|
+
End with:
|
|
81
|
+
|
|
82
|
+
- versions and environment detected;
|
|
83
|
+
- material claims verified and their primary links;
|
|
84
|
+
- implementation choices those claims changed;
|
|
85
|
+
- local evidence run;
|
|
86
|
+
- assumptions still **UNVERIFIED**, with the consequence if wrong.
|
|
87
|
+
|
|
88
|
+
Put source links in the final report, PR description or a durable decision record when the rationale must outlive the task. Do not scatter documentation URLs through source comments unless the code’s invariant genuinely cannot be understood without that external contract.
|
|
89
|
+
|
|
90
|
+
## Stop conditions
|
|
91
|
+
|
|
92
|
+
Stop searching when every decision-sensitive claim is verified, contradicted or explicitly marked unverified and the local proof is green. Search breadth is not quality. A fifth article repeating the official reference adds no evidence.
|
|
93
|
+
|
|
94
|
+
Do not use this skill as a reason to delay reversible work indefinitely. If a primary source is unavailable, name the uncertainty, choose the safest reversible option, add a proof/guard, and state what would trigger revisiting it.
|
|
95
|
+
|
|
96
|
+
## Anti-patterns
|
|
97
|
+
|
|
98
|
+
| Smell | Correction |
|
|
99
|
+
|---|---|
|
|
100
|
+
| A reproduced bug is replaced by a docs tour | `debug` owns diagnosis; use this only for the disputed versioned claim. |
|
|
101
|
+
| A broad open question is forced into implementation | Route the cited memo to `research-ops`. |
|
|
102
|
+
| A lasting architecture choice disappears into a task summary | Preserve it in `decision-records` with this ledger as evidence. |
|
|
103
|
+
| A search result or secondary article is presented as official | Follow it to the primary source or mark the claim unverified. |
|
|
104
|
+
| “Latest docs” are read before the lockfile | Detect the local version and mode first; then choose applicable docs. |
|
|
105
|
+
|
|
106
|
+
## Orientación (siempre)
|
|
107
|
+
|
|
108
|
+
Cierra cada turno con el **bloque-brújula** (📍 dónde estás · ✅ qué hiciste · 🧭 por qué · ➡️ siguiente, terminando en pregunta), calibrado al dial de `02-DOCS/wiki/harness/user-profile.md`. **Nunca termines en seco.** Protocolo completo: skill `orient` → `skills/orient/references/orientation-contract.md`. (Defiere a `suggest` el “¿instalo la skill que falta?”.)
|
|
@@ -0,0 +1,3 @@
|
|
|
1
|
+
# Evals — source-grounded-development
|
|
2
|
+
|
|
3
|
+
The routing cases separate version-sensitive implementation evidence from general research, debugging, ADRs, docs and security review. The capability rubric requires the full local-version → primary-source → applicability → local-proof chain and treats unverified deployment assumptions as visible risk.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
skill: source-grounded-development
|
|
2
|
+
|
|
3
|
+
should_trigger:
|
|
4
|
+
- prompt: "Before changing sessions, verify the installed Express and express-session versions against their official production guidance."
|
|
5
|
+
why: "A concrete implementation depends on current version-specific framework behaviour."
|
|
6
|
+
- prompt: "No te fíes de memoria: mira el lockfile y la documentación oficial de Stripe para comprobar la semántica de reintentos."
|
|
7
|
+
why: "Spanish request for local version detection and primary-source verification."
|
|
8
|
+
- prompt: "This SDK changed between majors; prove which callback contract applies to the version in this repo."
|
|
9
|
+
why: "A version boundary changes implementation correctness."
|
|
10
|
+
- prompt: "Comprova amb fonts primàries si aquesta opció està deprecada en la versió instal·lada i quin reemplaçament toca."
|
|
11
|
+
why: "Catalan request for official deprecation evidence tied to the installed version."
|
|
12
|
+
- prompt: "Implement the browser API only after checking its current specification and support boundary, then test the fallback."
|
|
13
|
+
why: "Protocol/browser behaviour requires primary evidence plus local proof."
|
|
14
|
+
- prompt: "Review this integration and cite the official pages behind every provider-specific claim that affects the verdict."
|
|
15
|
+
why: "A review depends on current provider contracts rather than recalled assumptions."
|
|
16
|
+
|
|
17
|
+
should_not_trigger:
|
|
18
|
+
- prompt: "Research whether passkeys improve consumer login conversion and deliver a cited memo."
|
|
19
|
+
route_to: research-ops
|
|
20
|
+
why: "The artifact is a standalone research memo, not an implementation grounded in an installed version."
|
|
21
|
+
- prompt: "The checkout test fails on every third retry; reproduce it and find the root cause."
|
|
22
|
+
route_to: debug
|
|
23
|
+
why: "A concrete failure needs diagnosis; documentation may support but does not own the workflow."
|
|
24
|
+
- prompt: "Record why we selected Kafka over SQS and preserve the rejected alternatives."
|
|
25
|
+
route_to: decision-records
|
|
26
|
+
why: "Durable decision rationale is an ADR."
|
|
27
|
+
- prompt: "Write the public setup guide for our SDK users."
|
|
28
|
+
route_to: technical-writing
|
|
29
|
+
why: "The deliverable is documentation, not source-grounded implementation."
|
|
30
|
+
- prompt: "Audit this webhook handler for SSRF and secret leakage."
|
|
31
|
+
route_to: secure-coding
|
|
32
|
+
why: "Security threat/control review owns the task."
|
|
33
|
+
|
|
34
|
+
capability:
|
|
35
|
+
- scenario: "Implement production session cookies for an application whose framework and session-library versions are pinned but whose deployment proxy behaviour is unclear."
|
|
36
|
+
must_include:
|
|
37
|
+
- "Reads lockfiles, runtime configuration, deployment target, and local wrappers before choosing documentation."
|
|
38
|
+
- "Creates a bounded claim ledger covering cookie defaults, proxy trust, secure transport, secret handling, and session-store production requirements."
|
|
39
|
+
- "Uses version-matching official docs, release/migration notes, specifications, or upstream source; secondary sources are not treated as final authority."
|
|
40
|
+
- "Explains why each material source applies to this runtime and flags deployment facts that remain unverified."
|
|
41
|
+
- "Adds or runs focused local proof plus the repository's normal verification gate instead of trusting documentation alone."
|
|
42
|
+
- "Reports citations in the result or decision record without filling ordinary source comments with URLs."
|
package/skills/specify/SKILL.md
CHANGED
|
@@ -130,6 +130,35 @@ irreversible, or large scope), dispatch a **fresh-context subagent** to read the
|
|
|
130
130
|
**Skip it** for L0/L1 on a small, low-risk spec — the self-review scan is enough there; don't spin up
|
|
131
131
|
a subagent to vet a two-paragraph spec. Like the rest of the chain, ceremony scales to the stakes.
|
|
132
132
|
|
|
133
|
+
### Approval is its own exchange (step 9)
|
|
134
|
+
|
|
135
|
+
**An answer to a question is not an approval.** If you asked the user to decide something, they
|
|
136
|
+
answered *that question* and nothing else. Their answer is an **input** to the spec — and it
|
|
137
|
+
*changes* the spec, which means any approval you were holding before the question is the approval
|
|
138
|
+
of a document that no longer exists.
|
|
139
|
+
|
|
140
|
+
Questions and approval are two exchanges, in that order:
|
|
141
|
+
|
|
142
|
+
```text
|
|
143
|
+
ask → user answers → fold it in → say what changed → show the revised spec → ask for approval
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
- **The recommended-option shape is where this fails most easily.** When the user picks the option
|
|
147
|
+
you recommended, the spec *looks* unchanged and consent *looks* implied. Neither is true — you
|
|
148
|
+
proposed, they chose, the spec moved.
|
|
149
|
+
- **None of these are approval:** an answer to your question; a "go ahead" about some other step;
|
|
150
|
+
silence; the request that started the task. If you cannot quote the words that approved **this**
|
|
151
|
+
spec, you do not have approval.
|
|
152
|
+
- **Autopilot is still valid** — an up-front "run it all yourself" authorizes the whole run, and
|
|
153
|
+
that is a real answer to a real question. What changes is the bookkeeping: record it as
|
|
154
|
+
**approved in autopilot, not item by item**, in the spec's `status:`. The spec then becomes the
|
|
155
|
+
artifact the user reviews *after the fact*, and the work claims correspondingly less certainty
|
|
156
|
+
about having captured their intent. Writing "aprobada" flat, when what happened was a blanket
|
|
157
|
+
go-ahead, is the record telling a small lie about the strongest gate in the chain.
|
|
158
|
+
- **Rejection is information — keep it.** If the user turns the spec down, revise it in place and
|
|
159
|
+
append the reason to `## Revisions`. Never delete and start clean: what they rejected, and why,
|
|
160
|
+
is the most useful thing in the file.
|
|
161
|
+
|
|
133
162
|
## Worked shape (abridged)
|
|
134
163
|
|
|
135
164
|
```markdown
|
|
@@ -206,6 +235,8 @@ The proposal is allowed to mention options and tradeoffs; the spec that follows
|
|
|
206
235
|
| Skip the whole loop because "this is too simple to design" | That's the rationalization that wastes the most work. Every feature gets the loop; only a literal one-line, zero-risk change skips. |
|
|
207
236
|
| Present one approach as the answer | Offer 2-3 distinct directions with trade-offs and a recommendation; let the user choose before you write the spec. |
|
|
208
237
|
| Hand to `plan`/`implement` before the user approved the written spec | The approval at step 9 is the gate. No design approved → nothing gets built. |
|
|
238
|
+
| Treat the answer to your question as the approval of the spec | They answered the question, and the answer *changed* the spec. Fold it in, say what changed, show it, ask again. |
|
|
239
|
+
| Write `status: aprobada` after a blanket "just run it" | That is autopilot. Record it as such — the record must not overstate the strongest gate in the chain. |
|
|
209
240
|
| Skip non-goals because "it's obvious" | Unsaid scope becomes assumed scope. State what you are *not* doing. |
|
|
210
241
|
| Resolve every ambiguity yourself to look finished | Inventing answers is worse than naming gaps. List them in Points to clarify. |
|
|
211
242
|
| Start designing the solution because it's clearer | Stay on WHAT/WHY. The plan is a later, separate phase. |
|
|
@@ -59,7 +59,7 @@ capability:
|
|
|
59
59
|
- "Checks what is already installed (npx @ericrisco/rsc list) and never proposes a skill the user already has"
|
|
60
60
|
- "Makes at most one suggestion and does not interrupt the flow for nice-to-haves"
|
|
61
61
|
|
|
62
|
-
- scenario: "Mid-conversation, after base install, the user says 'quiero montar una pagina web para vender cursos online'. `init`, `harness`, `orient`, and `
|
|
62
|
+
- scenario: "Mid-conversation, after base install, the user says 'quiero montar una pagina web para vender cursos online'. `init`, `harness`, `orient`, `suggest`, and `bro` are already installed; `nextjs` is not. Show how rsc-suggest behaves."
|
|
63
63
|
must_include:
|
|
64
64
|
- "Treats the message as in-agent capability intent, not as a CLI-only consult task"
|
|
65
65
|
- "Runs `npx @ericrisco/rsc catalog --available` to see the not-installed skills (already excludes installed `init`/`harness`)"
|
|
@@ -169,6 +169,23 @@ go build -cover -o ./bin/app . # coverage for integratio
|
|
|
169
169
|
|
|
170
170
|
A coverage **percentage is a smoke alarm, not a goal.** 100% line coverage with no assertions on the values is worthless; 70% on the branches that actually carry logic is fine. Use `-coverpkg=./...` when your tests live in a separate package and exercise code across the module, or the number undercounts. Profiles, integration-binary coverage, and reading the HTML report are in `references/coverage-and-benchmarks.md`.
|
|
171
171
|
|
|
172
|
+
## Mutation: does the suite actually notice?
|
|
173
|
+
|
|
174
|
+
Coverage tells you which code ran; it cannot tell you whether any test would have **noticed** if that code were wrong. A table-driven test whose cases all assert `err == nil` and nothing else raises coverage without detecting anything. Mutation testing plants bugs on purpose — flip a `<` to `<=`, drop an early return, replace a returned value with the zero value — and a mutant the suite still passes is a test that asserts nothing.
|
|
175
|
+
|
|
176
|
+
This is **not** fuzzing. Fuzzing hunts inputs your code mishandles; mutation hunts tests that would not catch a bug you already know about. They find different things and neither substitutes for the other.
|
|
177
|
+
|
|
178
|
+
Go has **no mature mutation tool** — `go-mutesting` and `gremlins` exist but neither is a dependable default — so this is the manual procedure, and that makes the discipline stricter here, not looser:
|
|
179
|
+
|
|
180
|
+
1. Pick the changed logic. Script the run and **commit the script** (e.g. `tools/mutants.go` or a shell driver); do not hand-edit N times, because you will need to rerun it and a restore mistake is silent.
|
|
181
|
+
2. One mutant at a time: flip a comparison, off-by-one a bound or slice index, delete one branch, swap `&&`/`||`, return the zero value. `go test ./...` must fail for every one.
|
|
182
|
+
3. Restore, confirm with `git diff` that nothing else moved, and rerun the suite green.
|
|
183
|
+
4. Report as `N/N killed`, naming the mutants.
|
|
184
|
+
|
|
185
|
+
**The runner must prove it executed each mutant.** A hand-rolled driver that can report a kill it never ran only ever *inflates* the score — so that defect can never surface as a red run, and no failing test will ever tell you about it. Assert the mutated file changed on disk (compare hashes before and after) and that the test command actually exited non-zero for the reason you expect, rather than trusting the loop.
|
|
186
|
+
|
|
187
|
+
**A survivor is not automatically a failure.** Some mutants are semantically equivalent and cannot be killed; classify those with the reason instead of writing a test that asserts non-behaviour. Scale the whole layer to risk (P7): reach for it where a bug is expensive — money, auth, data loss, concurrency, a public API — not on every change.
|
|
188
|
+
|
|
172
189
|
## Benchmarks
|
|
173
190
|
|
|
174
191
|
Use `for b.Loop()` (Go 1.24+), not the legacy `for i := 0; i < b.N; i++`. `b.Loop` manages the timer itself (setup before the loop and teardown after are excluded automatically), runs the body the right number of times, and the compiler is taught **not** to dead-code-eliminate calls inside it.
|
|
@@ -139,6 +139,24 @@ pytest --cov=mypkg --cov-branch --cov-report=term-missing --cov-fail-under=85
|
|
|
139
139
|
|
|
140
140
|
Line coverage lies because executing a line is not the same as testing its outcomes. A function with `if x: a() else: b()` reaches 100% line coverage from a single test that only takes the `if` arm — the `else` ships untested. `--cov-branch` (or `branch = true`) forces both arms to be exercised, which is exactly where happy-path-only suites leak bugs. `term-missing` prints the unhit line and branch numbers so you know what to write next. Exclude only code that is correctly never run by tests (`if TYPE_CHECKING:`, `raise NotImplementedError`, `__repr__`) — never exclude a branch just to make the number go up.
|
|
141
141
|
|
|
142
|
+
## Mutation: does the suite actually notice?
|
|
143
|
+
|
|
144
|
+
Branch coverage tells you which code ran. It cannot tell you whether any test would have **noticed** if that code were wrong — and a test with no assertion, or an assertion that cannot fail, raises coverage without detecting anything. Mutation testing closes that gap by planting bugs on purpose: if the suite still passes, the mutant *survived* and you have found a test that asserts nothing.
|
|
145
|
+
|
|
146
|
+
```bash
|
|
147
|
+
# pyproject.toml → [tool.mutmut] source_paths = ["src/"]
|
|
148
|
+
mutmut run # whole source tree
|
|
149
|
+
mutmut run "mypkg.core*" # scope it — this is the normal case
|
|
150
|
+
mutmut results # what survived
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
Reach for the real tool, not a hand-written mutant list: mutmut generates mutants from the syntax tree, so it cannot apply one to code that moved and cannot report one it never ran.
|
|
154
|
+
|
|
155
|
+
- **Scale it to risk.** This is not a default toll — it is minutes-to-hours on a real tree. Run it where a bug is expensive (money, auth, data loss, concurrency, a public API) or where a green suite keeps shipping bugs anyway. Always scope to the code you changed.
|
|
156
|
+
- **A survivor is not automatically a failure.** Some mutants are semantically equivalent to the original and cannot be killed. Classify those with the reason ("equivalent: both branches return the same value for every reachable input"). Never add a test that asserts non-behaviour just to kill one — that is the same gaming that coverage-chasing is.
|
|
157
|
+
- **A survivor that is a real bug gets a test, not an excuse.** Write the assertion that kills it, then rerun.
|
|
158
|
+
- **If you hand-roll a runner, prove it executed every mutant.** Clear `__pycache__` and set `PYTHONDONTWRITEBYTECODE` per mutant: two same-size mutants written in the same second can share a bytecode cache, and the runner then reports a kill for code it never ran. That defect can only ever *inflate* the score, so it will never show up as a red run — which is why the proof has to be built in rather than assumed.
|
|
159
|
+
|
|
142
160
|
## Property testing
|
|
143
161
|
|
|
144
162
|
Reach for Hypothesis when a fixed example matrix can't cover the input space: parsers, serializers, encoders, math, anything with an invariant. You assert a *property* that must hold for all inputs and let Hypothesis hunt counterexamples.
|
|
@@ -194,6 +194,26 @@ didn't change, and a giant DOM snapshot gets blindly `--updated` the first time
|
|
|
194
194
|
small, stable, serializable output (a formatted currency string, a normalized config object). Never
|
|
195
195
|
snapshot a full component tree as your primary assertion.
|
|
196
196
|
|
|
197
|
+
## Mutation: does the suite actually notice?
|
|
198
|
+
|
|
199
|
+
Coverage tells you which code ran. It cannot tell you whether any test would have **noticed** if that code were wrong — and a `render()` with no assertion, or a snapshot nobody reads, raises coverage while detecting nothing. Mutation testing plants bugs on purpose: if the suite still passes, the mutant *survived* and you have found a test that asserts nothing.
|
|
200
|
+
|
|
201
|
+
```jsonc
|
|
202
|
+
// stryker.config.json — scope is not optional here
|
|
203
|
+
{ "testRunner": "vitest", "mutate": ["src/cart/total.ts", "src/cart/discount.ts"] }
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
```bash
|
|
207
|
+
npx stryker run
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Reach for the real tool over a hand-written mutant list: Stryker generates mutants from the syntax tree, so it cannot apply one to code that moved and cannot report one it never ran.
|
|
211
|
+
|
|
212
|
+
- **Always scope `mutate` to the files you changed.** A whole-project Stryker run on a real front-end does not finish in a useful amount of time; that is the main reason teams try it once and abandon it.
|
|
213
|
+
- **Scale it to risk.** Not a default toll. Run it on the logic where a bug is expensive — pricing, totals, permissions, anything money- or auth-shaped — not on presentational components, where a surviving mutant usually just means the DOM detail genuinely does not matter.
|
|
214
|
+
- **A survivor is not automatically a failure.** Some mutants are semantically equivalent to the original and cannot be killed; classify those with the reason. Never add an assertion about non-behaviour just to kill one — that is coverage-chasing wearing a different hat.
|
|
215
|
+
- **A survivor that is a real bug gets an assertion, not an excuse.** Write it, then rerun.
|
|
216
|
+
|
|
197
217
|
## Anti-patterns
|
|
198
218
|
|
|
199
219
|
| Anti-pattern | Why it's wrong | Do instead |
|
package/skills/verify/SKILL.md
CHANGED
|
@@ -84,6 +84,15 @@ Rules for RUN:
|
|
|
84
84
|
- Run from the directory the stack skill documents (Go runs from the module root; others from the subproject root). Don't guess paths.
|
|
85
85
|
- If no `scripts/verify.sh` exists for a touched stack, that's a gap — say the gate is incomplete and recommend the user add the stack skill, rather than hand-rolling a one-off check that drifts from the real gate.
|
|
86
86
|
|
|
87
|
+
#### A home-grown gate must prove it can fail
|
|
88
|
+
|
|
89
|
+
The dangerous way a checker breaks is not a crash — it is **fail-open**: nothing errors, the layer prints pass, and it prints pass forever. That failure can only ever produce green, so no failing run will ever surface it. The rule follows: **before you trust a home-grown gate's pass, watch it fail** on a known-bad input. It is the RED principle applied to the checker.
|
|
90
|
+
|
|
91
|
+
- **Third-party tools are exempt.** pytest, mypy, tsc, eslint, ruff, `go vet` have earned their failure behavior. This rule is for what we wrote: grep gates, one-off scripts, custom guards, anything whose exit codes nobody has tested.
|
|
92
|
+
- **A must-find-nothing grep has three outcomes, not two.** No matches = pass. A match = fail. **The scan itself breaking** (unreadable input, bad pattern) = fail too. Left unhandled, an unreadable file becomes a silent pass. No `|| true`, no `2>/dev/null`, no bare fallthrough.
|
|
93
|
+
- **State the limit where you state the pass.** Watching a gate fail once proves that *one* known-bad case reaches its failure path. It does **not** prove the gate recognizes every violation of the rule it claims to enforce — a gate can fail closed perfectly and still guard a spelling rather than a behavior. Record the control, and record what it does not buy.
|
|
94
|
+
- **Our own known fail-open:** the stack `verify.sh` scripts SKIP a missing tool instead of failing. That is a documented gap, not a pass — so it goes in the record's *Capas no ejecutadas* under `HERRAMIENTA-AUSENTE`, never left implied.
|
|
95
|
+
|
|
87
96
|
### 3 — WALK the done-checks and acceptance criteria
|
|
88
97
|
|
|
89
98
|
The stack gate proves the code is *clean and tested*. It does **not** prove the feature does what the spec asked. Walk both lists explicitly:
|
|
@@ -106,6 +115,7 @@ tags: [sdd, verification]
|
|
|
106
115
|
timestamp: YYYY-MM-DDTHH:MM:SSZ
|
|
107
116
|
topic: sdd
|
|
108
117
|
slug: <slug>
|
|
118
|
+
source_state: 3f9a1c2 (clean)
|
|
109
119
|
---
|
|
110
120
|
|
|
111
121
|
# Verification — <slug> — YYYY-MM-DD
|
|
@@ -122,9 +132,33 @@ slug: <slug>
|
|
|
122
132
|
- [x] AC1 user can place an order — evidence: e2e test green
|
|
123
133
|
- [ ] AC3 duplicate submit is a no-op — UNVERIFIED: no observable proof
|
|
124
134
|
|
|
135
|
+
## Capas no ejecutadas
|
|
136
|
+
- **NO-APLICA** — mutation: no logic module changed, only templates.
|
|
137
|
+
- **HERRAMIENTA-AUSENTE** — mypy: not installed in this environment; verify.sh printed SKIP and **nothing ran in its place**.
|
|
138
|
+
- **SUSTITUIDA** — suite health: ran the suite twice in a fixed order instead of randomized. Cannot detect whole-suite order dependence.
|
|
139
|
+
|
|
140
|
+
## Hallazgos descartados
|
|
141
|
+
- "the retry loop can spin forever" — dismissed: `client.py:88` caps it at 3 attempts; test_client.py::test_retry_cap covers it.
|
|
142
|
+
|
|
125
143
|
## Verdict: FAIL — 2 open items (T4 done-check, AC3). Not ready for review.
|
|
126
144
|
```
|
|
127
145
|
|
|
146
|
+
**`source_state:` is not decoration.** A verdict belongs to the commit it was measured against, not to the project: without it, nobody can tell whether the record describes the code that shipped. Record the short SHA plus `clean` or `dirty` — a dirty tree means the record describes bytes that exist on no commit, which is worth knowing when someone reads it three weeks later.
|
|
147
|
+
|
|
148
|
+
**All numbers come from one fresh run made after the last edit.** A figure from mid-task is stale and does not go in the record, however true it was when you saw it.
|
|
149
|
+
|
|
150
|
+
**Split "didn't run" three ways**, because one `SKIP` list collapses three different confidence claims and they read identically:
|
|
151
|
+
|
|
152
|
+
| Label | Means | Reader should conclude |
|
|
153
|
+
|---|---|---|
|
|
154
|
+
| `NO-APLICA` | the project has no such surface | nothing missing; this describes the project |
|
|
155
|
+
| `HERRAMIENTA-AUSENTE` | the surface exists, the tool was missing, **nothing ran** | that failure class was not looked for at all |
|
|
156
|
+
| `SUSTITUIDA` | something else ran instead | say what the substitute **cannot** detect; never write this as a pass |
|
|
157
|
+
|
|
158
|
+
`SUSTITUIDA` is the dangerous one. "Ran the suite twice" is not *suite health: stable* — it is a substitute that cannot see whole-suite order dependence, and a reader who can't tell the two apart reads "found nothing" where the truth is "did not look with that instrument".
|
|
159
|
+
|
|
160
|
+
**A dismissed finding carries evidence, one line each.** A fix is self-evidencing — the test now passes. A dismissal is not: "not a real problem" is indistinguishable from "did not check". Name the command, the `file:line`, or the test that disproves it, or write "ninguno".
|
|
161
|
+
|
|
128
162
|
Append-only spirit: don't overwrite a prior run's record; a new run is a new dated file. The record is the receipt the `review` and `ship` phases trust.
|
|
129
163
|
|
|
130
164
|
### 5 — VERDICT
|
|
@@ -146,6 +180,9 @@ Append-only spirit: don't overwrite a prior run's record; a new run is a new dat
|
|
|
146
180
|
| "A test is failing — let me just fix it real quick." | That's `debug`, a different discipline. `verify` reports the failure and hands off; it does not patch mid-gate. |
|
|
147
181
|
| "I'll write the verdict, then run the checks to confirm." | Backwards. RUN and WALK first; the verdict is the *consequence* of output you already read. |
|
|
148
182
|
| "The gate is mostly green, I'll call it done." | If you cannot point at the line of command output that proves a claim, you have no claim. Go run it. |
|
|
183
|
+
| "I looked at that finding; it's not a real problem." | A dismissal with no evidence is indistinguishable from a check you never made. Name the command, `file:line`, or test that disproves it — or fix it. |
|
|
184
|
+
| "The record is dated, that's enough to trace it." | A date does not identify code. Without `source_state`, nobody can tell whether the record describes what shipped. |
|
|
185
|
+
| "My grep gate passed, so the tree is clean." | Did you ever watch it fail? An unhandled broken scan exits like a pass. Prove it can fail before you trust its green. |
|
|
149
186
|
|
|
150
187
|
## Result envelope
|
|
151
188
|
|