@codyswann/lisa 2.313.1 → 2.314.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (82) hide show
  1. package/dist/core/upstream-evidence-manifest.d.ts.map +1 -1
  2. package/dist/core/upstream-evidence-manifest.js +15 -4
  3. package/dist/core/upstream-evidence-manifest.js.map +1 -1
  4. package/package.json +1 -1
  5. package/plugins/lisa/.claude-plugin/plugin.json +1 -1
  6. package/plugins/lisa/.codex-plugin/plugin.json +1 -1
  7. package/plugins/lisa/.codex-plugin/skills/lisa-codify-verification/SKILL.md +23 -7
  8. package/plugins/lisa/agents/test-specialist.md +3 -1
  9. package/plugins/lisa/agents/verification-specialist.md +4 -1
  10. package/plugins/lisa/rules/eager/falsifiable-checks.md +26 -0
  11. package/plugins/lisa/rules/eager/verification.md +1 -0
  12. package/plugins/lisa/rules/reference/falsifiable-checks.md +91 -0
  13. package/plugins/lisa/skills/lisa-codify-verification/SKILL.md +23 -7
  14. package/plugins/lisa-agy/agents/test-specialist.md +3 -1
  15. package/plugins/lisa-agy/agents/verification-specialist.md +4 -1
  16. package/plugins/lisa-agy/plugin.json +1 -1
  17. package/plugins/lisa-agy/skills/lisa-codify-verification/SKILL.md +23 -7
  18. package/plugins/lisa-cdk/.claude-plugin/plugin.json +1 -1
  19. package/plugins/lisa-cdk/.codex-plugin/plugin.json +1 -1
  20. package/plugins/lisa-cdk-agy/plugin.json +1 -1
  21. package/plugins/lisa-cdk-copilot/.claude-plugin/plugin.json +1 -1
  22. package/plugins/lisa-cdk-cursor/.claude-plugin/plugin.json +1 -1
  23. package/plugins/lisa-copilot/.claude-plugin/plugin.json +1 -1
  24. package/plugins/lisa-copilot/agents/test-specialist.agent.md +3 -1
  25. package/plugins/lisa-copilot/agents/verification-specialist.agent.md +4 -1
  26. package/plugins/lisa-copilot/rules/eager/falsifiable-checks.md +26 -0
  27. package/plugins/lisa-copilot/rules/eager/verification.md +1 -0
  28. package/plugins/lisa-copilot/rules/reference/falsifiable-checks.md +91 -0
  29. package/plugins/lisa-copilot/skills/lisa-codify-verification/SKILL.md +23 -7
  30. package/plugins/lisa-cursor/.claude-plugin/plugin.json +1 -1
  31. package/plugins/lisa-cursor/agents/test-specialist.md +3 -1
  32. package/plugins/lisa-cursor/agents/verification-specialist.md +4 -1
  33. package/plugins/lisa-cursor/rules/falsifiable-checks-reference.mdc +96 -0
  34. package/plugins/lisa-cursor/rules/falsifiable-checks.mdc +31 -0
  35. package/plugins/lisa-cursor/rules/verification.mdc +1 -0
  36. package/plugins/lisa-cursor/skills/lisa-codify-verification/SKILL.md +23 -7
  37. package/plugins/lisa-expo/.claude-plugin/plugin.json +1 -1
  38. package/plugins/lisa-expo/.codex-plugin/plugin.json +1 -1
  39. package/plugins/lisa-expo-agy/plugin.json +1 -1
  40. package/plugins/lisa-expo-copilot/.claude-plugin/plugin.json +1 -1
  41. package/plugins/lisa-expo-cursor/.claude-plugin/plugin.json +1 -1
  42. package/plugins/lisa-harper-fabric/.claude-plugin/plugin.json +1 -1
  43. package/plugins/lisa-harper-fabric/.codex-plugin/plugin.json +1 -1
  44. package/plugins/lisa-harper-fabric-agy/plugin.json +1 -1
  45. package/plugins/lisa-harper-fabric-copilot/.claude-plugin/plugin.json +1 -1
  46. package/plugins/lisa-harper-fabric-cursor/.claude-plugin/plugin.json +1 -1
  47. package/plugins/lisa-nestjs/.claude-plugin/plugin.json +1 -1
  48. package/plugins/lisa-nestjs/.codex-plugin/plugin.json +1 -1
  49. package/plugins/lisa-nestjs-agy/plugin.json +1 -1
  50. package/plugins/lisa-nestjs-copilot/.claude-plugin/plugin.json +1 -1
  51. package/plugins/lisa-nestjs-cursor/.claude-plugin/plugin.json +1 -1
  52. package/plugins/lisa-openclaw/.claude-plugin/plugin.json +1 -1
  53. package/plugins/lisa-openclaw/.codex-plugin/plugin.json +1 -1
  54. package/plugins/lisa-openclaw-agy/plugin.json +1 -1
  55. package/plugins/lisa-openclaw-copilot/.claude-plugin/plugin.json +1 -1
  56. package/plugins/lisa-openclaw-cursor/.claude-plugin/plugin.json +1 -1
  57. package/plugins/lisa-phaser/.claude-plugin/plugin.json +1 -1
  58. package/plugins/lisa-phaser/.codex-plugin/plugin.json +1 -1
  59. package/plugins/lisa-phaser-agy/plugin.json +1 -1
  60. package/plugins/lisa-phaser-copilot/.claude-plugin/plugin.json +1 -1
  61. package/plugins/lisa-phaser-cursor/.claude-plugin/plugin.json +1 -1
  62. package/plugins/lisa-rails/.claude-plugin/plugin.json +1 -1
  63. package/plugins/lisa-rails/.codex-plugin/plugin.json +1 -1
  64. package/plugins/lisa-rails-agy/plugin.json +1 -1
  65. package/plugins/lisa-rails-copilot/.claude-plugin/plugin.json +1 -1
  66. package/plugins/lisa-rails-cursor/.claude-plugin/plugin.json +1 -1
  67. package/plugins/lisa-typescript/.claude-plugin/plugin.json +1 -1
  68. package/plugins/lisa-typescript/.codex-plugin/plugin.json +1 -1
  69. package/plugins/lisa-typescript-agy/plugin.json +1 -1
  70. package/plugins/lisa-typescript-copilot/.claude-plugin/plugin.json +1 -1
  71. package/plugins/lisa-typescript-cursor/.claude-plugin/plugin.json +1 -1
  72. package/plugins/lisa-wiki/.claude-plugin/plugin.json +1 -1
  73. package/plugins/lisa-wiki/.codex-plugin/plugin.json +1 -1
  74. package/plugins/lisa-wiki-agy/plugin.json +1 -1
  75. package/plugins/lisa-wiki-copilot/.claude-plugin/plugin.json +1 -1
  76. package/plugins/lisa-wiki-cursor/.claude-plugin/plugin.json +1 -1
  77. package/plugins/src/base/agents/test-specialist.md +3 -1
  78. package/plugins/src/base/agents/verification-specialist.md +4 -1
  79. package/plugins/src/base/rules/eager/falsifiable-checks.md +26 -0
  80. package/plugins/src/base/rules/eager/verification.md +1 -0
  81. package/plugins/src/base/rules/reference/falsifiable-checks.md +91 -0
  82. package/plugins/src/base/skills/lisa-codify-verification/SKILL.md +23 -7
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-phaser",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "Phaser 4 game-development rules for TypeScript projects",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-rails",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "Ruby on Rails-specific hooks — RuboCop linting/formatting and ast-grep scanning on edit",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-rails",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "Ruby on Rails-specific skills and hooks for RuboCop and ast-grep scanning on edit.",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-rails",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "Ruby on Rails-specific hooks — RuboCop linting/formatting and ast-grep scanning on edit",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-rails",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "Ruby on Rails-specific hooks — RuboCop linting/formatting and ast-grep scanning on edit",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-rails",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "Ruby on Rails-specific hooks — RuboCop linting/formatting and ast-grep scanning on edit",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-typescript",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "TypeScript-specific hooks — Prettier formatting, ESLint linting, ast-grep scanning, and error-suppression blocking on edit",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-typescript",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "TypeScript-specific hooks for formatting, linting, and ast-grep scanning on edit.",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-typescript",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "TypeScript-specific hooks — Prettier formatting, ESLint linting, ast-grep scanning, and error-suppression blocking on edit",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-typescript",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "TypeScript-specific hooks — Prettier formatting, ESLint linting, ast-grep scanning, and error-suppression blocking on edit",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-typescript",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "TypeScript-specific hooks — Prettier formatting, ESLint linting, ast-grep scanning, and error-suppression blocking on edit",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-wiki",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "LLM Wiki — a distributable, git-native markdown knowledge base for Claude Code and Codex",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-wiki",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "Distributable LLM Wiki kernel — ingest, query, lint, and maintain a git-native markdown knowledge base across Claude and Codex.",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-wiki",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "LLM Wiki — a distributable, git-native markdown knowledge base for Claude Code and Codex",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-wiki",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "LLM Wiki — a distributable, git-native markdown knowledge base for Claude Code and Codex",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "lisa-wiki",
3
- "version": "2.313.1",
3
+ "version": "2.314.0",
4
4
  "description": "LLM Wiki — a distributable, git-native markdown knowledge base for Claude Code and Codex",
5
5
  "author": {
6
6
  "name": "Cody Swann"
@@ -21,6 +21,8 @@ You decide what has to be true for this change to be trusted, and design the tes
21
21
 
22
22
  Do not write tests against the implementation's shape — they pass through a rewrite that breaks behaviour, which is the opposite of the job. Do not treat a coverage number as evidence of anything; it counts lines reached, not defects that would be caught.
23
23
 
24
+ Do not hand on a test you have not watched fail. A test that cannot fail is worse than a missing one: it reports the defect as absent and ends the search. Break the behaviour, watch the assertion fail and name the right place, restore. Watch especially for the assertion that is satisfiable by the test's own fixture rather than by the artifact under test — that one passes no matter what the production code does. `.claude/rules/falsifiable-checks.md` has the four observed shapes.
25
+
24
26
  ## What you hand on
25
27
 
26
- The matrix, the edge cases with the reason each is interesting, the TDD sequence, and the commands that run it all. Where behaviour is user-visible, say which runner proves it end to end.
28
+ The matrix, the edge cases with the reason each is interesting, the TDD sequence, and the commands that run it all. Where behaviour is user-visible, say which runner proves it end to end. For each test, what break makes it fail — an assertion whose failure mode you cannot name is not yet designed.
@@ -12,7 +12,7 @@ skills:
12
12
 
13
13
  You are a verification specialist. Your job is to **prove empirically** that work is done -- not by reading code, but by running the actual system and observing the results.
14
14
 
15
- Read `.claude/rules/verification.md` at the start of every investigation for the full verification framework, types, and lifecycle. Read `.claude/rules/claim-evidence-mapping.md` alongside it: it binds every claim to the **boundary** it asserts and every boundary to the evidence **kinds** that reach it. The verdict you write is what `spec-conformance-specialist` cross-checks — record each claim's `boundary`, its `required_evidence_kinds`, its `evidence_refs`, and its `not_established` list so a boundary mismatch is catchable rather than invisible.
15
+ Read `.claude/rules/verification.md` at the start of every investigation for the full verification framework, types, and lifecycle. Read `.claude/rules/falsifiable-checks.md` alongside it: every check YOU author — probe, script, codified spec, sweep — is subject to it, and a check that has not been shown capable of failing is reported as *unvalidated*, never as passing. Read `.claude/rules/claim-evidence-mapping.md` too: it binds every claim to the **boundary** it asserts and every boundary to the evidence **kinds** that reach it. The verdict you write is what `spec-conformance-specialist` cross-checks — record each claim's `boundary`, its `required_evidence_kinds`, its `evidence_refs`, and its `not_established` list so a boundary mismatch is catchable rather than invisible.
16
16
 
17
17
  ## Core Philosophy
18
18
 
@@ -128,6 +128,9 @@ For every empirical verification that produced PASS evidence, invoke the `codify
128
128
  - Follow the verification lifecycle: confirm quality gates, classify, check tooling, fail fast, plan, execute, codify, spec conformance, loop
129
129
  - Every passing empirical verification must be codified as a regression test via `codify-verification` before declaring done (skip allowed only for PR / Documentation / Deploy / Investigate-Only)
130
130
  - Tests, typecheck, lint, and format are quality gates (prerequisites), NOT verification — never report them as verification evidence
131
+ - Falsify every check you author before reporting a clean result: break the guarded property, confirm the check fails and NAMES the right location, restore. Report what you broke alongside the result — see `.claude/rules/falsifiable-checks.md`
132
+ - A zero-hit sweep is meaningless until the detector has found known instances on a ref where the defect still exists; a passing probe needs a deliberate bite control that MUST report a problem
133
+ - State each clean result's blind spot (presence vs. value, reachability, class completeness) — a negative result describes what the check can perceive, not the code
131
134
  - Discover existing project scripts and tools before creating new ones
132
135
  - Every verification must produce observable output -- a status code, a response body, a UI state, a test result
133
136
  - Verification scripts must be runnable locally without CI/CD dependencies
@@ -0,0 +1,26 @@
1
+ # Falsifiable Checks — A Check That Cannot Fail Is Not Evidence (load-bearing)
2
+
3
+ A passing test, a clean lint run, a zero-hit sweep, a green ratchet: each is evidence **only if that check is known to be capable of failing.** Otherwise it is a checkbox, and a green suite that protects nothing is worse than no suite — it actively suppresses the search for the defect.
4
+
5
+ **Before reporting any clean, zero, or passing result from a check you authored or modified, prove it fails on known-bad input.** Break the thing being guarded, confirm the check fails *and names the right location*, then restore. That falsification is part of the deliverable, not an optional extra.
6
+
7
+ This is the instrument-validity counterpart to the `verification` rule (which proves the *software* behaves) and `empirical-inquiry` (which proves a *fact*). All three reject "it looks correct." This one rejects "the check said so."
8
+
9
+ ## The four ways a check silently measures nothing
10
+
11
+ Each has been observed in real runs; each reported success while asserting nothing:
12
+
13
+ 1. **Self-matching guard** — the check's own explanatory comment, docstring, or ticket prose contains the token it searches for, so it matches itself. It passes with the guarded field deleted.
14
+ 2. **Fixture-validated assertion** — the assertion reads the test's own fixture rather than the artifact under test. Common when the production path consumes a *raw* input the fixture supplies directly.
15
+ 3. **Stale-artifact pass** — the revert-to-verify step silently failed (generator errored, cache served old output, build skipped), so the check re-read unchanged input and "passed".
16
+ 4. **Wrong-baseline sweep** — the detector ran against already-fixed state, so its zero is uninformative. Validate detectors against a ref where the defect still exists.
17
+
18
+ ## Mandatory
19
+
20
+ - **Falsify before reporting.** No clean result is reportable until the check has been shown to fail on a deliberate break. **"Mentally reverting" does not count** — reasoning that the assertion *would* fail is precisely the step that lets a non-functional guard ship, because the author already believes it is load-bearing. Run the break.
21
+ - **Say how it was falsified.** "0 findings" alone is not a result; state what you broke and that the check caught it. A gate whose falsification is untested must be reported as *unvalidated*, not as passing.
22
+ - **Prefer structural over textual checks.** Parse the AST/structure instead of matching source text: text matching cannot distinguish a field from a comment, an alias, or a nested occurrence, and it produces false positives that mask the real ones.
23
+ - **A negative result is scoped to what the check can see.** State the blind spot. A presence check cannot see a wrong value; a per-file check cannot see a cross-file interaction; fixing one instance of a class is not fixing the class — sweep the class.
24
+ - **When revert-to-verify is unreliable** (generated artifacts, schema-validated inputs, caches), unit-test the checker directly against synthetic known-bad input instead.
25
+
26
+ Full prose, worked examples, and the reporting template: [reference/falsifiable-checks.md](../reference/falsifiable-checks.md).
@@ -10,6 +10,7 @@
10
10
 
11
11
  - **Never claim success without runtime evidence.** "The code looks correct" is not evidence.
12
12
  - **If all you did was run tests, typecheck, and lint — you have NOT verified.**
13
+ - **A check that cannot fail is not evidence either.** Every gate, probe, sweep, and codified spec you author is subject to the `falsifiable-checks` rule: break the guarded property, observe the check fail and name the location, then restore — and report that falsification with the result. "Mentally reverting" does not count, and an unfalsified gate is reported as *unvalidated*, never as passing.
13
14
  - **Browser-controller neutrality.** For UI work, control a live browser and perform the Validation Journey as a human would. An in-app Browser/Chrome tool, interactive Playwright control (MCP, API, or ad hoc script), CDP, computer use, the optional Lisa-owned Kane adapter, or an equivalent controller is acceptable. Kane requires explicit upload approval, a passing `lisa kane probe`, an allow-listed non-production environment, and mutation policy `full`; its provider failure is not a product failure. Do not block merely because one preferred backend is unavailable when another interactive controller can drive the browser. Running an automated Playwright or Maestro test alone is still a quality gate, not the initial empirical evidence; after the live journey passes, codify it in the applicable native runner(s). Kane never replaces those regression gates.
14
15
  - **Before starting implementation, state your verification plan** — how you will USE the resulting software to prove it works. A plan that only lists `test`/`typecheck`/`lint` commands is not a plan. Do not begin until confirmed.
15
16
  - **After verifying empirically, codify it as a regression test** via the `codify-verification` skill — Playwright for UI, integration test for API/DB/auth, benchmark for performance. Codification is mandatory for every verification type except PR/Documentation/Deploy and Investigate-Only spikes. For **frontend work**, codification is dual-runner: a Playwright spec in the project's Playwright test runner AND a Maestro flow in the Maestro test runner whenever the project supports Maestro (`.maestro/` directory, `maestro:test` script, or Maestro CI workflow) — both encoding the same verified journey, neither a substitute for the other.
@@ -0,0 +1,91 @@
1
+ # Falsifiable Checks — Reference
2
+
3
+ Eager head: [eager/falsifiable-checks.md](../eager/falsifiable-checks.md).
4
+
5
+ ## Why this rule exists
6
+
7
+ The `verification` rule prevents the failure "I claimed it works without using it." This rule prevents a subtler one, one layer down: **I used a check, the check said clean, and the check was incapable of saying anything else.**
8
+
9
+ It is the more dangerous failure of the two, because the first leaves you uncertain while the second leaves you *confidently wrong* — and it terminates the investigation. A guard that cannot fail does not merely omit protection; it manufactures evidence that the defect is absent, so nobody looks again.
10
+
11
+ The empirical origin: a single defect-sweep run produced four false-passing checks and zero false fixes. Every fix was correct; every one of the four *instruments* was broken. Reviewers then found three real defects the checks had cleared. The error concentrated entirely in verification, which is why the countermeasure belongs at the check level rather than the code level.
12
+
13
+ ## The four failure modes, in detail
14
+
15
+ ### 1. Self-matching guard
16
+
17
+ A guard searches source text for the token it protects. The fix that satisfies the guard carries a comment explaining *why* that token is required — and the comment contains the token. The guard matches its own justification.
18
+
19
+ Observed: a guard asserting every `bioData` GraphQL selection includes `ggPlayerKey` passed with `ggPlayerKey` **deleted**, because the explanatory comment above the selection said the word. It was caught only by reverting the fix.
20
+
21
+ Countermeasures, in order of preference:
22
+ - Parse structurally (comments never enter an AST).
23
+ - Failing that, strip comments before matching — but note this is itself error-prone: naive `#`-to-end-of-line stripping also destroys `#` inside string literals.
24
+
25
+ ### 2. Fixture-validated assertion
26
+
27
+ The test writes a fixture through the production path and asserts on the result — but the production path reads the *fixture* for the property under test, not the artifact the test claims to be guarding.
28
+
29
+ Observed: a cache-normalization test asserted that two entities keyed apart. It passed with the key field deleted from the query, because the cache computes its key from the **raw incoming object** (which the fixture supplied complete) rather than from the query's selection set. The test validated its own fixture.
30
+
31
+ Countermeasures:
32
+ - Add an explicit assertion binding the test to the real artifact (the document, the config, the container's actual output).
33
+ - Prefer an existing source-bound test when one exists — often a config/props snapshot already reads the real object.
34
+ - Say so in the file: label a fixture as a fixture, so the next reader does not mistake it for the binding.
35
+
36
+ ### 3. Stale-artifact pass
37
+
38
+ The revert-to-verify step appears to run but silently does not change the input the check reads.
39
+
40
+ Observed: proving a guard bites by mutating a `.graphql` document and re-running codegen produced a **false pass** — the mutation made the document invalid against the schema, so codegen errored, left the previously generated file in place, and the test re-read unchanged input. The falsification attempt itself was the thing that failed.
41
+
42
+ Countermeasures:
43
+ - Assert the input actually changed (diff the generated artifact, check the generator's exit code, confirm the digest moved).
44
+ - When the artifact is generated, schema-validated, or cached, do not rely on revert-to-verify at all — unit-test the checker against synthetic known-bad input you construct in-process.
45
+
46
+ ### 4. Wrong-baseline sweep
47
+
48
+ A detector reports zero hits, but ran against state where the defect was already fixed — so zero carries no information.
49
+
50
+ Observed: an uncalled-method detector returned 0 hits across 4,656 files, run on the branch where both instances were already fixed. Re-run against the pre-fix ref it found exactly the 2 known instances, which is what made the zero meaningful.
51
+
52
+ Countermeasure: every detector reporting a zero must first be shown to find known instances on a ref where they exist (`origin/<base>`, the pre-fix commit, or a synthetic fixture).
53
+
54
+ ## What a negative result is scoped to
55
+
56
+ A clean result is a statement about what the check can perceive, not about the code. State the boundary:
57
+
58
+ - **Presence vs. value.** A check that a field exists cannot see that its value is wrong. A field whose value is a stringified function is *present*. Real instance: `new Date().toISOString` (missing call parens) passed a presence-diff, produced a byte-identical constant id on every call, and collided every optimistic cache entry.
59
+ - **Reachability.** A textual match is a candidate, not a defect. Candidates die on: dead code (nothing references the fragment/function), configuration that bypasses the mechanism (`fetchPolicy: "no-cache"` never normalizes), and upstream guards (a button that disables itself makes a state-based re-entry guard redundant). In one sweep, 14 candidates reduced to 3 real and then to 1 user-facing.
60
+ - **Class completeness.** Fixing one instance of a defect class is not fixing the class. Real instance: a cache-id collision was fixed in one file while an instance of the *same class* sat two lines from an active edit in another; a reviewer caught it. After identifying a class, sweep for it — and prefer a repo-wide guard over a local fix so the class cannot regrow.
61
+
62
+ ## How to apply
63
+
64
+ 1. Author the check.
65
+ 2. Deliberately break the guarded property.
66
+ 3. Confirm the check **fails and names the right file/line**. A failure that does not localize is weak evidence the check is measuring the right thing.
67
+ 4. Restore, and confirm green again.
68
+ 5. Report the falsification alongside the result.
69
+
70
+ For generated or validated inputs, replace steps 2–4 with a direct unit test of the checker against synthetic bad input.
71
+
72
+ **A check whose failure has never been observed is reported as `unvalidated`, not as passing.** "Mentally reverting" does not satisfy step 3 — the author of a guard already believes it is load-bearing, so reasoning about the failure reproduces the belief rather than testing it. `unvalidated` is a legitimate state to report and land; silently presenting an unfalsified gate as a passing one is not.
73
+
74
+ ## Reporting template
75
+
76
+ > `<check name>`: <result>. Falsified by <the deliberate break>, which failed as
77
+ > `<observed failure, ideally the located name>`. Blind spots: <what this check
78
+ > cannot see>.
79
+
80
+ Concretely:
81
+
82
+ > Repo-wide keyFields guard: 0 violations across 27 documents. Falsified by moving
83
+ > `Username` one level deeper into `Attributes`; the guard failed naming
84
+ > `activity-feeds/operations.graphql:28`. Blind spot: matches by field name, not
85
+ > resolved schema type.
86
+
87
+ ## Interaction with other rules
88
+
89
+ - **`verification`** — proves the software behaves as a user needs. This rule proves the proof is real. Codified regression tests added under `codify-verification` are subject to this rule: a codified spec that cannot fail is not a regression gate.
90
+ - **`empirical-inquiry`** — settles an uncertain fact with the cheapest probe. A probe is a check, so it inherits the falsification requirement, including deliberate bite controls (cases that MUST report a problem) when the probe's job is to detect problems.
91
+ - **`claim-evidence-mapping`** — an unfalsified gate cannot back a claim.
@@ -135,9 +135,22 @@ Run only the new test, using whatever per-test invocation the project supports:
135
135
 
136
136
  Confirm:
137
137
  1. The test PASSES against the current code (the change being shipped)
138
- 2. The test would have FAILED before the change (sanity check by mentally reverting, or for bug fixes, by running against the pre-fix commit if cheap)
138
+ 2. The test ACTUALLY FAILS without the change observed, not reasoned about
139
139
 
140
- For a bug fix, step 2 is mandatory and easy: check out the failing commit, run the new test, see it fail, return to the fix branch. This proves the test actually guards the regression.
140
+ **Step 2 is mandatory for every codified test, and "mentally reverting" does not satisfy it.** Mental reversion is the exact mechanism by which non-functional guards ship: the author believes the assertion is load-bearing, and it is not. Break the guarded property for real, run the test, and read the failure. See `.claude/rules/falsifiable-checks.md` for the four observed ways a check passes while asserting nothing.
141
+
142
+ Do it one of these ways, in order of preference:
143
+
144
+ - **Run against the pre-fix commit** (bug fixes): check out the failing commit, run the new test, see it fail, return to the fix branch.
145
+ - **Break the property in place**: delete the field, revert the line, flip the condition; run; restore. Prefer this when there is no single pre-fix commit.
146
+ - **Unit-test the checker against synthetic bad input**: required when the input is generated, schema-validated, or cached — a revert can silently fail to change what the test reads (a generator that errors leaves the previous artifact in place, and the test then "passes" on stale input).
147
+
148
+ Two properties the failure itself must have:
149
+
150
+ - It must **name the right location**. A failure that does not localize is weak evidence the test is measuring the intended thing.
151
+ - It must not be satisfiable by the test's own fixture. If the assertion can be met by data the test supplies rather than by the artifact under test, add an explicit assertion against the real artifact (the document, the config, the component's actual output) — or reuse an existing source-bound test.
152
+
153
+ Record the falsification in the codification report (what you broke, how it failed). A codified test whose failure has not been observed is reported as **unvalidated**, not as a regression gate.
141
154
 
142
155
  ### 5. Wire it into the suite
143
156
 
@@ -162,13 +175,13 @@ Append to the verification report (or PR description):
162
175
  ```markdown
163
176
  ### Codified Verifications
164
177
 
165
- | # | Verification | Framework | Test file | Status |
166
- |---|--------------|-----------|-----------|--------|
167
- | 1 | <description> | Playwright | `e2e/checkout.spec.ts::displays order confirmation after checkout` | PASS |
168
- | 2 | <same journey, native surface> | Maestro | `.maestro/flows/checkout-confirmation.yaml` | PASS |
178
+ | # | Verification | Framework | Test file | Status | Falsified by |
179
+ |---|--------------|-----------|-----------|--------|--------------|
180
+ | 1 | <description> | Playwright | `e2e/checkout.spec.ts::displays order confirmation after checkout` | PASS | removed the confirmation render → failed at `checkout.spec.ts:42` |
181
+ | 2 | <same journey, native surface> | Maestro | `.maestro/flows/checkout-confirmation.yaml` | PASS | same break → flow failed on the confirmation assertion |
169
182
  ```
170
183
 
171
- This evidence shows the verification is now guarded.
184
+ This evidence shows the verification is now guarded. **The `Falsified by` column is required** — it names the deliberate break and the observed failure. `UNVALIDATED` is the only permitted alternative, and it means the test is not yet a regression gate.
172
185
 
173
186
  ## Output
174
187
 
@@ -183,6 +196,9 @@ If codification was skipped, an explicit reason recorded in the report (one of t
183
196
  ## Rules
184
197
 
185
198
  - Never claim a verification is codified without running the new test and observing it pass
199
+ - Never claim it is codified without observing it **FAIL** on a real break — mental reversion is not observation, and a test whose failure was never seen is unvalidated, not a gate (`.claude/rules/falsifiable-checks.md`)
200
+ - Never let the assertion be satisfiable by the test's own fixture instead of the artifact under test — bind it to the real document/config/output
201
+ - Never trust a revert-to-verify on generated, schema-validated, or cached input without confirming the input actually changed; a failed generator silently leaves the old artifact and the test "passes" on stale bytes
186
202
  - Never disable, skip, or `.skip()` the new test "temporarily" to make CI green — fix the test or fix the underlying change
187
203
  - Never use `expect(true).toBe(true)` placeholders or smoke-only assertions that don't actually exercise the verified behavior
188
204
  - Never reuse the verification's manual artifact (screenshot, curl output) as a "test" — those are evidence, not regression coverage