@tamsensedev/runner 0.0.0-stage → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (79) hide show
  1. package/LICENSE +5 -0
  2. package/README.md +45 -2
  3. package/dist/cli.js +2234 -0
  4. package/dist/cli.js.map +1 -0
  5. package/library/.claude-plugin/plugin.json +5 -0
  6. package/library/THIRD_PARTY_NOTICES.md +123 -0
  7. package/library/agents/code-architect.md +67 -0
  8. package/library/agents/code-explorer.md +61 -0
  9. package/library/agents/code-reviewer.md +73 -0
  10. package/library/agents/code-simplifier.md +62 -0
  11. package/library/agents/design-critic.md +52 -0
  12. package/library/agents/second-opinion.md +64 -0
  13. package/library/agents/security-auditor.md +86 -0
  14. package/library/agents/silent-failure-hunter.md +70 -0
  15. package/library/agents/test-coverage-analyzer.md +78 -0
  16. package/library/agents/test-runner.md +48 -0
  17. package/library/agents/ui-tester.md +47 -0
  18. package/library/licenses/Apache-2.0.txt +202 -0
  19. package/library/skills/accessibility/SKILL.md +144 -0
  20. package/library/skills/accessibility/references/wcag-checks.md +168 -0
  21. package/library/skills/addressing-review-feedback/SKILL.md +109 -0
  22. package/library/skills/app-screens/SKILL.md +96 -0
  23. package/library/skills/app-screens/references/shell-kpis-charts.md +108 -0
  24. package/library/skills/app-screens/references/states-density-phone.md +54 -0
  25. package/library/skills/app-screens/references/tables-forms-settings.md +104 -0
  26. package/library/skills/avoiding-generic-ai-design/SKILL.md +92 -0
  27. package/library/skills/avoiding-generic-ai-design/references/tells.md +112 -0
  28. package/library/skills/clean-commits/SKILL.md +85 -0
  29. package/library/skills/design-review/SKILL.md +107 -0
  30. package/library/skills/design-review/references/critique-passes.md +101 -0
  31. package/library/skills/design-review/references/screenshot-procedure.md +113 -0
  32. package/library/skills/differential-security-review/LICENSE +427 -0
  33. package/library/skills/differential-security-review/SKILL.md +105 -0
  34. package/library/skills/differential-security-review/references/adversarial.md +115 -0
  35. package/library/skills/differential-security-review/references/methodology.md +145 -0
  36. package/library/skills/differential-security-review/references/patterns.md +108 -0
  37. package/library/skills/false-positive-check/LICENSE +427 -0
  38. package/library/skills/false-positive-check/SKILL.md +95 -0
  39. package/library/skills/false-positive-check/references/bug-class-verification.md +128 -0
  40. package/library/skills/false-positive-check/references/deep-verification.md +113 -0
  41. package/library/skills/false-positive-check/references/evidence-templates.md +93 -0
  42. package/library/skills/false-positive-check/references/false-positive-patterns.md +122 -0
  43. package/library/skills/false-positive-check/references/gate-reviews.md +36 -0
  44. package/library/skills/false-positive-check/references/standard-verification.md +80 -0
  45. package/library/skills/focused-implementation/SKILL.md +104 -0
  46. package/library/skills/implementation-plan/SKILL.md +66 -0
  47. package/library/skills/interface-design/SKILL.md +144 -0
  48. package/library/skills/interface-design/references/craft-defaults.md +130 -0
  49. package/library/skills/interface-design/references/new-direction.md +62 -0
  50. package/library/skills/interface-design/references/writing.md +52 -0
  51. package/library/skills/interface-polish/SKILL.md +103 -0
  52. package/library/skills/interface-polish/references/detail-rules.md +154 -0
  53. package/library/skills/interface-polish/references/worst-case-data.md +158 -0
  54. package/library/skills/motion-design/SKILL.md +106 -0
  55. package/library/skills/motion-design/references/recipes.md +243 -0
  56. package/library/skills/postgres-performance/SKILL.md +97 -0
  57. package/library/skills/postgres-performance/references/rules.md +265 -0
  58. package/library/skills/project-orientation/SKILL.md +77 -0
  59. package/library/skills/reviewing-changes/SKILL.md +183 -0
  60. package/library/skills/safe-database-changes/SKILL.md +111 -0
  61. package/library/skills/safe-database-changes/references/backfills.md +75 -0
  62. package/library/skills/safe-database-changes/references/migration-tools.md +69 -0
  63. package/library/skills/safe-database-changes/references/postgres-ddl.md +199 -0
  64. package/library/skills/security-review/SKILL.md +132 -0
  65. package/library/skills/security-review/references/checks-by-area.md +107 -0
  66. package/library/skills/security-review/references/do-not-report.md +86 -0
  67. package/library/skills/spec-writing/SKILL.md +144 -0
  68. package/library/skills/systematic-debugging/SKILL.md +130 -0
  69. package/library/skills/systematic-debugging/references/flaky-tests.md +51 -0
  70. package/library/skills/systematic-debugging/references/root-cause-tracing.md +55 -0
  71. package/library/skills/task-interview/SKILL.md +84 -0
  72. package/library/skills/test-driven-development/SKILL.md +116 -0
  73. package/library/skills/test-driven-development/references/writing-good-tests.md +85 -0
  74. package/library/skills/ui-testing/SKILL.md +99 -0
  75. package/library/skills/ui-testing/references/headless-check.md +262 -0
  76. package/library/skills/ui-testing/references/playwright-patterns.md +134 -0
  77. package/library/skills/verification-before-completion/SKILL.md +113 -0
  78. package/library/skills/web-interface-guidelines/SKILL.md +202 -0
  79. package/package.json +44 -5
@@ -0,0 +1,113 @@
1
+ # Deep Verification
2
+
3
+ Full verification for complex bugs. Use when routing in SKILL.md selects the deep path, or when standard verification escalates.
4
+
5
+ ## If Escalated from Standard
6
+
7
+ 1. Review all evidence gathered during the standard pass; do not repeat completed work.
8
+ 2. Identify which phases below are already satisfied by existing evidence.
9
+ 3. Continue from where standard verification left off.
10
+ 4. Keep and reference all prior findings.
11
+
12
+ ## Phases
13
+
14
+ Work through the phases in order for each bug (Bug #N). Keep a checklist in your notes and mark a phase done only when you have concrete evidence for it. Sub-steps within a phase are independent; do them in any order, but never start a phase before the one it depends on is complete. If the bug path is large, you may hand a phase's data-flow tracing to the `code-explorer` agent, with the bug description and prior evidence in the prompt; you still judge the result.
15
+
16
+ ```
17
+ Phase 1: Data flow analysis
18
+ 1.1 Map trust boundaries and trace data flow
19
+ 1.2 Research API contracts and safety guarantees
20
+ 1.3 Environment protection analysis
21
+ 1.4 Cross-reference analysis
22
+ Phase 2: Exploitability verification (after phase 1)
23
+ 2.1 Confirm attacker controls input data
24
+ 2.2 Mathematical bounds verification (or N/A)
25
+ 2.3 Race condition feasibility proof (or N/A)
26
+ 2.4 Adversarial analysis (after 2.1 to 2.3)
27
+ Phase 3: Impact assessment (after phase 2)
28
+ 3.1 Demonstrate real security impact
29
+ 3.2 Primary control vs defense in depth
30
+ Phase 4: PoC creation (after phase 3)
31
+ 4.1 Pseudocode PoC with data flow diagram
32
+ 4.2 Executable PoC if feasible
33
+ 4.3 Unit test PoC if feasible
34
+ 4.4 Negative PoC: exploit preconditions
35
+ 4.5 Verify the PoC demonstrates the vulnerability
36
+ Phase 5: Devil's advocate review (after phase 4)
37
+ Gate review (after phase 5)
38
+ ```
39
+
40
+ Apply all 13 checklist items from [false-positive-patterns.md](false-positive-patterns.md) to each bug.
41
+
42
+ ## Phase Requirements
43
+
44
+ The list above names every phase. Below are the key pitfalls and decision criteria for each: focus on what you might get wrong.
45
+
46
+ ### Phase 1: Data Flow Analysis
47
+
48
+ **1.1**: Map trust boundaries (internal/trusted vs external/untrusted) and trace data from source to the alleged vulnerability. Apply class-specific verification from [bug-class-verification.md](bug-class-verification.md). **Key pitfall**: analyzing code in isolation without tracing the full validation chain. Conditional logic upstream may make the vulnerable code mathematically unreachable (see [false-positive-patterns.md](false-positive-patterns.md) items 1 and 1a).
49
+
50
+ **1.2**: Check API contracts before claiming overflows or injections: many APIs have built-in protection (bounds checks, parameter binding, escaping) that prevents the alleged issue regardless of inputs.
51
+
52
+ **1.3**: Before concluding vulnerability, verify that no compiler, runtime, OS, or framework protection prevents exploitation. Mitigations like ASLR and stack canaries raise the bar but do not eliminate the vulnerability itself. Distinguish "prevents exploitation entirely" (Rust's safe type system, parameterized queries, framework auto-escaping) from "makes exploitation harder" (ASLR, a WAF).
53
+
54
+ **1.4**: Check whether similar code patterns exist elsewhere and are handled safely. Review test coverage, code review history (`git log -p`), and design documentation for this area.
55
+
56
+ ### Phase 2: Exploitability Verification
57
+
58
+ **2.1**: Prove the attacker controls the data reaching the vulnerability. **Key pitfall**: assuming external data reaches the operation without tracing the actual path. Internal storage set by trusted components is not attacker-controlled.
59
+
60
+ **2.2**: Create explicit algebraic proofs for bounds-related issues. Use the template in [evidence-templates.md](evidence-templates.md). Verify: IF validation_check_passes THEN bounds_guarantee_holds. Mark N/A when the bug is not about bounds or arithmetic.
61
+
62
+ **2.3**: For race conditions, prove concurrent access is actually possible (two requests, two workers, two processes). **Key pitfall**: assuming races in single-threaded initialization or in code already serialized by a lock, transaction or unique constraint. Mark N/A when no concurrency is involved.
63
+
64
+ **2.4**: Assess the full attack surface: input control, validation bypass paths, timing dependencies, and state manipulation.
65
+
66
+ ### Phase 3: Impact Assessment
67
+
68
+ **3.1**: Distinguish real security impact (code execution, privilege escalation, information disclosure, unauthorized data changes, money loss) from operational robustness issues.
69
+
70
+ **3.2**: Distinguish primary security controls from defense in depth. Failure of a defense-in-depth measure is not a vulnerability if primary protections remain intact.
71
+
72
+ ### Phase 4: PoC Creation
73
+
74
+ **Always create a pseudocode PoC.** Create executable and/or unit test PoCs when feasible and allowed:
75
+
76
+ 1. **Pseudocode with data flow diagram** showing the attack path (always).
77
+ 2. **Executable PoC** in the target language (if feasible). In review and test steps it runs only from a temp directory outside the repository, never against live services or production data, and is deleted afterwards.
78
+ 3. **Unit test PoC** exercising the vulnerable path with crafted inputs (if feasible). Same rule: never add it to the repository under review. In a code or bugfix step it may become the regression test.
79
+
80
+ See [evidence-templates.md](evidence-templates.md) for templates.
81
+
82
+ **Negative PoC (4.4)**: show the gap between normal operation and the exploit path: which preconditions must hold for the vulnerability to trigger, and why they do not hold under normal conditions.
83
+
84
+ ### Phase 5: Devil's Advocate Review
85
+
86
+ Before the verdict, challenge the claim systematically. Assume you are biased toward finding bugs and rating them as critical, and work against that bias.
87
+
88
+ **Challenges arguing AGAINST the vulnerability:**
89
+
90
+ 1. What non-vulnerability explanations exist for this code pattern?
91
+ 2. How would the original developers justify this implementation?
92
+ 3. What crucial system architecture context might be missing?
93
+ 4. Am I seeing a vulnerability because the pattern "looks dangerous" rather than because it actually is?
94
+ 5. Even if validation looks insufficient, does it actually prevent the claimed condition?
95
+ 6. Am I incorrectly assuming attacker control over trusted data?
96
+ 7. Have I rigorously proven the mathematical condition for vulnerability can occur?
97
+ 8. Beyond theoretical possibility, is this practically exploitable?
98
+ 9. Am I confusing defense-in-depth failure with a primary security vulnerability?
99
+ 10. What compiler, runtime, OS or framework protections might prevent exploitation?
100
+ 11. Am I hallucinating this vulnerability? LLMs are biased toward seeing bugs everywhere and rating every finding as critical. Is this a real, exploitable issue, or am I pattern-matching on scary-looking code?
101
+
102
+ **Challenges arguing FOR the vulnerability (false-negative protection):**
103
+
104
+ 12. Am I dismissing a real vulnerability because the exploit seems complex or unlikely?
105
+ 13. Am I inventing mitigations or validation logic that I have not verified in the actual source code? Re-read the code after reaching a conclusion.
106
+
107
+ See [evidence-templates.md](evidence-templates.md) for the documentation template.
108
+
109
+ ## Gate Review
110
+
111
+ Apply the six gates from [gate-reviews.md](gate-reviews.md) to reach a verdict.
112
+
113
+ Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: task tracking and plugin agents replaced by an in-order checklist (optionally the `code-explorer` agent), read-only rules for proofs of concept, N/A rules for 2.2 and 2.3, web examples of environment protections and impact.
@@ -0,0 +1,93 @@
1
+ # Evidence Templates
2
+
3
+ Use these templates when documenting verification evidence for each bug.
4
+
5
+ ## Data Flow Documentation
6
+
7
+ ```
8
+ Bug #N Data Flow Analysis
9
+ Source: [exact location] - Trust Level: [trusted/untrusted]
10
+ Path: Source → Validation1[file:line] → Transform[file:line] → Vulnerability[file:line]
11
+ Validation Points:
12
+ - Check1: [condition] at [file:line] - [passes/fails/bypassed]
13
+ - Check2: [condition] at [file:line] - [passes/fails/bypassed]
14
+ ```
15
+
16
+ ## Mathematical Bounds Proof
17
+
18
+ ```
19
+ Bug #N Mathematical Analysis
20
+ Claim: Operation X is vulnerable to [overflow/underflow/bounds violation]
21
+ Given Constraints: [list all validation conditions]
22
+
23
+ Algebraic Proof:
24
+ 1. [first constraint from validation]
25
+ 2. [constant or known value]
26
+ 3. [derived inequality]
27
+ ...
28
+ N. Therefore: [vulnerability confirmed/debunked] (Q.E.D.)
29
+
30
+ Conclusion: [vulnerability is/is not mathematically possible]
31
+ ```
32
+
33
+ **Example:**
34
+
35
+ ```
36
+ Given: validation ensures (input_size >= MIN_SIZE)
37
+ Given: MIN_SIZE = 16, header_size = 8
38
+ Prove: (input_size - header_size) cannot underflow
39
+
40
+ 1. input_size >= MIN_SIZE (from validation)
41
+ 2. MIN_SIZE = 16 (constant)
42
+ 3. header_size = 8 (constant)
43
+ 4. input_size >= 16 (substitution of 1,2)
44
+ 5. input_size - 8 >= 16 - 8 (subtract header_size from both sides)
45
+ 6. input_size - header_size >= 8 (simplification)
46
+ 7. Therefore: underflow impossible (Q.E.D.)
47
+ ```
48
+
49
+ ## Attacker Control Analysis
50
+
51
+ ```
52
+ Bug #N Attacker Control Analysis
53
+ Input Vector: [how attacker provides input]
54
+ Control Level: [full/partial/none]
55
+ Constraints: [what limits exist on attacker input]
56
+ Reachability: [can attacker-controlled data reach vulnerable operation?]
57
+ ```
58
+
59
+ ## PoC - Pseudocode with Data Flow Diagram
60
+
61
+ ```
62
+ PoC for Bug #N: [Brief Description]
63
+
64
+ Data Flow Diagram:
65
+
66
+ [External Input] → [Validation Point] → [Processing] → [Vulnerable Operation]
67
+ | | | |
68
+ Attacker (May be bypassed) (Transforms data) (Unsafe operation)
69
+ Controlled | | |
70
+ | v v v
71
+ [Malicious Data] → [Insufficient Check] → [Processed Data] → [Impact]
72
+
73
+ PSEUDOCODE:
74
+ function vulnerable_operation(user_data):
75
+ validation_result = weak_validation(user_data) // Explain why this fails
76
+ processed_data = transform_data(user_data) // Show transformation
77
+ unsafe_operation(processed_data) // Show vulnerability trigger
78
+ ```
79
+
80
+ ## Devil's Advocate Review
81
+
82
+ ```
83
+ Bug #N Devil's Advocate Review
84
+ Vulnerability Claim: [brief description]
85
+
86
+ For each devil's advocate question (7 in standard verification, 13 in deep verification), document your answer:
87
+ Questions against the vulnerability: [answers]
88
+ Questions for the vulnerability (false-negative protection): [answers]
89
+
90
+ Final Assessment: [Vulnerability confirmed/debunked with reasoning]
91
+ ```
92
+
93
+ Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: punctuation and small wording changes for read-only runs.
@@ -0,0 +1,122 @@
1
+ # False Positive Patterns: Lessons Learned
2
+
3
+ Apply ALL items in this checklist to EACH potential bug during verification.
4
+
5
+ Contents:
6
+
7
+ - [Checklist](#checklist): 13 items to apply to every bug (trace validation, confirm data paths, bounds logic, TOCTOU, trust boundaries, real vs theoretical impact, defense-in-depth)
8
+ - [Red Flags for False Positives](#red-flags-for-false-positives): recurring failure patterns (pattern-based, context-blind, math/bounds, API-contract)
9
+
10
+ ## Checklist
11
+
12
+ ### 1. Trace Full Validation Chain
13
+
14
+ Don't analyze isolated code snippets. Trace backwards to find ALL validation that precedes potentially dangerous operations. Network packet size operations may look dangerous but often have bounds validation earlier in the function.
15
+
16
+ ### 1a. Map Complete Conditional Logic Flow
17
+
18
+ Vulnerable-looking code may be unreachable due to conditional logic that creates mathematical guarantees. Example: array access `buffer[length-4]` appears unsafe when `length < 4`, but if the code is only reachable when `length > 12` due to earlier validation, the vulnerability is impossible.
19
+
20
+ **Verify:**
21
+
22
+ - What conditions must be met for execution to reach the alleged vulnerability?
23
+ - Do those conditions mathematically prevent the vulnerability scenario?
24
+ - Are there minimum size/length requirements that guarantee safe access?
25
+ - Does the conditional flow create impossible-to-violate bounds?
26
+
27
+ ### 2. Identify Defensive Programming Patterns
28
+
29
+ Distinguish between actual vulnerabilities and defensive assertions/validations. `ASSERT(size == expected_size)` followed by size-controlled operations is defensive, not vulnerable. Verify that checks actually prevent the alleged vulnerability.
30
+
31
+ ### 3. Confirm Exploitable Data Paths
32
+
33
+ Only report vulnerabilities with CONFIRMED exploitable data flow paths. Don't assume network-controlled data reaches dangerous functions without tracing the actual path step by step.
34
+
35
+ ### 4. Understand Data Source Context
36
+
37
+ Distinguish between data sources and their trust levels. API return values, compile-time constants, and network data have different risk profiles. Determine the actual source and whether it is attacker-controlled.
38
+
39
+ ### 5. Analyze Bounds Validation Logic
40
+
41
+ Look for mathematical relationships between validation checks and subsequent operations. If `packet_size >= MIN_SIZE` is checked and `MIN_SIZE >= sizeof(header)`, then `packet_size - sizeof(header)` cannot underflow.
42
+
43
+ ### 6. Verify TOCTOU Claims
44
+
45
+ Time-of-check-time-of-use issues require proof that the checked value can change between check and use. If a size is checked and immediately used in the same function with no external modification possible, there is no TOCTOU.
46
+
47
+ ### 7. Understand API Contract and Trust Boundaries
48
+
49
+ Always understand API contracts before claiming buffer overflows. Some APIs have built-in bounds protection and cannot write beyond the buffer regardless of input parameters.
50
+
51
+ ### 8. Distinguish Internal Storage from External Input
52
+
53
+ Internal storage systems (configuration stores, registries) are controlled by trusted components, not attackers. Values set during installation by trusted components are not attacker-controlled.
54
+
55
+ ### 9. Don't Confuse Pattern Recognition with Vulnerability Analysis
56
+
57
+ Code patterns that "look vulnerable" may be safely implemented due to context and API contracts. Size parameters being modified doesn't mean buffer overflow if the API prevents writing beyond bounds.
58
+
59
+ ### 10. Verify Concurrent Access is Actually Possible
60
+
61
+ Don't assume race conditions exist without proving concurrent access patterns. Single-threaded initialization contexts cannot have race conditions. Verify the threading model and synchronization mechanisms.
62
+
63
+ ### 11. Assess Real vs Theoretical Security Impact
64
+
65
+ Focus on vulnerabilities with actual security impact. Storage failure for non-critical data is an operational issue, not a security vulnerability. Ask: would this lead to code execution, privilege escalation, or information disclosure?
66
+
67
+ ### 12. Understand Defense-in-Depth vs Primary Controls
68
+
69
+ Failure of defense-in-depth mechanisms is not always a vulnerability if primary protections exist. Token cleanup failure is not critical if tokens are single-use by design at the server.
70
+
71
+ ### 13. Apply the Checklist Rigorously, Not Superficially
72
+
73
+ Having a checklist doesn't prevent false positives if it isn't applied systematically. For EVERY potential vulnerability, work through ALL checklist items before concluding.
74
+
75
+ ---
76
+
77
+ ## Red Flags for False Positives
78
+
79
+ ### Pattern-Based False Positives
80
+
81
+ - Reporting vulnerabilities in validation/bounds-checking code itself
82
+ - Claiming TOCTOU without proving the value can change
83
+ - Ignoring preceding validation logic
84
+ - Assuming network data reaches operations without tracing the path
85
+ - Confusing defensive programming (assertions/checks) with vulnerabilities
86
+ - Analyzing vulnerable-looking patterns without tracing conditional logic that controls reachability
87
+ - Reporting "vulnerabilities" in error handling or cleanup code
88
+ - Flagging size calculations without understanding mathematical constraints
89
+ - Identifying "dangerous" functions without checking if inputs are bounded
90
+ - Claiming buffer overflows in fixed-size operations with compile-time bounds
91
+ - Reporting race conditions in single-threaded or synchronized contexts
92
+
93
+ ### Context-Blind Analysis False Positives
94
+
95
+ - Analyzing code snippets without understanding broader system design
96
+ - Ignoring architectural guarantees (single-writer, trusted input sources)
97
+ - Missing that "vulnerable" code is unreachable due to earlier validation
98
+ - Confusing debug/development code paths with production paths
99
+ - Reporting issues in code that only runs during trusted installation/setup
100
+ - Flagging theoretical issues that cannot occur due to system architecture
101
+ - Missing that alleged vulnerabilities are prevented by framework or language guarantees
102
+ - Reporting issues in test-only or debug-only code paths as production vulnerabilities
103
+
104
+ ### Mathematical/Bounds Analysis False Positives
105
+
106
+ - Reporting integer underflow without proving the mathematical condition can occur
107
+ - Claiming buffer overflow when bounds are mathematically guaranteed by validation
108
+ - Missing that conditional logic creates mathematical impossibility of vulnerable conditions
109
+ - Reporting off-by-one errors without checking if loop bounds prevent the condition
110
+ - Claiming memory corruption when allocation sizes are verified sufficient
111
+ - Reporting arithmetic overflow without checking if input ranges prevent the condition
112
+
113
+ ### API Contract Misunderstanding False Positives
114
+
115
+ - Claiming buffer overflows when APIs have built-in bounds checking
116
+ - Reporting memory corruption for APIs that manage their own memory safely
117
+ - Missing that return values are already validated by the API contract
118
+ - Confusing API parameter modification with vulnerability when API prevents unsafe modification
119
+ - Reporting issues explicitly handled by the API's safety guarantees
120
+ - Missing that seemingly dangerous operations are safe due to API implementation details
121
+
122
+ Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: punctuation and small wording changes for read-only runs.
@@ -0,0 +1,36 @@
1
+ # Gate Reviews and Verdicts
2
+
3
+ Before reporting ANY bug as a vulnerability, all six gate reviews must pass. Evaluate them at the end, after all phases are complete:
4
+
5
+ | Gate | Criterion | Pass | Fail |
6
+ |------|-----------|------|------|
7
+ | **1. Process** | All phases completed with documented evidence | Evidence exists for every phase | Phases lack concrete evidence |
8
+ | **2. Reachability** | Attacker can reach and control data at the vulnerability | Clear evidence of attacker-controlled path + PoC confirms | Cannot demonstrate attacker control or reachability |
9
+ | **3. Real Impact** | Exploitation leads to code execution, privilege escalation, information disclosure, unauthorized changes to data, or loss of money | Direct impact with concrete scenarios | Only operational robustness issue |
10
+ | **4. PoC Validation** | PoC (pseudocode, executable, or unit test) demonstrates the attack path | Shows attacker control, trigger, and impact | PoC fails to show attack path or impact |
11
+ | **5. Math Bounds** | Mathematical analysis confirms vulnerable condition is possible (N/A, counted as pass, when the bug is not about bounds or arithmetic) | Algebraic proof shows condition is possible | Math proves validation prevents it |
12
+ | **6. Environment** | No environmental protections entirely prevent exploitation | Protections do not eliminate vulnerability | Environmental protections block it entirely |
13
+
14
+ For a suspected correctness bug rather than a vulnerability: gate 2 asks whether a realistic input or state reaches the faulty code, and gate 3 whether the outcome is a wrong result, lost or corrupted data, or a crash that users hit.
15
+
16
+ ## Verdict Format
17
+
18
+ - **TRUE POSITIVE**: All gate reviews pass → `BUG #N TRUE POSITIVE - [brief vulnerability description]`
19
+ - **FALSE POSITIVE**: Any gate review fails → `BUG #N FALSE POSITIVE - [brief reason for rejection]`
20
+ - **INCONCLUSIVE**: A gate cannot be decided from the code available (the caller is in another repository, the configuration is not in the repo) → `BUG #N NEEDS VERIFICATION - [the exact open question]`. Reported as a note, never as blocking.
21
+
22
+ If any phase fails verification, document the failure with evidence and continue all remaining phases. Issue the FALSE POSITIVE verdict only after all phases are complete.
23
+
24
+ ## Example Verdict
25
+
26
+ ```
27
+ BUG #3 FALSE POSITIVE - Integer underflow in packet_handler.c:142
28
+ Gate 5 (Math Bounds) FAIL: validation at line 98 ensures packet_size >= 16,
29
+ making (packet_size - header_size) >= 8. Underflow is mathematically impossible.
30
+
31
+ BUG #5 FALSE POSITIVE - IDOR in src/api/files.ts:19
32
+ Gate 2 (Reachability) FAIL: the route is mounted under accountRouter (src/app.ts:52),
33
+ whose middleware loads the file with the session's account id; other accounts' files return 404.
34
+ ```
35
+
36
+ Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: wider impact list for web applications, N/A rule for gate 5, correctness-bug reading of gates 2 and 3, inconclusive verdict, second example.
@@ -0,0 +1,80 @@
1
+ # Standard Verification
2
+
3
+ Linear single-pass checklist for straightforward bugs. No task tracking: work through each step sequentially and document findings inline.
4
+
5
+ ## Escalation Checkpoints
6
+
7
+ Two points in this checklist may trigger escalation to [deep-verification.md](deep-verification.md):
8
+
9
+ 1. **After Step 1 (Data Flow)**: Escalate if 3+ trust boundaries, callbacks/async control flow, or ambiguous validation chain
10
+ 2. **After Step 5 (Devil's Advocate)**: Escalate if any question produces genuine uncertainty you cannot resolve
11
+
12
+ When escalating, hand off all evidence gathered so far: deep verification will continue from where you left off.
13
+
14
+ ## Checklist
15
+
16
+ ### Step 1: Data Flow
17
+
18
+ Trace data from source to the alleged vulnerability sink.
19
+
20
+ - Map trust boundaries crossed (internal/trusted vs external/untrusted)
21
+ - Identify all validation and sanitization between source and sink
22
+ - Check API contracts: many APIs have built-in bounds protection that prevents the alleged issue
23
+ - Check for environmental protections (compiler, runtime, OS, framework) that prevent exploitation entirely (not just raise the bar)
24
+ - Apply class-specific checks from [bug-class-verification.md](bug-class-verification.md)
25
+
26
+ **Key pitfall**: Analyzing the vulnerable code in isolation. Conditional logic upstream may make the vulnerability mathematically unreachable. Trace the full validation chain.
27
+
28
+ **Escalation check**: If you found 3+ trust boundaries, callbacks or async control flow in the path, or an ambiguous validation chain: escalate to deep verification.
29
+
30
+ ### Step 2: Exploitability
31
+
32
+ Prove the attacker can trigger the vulnerability.
33
+
34
+ - **Attacker control**: Prove the attacker controls data reaching the vulnerable operation. Internal storage set by trusted components is not attacker-controlled.
35
+ - **Bounds proof**: For integer/bounds issues, create an explicit algebraic proof using the template in [evidence-templates.md](evidence-templates.md). Verify: IF validation_check_passes THEN bounds_guarantee_holds.
36
+ - **Race feasibility**: For race conditions, prove concurrent access is actually possible. Single-threaded initialization and synchronized contexts cannot have races.
37
+
38
+ ### Step 3: Impact
39
+
40
+ Determine whether exploitation has real security consequences.
41
+
42
+ - Distinguish real security impact (RCE, privesc, info disclosure) from operational robustness issues (crash recovery, cleanup failure)
43
+ - Distinguish primary security controls from defense-in-depth. Failure of a defense-in-depth measure is not a vulnerability if primary protections remain intact.
44
+
45
+ ### Step 4: PoC Sketch
46
+
47
+ Create a pseudocode PoC showing the attack path. Executable and unit test PoCs are optional for standard verification; in review and test steps they run only from a temp directory outside the repository and never against live services (see the ground rules in SKILL.md).
48
+
49
+ ```
50
+ Data Flow: [Source] → [Validation?] → [Transform?] → [Vulnerable Op] → [Impact]
51
+ Attacker controls: [what input, how]
52
+ Trigger: [pseudocode showing the exploit path]
53
+ ```
54
+
55
+ See [evidence-templates.md](evidence-templates.md) for the full PoC template.
56
+
57
+ ### Step 5: Devil's Advocate Spot-Check
58
+
59
+ Answer these 7 questions. If any produces genuine uncertainty, escalate to deep verification.
60
+
61
+ **Against the vulnerability:**
62
+
63
+ 1. Am I seeing a vulnerability because the pattern "looks dangerous" rather than because it actually is? (pattern-matching bias)
64
+ 2. Am I incorrectly assuming attacker control over trusted data? (trust boundary confusion)
65
+ 3. Have I rigorously proven the mathematical condition for vulnerability can occur? (proof rigor)
66
+ 4. Am I confusing defense-in-depth failure with a primary security vulnerability? (defense-in-depth confusion)
67
+ 5. Am I hallucinating this vulnerability? LLMs are biased toward seeing bugs everywhere: is this actually real or am I pattern-matching on scary-looking code? (LLM self-check)
68
+
69
+ **For the vulnerability (always ask: false-negative protection):**
70
+
71
+ 6. Am I dismissing a real vulnerability because the exploit seems complex or unlikely?
72
+ 7. Am I inventing mitigations or validation logic that I haven't verified in the actual source code? Re-read the code after reaching a conclusion.
73
+
74
+ **Escalation check**: If any question above produces genuine uncertainty you cannot resolve with the evidence at hand: escalate to deep verification.
75
+
76
+ ### Step 6: Gate Review
77
+
78
+ Apply all six gates from [gate-reviews.md](gate-reviews.md) and all 13 items from [false-positive-patterns.md](false-positive-patterns.md) to reach a verdict.
79
+
80
+ Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: read-only rule for proofs of concept, punctuation.
@@ -0,0 +1,104 @@
1
+ ---
2
+ name: focused-implementation
3
+ description: Use when implementing a feature, change or fix in the code step, especially one that touches several files: turn the task into checkable goals, build it in small working slices, keep every changed line tied to the task, and record assumptions and side findings for the report instead of asking.
4
+ ---
5
+
6
+ # Focused implementation
7
+
8
+ The reviewer should be able to read your diff and see the task in every line. Nobody answers questions during the run, so you decide, write the decision down, and keep going.
9
+
10
+ Start with `project-orientation` if you have not done it in this run. When the spec has several changes, write the plan first with `implementation-plan`.
11
+
12
+ ## 1. Turn the task into goals you can check
13
+
14
+ Before writing code, write a short list:
15
+
16
+ ```
17
+ Goal: invoices show the VAT amount per line
18
+ Checks:
19
+ - [ ] unit test: 3 lines at 20% VAT give per-line VAT 2.00, 4.00, 0.50
20
+ - [ ] invoice page renders a VAT column (component test or screenshot)
21
+ - [ ] existing invoice tests still pass
22
+ Assumptions:
23
+ - Rounding per line, half up, as in src/money/round.ts (spec does not say)
24
+ ```
25
+
26
+ - Take the acceptance criteria from the spec as they are. Each one becomes at least one check.
27
+ - A vague goal ("make it work", "improve") becomes a concrete check you can run.
28
+ - Where the task or spec is ambiguous, pick the reading that best matches the spec, the existing code and the smallest change. Write it under Assumptions. Do not stop to ask; `task-interview` says which open points deserve a question in the report.
29
+ - If the task as written is impossible or contradicts the code (an API that does not exist, a field that is not stored), do the closest useful thing that stays inside the task, and explain the gap in the report.
30
+
31
+ ## 2. Choose the simplest design that meets the goals
32
+
33
+ - Prefer the plain, obvious solution: a function call over an event bus, two similar components over a configurable framework for two cases. Three similar lines beat a premature abstraction.
34
+ - No speculative features, options, flags or extension points that the task does not need.
35
+ - Reuse what the project already has (helpers, components, error types, query builders) before writing new ones.
36
+ - For a change that spans many modules or needs a real design decision, ask the `code-architect` agent for one concrete blueprint, then follow it unless you find a reason not to (and say why).
37
+
38
+ ## 3. Build in thin slices
39
+
40
+ Work in slices that each leave the project working:
41
+
42
+ 1. Implement the smallest complete piece (vertical: from storage to API to UI for one case, rather than all of the storage first).
43
+ 2. Test it: write the test first where the project has tests (`test-driven-development`), then run the tests near the change.
44
+ 3. Check it builds and type-checks.
45
+ 4. Move to the next slice.
46
+
47
+ Rules:
48
+ - Do not write more than roughly 100 lines without running something.
49
+ - Put the riskiest or least certain piece first, so a dead end shows up early.
50
+ - Never leave the build broken between slices.
51
+ - Commit a slice when it is green if that makes the history clearer (`clean-commits`); one commit at the end is also fine.
52
+ - New behaviour defaults to the safe, conservative option (off, opt-in, read-only) unless the task says otherwise.
53
+
54
+ ## 4. Keep changes surgical
55
+
56
+ Every changed line should trace back to the task.
57
+
58
+ - Do not reformat, rename, reorder imports or "modernise" code you are not otherwise changing.
59
+ - Do not delete comments, code or tests you do not fully understand.
60
+ - Match the surrounding style even if you would write it differently.
61
+ - Clean up what your own change made unused (imports, helpers, variables). Leave older dead code alone.
62
+ - When you notice something wrong outside the task, do not fix it. Add it to a "Noticed, not changed" list for the report:
63
+
64
+ ```
65
+ Noticed, not changed:
66
+ - src/utils/format.ts: unused import of `dayjs` (unrelated)
67
+ - Login form shows the raw server error text (separate task)
68
+ ```
69
+
70
+ Exception: if the task cannot be done correctly without touching it (the bug you are fixing lives there), change it and say why.
71
+
72
+ ## 5. Flag these in the report
73
+
74
+ Do them only when the task needs them, choose the smaller option, and list them in the report so the reviewer looks closely:
75
+
76
+ - New dependencies, or upgrades of existing ones.
77
+ - Database schema changes or migrations (follow `safe-database-changes`).
78
+ - Changes to authentication, authorisation, sessions, CORS, uploads, rate limits, roles or permissions.
79
+ - New external services or new kinds of stored sensitive data.
80
+ - Changes to a public API, a shared contract, CI config or build config.
81
+ - Deleting files, features or tests.
82
+ - Changes in a repository that the task did not obviously involve.
83
+
84
+ Never: commit secrets, edit vendored or generated files by hand (regenerate them the way the project does), weaken or delete a failing test to get green, or disable lint rules to silence your own code.
85
+
86
+ ## 6. Finish
87
+
88
+ 1. Tidy your own diff: remove debug output, leftover TODOs you created, commented-out code. For a large change, the `code-simplifier` agent can do a behaviour-preserving pass over the changed files.
89
+ 2. Run `verification-before-completion`.
90
+ 3. Commit with `clean-commits`.
91
+ 4. Write the report: what changed, how it was checked (commands and results), assumptions, flagged items, noticed-not-changed list, anything left open.
92
+
93
+ ## Common mistakes
94
+
95
+ | Thought | What to do instead |
96
+ |---|---|
97
+ | "While I'm here I'll clean this up" | Note it under "Noticed, not changed". |
98
+ | "I'll test it all at the end" | Test each slice; a bug in slice 1 makes slices 2 to 5 wrong. |
99
+ | "Let me make it configurable, just in case" | Build what the task needs; add options when a second case exists. |
100
+ | "The spec is unclear, I'll wait" | Pick the most reasonable reading, write it under Assumptions, continue. |
101
+ | "This refactor is small enough to include" | Keep refactors out of feature changes unless the task needs them. |
102
+ | "Lint fails on my line, I'll add an ignore comment" | Fix the code; ignore comments only where the project already uses them for the same case, with a reason. |
103
+
104
+ Sources: addyosmani/agent-skills (MIT, commit aeac41d) incremental-implementation, plus ideas restated from the Karpathy coding guidelines. Changed: rewritten for unattended runs.
@@ -0,0 +1,66 @@
1
+ ---
2
+ name: implementation-plan
3
+ description: Use at the start of a code step when the spec or task touches more than one file or more than one repository: turn it into an ordered plan of small tasks with exact files, the interfaces passed between them, the checks that prove each one, and a review focus of the inputs no test covers yet.
4
+ ---
5
+
6
+ # Implementation plan
7
+
8
+ A plan is the set of decisions an implementer cannot make alone: which files, which names and signatures, which values from the spec, which tests prove each step. Write those down, nothing more. A plan longer than the code it describes has written the code instead.
9
+
10
+ Start with `project-orientation` if you have not done it in this run. Keep the plan in your working notes; it does not go into the repository unless the project asks for plans there.
11
+
12
+ ## 1. Read the spec as the contract
13
+
14
+ - The acceptance criteria (AC-1, AC-2 and so on) are the finish line. Each one must end up proven by a test or a check.
15
+ - Out of scope is a hard limit: nothing from it goes into the plan.
16
+ - Decisions, assumptions and the code map are settled: follow them. If the code contradicts one, note it for the report and pick the reading closest to the spec.
17
+ - Values the spec pins (texts, limits, names, status codes) are copied verbatim into a "Constraints" list at the top of the plan.
18
+
19
+ ## 2. Map the files
20
+
21
+ List every file you will create or change, with one line on its responsibility. Files that change together live together; follow the project's existing layout.
22
+
23
+ ## 3. Order the tasks
24
+
25
+ Each task is the smallest unit with its own test cycle that a reviewer could accept or reject on its own:
26
+
27
+ ```
28
+ Task 2: CSV export endpoint
29
+ Files: create server/api/boards/[id]/export.get.ts; test test/boardExport.test.ts
30
+ Consumes: canSeeBoard(userId, boardId) from server/utils/boardAccess.ts
31
+ Produces: GET /api/boards/:id/export -> text/csv, header "Title,Status,Owner"
32
+ Proves: AC-1, AC-2
33
+ Steps:
34
+ 1. Test: 3 tasks give 4 lines; no access gives 404 (fails first)
35
+ 2. Implement exportCsv(boardId: number): string and the route
36
+ 3. Run: pnpm test test/boardExport.test.ts -> 2 passed
37
+ ```
38
+
39
+ - Order by dependency: a task uses only what earlier tasks or the existing code provide. Put the riskiest or least certain task as early as its dependencies allow.
40
+ - **Interfaces:** write Consumes and Produces with exact names, parameters, return types and data shapes. This is how the next task learns the names; check that every name used later is produced earlier.
41
+ - **Steps:** each step is one action with a checkable result, and lets you write exactly one reasonable thing. Lines that decide nothing ("handle edge cases", "add validation", "write tests") are gaps: replace them with the case, the rule or the test name.
42
+ - **Tests at the highest seam** that covers the behaviour: an endpoint, a page or a public function, not private helpers. Follow `test-driven-development`.
43
+
44
+ ## 4. Verification
45
+
46
+ Split it in two:
47
+
48
+ - **Automated:** the exact commands (tests, type check, lint, build) and what passing looks like. Take them from the spec's test plan and from `project-orientation`.
49
+ - **Manual:** what a person does and sees, naming the page or command. For interface work, the screens to render with `ui-testing` or `design-review`.
50
+
51
+ Map every acceptance criterion to the task and the check that proves it. A criterion without a check is a gap: add the test.
52
+
53
+ ## 5. Review focus
54
+
55
+ Write up to five input classes or failure modes the spec implies but no planned test exercises, most likely to bite a person first: an empty list, a very long name, a second click, an expired session, a slow network. The spec's silence on an input is not permission for that input to break the program. For each line, add the test to the task that owns the code, or say in the report why it is left.
56
+
57
+ ## 6. Self-check before coding
58
+
59
+ 1. **Coverage:** every criterion has a task and a check; every task proves a criterion.
60
+ 2. **Names:** names and signatures match between Produces and Consumes.
61
+ 3. **Scope:** nothing from out of scope, nothing the spec did not ask for.
62
+ 4. **Proportion:** the plan is shorter than the code it describes.
63
+
64
+ Then build it with `focused-implementation`, one task at a time, and finish with `verification-before-completion`. Put the plan's Review Focus and any deviations from the spec in the report.
65
+
66
+ Sources: adapted from obra/superpowers writing-plans (MIT, commit b739a8a), the plan template and ready-for-development rules of bmad-code-org/BMAD-METHOD (MIT, commit 61e01f8), humanlayer/humanlayer create_plan (Apache-2.0, commit 99abe67) and mattpocock/skills to-spec testing seams (MIT, commit 5beb35b). Changed: condensed into one process for unattended code steps, the plan stays in working notes, execution choices removed.