@tamsensedev/runner 0.13.0 → 0.19.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (82) hide show
  1. package/LICENSE +5 -5
  2. package/README.md +76 -57
  3. package/dist/cli.js +757 -379
  4. package/dist/cli.js.map +1 -1
  5. package/dist/recorder.cjs.map +1 -1
  6. package/library/.claude-plugin/plugin.json +5 -5
  7. package/library/THIRD_PARTY_NOTICES.md +123 -123
  8. package/library/agents/code-architect.md +67 -67
  9. package/library/agents/code-explorer.md +61 -61
  10. package/library/agents/code-reviewer.md +73 -73
  11. package/library/agents/code-simplifier.md +62 -62
  12. package/library/agents/design-critic.md +52 -52
  13. package/library/agents/second-opinion.md +64 -64
  14. package/library/agents/security-auditor.md +86 -86
  15. package/library/agents/silent-failure-hunter.md +70 -70
  16. package/library/agents/test-coverage-analyzer.md +78 -78
  17. package/library/agents/test-runner.md +48 -48
  18. package/library/agents/ui-tester.md +50 -50
  19. package/library/licenses/Apache-2.0.txt +202 -202
  20. package/library/skills/accessibility/SKILL.md +144 -144
  21. package/library/skills/accessibility/references/wcag-checks.md +168 -168
  22. package/library/skills/addressing-review-feedback/SKILL.md +109 -109
  23. package/library/skills/app-screens/SKILL.md +96 -96
  24. package/library/skills/app-screens/references/shell-kpis-charts.md +108 -108
  25. package/library/skills/app-screens/references/states-density-phone.md +54 -54
  26. package/library/skills/app-screens/references/tables-forms-settings.md +104 -104
  27. package/library/skills/avoiding-generic-ai-design/SKILL.md +92 -92
  28. package/library/skills/avoiding-generic-ai-design/references/tells.md +112 -112
  29. package/library/skills/clean-commits/SKILL.md +85 -85
  30. package/library/skills/design-review/SKILL.md +107 -107
  31. package/library/skills/design-review/references/critique-passes.md +101 -101
  32. package/library/skills/design-review/references/screenshot-procedure.md +121 -121
  33. package/library/skills/differential-security-review/LICENSE +427 -427
  34. package/library/skills/differential-security-review/SKILL.md +105 -105
  35. package/library/skills/differential-security-review/references/adversarial.md +115 -115
  36. package/library/skills/differential-security-review/references/methodology.md +145 -145
  37. package/library/skills/differential-security-review/references/patterns.md +108 -108
  38. package/library/skills/false-positive-check/LICENSE +427 -427
  39. package/library/skills/false-positive-check/SKILL.md +95 -95
  40. package/library/skills/false-positive-check/references/bug-class-verification.md +128 -128
  41. package/library/skills/false-positive-check/references/deep-verification.md +113 -113
  42. package/library/skills/false-positive-check/references/evidence-templates.md +93 -93
  43. package/library/skills/false-positive-check/references/false-positive-patterns.md +122 -122
  44. package/library/skills/false-positive-check/references/gate-reviews.md +36 -36
  45. package/library/skills/false-positive-check/references/standard-verification.md +80 -80
  46. package/library/skills/focused-implementation/SKILL.md +104 -104
  47. package/library/skills/implementation-plan/SKILL.md +66 -66
  48. package/library/skills/interface-design/SKILL.md +144 -144
  49. package/library/skills/interface-design/references/craft-defaults.md +130 -130
  50. package/library/skills/interface-design/references/new-direction.md +62 -62
  51. package/library/skills/interface-design/references/writing.md +52 -52
  52. package/library/skills/interface-polish/SKILL.md +103 -103
  53. package/library/skills/interface-polish/references/detail-rules.md +154 -154
  54. package/library/skills/interface-polish/references/worst-case-data.md +158 -158
  55. package/library/skills/motion-design/SKILL.md +106 -106
  56. package/library/skills/motion-design/references/recipes.md +243 -243
  57. package/library/skills/postgres-performance/SKILL.md +97 -97
  58. package/library/skills/postgres-performance/references/rules.md +265 -265
  59. package/library/skills/project-orientation/SKILL.md +77 -77
  60. package/library/skills/reviewing-changes/SKILL.md +183 -183
  61. package/library/skills/safe-database-changes/SKILL.md +111 -111
  62. package/library/skills/safe-database-changes/references/backfills.md +75 -75
  63. package/library/skills/safe-database-changes/references/migration-tools.md +69 -69
  64. package/library/skills/safe-database-changes/references/postgres-ddl.md +199 -199
  65. package/library/skills/security-review/SKILL.md +132 -132
  66. package/library/skills/security-review/references/checks-by-area.md +107 -107
  67. package/library/skills/security-review/references/do-not-report.md +86 -86
  68. package/library/skills/spec-writing/SKILL.md +144 -144
  69. package/library/skills/systematic-debugging/SKILL.md +130 -130
  70. package/library/skills/systematic-debugging/references/flaky-tests.md +51 -51
  71. package/library/skills/systematic-debugging/references/root-cause-tracing.md +55 -55
  72. package/library/skills/task-interview/SKILL.md +84 -84
  73. package/library/skills/test-driven-development/SKILL.md +116 -116
  74. package/library/skills/test-driven-development/references/writing-good-tests.md +85 -85
  75. package/library/skills/ui-testing/SKILL.md +102 -102
  76. package/library/skills/ui-testing/references/headless-check.md +295 -295
  77. package/library/skills/ui-testing/references/local-stand.md +51 -51
  78. package/library/skills/ui-testing/references/playwright-patterns.md +134 -134
  79. package/library/skills/ui-testing/references/recording.md +61 -61
  80. package/library/skills/verification-before-completion/SKILL.md +113 -113
  81. package/library/skills/web-interface-guidelines/SKILL.md +202 -202
  82. package/package.json +46 -46
@@ -1,122 +1,122 @@
1
- # False Positive Patterns: Lessons Learned
2
-
3
- Apply ALL items in this checklist to EACH potential bug during verification.
4
-
5
- Contents:
6
-
7
- - [Checklist](#checklist): 13 items to apply to every bug (trace validation, confirm data paths, bounds logic, TOCTOU, trust boundaries, real vs theoretical impact, defense-in-depth)
8
- - [Red Flags for False Positives](#red-flags-for-false-positives): recurring failure patterns (pattern-based, context-blind, math/bounds, API-contract)
9
-
10
- ## Checklist
11
-
12
- ### 1. Trace Full Validation Chain
13
-
14
- Don't analyze isolated code snippets. Trace backwards to find ALL validation that precedes potentially dangerous operations. Network packet size operations may look dangerous but often have bounds validation earlier in the function.
15
-
16
- ### 1a. Map Complete Conditional Logic Flow
17
-
18
- Vulnerable-looking code may be unreachable due to conditional logic that creates mathematical guarantees. Example: array access `buffer[length-4]` appears unsafe when `length < 4`, but if the code is only reachable when `length > 12` due to earlier validation, the vulnerability is impossible.
19
-
20
- **Verify:**
21
-
22
- - What conditions must be met for execution to reach the alleged vulnerability?
23
- - Do those conditions mathematically prevent the vulnerability scenario?
24
- - Are there minimum size/length requirements that guarantee safe access?
25
- - Does the conditional flow create impossible-to-violate bounds?
26
-
27
- ### 2. Identify Defensive Programming Patterns
28
-
29
- Distinguish between actual vulnerabilities and defensive assertions/validations. `ASSERT(size == expected_size)` followed by size-controlled operations is defensive, not vulnerable. Verify that checks actually prevent the alleged vulnerability.
30
-
31
- ### 3. Confirm Exploitable Data Paths
32
-
33
- Only report vulnerabilities with CONFIRMED exploitable data flow paths. Don't assume network-controlled data reaches dangerous functions without tracing the actual path step by step.
34
-
35
- ### 4. Understand Data Source Context
36
-
37
- Distinguish between data sources and their trust levels. API return values, compile-time constants, and network data have different risk profiles. Determine the actual source and whether it is attacker-controlled.
38
-
39
- ### 5. Analyze Bounds Validation Logic
40
-
41
- Look for mathematical relationships between validation checks and subsequent operations. If `packet_size >= MIN_SIZE` is checked and `MIN_SIZE >= sizeof(header)`, then `packet_size - sizeof(header)` cannot underflow.
42
-
43
- ### 6. Verify TOCTOU Claims
44
-
45
- Time-of-check-time-of-use issues require proof that the checked value can change between check and use. If a size is checked and immediately used in the same function with no external modification possible, there is no TOCTOU.
46
-
47
- ### 7. Understand API Contract and Trust Boundaries
48
-
49
- Always understand API contracts before claiming buffer overflows. Some APIs have built-in bounds protection and cannot write beyond the buffer regardless of input parameters.
50
-
51
- ### 8. Distinguish Internal Storage from External Input
52
-
53
- Internal storage systems (configuration stores, registries) are controlled by trusted components, not attackers. Values set during installation by trusted components are not attacker-controlled.
54
-
55
- ### 9. Don't Confuse Pattern Recognition with Vulnerability Analysis
56
-
57
- Code patterns that "look vulnerable" may be safely implemented due to context and API contracts. Size parameters being modified doesn't mean buffer overflow if the API prevents writing beyond bounds.
58
-
59
- ### 10. Verify Concurrent Access is Actually Possible
60
-
61
- Don't assume race conditions exist without proving concurrent access patterns. Single-threaded initialization contexts cannot have race conditions. Verify the threading model and synchronization mechanisms.
62
-
63
- ### 11. Assess Real vs Theoretical Security Impact
64
-
65
- Focus on vulnerabilities with actual security impact. Storage failure for non-critical data is an operational issue, not a security vulnerability. Ask: would this lead to code execution, privilege escalation, or information disclosure?
66
-
67
- ### 12. Understand Defense-in-Depth vs Primary Controls
68
-
69
- Failure of defense-in-depth mechanisms is not always a vulnerability if primary protections exist. Token cleanup failure is not critical if tokens are single-use by design at the server.
70
-
71
- ### 13. Apply the Checklist Rigorously, Not Superficially
72
-
73
- Having a checklist doesn't prevent false positives if it isn't applied systematically. For EVERY potential vulnerability, work through ALL checklist items before concluding.
74
-
75
- ---
76
-
77
- ## Red Flags for False Positives
78
-
79
- ### Pattern-Based False Positives
80
-
81
- - Reporting vulnerabilities in validation/bounds-checking code itself
82
- - Claiming TOCTOU without proving the value can change
83
- - Ignoring preceding validation logic
84
- - Assuming network data reaches operations without tracing the path
85
- - Confusing defensive programming (assertions/checks) with vulnerabilities
86
- - Analyzing vulnerable-looking patterns without tracing conditional logic that controls reachability
87
- - Reporting "vulnerabilities" in error handling or cleanup code
88
- - Flagging size calculations without understanding mathematical constraints
89
- - Identifying "dangerous" functions without checking if inputs are bounded
90
- - Claiming buffer overflows in fixed-size operations with compile-time bounds
91
- - Reporting race conditions in single-threaded or synchronized contexts
92
-
93
- ### Context-Blind Analysis False Positives
94
-
95
- - Analyzing code snippets without understanding broader system design
96
- - Ignoring architectural guarantees (single-writer, trusted input sources)
97
- - Missing that "vulnerable" code is unreachable due to earlier validation
98
- - Confusing debug/development code paths with production paths
99
- - Reporting issues in code that only runs during trusted installation/setup
100
- - Flagging theoretical issues that cannot occur due to system architecture
101
- - Missing that alleged vulnerabilities are prevented by framework or language guarantees
102
- - Reporting issues in test-only or debug-only code paths as production vulnerabilities
103
-
104
- ### Mathematical/Bounds Analysis False Positives
105
-
106
- - Reporting integer underflow without proving the mathematical condition can occur
107
- - Claiming buffer overflow when bounds are mathematically guaranteed by validation
108
- - Missing that conditional logic creates mathematical impossibility of vulnerable conditions
109
- - Reporting off-by-one errors without checking if loop bounds prevent the condition
110
- - Claiming memory corruption when allocation sizes are verified sufficient
111
- - Reporting arithmetic overflow without checking if input ranges prevent the condition
112
-
113
- ### API Contract Misunderstanding False Positives
114
-
115
- - Claiming buffer overflows when APIs have built-in bounds checking
116
- - Reporting memory corruption for APIs that manage their own memory safely
117
- - Missing that return values are already validated by the API contract
118
- - Confusing API parameter modification with vulnerability when API prevents unsafe modification
119
- - Reporting issues explicitly handled by the API's safety guarantees
120
- - Missing that seemingly dangerous operations are safe due to API implementation details
121
-
122
- Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: punctuation and small wording changes for read-only runs.
1
+ # False Positive Patterns: Lessons Learned
2
+
3
+ Apply ALL items in this checklist to EACH potential bug during verification.
4
+
5
+ Contents:
6
+
7
+ - [Checklist](#checklist): 13 items to apply to every bug (trace validation, confirm data paths, bounds logic, TOCTOU, trust boundaries, real vs theoretical impact, defense-in-depth)
8
+ - [Red Flags for False Positives](#red-flags-for-false-positives): recurring failure patterns (pattern-based, context-blind, math/bounds, API-contract)
9
+
10
+ ## Checklist
11
+
12
+ ### 1. Trace Full Validation Chain
13
+
14
+ Don't analyze isolated code snippets. Trace backwards to find ALL validation that precedes potentially dangerous operations. Network packet size operations may look dangerous but often have bounds validation earlier in the function.
15
+
16
+ ### 1a. Map Complete Conditional Logic Flow
17
+
18
+ Vulnerable-looking code may be unreachable due to conditional logic that creates mathematical guarantees. Example: array access `buffer[length-4]` appears unsafe when `length < 4`, but if the code is only reachable when `length > 12` due to earlier validation, the vulnerability is impossible.
19
+
20
+ **Verify:**
21
+
22
+ - What conditions must be met for execution to reach the alleged vulnerability?
23
+ - Do those conditions mathematically prevent the vulnerability scenario?
24
+ - Are there minimum size/length requirements that guarantee safe access?
25
+ - Does the conditional flow create impossible-to-violate bounds?
26
+
27
+ ### 2. Identify Defensive Programming Patterns
28
+
29
+ Distinguish between actual vulnerabilities and defensive assertions/validations. `ASSERT(size == expected_size)` followed by size-controlled operations is defensive, not vulnerable. Verify that checks actually prevent the alleged vulnerability.
30
+
31
+ ### 3. Confirm Exploitable Data Paths
32
+
33
+ Only report vulnerabilities with CONFIRMED exploitable data flow paths. Don't assume network-controlled data reaches dangerous functions without tracing the actual path step by step.
34
+
35
+ ### 4. Understand Data Source Context
36
+
37
+ Distinguish between data sources and their trust levels. API return values, compile-time constants, and network data have different risk profiles. Determine the actual source and whether it is attacker-controlled.
38
+
39
+ ### 5. Analyze Bounds Validation Logic
40
+
41
+ Look for mathematical relationships between validation checks and subsequent operations. If `packet_size >= MIN_SIZE` is checked and `MIN_SIZE >= sizeof(header)`, then `packet_size - sizeof(header)` cannot underflow.
42
+
43
+ ### 6. Verify TOCTOU Claims
44
+
45
+ Time-of-check-time-of-use issues require proof that the checked value can change between check and use. If a size is checked and immediately used in the same function with no external modification possible, there is no TOCTOU.
46
+
47
+ ### 7. Understand API Contract and Trust Boundaries
48
+
49
+ Always understand API contracts before claiming buffer overflows. Some APIs have built-in bounds protection and cannot write beyond the buffer regardless of input parameters.
50
+
51
+ ### 8. Distinguish Internal Storage from External Input
52
+
53
+ Internal storage systems (configuration stores, registries) are controlled by trusted components, not attackers. Values set during installation by trusted components are not attacker-controlled.
54
+
55
+ ### 9. Don't Confuse Pattern Recognition with Vulnerability Analysis
56
+
57
+ Code patterns that "look vulnerable" may be safely implemented due to context and API contracts. Size parameters being modified doesn't mean buffer overflow if the API prevents writing beyond bounds.
58
+
59
+ ### 10. Verify Concurrent Access is Actually Possible
60
+
61
+ Don't assume race conditions exist without proving concurrent access patterns. Single-threaded initialization contexts cannot have race conditions. Verify the threading model and synchronization mechanisms.
62
+
63
+ ### 11. Assess Real vs Theoretical Security Impact
64
+
65
+ Focus on vulnerabilities with actual security impact. Storage failure for non-critical data is an operational issue, not a security vulnerability. Ask: would this lead to code execution, privilege escalation, or information disclosure?
66
+
67
+ ### 12. Understand Defense-in-Depth vs Primary Controls
68
+
69
+ Failure of defense-in-depth mechanisms is not always a vulnerability if primary protections exist. Token cleanup failure is not critical if tokens are single-use by design at the server.
70
+
71
+ ### 13. Apply the Checklist Rigorously, Not Superficially
72
+
73
+ Having a checklist doesn't prevent false positives if it isn't applied systematically. For EVERY potential vulnerability, work through ALL checklist items before concluding.
74
+
75
+ ---
76
+
77
+ ## Red Flags for False Positives
78
+
79
+ ### Pattern-Based False Positives
80
+
81
+ - Reporting vulnerabilities in validation/bounds-checking code itself
82
+ - Claiming TOCTOU without proving the value can change
83
+ - Ignoring preceding validation logic
84
+ - Assuming network data reaches operations without tracing the path
85
+ - Confusing defensive programming (assertions/checks) with vulnerabilities
86
+ - Analyzing vulnerable-looking patterns without tracing conditional logic that controls reachability
87
+ - Reporting "vulnerabilities" in error handling or cleanup code
88
+ - Flagging size calculations without understanding mathematical constraints
89
+ - Identifying "dangerous" functions without checking if inputs are bounded
90
+ - Claiming buffer overflows in fixed-size operations with compile-time bounds
91
+ - Reporting race conditions in single-threaded or synchronized contexts
92
+
93
+ ### Context-Blind Analysis False Positives
94
+
95
+ - Analyzing code snippets without understanding broader system design
96
+ - Ignoring architectural guarantees (single-writer, trusted input sources)
97
+ - Missing that "vulnerable" code is unreachable due to earlier validation
98
+ - Confusing debug/development code paths with production paths
99
+ - Reporting issues in code that only runs during trusted installation/setup
100
+ - Flagging theoretical issues that cannot occur due to system architecture
101
+ - Missing that alleged vulnerabilities are prevented by framework or language guarantees
102
+ - Reporting issues in test-only or debug-only code paths as production vulnerabilities
103
+
104
+ ### Mathematical/Bounds Analysis False Positives
105
+
106
+ - Reporting integer underflow without proving the mathematical condition can occur
107
+ - Claiming buffer overflow when bounds are mathematically guaranteed by validation
108
+ - Missing that conditional logic creates mathematical impossibility of vulnerable conditions
109
+ - Reporting off-by-one errors without checking if loop bounds prevent the condition
110
+ - Claiming memory corruption when allocation sizes are verified sufficient
111
+ - Reporting arithmetic overflow without checking if input ranges prevent the condition
112
+
113
+ ### API Contract Misunderstanding False Positives
114
+
115
+ - Claiming buffer overflows when APIs have built-in bounds checking
116
+ - Reporting memory corruption for APIs that manage their own memory safely
117
+ - Missing that return values are already validated by the API contract
118
+ - Confusing API parameter modification with vulnerability when API prevents unsafe modification
119
+ - Reporting issues explicitly handled by the API's safety guarantees
120
+ - Missing that seemingly dangerous operations are safe due to API implementation details
121
+
122
+ Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: punctuation and small wording changes for read-only runs.
@@ -1,36 +1,36 @@
1
- # Gate Reviews and Verdicts
2
-
3
- Before reporting ANY bug as a vulnerability, all six gate reviews must pass. Evaluate them at the end, after all phases are complete:
4
-
5
- | Gate | Criterion | Pass | Fail |
6
- |------|-----------|------|------|
7
- | **1. Process** | All phases completed with documented evidence | Evidence exists for every phase | Phases lack concrete evidence |
8
- | **2. Reachability** | Attacker can reach and control data at the vulnerability | Clear evidence of attacker-controlled path + PoC confirms | Cannot demonstrate attacker control or reachability |
9
- | **3. Real Impact** | Exploitation leads to code execution, privilege escalation, information disclosure, unauthorized changes to data, or loss of money | Direct impact with concrete scenarios | Only operational robustness issue |
10
- | **4. PoC Validation** | PoC (pseudocode, executable, or unit test) demonstrates the attack path | Shows attacker control, trigger, and impact | PoC fails to show attack path or impact |
11
- | **5. Math Bounds** | Mathematical analysis confirms vulnerable condition is possible (N/A, counted as pass, when the bug is not about bounds or arithmetic) | Algebraic proof shows condition is possible | Math proves validation prevents it |
12
- | **6. Environment** | No environmental protections entirely prevent exploitation | Protections do not eliminate vulnerability | Environmental protections block it entirely |
13
-
14
- For a suspected correctness bug rather than a vulnerability: gate 2 asks whether a realistic input or state reaches the faulty code, and gate 3 whether the outcome is a wrong result, lost or corrupted data, or a crash that users hit.
15
-
16
- ## Verdict Format
17
-
18
- - **TRUE POSITIVE**: All gate reviews pass → `BUG #N TRUE POSITIVE - [brief vulnerability description]`
19
- - **FALSE POSITIVE**: Any gate review fails → `BUG #N FALSE POSITIVE - [brief reason for rejection]`
20
- - **INCONCLUSIVE**: A gate cannot be decided from the code available (the caller is in another repository, the configuration is not in the repo) → `BUG #N NEEDS VERIFICATION - [the exact open question]`. Reported as a note, never as blocking.
21
-
22
- If any phase fails verification, document the failure with evidence and continue all remaining phases. Issue the FALSE POSITIVE verdict only after all phases are complete.
23
-
24
- ## Example Verdict
25
-
26
- ```
27
- BUG #3 FALSE POSITIVE - Integer underflow in packet_handler.c:142
28
- Gate 5 (Math Bounds) FAIL: validation at line 98 ensures packet_size >= 16,
29
- making (packet_size - header_size) >= 8. Underflow is mathematically impossible.
30
-
31
- BUG #5 FALSE POSITIVE - IDOR in src/api/files.ts:19
32
- Gate 2 (Reachability) FAIL: the route is mounted under accountRouter (src/app.ts:52),
33
- whose middleware loads the file with the session's account id; other accounts' files return 404.
34
- ```
35
-
36
- Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: wider impact list for web applications, N/A rule for gate 5, correctness-bug reading of gates 2 and 3, inconclusive verdict, second example.
1
+ # Gate Reviews and Verdicts
2
+
3
+ Before reporting ANY bug as a vulnerability, all six gate reviews must pass. Evaluate them at the end, after all phases are complete:
4
+
5
+ | Gate | Criterion | Pass | Fail |
6
+ |------|-----------|------|------|
7
+ | **1. Process** | All phases completed with documented evidence | Evidence exists for every phase | Phases lack concrete evidence |
8
+ | **2. Reachability** | Attacker can reach and control data at the vulnerability | Clear evidence of attacker-controlled path + PoC confirms | Cannot demonstrate attacker control or reachability |
9
+ | **3. Real Impact** | Exploitation leads to code execution, privilege escalation, information disclosure, unauthorized changes to data, or loss of money | Direct impact with concrete scenarios | Only operational robustness issue |
10
+ | **4. PoC Validation** | PoC (pseudocode, executable, or unit test) demonstrates the attack path | Shows attacker control, trigger, and impact | PoC fails to show attack path or impact |
11
+ | **5. Math Bounds** | Mathematical analysis confirms vulnerable condition is possible (N/A, counted as pass, when the bug is not about bounds or arithmetic) | Algebraic proof shows condition is possible | Math proves validation prevents it |
12
+ | **6. Environment** | No environmental protections entirely prevent exploitation | Protections do not eliminate vulnerability | Environmental protections block it entirely |
13
+
14
+ For a suspected correctness bug rather than a vulnerability: gate 2 asks whether a realistic input or state reaches the faulty code, and gate 3 whether the outcome is a wrong result, lost or corrupted data, or a crash that users hit.
15
+
16
+ ## Verdict Format
17
+
18
+ - **TRUE POSITIVE**: All gate reviews pass → `BUG #N TRUE POSITIVE - [brief vulnerability description]`
19
+ - **FALSE POSITIVE**: Any gate review fails → `BUG #N FALSE POSITIVE - [brief reason for rejection]`
20
+ - **INCONCLUSIVE**: A gate cannot be decided from the code available (the caller is in another repository, the configuration is not in the repo) → `BUG #N NEEDS VERIFICATION - [the exact open question]`. Reported as a note, never as blocking.
21
+
22
+ If any phase fails verification, document the failure with evidence and continue all remaining phases. Issue the FALSE POSITIVE verdict only after all phases are complete.
23
+
24
+ ## Example Verdict
25
+
26
+ ```
27
+ BUG #3 FALSE POSITIVE - Integer underflow in packet_handler.c:142
28
+ Gate 5 (Math Bounds) FAIL: validation at line 98 ensures packet_size >= 16,
29
+ making (packet_size - header_size) >= 8. Underflow is mathematically impossible.
30
+
31
+ BUG #5 FALSE POSITIVE - IDOR in src/api/files.ts:19
32
+ Gate 2 (Reachability) FAIL: the route is mounted under accountRouter (src/app.ts:52),
33
+ whose middleware loads the file with the session's account id; other accounts' files return 404.
34
+ ```
35
+
36
+ Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: wider impact list for web applications, N/A rule for gate 5, correctness-bug reading of gates 2 and 3, inconclusive verdict, second example.
@@ -1,80 +1,80 @@
1
- # Standard Verification
2
-
3
- Linear single-pass checklist for straightforward bugs. No task tracking: work through each step sequentially and document findings inline.
4
-
5
- ## Escalation Checkpoints
6
-
7
- Two points in this checklist may trigger escalation to [deep-verification.md](deep-verification.md):
8
-
9
- 1. **After Step 1 (Data Flow)**: Escalate if 3+ trust boundaries, callbacks/async control flow, or ambiguous validation chain
10
- 2. **After Step 5 (Devil's Advocate)**: Escalate if any question produces genuine uncertainty you cannot resolve
11
-
12
- When escalating, hand off all evidence gathered so far: deep verification will continue from where you left off.
13
-
14
- ## Checklist
15
-
16
- ### Step 1: Data Flow
17
-
18
- Trace data from source to the alleged vulnerability sink.
19
-
20
- - Map trust boundaries crossed (internal/trusted vs external/untrusted)
21
- - Identify all validation and sanitization between source and sink
22
- - Check API contracts: many APIs have built-in bounds protection that prevents the alleged issue
23
- - Check for environmental protections (compiler, runtime, OS, framework) that prevent exploitation entirely (not just raise the bar)
24
- - Apply class-specific checks from [bug-class-verification.md](bug-class-verification.md)
25
-
26
- **Key pitfall**: Analyzing the vulnerable code in isolation. Conditional logic upstream may make the vulnerability mathematically unreachable. Trace the full validation chain.
27
-
28
- **Escalation check**: If you found 3+ trust boundaries, callbacks or async control flow in the path, or an ambiguous validation chain: escalate to deep verification.
29
-
30
- ### Step 2: Exploitability
31
-
32
- Prove the attacker can trigger the vulnerability.
33
-
34
- - **Attacker control**: Prove the attacker controls data reaching the vulnerable operation. Internal storage set by trusted components is not attacker-controlled.
35
- - **Bounds proof**: For integer/bounds issues, create an explicit algebraic proof using the template in [evidence-templates.md](evidence-templates.md). Verify: IF validation_check_passes THEN bounds_guarantee_holds.
36
- - **Race feasibility**: For race conditions, prove concurrent access is actually possible. Single-threaded initialization and synchronized contexts cannot have races.
37
-
38
- ### Step 3: Impact
39
-
40
- Determine whether exploitation has real security consequences.
41
-
42
- - Distinguish real security impact (RCE, privesc, info disclosure) from operational robustness issues (crash recovery, cleanup failure)
43
- - Distinguish primary security controls from defense-in-depth. Failure of a defense-in-depth measure is not a vulnerability if primary protections remain intact.
44
-
45
- ### Step 4: PoC Sketch
46
-
47
- Create a pseudocode PoC showing the attack path. Executable and unit test PoCs are optional for standard verification; in review and test steps they run only from a temp directory outside the repository and never against live services (see the ground rules in SKILL.md).
48
-
49
- ```
50
- Data Flow: [Source] → [Validation?] → [Transform?] → [Vulnerable Op] → [Impact]
51
- Attacker controls: [what input, how]
52
- Trigger: [pseudocode showing the exploit path]
53
- ```
54
-
55
- See [evidence-templates.md](evidence-templates.md) for the full PoC template.
56
-
57
- ### Step 5: Devil's Advocate Spot-Check
58
-
59
- Answer these 7 questions. If any produces genuine uncertainty, escalate to deep verification.
60
-
61
- **Against the vulnerability:**
62
-
63
- 1. Am I seeing a vulnerability because the pattern "looks dangerous" rather than because it actually is? (pattern-matching bias)
64
- 2. Am I incorrectly assuming attacker control over trusted data? (trust boundary confusion)
65
- 3. Have I rigorously proven the mathematical condition for vulnerability can occur? (proof rigor)
66
- 4. Am I confusing defense-in-depth failure with a primary security vulnerability? (defense-in-depth confusion)
67
- 5. Am I hallucinating this vulnerability? LLMs are biased toward seeing bugs everywhere: is this actually real or am I pattern-matching on scary-looking code? (LLM self-check)
68
-
69
- **For the vulnerability (always ask: false-negative protection):**
70
-
71
- 6. Am I dismissing a real vulnerability because the exploit seems complex or unlikely?
72
- 7. Am I inventing mitigations or validation logic that I haven't verified in the actual source code? Re-read the code after reaching a conclusion.
73
-
74
- **Escalation check**: If any question above produces genuine uncertainty you cannot resolve with the evidence at hand: escalate to deep verification.
75
-
76
- ### Step 6: Gate Review
77
-
78
- Apply all six gates from [gate-reviews.md](gate-reviews.md) and all 13 items from [false-positive-patterns.md](false-positive-patterns.md) to reach a verdict.
79
-
80
- Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: read-only rule for proofs of concept, punctuation.
1
+ # Standard Verification
2
+
3
+ Linear single-pass checklist for straightforward bugs. No task tracking: work through each step sequentially and document findings inline.
4
+
5
+ ## Escalation Checkpoints
6
+
7
+ Two points in this checklist may trigger escalation to [deep-verification.md](deep-verification.md):
8
+
9
+ 1. **After Step 1 (Data Flow)**: Escalate if 3+ trust boundaries, callbacks/async control flow, or ambiguous validation chain
10
+ 2. **After Step 5 (Devil's Advocate)**: Escalate if any question produces genuine uncertainty you cannot resolve
11
+
12
+ When escalating, hand off all evidence gathered so far: deep verification will continue from where you left off.
13
+
14
+ ## Checklist
15
+
16
+ ### Step 1: Data Flow
17
+
18
+ Trace data from source to the alleged vulnerability sink.
19
+
20
+ - Map trust boundaries crossed (internal/trusted vs external/untrusted)
21
+ - Identify all validation and sanitization between source and sink
22
+ - Check API contracts: many APIs have built-in bounds protection that prevents the alleged issue
23
+ - Check for environmental protections (compiler, runtime, OS, framework) that prevent exploitation entirely (not just raise the bar)
24
+ - Apply class-specific checks from [bug-class-verification.md](bug-class-verification.md)
25
+
26
+ **Key pitfall**: Analyzing the vulnerable code in isolation. Conditional logic upstream may make the vulnerability mathematically unreachable. Trace the full validation chain.
27
+
28
+ **Escalation check**: If you found 3+ trust boundaries, callbacks or async control flow in the path, or an ambiguous validation chain: escalate to deep verification.
29
+
30
+ ### Step 2: Exploitability
31
+
32
+ Prove the attacker can trigger the vulnerability.
33
+
34
+ - **Attacker control**: Prove the attacker controls data reaching the vulnerable operation. Internal storage set by trusted components is not attacker-controlled.
35
+ - **Bounds proof**: For integer/bounds issues, create an explicit algebraic proof using the template in [evidence-templates.md](evidence-templates.md). Verify: IF validation_check_passes THEN bounds_guarantee_holds.
36
+ - **Race feasibility**: For race conditions, prove concurrent access is actually possible. Single-threaded initialization and synchronized contexts cannot have races.
37
+
38
+ ### Step 3: Impact
39
+
40
+ Determine whether exploitation has real security consequences.
41
+
42
+ - Distinguish real security impact (RCE, privesc, info disclosure) from operational robustness issues (crash recovery, cleanup failure)
43
+ - Distinguish primary security controls from defense-in-depth. Failure of a defense-in-depth measure is not a vulnerability if primary protections remain intact.
44
+
45
+ ### Step 4: PoC Sketch
46
+
47
+ Create a pseudocode PoC showing the attack path. Executable and unit test PoCs are optional for standard verification; in review and test steps they run only from a temp directory outside the repository and never against live services (see the ground rules in SKILL.md).
48
+
49
+ ```
50
+ Data Flow: [Source] → [Validation?] → [Transform?] → [Vulnerable Op] → [Impact]
51
+ Attacker controls: [what input, how]
52
+ Trigger: [pseudocode showing the exploit path]
53
+ ```
54
+
55
+ See [evidence-templates.md](evidence-templates.md) for the full PoC template.
56
+
57
+ ### Step 5: Devil's Advocate Spot-Check
58
+
59
+ Answer these 7 questions. If any produces genuine uncertainty, escalate to deep verification.
60
+
61
+ **Against the vulnerability:**
62
+
63
+ 1. Am I seeing a vulnerability because the pattern "looks dangerous" rather than because it actually is? (pattern-matching bias)
64
+ 2. Am I incorrectly assuming attacker control over trusted data? (trust boundary confusion)
65
+ 3. Have I rigorously proven the mathematical condition for vulnerability can occur? (proof rigor)
66
+ 4. Am I confusing defense-in-depth failure with a primary security vulnerability? (defense-in-depth confusion)
67
+ 5. Am I hallucinating this vulnerability? LLMs are biased toward seeing bugs everywhere: is this actually real or am I pattern-matching on scary-looking code? (LLM self-check)
68
+
69
+ **For the vulnerability (always ask: false-negative protection):**
70
+
71
+ 6. Am I dismissing a real vulnerability because the exploit seems complex or unlikely?
72
+ 7. Am I inventing mitigations or validation logic that I haven't verified in the actual source code? Re-read the code after reaching a conclusion.
73
+
74
+ **Escalation check**: If any question above produces genuine uncertainty you cannot resolve with the evidence at hand: escalate to deep verification.
75
+
76
+ ### Step 6: Gate Review
77
+
78
+ Apply all six gates from [gate-reviews.md](gate-reviews.md) and all 13 items from [false-positive-patterns.md](false-positive-patterns.md) to reach a verdict.
79
+
80
+ Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: read-only rule for proofs of concept, punctuation.