@tamsensedev/runner 0.13.0 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +5 -5
- package/README.md +76 -57
- package/dist/cli.js +757 -379
- package/dist/cli.js.map +1 -1
- package/dist/recorder.cjs.map +1 -1
- package/library/.claude-plugin/plugin.json +5 -5
- package/library/THIRD_PARTY_NOTICES.md +123 -123
- package/library/agents/code-architect.md +67 -67
- package/library/agents/code-explorer.md +61 -61
- package/library/agents/code-reviewer.md +73 -73
- package/library/agents/code-simplifier.md +62 -62
- package/library/agents/design-critic.md +52 -52
- package/library/agents/second-opinion.md +64 -64
- package/library/agents/security-auditor.md +86 -86
- package/library/agents/silent-failure-hunter.md +70 -70
- package/library/agents/test-coverage-analyzer.md +78 -78
- package/library/agents/test-runner.md +48 -48
- package/library/agents/ui-tester.md +50 -50
- package/library/licenses/Apache-2.0.txt +202 -202
- package/library/skills/accessibility/SKILL.md +144 -144
- package/library/skills/accessibility/references/wcag-checks.md +168 -168
- package/library/skills/addressing-review-feedback/SKILL.md +109 -109
- package/library/skills/app-screens/SKILL.md +96 -96
- package/library/skills/app-screens/references/shell-kpis-charts.md +108 -108
- package/library/skills/app-screens/references/states-density-phone.md +54 -54
- package/library/skills/app-screens/references/tables-forms-settings.md +104 -104
- package/library/skills/avoiding-generic-ai-design/SKILL.md +92 -92
- package/library/skills/avoiding-generic-ai-design/references/tells.md +112 -112
- package/library/skills/clean-commits/SKILL.md +85 -85
- package/library/skills/design-review/SKILL.md +107 -107
- package/library/skills/design-review/references/critique-passes.md +101 -101
- package/library/skills/design-review/references/screenshot-procedure.md +121 -121
- package/library/skills/differential-security-review/LICENSE +427 -427
- package/library/skills/differential-security-review/SKILL.md +105 -105
- package/library/skills/differential-security-review/references/adversarial.md +115 -115
- package/library/skills/differential-security-review/references/methodology.md +145 -145
- package/library/skills/differential-security-review/references/patterns.md +108 -108
- package/library/skills/false-positive-check/LICENSE +427 -427
- package/library/skills/false-positive-check/SKILL.md +95 -95
- package/library/skills/false-positive-check/references/bug-class-verification.md +128 -128
- package/library/skills/false-positive-check/references/deep-verification.md +113 -113
- package/library/skills/false-positive-check/references/evidence-templates.md +93 -93
- package/library/skills/false-positive-check/references/false-positive-patterns.md +122 -122
- package/library/skills/false-positive-check/references/gate-reviews.md +36 -36
- package/library/skills/false-positive-check/references/standard-verification.md +80 -80
- package/library/skills/focused-implementation/SKILL.md +104 -104
- package/library/skills/implementation-plan/SKILL.md +66 -66
- package/library/skills/interface-design/SKILL.md +144 -144
- package/library/skills/interface-design/references/craft-defaults.md +130 -130
- package/library/skills/interface-design/references/new-direction.md +62 -62
- package/library/skills/interface-design/references/writing.md +52 -52
- package/library/skills/interface-polish/SKILL.md +103 -103
- package/library/skills/interface-polish/references/detail-rules.md +154 -154
- package/library/skills/interface-polish/references/worst-case-data.md +158 -158
- package/library/skills/motion-design/SKILL.md +106 -106
- package/library/skills/motion-design/references/recipes.md +243 -243
- package/library/skills/postgres-performance/SKILL.md +97 -97
- package/library/skills/postgres-performance/references/rules.md +265 -265
- package/library/skills/project-orientation/SKILL.md +77 -77
- package/library/skills/reviewing-changes/SKILL.md +183 -183
- package/library/skills/safe-database-changes/SKILL.md +111 -111
- package/library/skills/safe-database-changes/references/backfills.md +75 -75
- package/library/skills/safe-database-changes/references/migration-tools.md +69 -69
- package/library/skills/safe-database-changes/references/postgres-ddl.md +199 -199
- package/library/skills/security-review/SKILL.md +132 -132
- package/library/skills/security-review/references/checks-by-area.md +107 -107
- package/library/skills/security-review/references/do-not-report.md +86 -86
- package/library/skills/spec-writing/SKILL.md +144 -144
- package/library/skills/systematic-debugging/SKILL.md +130 -130
- package/library/skills/systematic-debugging/references/flaky-tests.md +51 -51
- package/library/skills/systematic-debugging/references/root-cause-tracing.md +55 -55
- package/library/skills/task-interview/SKILL.md +84 -84
- package/library/skills/test-driven-development/SKILL.md +116 -116
- package/library/skills/test-driven-development/references/writing-good-tests.md +85 -85
- package/library/skills/ui-testing/SKILL.md +102 -102
- package/library/skills/ui-testing/references/headless-check.md +295 -295
- package/library/skills/ui-testing/references/local-stand.md +51 -51
- package/library/skills/ui-testing/references/playwright-patterns.md +134 -134
- package/library/skills/ui-testing/references/recording.md +61 -61
- package/library/skills/verification-before-completion/SKILL.md +113 -113
- package/library/skills/web-interface-guidelines/SKILL.md +202 -202
- package/package.json +46 -46
|
@@ -1,122 +1,122 @@
|
|
|
1
|
-
# False Positive Patterns: Lessons Learned
|
|
2
|
-
|
|
3
|
-
Apply ALL items in this checklist to EACH potential bug during verification.
|
|
4
|
-
|
|
5
|
-
Contents:
|
|
6
|
-
|
|
7
|
-
- [Checklist](#checklist): 13 items to apply to every bug (trace validation, confirm data paths, bounds logic, TOCTOU, trust boundaries, real vs theoretical impact, defense-in-depth)
|
|
8
|
-
- [Red Flags for False Positives](#red-flags-for-false-positives): recurring failure patterns (pattern-based, context-blind, math/bounds, API-contract)
|
|
9
|
-
|
|
10
|
-
## Checklist
|
|
11
|
-
|
|
12
|
-
### 1. Trace Full Validation Chain
|
|
13
|
-
|
|
14
|
-
Don't analyze isolated code snippets. Trace backwards to find ALL validation that precedes potentially dangerous operations. Network packet size operations may look dangerous but often have bounds validation earlier in the function.
|
|
15
|
-
|
|
16
|
-
### 1a. Map Complete Conditional Logic Flow
|
|
17
|
-
|
|
18
|
-
Vulnerable-looking code may be unreachable due to conditional logic that creates mathematical guarantees. Example: array access `buffer[length-4]` appears unsafe when `length < 4`, but if the code is only reachable when `length > 12` due to earlier validation, the vulnerability is impossible.
|
|
19
|
-
|
|
20
|
-
**Verify:**
|
|
21
|
-
|
|
22
|
-
- What conditions must be met for execution to reach the alleged vulnerability?
|
|
23
|
-
- Do those conditions mathematically prevent the vulnerability scenario?
|
|
24
|
-
- Are there minimum size/length requirements that guarantee safe access?
|
|
25
|
-
- Does the conditional flow create impossible-to-violate bounds?
|
|
26
|
-
|
|
27
|
-
### 2. Identify Defensive Programming Patterns
|
|
28
|
-
|
|
29
|
-
Distinguish between actual vulnerabilities and defensive assertions/validations. `ASSERT(size == expected_size)` followed by size-controlled operations is defensive, not vulnerable. Verify that checks actually prevent the alleged vulnerability.
|
|
30
|
-
|
|
31
|
-
### 3. Confirm Exploitable Data Paths
|
|
32
|
-
|
|
33
|
-
Only report vulnerabilities with CONFIRMED exploitable data flow paths. Don't assume network-controlled data reaches dangerous functions without tracing the actual path step by step.
|
|
34
|
-
|
|
35
|
-
### 4. Understand Data Source Context
|
|
36
|
-
|
|
37
|
-
Distinguish between data sources and their trust levels. API return values, compile-time constants, and network data have different risk profiles. Determine the actual source and whether it is attacker-controlled.
|
|
38
|
-
|
|
39
|
-
### 5. Analyze Bounds Validation Logic
|
|
40
|
-
|
|
41
|
-
Look for mathematical relationships between validation checks and subsequent operations. If `packet_size >= MIN_SIZE` is checked and `MIN_SIZE >= sizeof(header)`, then `packet_size - sizeof(header)` cannot underflow.
|
|
42
|
-
|
|
43
|
-
### 6. Verify TOCTOU Claims
|
|
44
|
-
|
|
45
|
-
Time-of-check-time-of-use issues require proof that the checked value can change between check and use. If a size is checked and immediately used in the same function with no external modification possible, there is no TOCTOU.
|
|
46
|
-
|
|
47
|
-
### 7. Understand API Contract and Trust Boundaries
|
|
48
|
-
|
|
49
|
-
Always understand API contracts before claiming buffer overflows. Some APIs have built-in bounds protection and cannot write beyond the buffer regardless of input parameters.
|
|
50
|
-
|
|
51
|
-
### 8. Distinguish Internal Storage from External Input
|
|
52
|
-
|
|
53
|
-
Internal storage systems (configuration stores, registries) are controlled by trusted components, not attackers. Values set during installation by trusted components are not attacker-controlled.
|
|
54
|
-
|
|
55
|
-
### 9. Don't Confuse Pattern Recognition with Vulnerability Analysis
|
|
56
|
-
|
|
57
|
-
Code patterns that "look vulnerable" may be safely implemented due to context and API contracts. Size parameters being modified doesn't mean buffer overflow if the API prevents writing beyond bounds.
|
|
58
|
-
|
|
59
|
-
### 10. Verify Concurrent Access is Actually Possible
|
|
60
|
-
|
|
61
|
-
Don't assume race conditions exist without proving concurrent access patterns. Single-threaded initialization contexts cannot have race conditions. Verify the threading model and synchronization mechanisms.
|
|
62
|
-
|
|
63
|
-
### 11. Assess Real vs Theoretical Security Impact
|
|
64
|
-
|
|
65
|
-
Focus on vulnerabilities with actual security impact. Storage failure for non-critical data is an operational issue, not a security vulnerability. Ask: would this lead to code execution, privilege escalation, or information disclosure?
|
|
66
|
-
|
|
67
|
-
### 12. Understand Defense-in-Depth vs Primary Controls
|
|
68
|
-
|
|
69
|
-
Failure of defense-in-depth mechanisms is not always a vulnerability if primary protections exist. Token cleanup failure is not critical if tokens are single-use by design at the server.
|
|
70
|
-
|
|
71
|
-
### 13. Apply the Checklist Rigorously, Not Superficially
|
|
72
|
-
|
|
73
|
-
Having a checklist doesn't prevent false positives if it isn't applied systematically. For EVERY potential vulnerability, work through ALL checklist items before concluding.
|
|
74
|
-
|
|
75
|
-
---
|
|
76
|
-
|
|
77
|
-
## Red Flags for False Positives
|
|
78
|
-
|
|
79
|
-
### Pattern-Based False Positives
|
|
80
|
-
|
|
81
|
-
- Reporting vulnerabilities in validation/bounds-checking code itself
|
|
82
|
-
- Claiming TOCTOU without proving the value can change
|
|
83
|
-
- Ignoring preceding validation logic
|
|
84
|
-
- Assuming network data reaches operations without tracing the path
|
|
85
|
-
- Confusing defensive programming (assertions/checks) with vulnerabilities
|
|
86
|
-
- Analyzing vulnerable-looking patterns without tracing conditional logic that controls reachability
|
|
87
|
-
- Reporting "vulnerabilities" in error handling or cleanup code
|
|
88
|
-
- Flagging size calculations without understanding mathematical constraints
|
|
89
|
-
- Identifying "dangerous" functions without checking if inputs are bounded
|
|
90
|
-
- Claiming buffer overflows in fixed-size operations with compile-time bounds
|
|
91
|
-
- Reporting race conditions in single-threaded or synchronized contexts
|
|
92
|
-
|
|
93
|
-
### Context-Blind Analysis False Positives
|
|
94
|
-
|
|
95
|
-
- Analyzing code snippets without understanding broader system design
|
|
96
|
-
- Ignoring architectural guarantees (single-writer, trusted input sources)
|
|
97
|
-
- Missing that "vulnerable" code is unreachable due to earlier validation
|
|
98
|
-
- Confusing debug/development code paths with production paths
|
|
99
|
-
- Reporting issues in code that only runs during trusted installation/setup
|
|
100
|
-
- Flagging theoretical issues that cannot occur due to system architecture
|
|
101
|
-
- Missing that alleged vulnerabilities are prevented by framework or language guarantees
|
|
102
|
-
- Reporting issues in test-only or debug-only code paths as production vulnerabilities
|
|
103
|
-
|
|
104
|
-
### Mathematical/Bounds Analysis False Positives
|
|
105
|
-
|
|
106
|
-
- Reporting integer underflow without proving the mathematical condition can occur
|
|
107
|
-
- Claiming buffer overflow when bounds are mathematically guaranteed by validation
|
|
108
|
-
- Missing that conditional logic creates mathematical impossibility of vulnerable conditions
|
|
109
|
-
- Reporting off-by-one errors without checking if loop bounds prevent the condition
|
|
110
|
-
- Claiming memory corruption when allocation sizes are verified sufficient
|
|
111
|
-
- Reporting arithmetic overflow without checking if input ranges prevent the condition
|
|
112
|
-
|
|
113
|
-
### API Contract Misunderstanding False Positives
|
|
114
|
-
|
|
115
|
-
- Claiming buffer overflows when APIs have built-in bounds checking
|
|
116
|
-
- Reporting memory corruption for APIs that manage their own memory safely
|
|
117
|
-
- Missing that return values are already validated by the API contract
|
|
118
|
-
- Confusing API parameter modification with vulnerability when API prevents unsafe modification
|
|
119
|
-
- Reporting issues explicitly handled by the API's safety guarantees
|
|
120
|
-
- Missing that seemingly dangerous operations are safe due to API implementation details
|
|
121
|
-
|
|
122
|
-
Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: punctuation and small wording changes for read-only runs.
|
|
1
|
+
# False Positive Patterns: Lessons Learned
|
|
2
|
+
|
|
3
|
+
Apply ALL items in this checklist to EACH potential bug during verification.
|
|
4
|
+
|
|
5
|
+
Contents:
|
|
6
|
+
|
|
7
|
+
- [Checklist](#checklist): 13 items to apply to every bug (trace validation, confirm data paths, bounds logic, TOCTOU, trust boundaries, real vs theoretical impact, defense-in-depth)
|
|
8
|
+
- [Red Flags for False Positives](#red-flags-for-false-positives): recurring failure patterns (pattern-based, context-blind, math/bounds, API-contract)
|
|
9
|
+
|
|
10
|
+
## Checklist
|
|
11
|
+
|
|
12
|
+
### 1. Trace Full Validation Chain
|
|
13
|
+
|
|
14
|
+
Don't analyze isolated code snippets. Trace backwards to find ALL validation that precedes potentially dangerous operations. Network packet size operations may look dangerous but often have bounds validation earlier in the function.
|
|
15
|
+
|
|
16
|
+
### 1a. Map Complete Conditional Logic Flow
|
|
17
|
+
|
|
18
|
+
Vulnerable-looking code may be unreachable due to conditional logic that creates mathematical guarantees. Example: array access `buffer[length-4]` appears unsafe when `length < 4`, but if the code is only reachable when `length > 12` due to earlier validation, the vulnerability is impossible.
|
|
19
|
+
|
|
20
|
+
**Verify:**
|
|
21
|
+
|
|
22
|
+
- What conditions must be met for execution to reach the alleged vulnerability?
|
|
23
|
+
- Do those conditions mathematically prevent the vulnerability scenario?
|
|
24
|
+
- Are there minimum size/length requirements that guarantee safe access?
|
|
25
|
+
- Does the conditional flow create impossible-to-violate bounds?
|
|
26
|
+
|
|
27
|
+
### 2. Identify Defensive Programming Patterns
|
|
28
|
+
|
|
29
|
+
Distinguish between actual vulnerabilities and defensive assertions/validations. `ASSERT(size == expected_size)` followed by size-controlled operations is defensive, not vulnerable. Verify that checks actually prevent the alleged vulnerability.
|
|
30
|
+
|
|
31
|
+
### 3. Confirm Exploitable Data Paths
|
|
32
|
+
|
|
33
|
+
Only report vulnerabilities with CONFIRMED exploitable data flow paths. Don't assume network-controlled data reaches dangerous functions without tracing the actual path step by step.
|
|
34
|
+
|
|
35
|
+
### 4. Understand Data Source Context
|
|
36
|
+
|
|
37
|
+
Distinguish between data sources and their trust levels. API return values, compile-time constants, and network data have different risk profiles. Determine the actual source and whether it is attacker-controlled.
|
|
38
|
+
|
|
39
|
+
### 5. Analyze Bounds Validation Logic
|
|
40
|
+
|
|
41
|
+
Look for mathematical relationships between validation checks and subsequent operations. If `packet_size >= MIN_SIZE` is checked and `MIN_SIZE >= sizeof(header)`, then `packet_size - sizeof(header)` cannot underflow.
|
|
42
|
+
|
|
43
|
+
### 6. Verify TOCTOU Claims
|
|
44
|
+
|
|
45
|
+
Time-of-check-time-of-use issues require proof that the checked value can change between check and use. If a size is checked and immediately used in the same function with no external modification possible, there is no TOCTOU.
|
|
46
|
+
|
|
47
|
+
### 7. Understand API Contract and Trust Boundaries
|
|
48
|
+
|
|
49
|
+
Always understand API contracts before claiming buffer overflows. Some APIs have built-in bounds protection and cannot write beyond the buffer regardless of input parameters.
|
|
50
|
+
|
|
51
|
+
### 8. Distinguish Internal Storage from External Input
|
|
52
|
+
|
|
53
|
+
Internal storage systems (configuration stores, registries) are controlled by trusted components, not attackers. Values set during installation by trusted components are not attacker-controlled.
|
|
54
|
+
|
|
55
|
+
### 9. Don't Confuse Pattern Recognition with Vulnerability Analysis
|
|
56
|
+
|
|
57
|
+
Code patterns that "look vulnerable" may be safely implemented due to context and API contracts. Size parameters being modified doesn't mean buffer overflow if the API prevents writing beyond bounds.
|
|
58
|
+
|
|
59
|
+
### 10. Verify Concurrent Access is Actually Possible
|
|
60
|
+
|
|
61
|
+
Don't assume race conditions exist without proving concurrent access patterns. Single-threaded initialization contexts cannot have race conditions. Verify the threading model and synchronization mechanisms.
|
|
62
|
+
|
|
63
|
+
### 11. Assess Real vs Theoretical Security Impact
|
|
64
|
+
|
|
65
|
+
Focus on vulnerabilities with actual security impact. Storage failure for non-critical data is an operational issue, not a security vulnerability. Ask: would this lead to code execution, privilege escalation, or information disclosure?
|
|
66
|
+
|
|
67
|
+
### 12. Understand Defense-in-Depth vs Primary Controls
|
|
68
|
+
|
|
69
|
+
Failure of defense-in-depth mechanisms is not always a vulnerability if primary protections exist. Token cleanup failure is not critical if tokens are single-use by design at the server.
|
|
70
|
+
|
|
71
|
+
### 13. Apply the Checklist Rigorously, Not Superficially
|
|
72
|
+
|
|
73
|
+
Having a checklist doesn't prevent false positives if it isn't applied systematically. For EVERY potential vulnerability, work through ALL checklist items before concluding.
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## Red Flags for False Positives
|
|
78
|
+
|
|
79
|
+
### Pattern-Based False Positives
|
|
80
|
+
|
|
81
|
+
- Reporting vulnerabilities in validation/bounds-checking code itself
|
|
82
|
+
- Claiming TOCTOU without proving the value can change
|
|
83
|
+
- Ignoring preceding validation logic
|
|
84
|
+
- Assuming network data reaches operations without tracing the path
|
|
85
|
+
- Confusing defensive programming (assertions/checks) with vulnerabilities
|
|
86
|
+
- Analyzing vulnerable-looking patterns without tracing conditional logic that controls reachability
|
|
87
|
+
- Reporting "vulnerabilities" in error handling or cleanup code
|
|
88
|
+
- Flagging size calculations without understanding mathematical constraints
|
|
89
|
+
- Identifying "dangerous" functions without checking if inputs are bounded
|
|
90
|
+
- Claiming buffer overflows in fixed-size operations with compile-time bounds
|
|
91
|
+
- Reporting race conditions in single-threaded or synchronized contexts
|
|
92
|
+
|
|
93
|
+
### Context-Blind Analysis False Positives
|
|
94
|
+
|
|
95
|
+
- Analyzing code snippets without understanding broader system design
|
|
96
|
+
- Ignoring architectural guarantees (single-writer, trusted input sources)
|
|
97
|
+
- Missing that "vulnerable" code is unreachable due to earlier validation
|
|
98
|
+
- Confusing debug/development code paths with production paths
|
|
99
|
+
- Reporting issues in code that only runs during trusted installation/setup
|
|
100
|
+
- Flagging theoretical issues that cannot occur due to system architecture
|
|
101
|
+
- Missing that alleged vulnerabilities are prevented by framework or language guarantees
|
|
102
|
+
- Reporting issues in test-only or debug-only code paths as production vulnerabilities
|
|
103
|
+
|
|
104
|
+
### Mathematical/Bounds Analysis False Positives
|
|
105
|
+
|
|
106
|
+
- Reporting integer underflow without proving the mathematical condition can occur
|
|
107
|
+
- Claiming buffer overflow when bounds are mathematically guaranteed by validation
|
|
108
|
+
- Missing that conditional logic creates mathematical impossibility of vulnerable conditions
|
|
109
|
+
- Reporting off-by-one errors without checking if loop bounds prevent the condition
|
|
110
|
+
- Claiming memory corruption when allocation sizes are verified sufficient
|
|
111
|
+
- Reporting arithmetic overflow without checking if input ranges prevent the condition
|
|
112
|
+
|
|
113
|
+
### API Contract Misunderstanding False Positives
|
|
114
|
+
|
|
115
|
+
- Claiming buffer overflows when APIs have built-in bounds checking
|
|
116
|
+
- Reporting memory corruption for APIs that manage their own memory safely
|
|
117
|
+
- Missing that return values are already validated by the API contract
|
|
118
|
+
- Confusing API parameter modification with vulnerability when API prevents unsafe modification
|
|
119
|
+
- Reporting issues explicitly handled by the API's safety guarantees
|
|
120
|
+
- Missing that seemingly dangerous operations are safe due to API implementation details
|
|
121
|
+
|
|
122
|
+
Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: punctuation and small wording changes for read-only runs.
|
|
@@ -1,36 +1,36 @@
|
|
|
1
|
-
# Gate Reviews and Verdicts
|
|
2
|
-
|
|
3
|
-
Before reporting ANY bug as a vulnerability, all six gate reviews must pass. Evaluate them at the end, after all phases are complete:
|
|
4
|
-
|
|
5
|
-
| Gate | Criterion | Pass | Fail |
|
|
6
|
-
|------|-----------|------|------|
|
|
7
|
-
| **1. Process** | All phases completed with documented evidence | Evidence exists for every phase | Phases lack concrete evidence |
|
|
8
|
-
| **2. Reachability** | Attacker can reach and control data at the vulnerability | Clear evidence of attacker-controlled path + PoC confirms | Cannot demonstrate attacker control or reachability |
|
|
9
|
-
| **3. Real Impact** | Exploitation leads to code execution, privilege escalation, information disclosure, unauthorized changes to data, or loss of money | Direct impact with concrete scenarios | Only operational robustness issue |
|
|
10
|
-
| **4. PoC Validation** | PoC (pseudocode, executable, or unit test) demonstrates the attack path | Shows attacker control, trigger, and impact | PoC fails to show attack path or impact |
|
|
11
|
-
| **5. Math Bounds** | Mathematical analysis confirms vulnerable condition is possible (N/A, counted as pass, when the bug is not about bounds or arithmetic) | Algebraic proof shows condition is possible | Math proves validation prevents it |
|
|
12
|
-
| **6. Environment** | No environmental protections entirely prevent exploitation | Protections do not eliminate vulnerability | Environmental protections block it entirely |
|
|
13
|
-
|
|
14
|
-
For a suspected correctness bug rather than a vulnerability: gate 2 asks whether a realistic input or state reaches the faulty code, and gate 3 whether the outcome is a wrong result, lost or corrupted data, or a crash that users hit.
|
|
15
|
-
|
|
16
|
-
## Verdict Format
|
|
17
|
-
|
|
18
|
-
- **TRUE POSITIVE**: All gate reviews pass → `BUG #N TRUE POSITIVE - [brief vulnerability description]`
|
|
19
|
-
- **FALSE POSITIVE**: Any gate review fails → `BUG #N FALSE POSITIVE - [brief reason for rejection]`
|
|
20
|
-
- **INCONCLUSIVE**: A gate cannot be decided from the code available (the caller is in another repository, the configuration is not in the repo) → `BUG #N NEEDS VERIFICATION - [the exact open question]`. Reported as a note, never as blocking.
|
|
21
|
-
|
|
22
|
-
If any phase fails verification, document the failure with evidence and continue all remaining phases. Issue the FALSE POSITIVE verdict only after all phases are complete.
|
|
23
|
-
|
|
24
|
-
## Example Verdict
|
|
25
|
-
|
|
26
|
-
```
|
|
27
|
-
BUG #3 FALSE POSITIVE - Integer underflow in packet_handler.c:142
|
|
28
|
-
Gate 5 (Math Bounds) FAIL: validation at line 98 ensures packet_size >= 16,
|
|
29
|
-
making (packet_size - header_size) >= 8. Underflow is mathematically impossible.
|
|
30
|
-
|
|
31
|
-
BUG #5 FALSE POSITIVE - IDOR in src/api/files.ts:19
|
|
32
|
-
Gate 2 (Reachability) FAIL: the route is mounted under accountRouter (src/app.ts:52),
|
|
33
|
-
whose middleware loads the file with the session's account id; other accounts' files return 404.
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: wider impact list for web applications, N/A rule for gate 5, correctness-bug reading of gates 2 and 3, inconclusive verdict, second example.
|
|
1
|
+
# Gate Reviews and Verdicts
|
|
2
|
+
|
|
3
|
+
Before reporting ANY bug as a vulnerability, all six gate reviews must pass. Evaluate them at the end, after all phases are complete:
|
|
4
|
+
|
|
5
|
+
| Gate | Criterion | Pass | Fail |
|
|
6
|
+
|------|-----------|------|------|
|
|
7
|
+
| **1. Process** | All phases completed with documented evidence | Evidence exists for every phase | Phases lack concrete evidence |
|
|
8
|
+
| **2. Reachability** | Attacker can reach and control data at the vulnerability | Clear evidence of attacker-controlled path + PoC confirms | Cannot demonstrate attacker control or reachability |
|
|
9
|
+
| **3. Real Impact** | Exploitation leads to code execution, privilege escalation, information disclosure, unauthorized changes to data, or loss of money | Direct impact with concrete scenarios | Only operational robustness issue |
|
|
10
|
+
| **4. PoC Validation** | PoC (pseudocode, executable, or unit test) demonstrates the attack path | Shows attacker control, trigger, and impact | PoC fails to show attack path or impact |
|
|
11
|
+
| **5. Math Bounds** | Mathematical analysis confirms vulnerable condition is possible (N/A, counted as pass, when the bug is not about bounds or arithmetic) | Algebraic proof shows condition is possible | Math proves validation prevents it |
|
|
12
|
+
| **6. Environment** | No environmental protections entirely prevent exploitation | Protections do not eliminate vulnerability | Environmental protections block it entirely |
|
|
13
|
+
|
|
14
|
+
For a suspected correctness bug rather than a vulnerability: gate 2 asks whether a realistic input or state reaches the faulty code, and gate 3 whether the outcome is a wrong result, lost or corrupted data, or a crash that users hit.
|
|
15
|
+
|
|
16
|
+
## Verdict Format
|
|
17
|
+
|
|
18
|
+
- **TRUE POSITIVE**: All gate reviews pass → `BUG #N TRUE POSITIVE - [brief vulnerability description]`
|
|
19
|
+
- **FALSE POSITIVE**: Any gate review fails → `BUG #N FALSE POSITIVE - [brief reason for rejection]`
|
|
20
|
+
- **INCONCLUSIVE**: A gate cannot be decided from the code available (the caller is in another repository, the configuration is not in the repo) → `BUG #N NEEDS VERIFICATION - [the exact open question]`. Reported as a note, never as blocking.
|
|
21
|
+
|
|
22
|
+
If any phase fails verification, document the failure with evidence and continue all remaining phases. Issue the FALSE POSITIVE verdict only after all phases are complete.
|
|
23
|
+
|
|
24
|
+
## Example Verdict
|
|
25
|
+
|
|
26
|
+
```
|
|
27
|
+
BUG #3 FALSE POSITIVE - Integer underflow in packet_handler.c:142
|
|
28
|
+
Gate 5 (Math Bounds) FAIL: validation at line 98 ensures packet_size >= 16,
|
|
29
|
+
making (packet_size - header_size) >= 8. Underflow is mathematically impossible.
|
|
30
|
+
|
|
31
|
+
BUG #5 FALSE POSITIVE - IDOR in src/api/files.ts:19
|
|
32
|
+
Gate 2 (Reachability) FAIL: the route is mounted under accountRouter (src/app.ts:52),
|
|
33
|
+
whose middleware loads the file with the session's account id; other accounts' files return 404.
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: wider impact list for web applications, N/A rule for gate 5, correctness-bug reading of gates 2 and 3, inconclusive verdict, second example.
|
|
@@ -1,80 +1,80 @@
|
|
|
1
|
-
# Standard Verification
|
|
2
|
-
|
|
3
|
-
Linear single-pass checklist for straightforward bugs. No task tracking: work through each step sequentially and document findings inline.
|
|
4
|
-
|
|
5
|
-
## Escalation Checkpoints
|
|
6
|
-
|
|
7
|
-
Two points in this checklist may trigger escalation to [deep-verification.md](deep-verification.md):
|
|
8
|
-
|
|
9
|
-
1. **After Step 1 (Data Flow)**: Escalate if 3+ trust boundaries, callbacks/async control flow, or ambiguous validation chain
|
|
10
|
-
2. **After Step 5 (Devil's Advocate)**: Escalate if any question produces genuine uncertainty you cannot resolve
|
|
11
|
-
|
|
12
|
-
When escalating, hand off all evidence gathered so far: deep verification will continue from where you left off.
|
|
13
|
-
|
|
14
|
-
## Checklist
|
|
15
|
-
|
|
16
|
-
### Step 1: Data Flow
|
|
17
|
-
|
|
18
|
-
Trace data from source to the alleged vulnerability sink.
|
|
19
|
-
|
|
20
|
-
- Map trust boundaries crossed (internal/trusted vs external/untrusted)
|
|
21
|
-
- Identify all validation and sanitization between source and sink
|
|
22
|
-
- Check API contracts: many APIs have built-in bounds protection that prevents the alleged issue
|
|
23
|
-
- Check for environmental protections (compiler, runtime, OS, framework) that prevent exploitation entirely (not just raise the bar)
|
|
24
|
-
- Apply class-specific checks from [bug-class-verification.md](bug-class-verification.md)
|
|
25
|
-
|
|
26
|
-
**Key pitfall**: Analyzing the vulnerable code in isolation. Conditional logic upstream may make the vulnerability mathematically unreachable. Trace the full validation chain.
|
|
27
|
-
|
|
28
|
-
**Escalation check**: If you found 3+ trust boundaries, callbacks or async control flow in the path, or an ambiguous validation chain: escalate to deep verification.
|
|
29
|
-
|
|
30
|
-
### Step 2: Exploitability
|
|
31
|
-
|
|
32
|
-
Prove the attacker can trigger the vulnerability.
|
|
33
|
-
|
|
34
|
-
- **Attacker control**: Prove the attacker controls data reaching the vulnerable operation. Internal storage set by trusted components is not attacker-controlled.
|
|
35
|
-
- **Bounds proof**: For integer/bounds issues, create an explicit algebraic proof using the template in [evidence-templates.md](evidence-templates.md). Verify: IF validation_check_passes THEN bounds_guarantee_holds.
|
|
36
|
-
- **Race feasibility**: For race conditions, prove concurrent access is actually possible. Single-threaded initialization and synchronized contexts cannot have races.
|
|
37
|
-
|
|
38
|
-
### Step 3: Impact
|
|
39
|
-
|
|
40
|
-
Determine whether exploitation has real security consequences.
|
|
41
|
-
|
|
42
|
-
- Distinguish real security impact (RCE, privesc, info disclosure) from operational robustness issues (crash recovery, cleanup failure)
|
|
43
|
-
- Distinguish primary security controls from defense-in-depth. Failure of a defense-in-depth measure is not a vulnerability if primary protections remain intact.
|
|
44
|
-
|
|
45
|
-
### Step 4: PoC Sketch
|
|
46
|
-
|
|
47
|
-
Create a pseudocode PoC showing the attack path. Executable and unit test PoCs are optional for standard verification; in review and test steps they run only from a temp directory outside the repository and never against live services (see the ground rules in SKILL.md).
|
|
48
|
-
|
|
49
|
-
```
|
|
50
|
-
Data Flow: [Source] → [Validation?] → [Transform?] → [Vulnerable Op] → [Impact]
|
|
51
|
-
Attacker controls: [what input, how]
|
|
52
|
-
Trigger: [pseudocode showing the exploit path]
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
See [evidence-templates.md](evidence-templates.md) for the full PoC template.
|
|
56
|
-
|
|
57
|
-
### Step 5: Devil's Advocate Spot-Check
|
|
58
|
-
|
|
59
|
-
Answer these 7 questions. If any produces genuine uncertainty, escalate to deep verification.
|
|
60
|
-
|
|
61
|
-
**Against the vulnerability:**
|
|
62
|
-
|
|
63
|
-
1. Am I seeing a vulnerability because the pattern "looks dangerous" rather than because it actually is? (pattern-matching bias)
|
|
64
|
-
2. Am I incorrectly assuming attacker control over trusted data? (trust boundary confusion)
|
|
65
|
-
3. Have I rigorously proven the mathematical condition for vulnerability can occur? (proof rigor)
|
|
66
|
-
4. Am I confusing defense-in-depth failure with a primary security vulnerability? (defense-in-depth confusion)
|
|
67
|
-
5. Am I hallucinating this vulnerability? LLMs are biased toward seeing bugs everywhere: is this actually real or am I pattern-matching on scary-looking code? (LLM self-check)
|
|
68
|
-
|
|
69
|
-
**For the vulnerability (always ask: false-negative protection):**
|
|
70
|
-
|
|
71
|
-
6. Am I dismissing a real vulnerability because the exploit seems complex or unlikely?
|
|
72
|
-
7. Am I inventing mitigations or validation logic that I haven't verified in the actual source code? Re-read the code after reaching a conclusion.
|
|
73
|
-
|
|
74
|
-
**Escalation check**: If any question above produces genuine uncertainty you cannot resolve with the evidence at hand: escalate to deep verification.
|
|
75
|
-
|
|
76
|
-
### Step 6: Gate Review
|
|
77
|
-
|
|
78
|
-
Apply all six gates from [gate-reviews.md](gate-reviews.md) and all 13 items from [false-positive-patterns.md](false-positive-patterns.md) to reach a verdict.
|
|
79
|
-
|
|
80
|
-
Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: read-only rule for proofs of concept, punctuation.
|
|
1
|
+
# Standard Verification
|
|
2
|
+
|
|
3
|
+
Linear single-pass checklist for straightforward bugs. No task tracking: work through each step sequentially and document findings inline.
|
|
4
|
+
|
|
5
|
+
## Escalation Checkpoints
|
|
6
|
+
|
|
7
|
+
Two points in this checklist may trigger escalation to [deep-verification.md](deep-verification.md):
|
|
8
|
+
|
|
9
|
+
1. **After Step 1 (Data Flow)**: Escalate if 3+ trust boundaries, callbacks/async control flow, or ambiguous validation chain
|
|
10
|
+
2. **After Step 5 (Devil's Advocate)**: Escalate if any question produces genuine uncertainty you cannot resolve
|
|
11
|
+
|
|
12
|
+
When escalating, hand off all evidence gathered so far: deep verification will continue from where you left off.
|
|
13
|
+
|
|
14
|
+
## Checklist
|
|
15
|
+
|
|
16
|
+
### Step 1: Data Flow
|
|
17
|
+
|
|
18
|
+
Trace data from source to the alleged vulnerability sink.
|
|
19
|
+
|
|
20
|
+
- Map trust boundaries crossed (internal/trusted vs external/untrusted)
|
|
21
|
+
- Identify all validation and sanitization between source and sink
|
|
22
|
+
- Check API contracts: many APIs have built-in bounds protection that prevents the alleged issue
|
|
23
|
+
- Check for environmental protections (compiler, runtime, OS, framework) that prevent exploitation entirely (not just raise the bar)
|
|
24
|
+
- Apply class-specific checks from [bug-class-verification.md](bug-class-verification.md)
|
|
25
|
+
|
|
26
|
+
**Key pitfall**: Analyzing the vulnerable code in isolation. Conditional logic upstream may make the vulnerability mathematically unreachable. Trace the full validation chain.
|
|
27
|
+
|
|
28
|
+
**Escalation check**: If you found 3+ trust boundaries, callbacks or async control flow in the path, or an ambiguous validation chain: escalate to deep verification.
|
|
29
|
+
|
|
30
|
+
### Step 2: Exploitability
|
|
31
|
+
|
|
32
|
+
Prove the attacker can trigger the vulnerability.
|
|
33
|
+
|
|
34
|
+
- **Attacker control**: Prove the attacker controls data reaching the vulnerable operation. Internal storage set by trusted components is not attacker-controlled.
|
|
35
|
+
- **Bounds proof**: For integer/bounds issues, create an explicit algebraic proof using the template in [evidence-templates.md](evidence-templates.md). Verify: IF validation_check_passes THEN bounds_guarantee_holds.
|
|
36
|
+
- **Race feasibility**: For race conditions, prove concurrent access is actually possible. Single-threaded initialization and synchronized contexts cannot have races.
|
|
37
|
+
|
|
38
|
+
### Step 3: Impact
|
|
39
|
+
|
|
40
|
+
Determine whether exploitation has real security consequences.
|
|
41
|
+
|
|
42
|
+
- Distinguish real security impact (RCE, privesc, info disclosure) from operational robustness issues (crash recovery, cleanup failure)
|
|
43
|
+
- Distinguish primary security controls from defense-in-depth. Failure of a defense-in-depth measure is not a vulnerability if primary protections remain intact.
|
|
44
|
+
|
|
45
|
+
### Step 4: PoC Sketch
|
|
46
|
+
|
|
47
|
+
Create a pseudocode PoC showing the attack path. Executable and unit test PoCs are optional for standard verification; in review and test steps they run only from a temp directory outside the repository and never against live services (see the ground rules in SKILL.md).
|
|
48
|
+
|
|
49
|
+
```
|
|
50
|
+
Data Flow: [Source] → [Validation?] → [Transform?] → [Vulnerable Op] → [Impact]
|
|
51
|
+
Attacker controls: [what input, how]
|
|
52
|
+
Trigger: [pseudocode showing the exploit path]
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
See [evidence-templates.md](evidence-templates.md) for the full PoC template.
|
|
56
|
+
|
|
57
|
+
### Step 5: Devil's Advocate Spot-Check
|
|
58
|
+
|
|
59
|
+
Answer these 7 questions. If any produces genuine uncertainty, escalate to deep verification.
|
|
60
|
+
|
|
61
|
+
**Against the vulnerability:**
|
|
62
|
+
|
|
63
|
+
1. Am I seeing a vulnerability because the pattern "looks dangerous" rather than because it actually is? (pattern-matching bias)
|
|
64
|
+
2. Am I incorrectly assuming attacker control over trusted data? (trust boundary confusion)
|
|
65
|
+
3. Have I rigorously proven the mathematical condition for vulnerability can occur? (proof rigor)
|
|
66
|
+
4. Am I confusing defense-in-depth failure with a primary security vulnerability? (defense-in-depth confusion)
|
|
67
|
+
5. Am I hallucinating this vulnerability? LLMs are biased toward seeing bugs everywhere: is this actually real or am I pattern-matching on scary-looking code? (LLM self-check)
|
|
68
|
+
|
|
69
|
+
**For the vulnerability (always ask: false-negative protection):**
|
|
70
|
+
|
|
71
|
+
6. Am I dismissing a real vulnerability because the exploit seems complex or unlikely?
|
|
72
|
+
7. Am I inventing mitigations or validation logic that I haven't verified in the actual source code? Re-read the code after reaching a conclusion.
|
|
73
|
+
|
|
74
|
+
**Escalation check**: If any question above produces genuine uncertainty you cannot resolve with the evidence at hand: escalate to deep verification.
|
|
75
|
+
|
|
76
|
+
### Step 6: Gate Review
|
|
77
|
+
|
|
78
|
+
Apply all six gates from [gate-reviews.md](gate-reviews.md) and all 13 items from [false-positive-patterns.md](false-positive-patterns.md) to reach a verdict.
|
|
79
|
+
|
|
80
|
+
Part of the `false-positive-check` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/fp-check (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: read-only rule for proofs of concept, punctuation.
|