@tamsensedev/runner 0.13.0 → 0.19.0

package/library/skills/differential-security-review/SKILL.md

@@ -1,105 +1,105 @@
----
-name: differential-security-review
-description: Use when a risky diff needs a deep security review against its base branch, such as auth or permission changes, removed validation, new external calls, money or data handling. Classifies risk per file, checks git history for regressions, counts callers for blast radius, checks tests on changed code, and builds concrete attack scenarios.
----
-
-# Differential security review
-
-Security review of a change, measured against what the code was before. Where `security-review` asks "is this code exploitable?", this skill adds "what did this change remove, whom does it affect, and has this bug been fixed here before?"
-
-Use it for pull requests and branches that touch authentication, authorization, cryptography, external calls, money or value transfer, or that remove validation. For a small change with no trust boundary, `security-review` alone is enough.
-
-## Core principles
-
-1. **Risk first.** Focus on auth, crypto, value transfer, external calls, validation.
-2. **Evidence based.** Every finding is backed by line numbers, git history and a concrete attack scenario.
-3. **Adaptive.** Scale depth to the codebase size and the risk.
-4. **Honest.** State what you covered, what you did not, and your confidence.
-5. **Read-only.** Never check out other commits, edit, commit, push or switch branches in the repository under review. Read old versions with `git show <ref>:<path>`. If you need a full old tree, add a detached worktree in a temp directory outside the repository (`git worktree add --detach <tmp> origin/<base>`) and remove it when done.
-
-## Rationalizations to reject
-
-| Rationalization | Why it is wrong | Required action |
-|---|---|---|
-| "Small PR, quick review" | Heartbleed was 2 lines | Classify by risk, not size |
-| "I know this codebase" | Familiarity breeds blind spots | Build explicit baseline context |
-| "Git history takes too long" | History reveals regressions | Never skip phase 1 |
-| "Blast radius is obvious" | You will miss transitive callers | Count them |
-| "No tests = not my problem" | Missing tests mean higher risk | Flag it, raise the risk rating |
-| "Just a refactor, no security impact" | Refactors break invariants | Treat as HIGH until proven LOW |
-| "I will just summarise" | Findings without evidence get lost | Write the full report into the verdict |
-
-## Quick reference
-
-**Codebase size strategy**
-
-| Codebase size | Strategy | Approach |
-|---|---|---|
-| Small (under 20 files) | Deep | Read all dependencies, full git blame |
-| Medium (20 to 200) | Focused | One hop of dependencies, priority files |
-| Large (over 200) | Surgical | Critical paths only |
-
-**Risk level per changed file**
-
-| Risk | Triggers |
-|---|---|
-| HIGH | Auth, crypto, external calls, value transfer, validation removal, permission changes |
-| MEDIUM | Business logic, state changes, new public APIs |
-| LOW | Comments, tests, UI text, logging |
-
-## Workflow
-
-```
-Phase 0: triage -> Phase 1: changed code -> Phase 2: test coverage -> Phase 3: blast radius
--> Phase 4: deep context (HIGH risk) -> Phase 5: adversarial analysis (HIGH risk) -> Phase 6: report
-```
-
-- Phases 0 to 4, with commands: [references/methodology.md](references/methodology.md). Read it at the start of every review.
-- Phase 5, attacker modelling and exploit scenarios: [references/adversarial.md](references/adversarial.md). Read it for every HIGH risk change.
-- Vulnerability patterns to check for in a diff: [references/patterns.md](references/patterns.md). Read it during phase 1.
-
-Before reporting any finding, verify it: apply the do-not-report rules of `security-review`, and run `false-positive-check` on each finding you would make blocking.
-
-## Red flags (stop and investigate)
-
-These need adversarial analysis even in a quick triage:
-
-- Code removed that came from commits mentioning "security", "CVE", "fix" or "vulnerability".
-- Access control removed or loosened: a guard, decorator, middleware, policy or role check deleted; a private function made public.
-- Validation removed without replacement.
-- External calls added without checks on the response.
-- High blast radius (more than 50 callers) combined with a HIGH risk change.
-
-## Phase 6: report
-
-The report goes into the verdict, not into a file in the repository. If a `submit_verdict` tool is available, that is how the result is returned: call it once, `passed: true` only if nothing blocks. Otherwise, or when you are a helper inside another step, put the report at the end of your reply. Sections:
-
-1. **Summary.** Count by severity, overall risk, recommendation: approve (passed) or reject (failed).
-2. **What changed.** Commit range, number of commits, a table of changed files with risk and blast radius.
-3. **Findings**, highest first. For each: severity, file and line, commit, blast radius (number of callers), test coverage (yes, no, partial), what is wrong, historical context (when and why the removed code was added), the attack scenario, and the fix.
-4. **Test coverage gaps** in changed code, with risk.
-5. **Blast radius** of HIGH risk functions.
-6. **Historical context:** security-related removals and regressions found.
-7. **Method and limits:** strategy used, files analysed versus total, techniques applied, what was not analysed, confidence.
-
-What blocks (passed is false): any confirmed vulnerability with a concrete attack path (Critical or High, or Medium with confidence 8 or more), a regression of an earlier security fix, removed validation or access control without an equivalent replacement. Missing tests, high blast radius without a vulnerability, and hardening suggestions are notes that raise the risk rating but do not fail the check on their own.
-
-Never put secret values in the report: mask them. Never post the report to GitHub.
-
-## Quality checklist
-
-- [ ] Every changed file was classified by risk and analysed at the depth its risk needs.
-- [ ] Git history was checked for every removal of security-relevant code.
-- [ ] Blast radius was counted for every HIGH risk change.
-- [ ] Attack scenarios are concrete: who, through which entry point, with which input, gaining what.
-- [ ] Findings cite file, line and commit.
-- [ ] Coverage limits and confidence are stated.
-- [ ] The repository is exactly as you found it (no checkouts, no temp files, worktree removed).
-- [ ] The verdict was returned once.
-
-## When not to use
-
-- Greenfield code with no baseline to compare against: use `security-review`.
-- Documentation-only or formatting-only changes.
-
-Adapted from https://github.com/trailofbits/skills/tree/main/plugins/differential-review (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), by Trail of Bits (Omar Inuwa), CC BY-SA 4.0. Changed: rewritten for unattended read-only runs (no checkout of other commits, report returned through the verdict instead of a file, no user notification, no external skills or agents), added verdict mapping, removed time estimates and emoji, generalised examples beyond smart contracts. This skill folder, including its references, is licensed under CC BY-SA 4.0; see LICENSE.
+---
+name: differential-security-review
+description: Use when a risky diff needs a deep security review against its base branch, such as auth or permission changes, removed validation, new external calls, money or data handling. Classifies risk per file, checks git history for regressions, counts callers for blast radius, checks tests on changed code, and builds concrete attack scenarios.
+---
+
+# Differential security review
+
+Security review of a change, measured against what the code was before. Where `security-review` asks "is this code exploitable?", this skill adds "what did this change remove, whom does it affect, and has this bug been fixed here before?"
+
+Use it for pull requests and branches that touch authentication, authorization, cryptography, external calls, money or value transfer, or that remove validation. For a small change with no trust boundary, `security-review` alone is enough.
+
+## Core principles
+
+1. **Risk first.** Focus on auth, crypto, value transfer, external calls, validation.
+2. **Evidence based.** Every finding is backed by line numbers, git history and a concrete attack scenario.
+3. **Adaptive.** Scale depth to the codebase size and the risk.
+4. **Honest.** State what you covered, what you did not, and your confidence.
+5. **Read-only.** Never check out other commits, edit, commit, push or switch branches in the repository under review. Read old versions with `git show <ref>:<path>`. If you need a full old tree, add a detached worktree in a temp directory outside the repository (`git worktree add --detach <tmp> origin/<base>`) and remove it when done.
+
+## Rationalizations to reject
+
+| Rationalization | Why it is wrong | Required action |
+|---|---|---|
+| "Small PR, quick review" | Heartbleed was 2 lines | Classify by risk, not size |
+| "I know this codebase" | Familiarity breeds blind spots | Build explicit baseline context |
+| "Git history takes too long" | History reveals regressions | Never skip phase 1 |
+| "Blast radius is obvious" | You will miss transitive callers | Count them |
+| "No tests = not my problem" | Missing tests mean higher risk | Flag it, raise the risk rating |
+| "Just a refactor, no security impact" | Refactors break invariants | Treat as HIGH until proven LOW |
+| "I will just summarise" | Findings without evidence get lost | Write the full report into the verdict |
+
+## Quick reference
+
+**Codebase size strategy**
+
+| Codebase size | Strategy | Approach |
+|---|---|---|
+| Small (under 20 files) | Deep | Read all dependencies, full git blame |
+| Medium (20 to 200) | Focused | One hop of dependencies, priority files |
+| Large (over 200) | Surgical | Critical paths only |
+
+**Risk level per changed file**
+
+| Risk | Triggers |
+|---|---|
+| HIGH | Auth, crypto, external calls, value transfer, validation removal, permission changes |
+| MEDIUM | Business logic, state changes, new public APIs |
+| LOW | Comments, tests, UI text, logging |
+
+## Workflow
+
+```
+Phase 0: triage -> Phase 1: changed code -> Phase 2: test coverage -> Phase 3: blast radius
+-> Phase 4: deep context (HIGH risk) -> Phase 5: adversarial analysis (HIGH risk) -> Phase 6: report
+```
+
+- Phases 0 to 4, with commands: [references/methodology.md](references/methodology.md). Read it at the start of every review.
+- Phase 5, attacker modelling and exploit scenarios: [references/adversarial.md](references/adversarial.md). Read it for every HIGH risk change.
+- Vulnerability patterns to check for in a diff: [references/patterns.md](references/patterns.md). Read it during phase 1.
+
+Before reporting any finding, verify it: apply the do-not-report rules of `security-review`, and run `false-positive-check` on each finding you would make blocking.
+
+## Red flags (stop and investigate)
+
+These need adversarial analysis even in a quick triage:
+
+- Code removed that came from commits mentioning "security", "CVE", "fix" or "vulnerability".
+- Access control removed or loosened: a guard, decorator, middleware, policy or role check deleted; a private function made public.
+- Validation removed without replacement.
+- External calls added without checks on the response.
+- High blast radius (more than 50 callers) combined with a HIGH risk change.
+
+## Phase 6: report
+
+The report goes into the verdict, not into a file in the repository. If a `submit_verdict` tool is available, that is how the result is returned: call it once, `passed: true` only if nothing blocks. Otherwise, or when you are a helper inside another step, put the report at the end of your reply. Sections:
+
+1. **Summary.** Count by severity, overall risk, recommendation: approve (passed) or reject (failed).
+2. **What changed.** Commit range, number of commits, a table of changed files with risk and blast radius.
+3. **Findings**, highest first. For each: severity, file and line, commit, blast radius (number of callers), test coverage (yes, no, partial), what is wrong, historical context (when and why the removed code was added), the attack scenario, and the fix.
+4. **Test coverage gaps** in changed code, with risk.
+5. **Blast radius** of HIGH risk functions.
+6. **Historical context:** security-related removals and regressions found.
+7. **Method and limits:** strategy used, files analysed versus total, techniques applied, what was not analysed, confidence.
+
+What blocks (passed is false): any confirmed vulnerability with a concrete attack path (Critical or High, or Medium with confidence 8 or more), a regression of an earlier security fix, removed validation or access control without an equivalent replacement. Missing tests, high blast radius without a vulnerability, and hardening suggestions are notes that raise the risk rating but do not fail the check on their own.
+
+Never put secret values in the report: mask them. Never post the report to GitHub.
+
+## Quality checklist
+
+- [ ] Every changed file was classified by risk and analysed at the depth its risk needs.
+- [ ] Git history was checked for every removal of security-relevant code.
+- [ ] Blast radius was counted for every HIGH risk change.
+- [ ] Attack scenarios are concrete: who, through which entry point, with which input, gaining what.
+- [ ] Findings cite file, line and commit.
+- [ ] Coverage limits and confidence are stated.
+- [ ] The repository is exactly as you found it (no checkouts, no temp files, worktree removed).
+- [ ] The verdict was returned once.
+
+## When not to use
+
+- Greenfield code with no baseline to compare against: use `security-review`.
+- Documentation-only or formatting-only changes.
+
+Adapted from https://github.com/trailofbits/skills/tree/main/plugins/differential-review (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), by Trail of Bits (Omar Inuwa), CC BY-SA 4.0. Changed: rewritten for unattended read-only runs (no checkout of other commits, report returned through the verdict instead of a file, no user notification, no external skills or agents), added verdict mapping, removed time estimates and emoji, generalised examples beyond smart contracts. This skill folder, including its references, is licensed under CC BY-SA 4.0; see LICENSE.

package/library/skills/differential-security-review/references/adversarial.md

@@ -1,115 +1,115 @@
-# Adversarial analysis (phase 5)
-
-Read after phase 4 for every HIGH risk change. The rule: concrete impact only, never "could cause issues". Every finding names the data exposed, the privilege gained, the money at risk or the invariant broken.
-
-Everything here is analysis on paper and reading code. Do not send requests to live services, and do not write files into the repository under review.
-
-## 1. Define a specific attacker model
-
-**Who is the attacker?**
-- Unauthenticated visitor
-- Authenticated regular user
-- User of another tenant or organisation
-- Malicious or compromised administrator
-- Compromised upstream service, webhook sender or dependency
-
-**What access do they have?**
-- Public API only
-- An authenticated role
-- Specific permissions or tokens
-- Ability to send webhooks, queue messages or files
-
-**Where do they interact with the system?**
-- Specific HTTP endpoints, pages or forms
-- RPC, GraphQL or WebSocket interfaces
-- Jobs triggered by their input
-- External APIs the system calls on their behalf
-
-## 2. Identify concrete attack vectors
-
-```
-ENTRY POINT: <exact endpoint or function the attacker can reach>
-
-ATTACK SEQUENCE:
-1. <specific request with parameters>
-2. <how it reaches the vulnerable code, file:line>
-3. <what happens in the vulnerable code>
-4. <impact achieved>
-
-PROOF OF ACCESSIBILITY:
-- The entry point is exposed (route registered, no guard in front)
-- The attacker has the required permissions
-- The path exists through real interfaces, not only internal calls
-```
-
-## 3. Rate realistic exploitability
-
-**EASY:** through public or ordinary user APIs, no special privileges, a single request, no special conditions.
-
-**MEDIUM:** needs specific conditions or elevated but obtainable privileges: several steps, timing, a particular system state.
-
-**HARD:** needs admin privileges, rare conditions or significant resources.
-
-## 4. Build the complete exploit scenario
-
-```
-ATTACKER STARTING POSITION:
-<what the attacker has at the beginning>
-
-STEP-BY-STEP EXPLOITATION:
-Step 1: <concrete action through an accessible interface>
-  - Request: <exact call>
-  - Parameters: <specific values>
-  - Result: <what happens>
-Step 2: <next action>
-  - Why this works: <reference to the code change>
-  - State change: <what changed>
-Step 3: <final impact>
-  - Result: <concrete harm>
-  - Evidence: <how to verify it from the code or a test>
-
-CONCRETE IMPACT:
-<specific and measurable: which records exposed, which privilege gained, how much money>
-```
-
-## 5. Cross-reference with the baseline
-
-From the baseline context in [methodology.md](methodology.md):
-
-- Does this violate a system-wide invariant?
-- Does it break a trust boundary?
-- Does it bypass a validation pattern used elsewhere?
-- Is it a regression of an earlier fix?
-
-## Finding template
-
-```
-[SEVERITY] <title>
-
-Attacker model:
-- WHO: <attacker type>
-- ACCESS: <privileges>
-- INTERFACE: <entry point>
-
-Attack vector: <step-by-step through accessible interfaces>
-Exploitability: EASY | MEDIUM | HARD, because <reason>
-Concrete impact: <specific harm>
-Proof of concept: <pseudocode or request sequence>
-Root cause: <file:line, commit>
-Blast radius: <N callers>
-Baseline violation: <invariant or pattern broken>
-```
-
-## Example
-
-**Change:** a refactor of the invoice routes moved handlers into a new router file. The old router applied `requireAccountMember` to the whole `/accounts/:accountId/*` group; the new file registers `GET /invoices/:id` without it and loads the invoice by id alone.
-
-1. **Attacker model.** WHO: any authenticated user. ACCESS: an ordinary account. INTERFACE: `GET /invoices/:id`.
-2. **Attack vector.** ENTRY POINT: `GET /invoices/:id`. SEQUENCE: log in with any account; request `/invoices/1001`, `/invoices/1002`, and so on; the handler calls `invoiceRepo.findById(id)` (src/invoices/routes.ts:24) with no account filter; the full invoice is returned. PROOF: the route is registered in src/app.ts:40 behind `requireLogin` only.
-3. **Exploitability.** EASY: one request per invoice, ordinary user, ids are sequential integers.
-4. **Impact.** Every invoice of every customer can be read: names, addresses, amounts, tax ids.
-5. **Baseline violation.** Breaks the invariant "a user only sees data of accounts they belong to". `requireAccountMember` guards every other account resource (12 routes). Git history: the guard was added in commit 4f2a9c1, "Fix: invoices readable across accounts". This is a regression.
-
-Recommendation: register the route inside the account-scoped router, or add the guard and filter the query by account id; add a test that a member of account A gets 404 for an invoice of account B.
-
-Part of the `differential-security-review` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/differential-review (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: attacker types and example rewritten for web applications, no live testing, no separate agent.
+# Adversarial analysis (phase 5)
+
+Read after phase 4 for every HIGH risk change. The rule: concrete impact only, never "could cause issues". Every finding names the data exposed, the privilege gained, the money at risk or the invariant broken.
+
+Everything here is analysis on paper and reading code. Do not send requests to live services, and do not write files into the repository under review.
+
+## 1. Define a specific attacker model
+
+**Who is the attacker?**
+- Unauthenticated visitor
+- Authenticated regular user
+- User of another tenant or organisation
+- Malicious or compromised administrator
+- Compromised upstream service, webhook sender or dependency
+
+**What access do they have?**
+- Public API only
+- An authenticated role
+- Specific permissions or tokens
+- Ability to send webhooks, queue messages or files
+
+**Where do they interact with the system?**
+- Specific HTTP endpoints, pages or forms
+- RPC, GraphQL or WebSocket interfaces
+- Jobs triggered by their input
+- External APIs the system calls on their behalf
+
+## 2. Identify concrete attack vectors
+
+```
+ENTRY POINT: <exact endpoint or function the attacker can reach>
+
+ATTACK SEQUENCE:
+1. <specific request with parameters>
+2. <how it reaches the vulnerable code, file:line>
+3. <what happens in the vulnerable code>
+4. <impact achieved>
+
+PROOF OF ACCESSIBILITY:
+- The entry point is exposed (route registered, no guard in front)
+- The attacker has the required permissions
+- The path exists through real interfaces, not only internal calls
+```
+
+## 3. Rate realistic exploitability
+
+**EASY:** through public or ordinary user APIs, no special privileges, a single request, no special conditions.
+
+**MEDIUM:** needs specific conditions or elevated but obtainable privileges: several steps, timing, a particular system state.
+
+**HARD:** needs admin privileges, rare conditions or significant resources.
+
+## 4. Build the complete exploit scenario
+
+```
+ATTACKER STARTING POSITION:
+<what the attacker has at the beginning>
+
+STEP-BY-STEP EXPLOITATION:
+Step 1: <concrete action through an accessible interface>
+  - Request: <exact call>
+  - Parameters: <specific values>
+  - Result: <what happens>
+Step 2: <next action>
+  - Why this works: <reference to the code change>
+  - State change: <what changed>
+Step 3: <final impact>
+  - Result: <concrete harm>
+  - Evidence: <how to verify it from the code or a test>
+
+CONCRETE IMPACT:
+<specific and measurable: which records exposed, which privilege gained, how much money>
+```
+
+## 5. Cross-reference with the baseline
+
+From the baseline context in [methodology.md](methodology.md):
+
+- Does this violate a system-wide invariant?
+- Does it break a trust boundary?
+- Does it bypass a validation pattern used elsewhere?
+- Is it a regression of an earlier fix?
+
+## Finding template
+
+```
+[SEVERITY] <title>
+
+Attacker model:
+- WHO: <attacker type>
+- ACCESS: <privileges>
+- INTERFACE: <entry point>
+
+Attack vector: <step-by-step through accessible interfaces>
+Exploitability: EASY | MEDIUM | HARD, because <reason>
+Concrete impact: <specific harm>
+Proof of concept: <pseudocode or request sequence>
+Root cause: <file:line, commit>
+Blast radius: <N callers>
+Baseline violation: <invariant or pattern broken>
+```
+
+## Example
+
+**Change:** a refactor of the invoice routes moved handlers into a new router file. The old router applied `requireAccountMember` to the whole `/accounts/:accountId/*` group; the new file registers `GET /invoices/:id` without it and loads the invoice by id alone.
+
+1. **Attacker model.** WHO: any authenticated user. ACCESS: an ordinary account. INTERFACE: `GET /invoices/:id`.
+2. **Attack vector.** ENTRY POINT: `GET /invoices/:id`. SEQUENCE: log in with any account; request `/invoices/1001`, `/invoices/1002`, and so on; the handler calls `invoiceRepo.findById(id)` (src/invoices/routes.ts:24) with no account filter; the full invoice is returned. PROOF: the route is registered in src/app.ts:40 behind `requireLogin` only.
+3. **Exploitability.** EASY: one request per invoice, ordinary user, ids are sequential integers.
+4. **Impact.** Every invoice of every customer can be read: names, addresses, amounts, tax ids.
+5. **Baseline violation.** Breaks the invariant "a user only sees data of accounts they belong to". `requireAccountMember` guards every other account resource (12 routes). Git history: the guard was added in commit 4f2a9c1, "Fix: invoices readable across accounts". This is a regression.
+
+Recommendation: register the route inside the account-scoped router, or add the guard and filter the query by account id; add a test that a member of account A gets 404 for an invoice of account B.
+
+Part of the `differential-security-review` skill, adapted from https://github.com/trailofbits/skills/tree/main/plugins/differential-review (commit 5c99c03fb8a9ea568432ecbcba5d86e9f4ac2bf0), CC BY-SA 4.0. Changed: attacker types and example rewritten for web applications, no live testing, no separate agent.