bug-bounty-intelligence-mcp 1.0.5 → 1.1.0

package/BENCHMARK.md

@@ -1,174 +1,174 @@
-# Benchmark: Bug Bounty Intelligence vs Slither
-
-**Target protocol:** 3FLabs/grunt — leveraged strategies for on-chain funds  
-**Protocol type:** ERC-4626/ERC-7540 vaults + Morpho lending + flash loan requests  
-**Scope:** `src/` (218 Solidity files)  
-**Slither version:** 0.10.x  
-**Date:** 2026-07-20
-
----
-
-## TL;DR
-
-| Metric | Slither | Bug Bounty Intelligence |
-|--------|---------|------------------------|
-| "High" findings reported | 27 | 0 |
-| True positives (real bugs) | 0 | 0 |
-| False positives | 27 (100%) | 0 |
-| Out-of-scope library noise | 24 (89%) | 0 |
-| Signals requiring human review | n/a | 17 |
-| Reading time to triage all "Highs" | ~4 hours | ~15 min |
-
-**Slither false positive rate on this codebase: 100%.**
-
----
-
-## Test setup
-
-We ran Slither on the full repo (including `lib/`), and Bug Bounty Intelligence on `src/` only. The scope for any real audit is `src/` — dependencies in `lib/` are battle-tested third-party code that has its own audit history.
-
----
-
-## Slither's 27 "High" findings — breakdown
-
-### Group A: 24 findings in `lib/solady/` (out of scope)
-
-Slither flagged 24 High-severity issues, all in the `lib/solady/` dependency:
-
-- 5x `incorrect-exp` — Solady uses XOR (`^`) for bitwise operations, not exponentiation. Slither confuses the two.
-- 19x `incorrect-shift` — Solady uses `<<` / `>>` inside assembly blocks. Slither's detector fires on these but they are intentional and correct.
-
-**Why these are not bugs:**
-
-Solady is one of the most audited Solidity libraries in existence, used by hundreds of protocols. Its math implementations deliberately use low-level bit operations for gas efficiency. Slither's `incorrect-exp` and `incorrect-shift` detectors are known to produce false positives on Solady. Every human auditor knows to ignore these. Slither does not.
-
-**Example:**
-
-```
-incorrect-shift | FixedPointMathLib.lnWad(int256) (lib/solady/src/utils/FixedPointMathLib.sol#269-341)
-contains an incorrect shift operation: ...
-```
-
-This is in `lib/solady/`, a third-party library. It is out of scope for any 3FLabs audit.
-
-**Update:** while triaging these, we noticed a related pattern on the same `lib/solady` code:
-`divide-before-multiply` fired 38 times on `FixedPointMathLib`'s `fullMulDivUnchecked`, `cbrt`,
-`invMod`, `rpow`, and `cbrtWad`. We initially described all 38 as the same false-positive class
-(divided variable fully reassigned before the multiply) and reported it upstream —
-[crytic/slither#3039](https://github.com/crytic/slither/issues/3039). That was an overclaim on our
-part. A maintainer merged a fix for the specific reassignment pattern we described
-([crytic/slither#3040](https://github.com/crytic/slither/pull/3040), not yet in a PyPI release as
-of 2026-08-01), and re-testing against the merged code ourselves shows it correctly resolves 11 of
-the 38 (down to 27 remaining on this file). The other 27 keep the divided variable in the same
-expression rather than through a clean reassignment, which is a different pattern this fix
-correctly leaves untouched — we haven't yet re-analyzed whether those are genuine false positives
-too or something else. Filed as a lesson: verify a fix against the actual code before describing
-it as resolving the full scope of what you reported.
-
-### Group B: 3 findings in `src/` — all false positives
-
-Slither flagged 3 uses of `safeTransferFrom(offer.maker, ...)` as "arbitrary from in transferFrom":
-
-```
-arbitrary-send-erc20 | Request.consume() uses arbitrary from in transferFrom:
-  _asset().safeTransferFrom(offer.maker, address(this), ptAmount)
-```
-
-**Why this is not a bug:**
-
-`offer.maker` is not arbitrary. The offer is validated via EIP-712 signature before the transfer (`_validateOffer(offer, signature)` is called on line 421). The maker explicitly signed the offer, authorizing exactly this transfer of exactly this amount. Additionally, `consume()` is gated by `onlyOwnerOrRoles(_ROLE_CONSUMER)` — only trusted consumers can call it.
-
-This is a correct and standard EIP-712 off-chain order pattern. Slither has no understanding of EIP-712 and cannot reason about signature-based authorization.
-
----
-
-## What Bug Bounty Intelligence flags instead
-
-After filtering to `src/` and applying Al-Mizaan validation gates, we identified **17 signals requiring human review** — none rising to a confirmed finding:
-
-| Category | Count | Description |
-|----------|-------|-------------|
-| `unsafe_keyword` | 15 | Explicit `forge-lint: disable-next-line(unsafe-typecast)` annotations — developer acknowledged unsafe casts. Require review for truncation risk. |
-| `delegatecall` | 2 | `MorphoFlashLoanRequest.execute()` delegatecalls to whitelisted scripts. By design — whitelist check present. |
-
-**What we do not flag:**
-- Library code in `lib/` (out of scope)
-- EIP-712 signed transfers (intentional pattern, not a vulnerability)
-- Standard Solady bit operations (well-audited library behavior)
-
----
-
-## Al-Mizaan filtering — why the 17 signals do not become findings
-
-The 15 `unsafe_keyword` signals (explicit forge-lint disables) were put through Al-Mizaan's reachability chain:
-
-1. **Code** — read the actual casts in `LibTokenController.sol`, `TokenController.sol`, and `PositionManagerLP.sol`
-2. **Reachability** — are the truncated values reachable by an untrusted caller?
-3. **Threat model** — who triggers these paths? In all cases, the entry points are role-gated (owner, ROLE_CONSUMER, ROLE_MANAGER)
-4. **Invariant** — does the truncation break any accounting invariant?
-
-Verdict: The typecasts are `uint256 → uint128` in share accounting. The values come from internal calculations that are capped by deposit limits. Overflow requires either extremely large TVL or a role-gated actor. **Trusted actor as sole trigger → ELIMINATE.**
-
-The 2 `delegatecall` signals: `MorphoFlashLoanRequest` maintains an explicit `isScriptWhitelisted` mapping. Only the owner can whitelist. **Trusted actor (owner) as sole attacker → ELIMINATE.**
-
-**Result: 0 confirmed findings.** This matches the expected difficulty level of a mature protocol with prior audits.
-
----
-
-## Why Slither's false positive rate matters
-
-A 100% false positive rate on High findings is not an edge case. It happens because:
-
-1. **No scope awareness** — Slither scans everything, including dependencies. A real audit scopes to `src/` only.
-2. **No protocol context** — Slither cannot distinguish between an arbitrary `transferFrom` and an EIP-712-authorized one.
-3. **No library knowledge** — Solady's assembly patterns trigger false positives on multiple detectors.
-
-For a developer running Slither for the first time, triaging 27 "High" findings (all false positives) takes hours. For a developer using Bug Bounty Intelligence, the 17 signals that remain after scope filtering take 15 minutes to review.
-
----
-
-## When Slither IS useful
-
-Slither is best used as a first-pass tool before human review, with explicit exclusions:
-
-```bash
-slither src/ --exclude incorrect-exp,incorrect-shift --filter-paths "lib/"
-```
-
-After filtering, Slither's Medium and Low findings become more useful starting points.
-
-Bug Bounty Intelligence is designed to do this filtering for you, and adds protocol-context reasoning that static analysis cannot.
-
----
-
-## Reproduce this benchmark
-
-The 3FLabs/grunt codebase is public at [github.com/3FLabs/grunt](https://github.com/3FLabs/grunt).
-
-To reproduce Slither's output:
-```bash
-git clone https://github.com/3FLabs/grunt
-cd grunt
-pip install slither-analyzer
-slither . --json slither-output.json
-```
-
-To use Bug Bounty Intelligence:
-```bash
-npx -y bug-bounty-intelligence-mcp@latest
-```
-
-Then in your MCP client:
-```
-scan_contract({ repo_url: "https://github.com/3FLabs/grunt", protocol_type: "LENDING" })
-```
-
----
-
-## About Bug Bounty Intelligence
-
-Pattern-aware from a corpus of 27,681 submitted Sherlock/Code4rena findings across 105 contests; acceptance-rate numbers (used by `list_vulnerability_patterns`) are limited to the 1,032 findings across 10 contests that could be exact-reconciled against contest outcomes (461 accepted) — see METHODOLOGY.md. Uses the Al-Mizaan v3 framework — a 7-gate validation chain (Code, Reachability, Threat-Model, Invariant, Protocol-Intent, Impact, Proof) — to filter signals into confirmed findings.
-
-**Cost:** $5 USDC on Base (eip155:8453) via x402.  
-**Install:** `npx -y bug-bounty-intelligence-mcp@latest`  
-**npm:** [npmjs.com/package/bug-bounty-intelligence-mcp](https://www.npmjs.com/package/bug-bounty-intelligence-mcp)
+# Benchmark: Bug Bounty Intelligence vs Slither
+
+**Target protocol:** 3FLabs/grunt — leveraged strategies for on-chain funds  
+**Protocol type:** ERC-4626/ERC-7540 vaults + Morpho lending + flash loan requests  
+**Scope:** `src/` (218 Solidity files)  
+**Slither version:** 0.10.x  
+**Date:** 2026-07-20
+
+---
+
+## TL;DR
+
+| Metric | Slither | Bug Bounty Intelligence |
+|--------|---------|------------------------|
+| "High" findings reported | 27 | 0 |
+| True positives (real bugs) | 0 | 0 |
+| False positives | 27 (100%) | 0 |
+| Out-of-scope library noise | 24 (89%) | 0 |
+| Signals requiring human review | n/a | 17 |
+| Reading time to triage all "Highs" | ~4 hours | ~15 min |
+
+**Slither false positive rate on this codebase: 100%.**
+
+---
+
+## Test setup
+
+We ran Slither on the full repo (including `lib/`), and Bug Bounty Intelligence on `src/` only. The scope for any real audit is `src/` — dependencies in `lib/` are battle-tested third-party code that has its own audit history.
+
+---
+
+## Slither's 27 "High" findings — breakdown
+
+### Group A: 24 findings in `lib/solady/` (out of scope)
+
+Slither flagged 24 High-severity issues, all in the `lib/solady/` dependency:
+
+- 5x `incorrect-exp` — Solady uses XOR (`^`) for bitwise operations, not exponentiation. Slither confuses the two.
+- 19x `incorrect-shift` — Solady uses `<<` / `>>` inside assembly blocks. Slither's detector fires on these but they are intentional and correct.
+
+**Why these are not bugs:**
+
+Solady is one of the most audited Solidity libraries in existence, used by hundreds of protocols. Its math implementations deliberately use low-level bit operations for gas efficiency. Slither's `incorrect-exp` and `incorrect-shift` detectors are known to produce false positives on Solady. Every human auditor knows to ignore these. Slither does not.
+
+**Example:**
+
+```
+incorrect-shift | FixedPointMathLib.lnWad(int256) (lib/solady/src/utils/FixedPointMathLib.sol#269-341)
+contains an incorrect shift operation: ...
+```
+
+This is in `lib/solady/`, a third-party library. It is out of scope for any 3FLabs audit.
+
+**Update:** while triaging these, we noticed a related pattern on the same `lib/solady` code:
+`divide-before-multiply` fired 38 times on `FixedPointMathLib`'s `fullMulDivUnchecked`, `cbrt`,
+`invMod`, `rpow`, and `cbrtWad`. We initially described all 38 as the same false-positive class
+(divided variable fully reassigned before the multiply) and reported it upstream —
+[crytic/slither#3039](https://github.com/crytic/slither/issues/3039). That was an overclaim on our
+part. A maintainer merged a fix for the specific reassignment pattern we described
+([crytic/slither#3040](https://github.com/crytic/slither/pull/3040), not yet in a PyPI release as
+of 2026-08-01), and re-testing against the merged code ourselves shows it correctly resolves 11 of
+the 38 (down to 27 remaining on this file). The other 27 keep the divided variable in the same
+expression rather than through a clean reassignment, which is a different pattern this fix
+correctly leaves untouched — we haven't yet re-analyzed whether those are genuine false positives
+too or something else. Filed as a lesson: verify a fix against the actual code before describing
+it as resolving the full scope of what you reported.
+
+### Group B: 3 findings in `src/` — all false positives
+
+Slither flagged 3 uses of `safeTransferFrom(offer.maker, ...)` as "arbitrary from in transferFrom":
+
+```
+arbitrary-send-erc20 | Request.consume() uses arbitrary from in transferFrom:
+  _asset().safeTransferFrom(offer.maker, address(this), ptAmount)
+```
+
+**Why this is not a bug:**
+
+`offer.maker` is not arbitrary. The offer is validated via EIP-712 signature before the transfer (`_validateOffer(offer, signature)` is called on line 421). The maker explicitly signed the offer, authorizing exactly this transfer of exactly this amount. Additionally, `consume()` is gated by `onlyOwnerOrRoles(_ROLE_CONSUMER)` — only trusted consumers can call it.
+
+This is a correct and standard EIP-712 off-chain order pattern. Slither has no understanding of EIP-712 and cannot reason about signature-based authorization.
+
+---
+
+## What Bug Bounty Intelligence flags instead
+
+After filtering to `src/` and applying Al-Mizaan validation gates, we identified **17 signals requiring human review** — none rising to a confirmed finding:
+
+| Category | Count | Description |
+|----------|-------|-------------|
+| `unsafe_keyword` | 15 | Explicit `forge-lint: disable-next-line(unsafe-typecast)` annotations — developer acknowledged unsafe casts. Require review for truncation risk. |
+| `delegatecall` | 2 | `MorphoFlashLoanRequest.execute()` delegatecalls to whitelisted scripts. By design — whitelist check present. |
+
+**What we do not flag:**
+- Library code in `lib/` (out of scope)
+- EIP-712 signed transfers (intentional pattern, not a vulnerability)
+- Standard Solady bit operations (well-audited library behavior)
+
+---
+
+## Al-Mizaan filtering — why the 17 signals do not become findings
+
+The 15 `unsafe_keyword` signals (explicit forge-lint disables) were put through Al-Mizaan's reachability chain:
+
+1. **Code** — read the actual casts in `LibTokenController.sol`, `TokenController.sol`, and `PositionManagerLP.sol`
+2. **Reachability** — are the truncated values reachable by an untrusted caller?
+3. **Threat model** — who triggers these paths? In all cases, the entry points are role-gated (owner, ROLE_CONSUMER, ROLE_MANAGER)
+4. **Invariant** — does the truncation break any accounting invariant?
+
+Verdict: The typecasts are `uint256 → uint128` in share accounting. The values come from internal calculations that are capped by deposit limits. Overflow requires either extremely large TVL or a role-gated actor. **Trusted actor as sole trigger → ELIMINATE.**
+
+The 2 `delegatecall` signals: `MorphoFlashLoanRequest` maintains an explicit `isScriptWhitelisted` mapping. Only the owner can whitelist. **Trusted actor (owner) as sole attacker → ELIMINATE.**
+
+**Result: 0 confirmed findings.** This matches the expected difficulty level of a mature protocol with prior audits.
+
+---
+
+## Why Slither's false positive rate matters
+
+A 100% false positive rate on High findings is not an edge case. It happens because:
+
+1. **No scope awareness** — Slither scans everything, including dependencies. A real audit scopes to `src/` only.
+2. **No protocol context** — Slither cannot distinguish between an arbitrary `transferFrom` and an EIP-712-authorized one.
+3. **No library knowledge** — Solady's assembly patterns trigger false positives on multiple detectors.
+
+For a developer running Slither for the first time, triaging 27 "High" findings (all false positives) takes hours. For a developer using Bug Bounty Intelligence, the 17 signals that remain after scope filtering take 15 minutes to review.
+
+---
+
+## When Slither IS useful
+
+Slither is best used as a first-pass tool before human review, with explicit exclusions:
+
+```bash
+slither src/ --exclude incorrect-exp,incorrect-shift --filter-paths "lib/"
+```
+
+After filtering, Slither's Medium and Low findings become more useful starting points.
+
+Bug Bounty Intelligence is designed to do this filtering for you, and adds protocol-context reasoning that static analysis cannot.
+
+---
+
+## Reproduce this benchmark
+
+The 3FLabs/grunt codebase is public at [github.com/3FLabs/grunt](https://github.com/3FLabs/grunt).
+
+To reproduce Slither's output:
+```bash
+git clone https://github.com/3FLabs/grunt
+cd grunt
+pip install slither-analyzer
+slither . --json slither-output.json
+```
+
+To use Bug Bounty Intelligence:
+```bash
+npx -y bug-bounty-intelligence-mcp@latest
+```
+
+Then in your MCP client:
+```
+scan_contract({ repo_url: "https://github.com/3FLabs/grunt", protocol_type: "LENDING" })
+```
+
+---
+
+## About Bug Bounty Intelligence
+
+Pattern-aware from a corpus of 56,657 submitted Sherlock findings across 212 measured contests, acceptance-rate numbers (used by `list_vulnerability_patterns`) are limited to the 7,388 findings across 35 contests that could be exact-reconciled against contest outcomes (3,694 accepted), see METHODOLOGY.md. Uses the Al-Mizaan v3 framework, a 7-gate validation chain (Code, Reachability, Threat-Model, Invariant, Protocol-Intent, Impact, Proof), to filter signals into confirmed findings.
+
+**Cost:** $5 USDC on Base (eip155:8453) via x402.  
+**Install:** `npx -y bug-bounty-intelligence-mcp@latest`  
+**npm:** [npmjs.com/package/bug-bounty-intelligence-mcp](https://www.npmjs.com/package/bug-bounty-intelligence-mcp)

package/LICENSE

@@ -1,21 +1,21 @@
-MIT License
-
-Copyright (c) 2026 Abdellah Ouadoudi
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License
+
+Copyright (c) 2026 Abdellah Ouadoudi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/METHODOLOGY.md

@@ -1,94 +1,109 @@
-# Methodology: acceptance-rate data behind `list_vulnerability_patterns`
-
-**Last updated: 2026-07-21.**
-
-## What this covers
-
-`list_vulnerability_patterns` returns, per vulnerability class, what fraction of *submitted*
-audit-competition findings were judged valid. This document explains exactly how those numbers
-are computed, what they don't claim, and how to reproduce them from public data.
-
-## Data source
-
-Sherlock runs public GitHub repos for judging each audit contest
-(`sherlock-audit/<contest>-judging`), where security researchers ("Watsons") open one GitHub
-issue per submitted finding. 105 of these judging repos were crawled (27,681 raw submitted
-findings total, 2024-02 through 2025-07).
-
-## The problem: "was this accepted?" is not reliably answerable from GitHub alone
-
-An early version of this pipeline inferred acceptance from GitHub issue state
-(`OPEN` = accepted, `CLOSED` = rejected). That heuristic turned out to be wrong, in both
-directions, depending on the contest — sometimes undercounting acceptance by an order of
-magnitude, sometimes overcounting it. GitHub issue labels (`Reward`, `Non-Reward`, `Invalid`,
-etc.) turned out to be applied inconsistently across contests too — some contests label every
-issue, some label almost none.
-
-## The fix: use Sherlock's own result structure, and only trust it when it reconciles
-
-Sherlock's judging repos also contain, at the repo root, a folder per accepted finding
-(`001-M/`, `002-H/`, etc. — number + severity), each containing one file per GitHub issue number
-that was judged to be that finding (the primary write-up plus any duplicates), and a separate
-`invalid/` folder for rejected issue numbers. This is Sherlock's own authoritative accounting,
-not an inference.
-
-We only trust this for a given contest if it **reconciles exactly**:
-`accepted_issue_numbers + invalid_issue_numbers === total_submitted_issues`.
-
-If a contest's folder structure doesn't exist, or doesn't reconcile exactly, that contest is
-**excluded entirely** — not estimated, not approximated. Of 105 crawled contests:
-
-- **10 reconcile exactly** and are used for `list_vulnerability_patterns`: 1,032 findings, 461
-  accepted.
-- **95 do not** — either no accepted/invalid folder structure exists in that judging repo (a
-  likely explanation: some contests use a different Sherlock judging workflow that doesn't
-  generate this structure), or the counts don't sum to the known total (a likely explanation for
-  many of these: the original crawl capped issue retrieval at 500 per contest, so large contests'
-  true totals exceed what was captured — a fixable crawler limitation, not a Sherlock data
-  problem, but not yet re-verified).
-
-## What the resulting numbers mean
-
-For each pattern (keyword-matched against finding titles/bodies — e.g. "reentrancy",
-"oracle manipulation"), the acceptance rate is: of all submissions whose issue number appears in
-*any* accepted-finding folder (including duplicates of the same underlying bug), what fraction
-of total tagged submissions that is. This measures **submission-level reliability** — "if
-someone writes up and submits a claim of this bug class, how often does it hold up" — not the
-number of distinct vulnerabilities found.
-
-## What this does not claim
-
-- Not a claim about real-world exploit frequency — this is pre-deployment audit-competition
-  data, not realized incidents.
-- Not a claim about detection coverage — it describes submitted reports, not silent misses.
-- The keyword-based pattern tagging is a simple substring matcher (see `detectPatterns()` in the
-  crawler scripts), not a semantic classifier — a manual audit found roughly 65% tagging
-  precision on a sample. Pattern labels are directional, not exact classifications.
-- The 10-contest sample is small. Some pattern rows have thin n (e.g. `liquidation`, n=14) —
-  treat low-n rows as lower-confidence than high-n rows (e.g. `trusted-actor`, n=311).
-
-## Reproducing this
-
-Everything below uses only public data — no private API, no authentication required beyond
-default `gh` CLI rate limits.
-
-```bash
-# 1. List Sherlock judging repos
-gh repo list sherlock-audit --limit 1000 --json name | jq -r '.[] | select(.name | endswith("-judging")) | .name'
-
-# 2. For a given contest, pull its full file tree and look for the NNN-severity / invalid folders
-gh api repos/sherlock-audit/<contest>-judging/git/trees/main?recursive=1 \
-  | jq '[.tree[].path | select(test("^[0-9]{3}-[HM]/|^invalid/"))]'
-
-# 3. Cross-check reconciliation against the contest's total issue count
-gh api "repos/sherlock-audit/<contest>-judging/issues?state=all&per_page=100" | jq 'length'
-```
-
-A finding's issue number appearing under any `NNN-[HM]/` path = accepted. Under `invalid/` =
-rejected. If `accepted + invalid !== total`, do not use that contest's data.
-
-## Why we publish this instead of just the bigger, older number
-
-An earlier draft of this tool advertised acceptance percentages computed from the full,
-unverified 27,681-finding corpus. Independent review caught that those specific percentages
-were unreliable. We would rather ship a smaller number we can prove than a bigger one we can't.
+# Methodology: acceptance-rate data behind `list_vulnerability_patterns`
+
+**Last updated: 2026-11-12.**
+
+## What this covers
+
+`list_vulnerability_patterns` returns, per vulnerability class, what fraction of *submitted*
+audit-competition findings were judged valid. This document explains exactly how those numbers
+are computed, what they don't claim, and how to reproduce them from public data.
+
+## Data source
+
+Sherlock runs public GitHub repos for judging each audit contest
+(`sherlock-audit/<contest>-judging`), where security researchers ("Watsons") open one GitHub
+issue per submitted finding. 218 of these judging repos were crawled; a real issue count could be
+confirmed for 212 of them, 56,657 raw submitted findings total (2022-09 through 2025-02).
+
+## The problem: "was this accepted?" is not reliably answerable from GitHub alone
+
+An early version of this pipeline inferred acceptance from GitHub issue state
+(`OPEN` = accepted, `CLOSED` = rejected). That heuristic turned out to be wrong, in both
+directions, depending on the contest — sometimes undercounting acceptance by an order of
+magnitude, sometimes overcounting it. GitHub issue labels (`Reward`, `Non-Reward`, `Invalid`,
+etc.) turned out to be applied inconsistently across contests too — some contests label every
+issue, some label almost none.
+
+## The fix: use Sherlock's own result structure, and only trust it when it reconciles
+
+Sherlock's judging repos also contain, at the repo root, a folder per accepted finding
+(`001-M/`, `002-H/`, etc. — number + severity), each containing one file per GitHub issue number
+that was judged to be that finding (the primary write-up plus any duplicates), and a separate
+`invalid/` folder for rejected issue numbers. This is Sherlock's own authoritative accounting,
+not an inference.
+
+We only trust this for a given contest if it **reconciles exactly**:
+`accepted_issue_numbers + invalid_issue_numbers === total_submitted_issues`.
+
+If a contest's folder structure doesn't exist, or doesn't reconcile exactly, that contest is
+**excluded entirely** — not estimated, not approximated. Of 218 crawled contests:
+
+- **35 reconcile exactly** and are used for `list_vulnerability_patterns`: 7,388 findings, 3,694
+  accepted.
+- **183 do not** — either no accepted/invalid folder structure exists in that judging repo (a
+  likely explanation: some contests use a different Sherlock judging workflow that doesn't
+  generate this structure), or the counts don't sum to the real total.
+
+**2026-11-12 re-verification.** The original 2026-07-21 run (105 contests, 1,032 reconciled
+findings) turned out to rest on a locally cached issue count that was itself wrong for most
+contests, not only for the handful capped at the documented 500-issue crawl limit. Spot check: the
+cache said `2022-09-knox-judging` had 12 submissions; the real count is 153.
+`2024-02-optimism-2024-judging` showed 4; the real count is 251. The fix was to stop trusting any
+locally cached total and re-derive it fresh from GitHub's search API for every contest, which is
+also how the reconciliation check itself works (see Reproducing this, below). Re-running the exact
+same reconciliation rule this way, with no local cache in the loop, is what produced the 35/7,388/
+3,694 figures above.
+
+## What the resulting numbers mean
+
+For each pattern (keyword-matched against finding titles/bodies — e.g. "reentrancy",
+"oracle manipulation"), the acceptance rate is: of all submissions whose issue number appears in
+*any* accepted-finding folder (including duplicates of the same underlying bug), what fraction
+of total tagged submissions that is. This measures **submission-level reliability** — "if
+someone writes up and submits a claim of this bug class, how often does it hold up" — not the
+number of distinct vulnerabilities found.
+
+## What this does not claim
+
+- Not a claim about real-world exploit frequency — this is pre-deployment audit-competition
+  data, not realized incidents.
+- Not a claim about detection coverage — it describes submitted reports, not silent misses.
+- The keyword-based pattern tagging is a simple substring matcher (see `detectPatterns()` in the
+  crawler scripts), not a semantic classifier — a manual audit found roughly 65% tagging
+  precision on a sample. Pattern labels are directional, not exact classifications.
+- The 35-contest sample is still small relative to the full Sherlock history. Some pattern rows
+  have thin n (e.g. `idor`, n=18) — treat low-n rows as lower-confidence than high-n rows (e.g.
+  `trusted-actor`, n=1,814).
+
+## Reproducing this
+
+Everything below uses only public data — no private API, no authentication required beyond
+default `gh` CLI rate limits.
+
+```bash
+# 1. List Sherlock judging repos
+gh repo list sherlock-audit --limit 1000 --json name | jq -r '.[] | select(.name | endswith("-judging")) | .name'
+
+# 2. For a given contest, pull its full file tree and look for the NNN-severity / invalid folders
+gh api repos/sherlock-audit/<contest>-judging/git/trees/main?recursive=1 \
+  | jq '[.tree[].path | select(test("^[0-9]{3}-[HM]/|^invalid/"))]'
+
+# 3. Cross-check reconciliation against the contest's REAL total issue count. Do not paginate
+#    an issues?state=all list and take its length: that silently undercounts the moment a
+#    contest has more issues than one page covers, which is most of them. The search API's
+#    own total_count does not have this problem.
+gh api "search/issues?q=repo:sherlock-audit/<contest>-judging+type:issue" --jq .total_count
+```
+
+A finding's issue number appearing under any `NNN-[HM]/` path = accepted (strip any leading
+zeros before comparing to the issue number from the GitHub API, the folder names are zero-padded
+and the API numbers are not). Under `invalid/` = rejected. If `accepted + invalid !== total`, do
+not use that contest's data.
+
+## Why we publish this instead of just the bigger, older number
+
+An earlier draft of this tool advertised acceptance percentages computed from the full,
+unverified raw corpus. Independent review caught that those specific percentages were
+unreliable. We would rather ship a smaller number we can prove than a bigger one we can't. That
+is still true today: the raw corpus is 56,657 findings, the reconciled, trustworthy number
+published here is 7,388.