bug-bounty-intelligence-mcp 1.0.1 → 1.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/METHODOLOGY.md +94 -0
- package/README.md +14 -9
- package/index.mjs +21 -15
- package/package.json +3 -3
package/METHODOLOGY.md
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# Methodology: acceptance-rate data behind `list_vulnerability_patterns`
|
|
2
|
+
|
|
3
|
+
**Last updated: 2026-07-21.**
|
|
4
|
+
|
|
5
|
+
## What this covers
|
|
6
|
+
|
|
7
|
+
`list_vulnerability_patterns` returns, per vulnerability class, what fraction of *submitted*
|
|
8
|
+
audit-competition findings were judged valid. This document explains exactly how those numbers
|
|
9
|
+
are computed, what they don't claim, and how to reproduce them from public data.
|
|
10
|
+
|
|
11
|
+
## Data source
|
|
12
|
+
|
|
13
|
+
Sherlock runs public GitHub repos for judging each audit contest
|
|
14
|
+
(`sherlock-audit/<contest>-judging`), where security researchers ("Watsons") open one GitHub
|
|
15
|
+
issue per submitted finding. 105 of these judging repos were crawled (27,681 raw submitted
|
|
16
|
+
findings total, 2024-02 through 2025-07).
|
|
17
|
+
|
|
18
|
+
## The problem: "was this accepted?" is not reliably answerable from GitHub alone
|
|
19
|
+
|
|
20
|
+
An early version of this pipeline inferred acceptance from GitHub issue state
|
|
21
|
+
(`OPEN` = accepted, `CLOSED` = rejected). That heuristic turned out to be wrong, in both
|
|
22
|
+
directions, depending on the contest — sometimes undercounting acceptance by an order of
|
|
23
|
+
magnitude, sometimes overcounting it. GitHub issue labels (`Reward`, `Non-Reward`, `Invalid`,
|
|
24
|
+
etc.) turned out to be applied inconsistently across contests too — some contests label every
|
|
25
|
+
issue, some label almost none.
|
|
26
|
+
|
|
27
|
+
## The fix: use Sherlock's own result structure, and only trust it when it reconciles
|
|
28
|
+
|
|
29
|
+
Sherlock's judging repos also contain, at the repo root, a folder per accepted finding
|
|
30
|
+
(`001-M/`, `002-H/`, etc. — number + severity), each containing one file per GitHub issue number
|
|
31
|
+
that was judged to be that finding (the primary write-up plus any duplicates), and a separate
|
|
32
|
+
`invalid/` folder for rejected issue numbers. This is Sherlock's own authoritative accounting,
|
|
33
|
+
not an inference.
|
|
34
|
+
|
|
35
|
+
We only trust this for a given contest if it **reconciles exactly**:
|
|
36
|
+
`accepted_issue_numbers + invalid_issue_numbers === total_submitted_issues`.
|
|
37
|
+
|
|
38
|
+
If a contest's folder structure doesn't exist, or doesn't reconcile exactly, that contest is
|
|
39
|
+
**excluded entirely** — not estimated, not approximated. Of 105 crawled contests:
|
|
40
|
+
|
|
41
|
+
- **10 reconcile exactly** and are used for `list_vulnerability_patterns`: 1,032 findings, 461
|
|
42
|
+
accepted.
|
|
43
|
+
- **95 do not** — either no accepted/invalid folder structure exists in that judging repo (a
|
|
44
|
+
likely explanation: some contests use a different Sherlock judging workflow that doesn't
|
|
45
|
+
generate this structure), or the counts don't sum to the known total (a likely explanation for
|
|
46
|
+
many of these: the original crawl capped issue retrieval at 500 per contest, so large contests'
|
|
47
|
+
true totals exceed what was captured — a fixable crawler limitation, not a Sherlock data
|
|
48
|
+
problem, but not yet re-verified).
|
|
49
|
+
|
|
50
|
+
## What the resulting numbers mean
|
|
51
|
+
|
|
52
|
+
For each pattern (keyword-matched against finding titles/bodies — e.g. "reentrancy",
|
|
53
|
+
"oracle manipulation"), the acceptance rate is: of all submissions whose issue number appears in
|
|
54
|
+
*any* accepted-finding folder (including duplicates of the same underlying bug), what fraction
|
|
55
|
+
of total tagged submissions that is. This measures **submission-level reliability** — "if
|
|
56
|
+
someone writes up and submits a claim of this bug class, how often does it hold up" — not the
|
|
57
|
+
number of distinct vulnerabilities found.
|
|
58
|
+
|
|
59
|
+
## What this does not claim
|
|
60
|
+
|
|
61
|
+
- Not a claim about real-world exploit frequency — this is pre-deployment audit-competition
|
|
62
|
+
data, not realized incidents.
|
|
63
|
+
- Not a claim about detection coverage — it describes submitted reports, not silent misses.
|
|
64
|
+
- The keyword-based pattern tagging is a simple substring matcher (see `detectPatterns()` in the
|
|
65
|
+
crawler scripts), not a semantic classifier — a manual audit found roughly 65% tagging
|
|
66
|
+
precision on a sample. Pattern labels are directional, not exact classifications.
|
|
67
|
+
- The 10-contest sample is small. Some pattern rows have thin n (e.g. `liquidation`, n=14) —
|
|
68
|
+
treat low-n rows as lower-confidence than high-n rows (e.g. `trusted-actor`, n=311).
|
|
69
|
+
|
|
70
|
+
## Reproducing this
|
|
71
|
+
|
|
72
|
+
Everything below uses only public data — no private API, no authentication required beyond
|
|
73
|
+
default `gh` CLI rate limits.
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
# 1. List Sherlock judging repos
|
|
77
|
+
gh repo list sherlock-audit --limit 1000 --json name | jq -r '.[] | select(.name | endswith("-judging")) | .name'
|
|
78
|
+
|
|
79
|
+
# 2. For a given contest, pull its full file tree and look for the NNN-severity / invalid folders
|
|
80
|
+
gh api repos/sherlock-audit/<contest>-judging/git/trees/main?recursive=1 \
|
|
81
|
+
| jq '[.tree[].path | select(test("^[0-9]{3}-[HM]/|^invalid/"))]'
|
|
82
|
+
|
|
83
|
+
# 3. Cross-check reconciliation against the contest's total issue count
|
|
84
|
+
gh api "repos/sherlock-audit/<contest>-judging/issues?state=all&per_page=100" | jq 'length'
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
A finding's issue number appearing under any `NNN-[HM]/` path = accepted. Under `invalid/` =
|
|
88
|
+
rejected. If `accepted + invalid !== total`, do not use that contest's data.
|
|
89
|
+
|
|
90
|
+
## Why we publish this instead of just the bigger, older number
|
|
91
|
+
|
|
92
|
+
An earlier draft of this tool advertised acceptance percentages computed from the full,
|
|
93
|
+
unverified 27,681-finding corpus. Independent review caught that those specific percentages
|
|
94
|
+
were unreliable. We would rather ship a smaller number we can prove than a bigger one we can't.
|
package/README.md
CHANGED
|
@@ -2,10 +2,15 @@
|
|
|
2
2
|
|
|
3
3
|
AI-powered smart contract security analysis for AI agents and developers.
|
|
4
4
|
|
|
5
|
-
**
|
|
5
|
+
**Drawn from a corpus of 27,681 submitted findings across 105 Sherlock and Code4rena contests.**
|
|
6
6
|
**Cost: $5 USDC on Base (eip155:8453) via x402.**
|
|
7
7
|
**Free tool: `list_vulnerability_patterns` — no payment needed.**
|
|
8
8
|
|
|
9
|
+
`list_vulnerability_patterns`' acceptance-rate numbers are computed only from the subset where
|
|
10
|
+
results could be exactly reconciled against Sherlock's own published outcomes — 1,032 findings
|
|
11
|
+
across 10 contests. We'd rather show fewer, verified numbers than a bigger set we can't stand
|
|
12
|
+
behind. See [METHODOLOGY.md](METHODOLOGY.md) for exactly why, and how to reproduce it.
|
|
13
|
+
|
|
9
14
|
## Benchmark: vs Slither
|
|
10
15
|
|
|
11
16
|
**[See the full benchmark →](BENCHMARK.md)**
|
|
@@ -24,7 +29,7 @@ Submit a public GitHub repo containing Solidity smart contracts. Receive a full
|
|
|
24
29
|
|------|-------------|------|
|
|
25
30
|
| `scan_contract` | Submit repo for security analysis | $5 USDC |
|
|
26
31
|
| `get_scan_report` | Poll status and get report URL | Free |
|
|
27
|
-
| `list_vulnerability_patterns` | Show
|
|
32
|
+
| `list_vulnerability_patterns` | Show acceptance rates from exact-reconciled Sherlock contests | Free |
|
|
28
33
|
|
|
29
34
|
## Quick start (Claude Desktop / Claude Code)
|
|
30
35
|
|
|
@@ -62,14 +67,14 @@ If `scan_contract` returns `PAYMENT REQUIRED`, send exactly $5 USDC on Base to t
|
|
|
62
67
|
```
|
|
63
68
|
> list_vulnerability_patterns({ protocol_type: "LENDING" })
|
|
64
69
|
|
|
65
|
-
Vulnerability patterns for LENDING protocols (from
|
|
70
|
+
Vulnerability patterns for LENDING protocols (from Sherlock audit findings, exact-reconciled contests only):
|
|
66
71
|
|
|
67
|
-
oracle-manipulation
|
|
68
|
-
liquidation
|
|
69
|
-
staleness
|
|
70
|
-
rounding
|
|
71
|
-
flash-loan
|
|
72
|
-
access-control
|
|
72
|
+
oracle-manipulation 36% accepted (131 cases)
|
|
73
|
+
liquidation 29% accepted (14 cases)
|
|
74
|
+
staleness 49% accepted (174 cases)
|
|
75
|
+
rounding 40% accepted (63 cases)
|
|
76
|
+
flash-loan 46% accepted (13 cases)
|
|
77
|
+
access-control 47% accepted (74 cases)
|
|
73
78
|
|
|
74
79
|
> scan_contract({ repo_url: "https://github.com/myprotocol/contracts", protocol_type: "LENDING" })
|
|
75
80
|
|
package/index.mjs
CHANGED
|
@@ -21,20 +21,25 @@ import {
|
|
|
21
21
|
} from '@modelcontextprotocol/sdk/types.js'
|
|
22
22
|
const API_URL = 'https://wazir-x402.duckdns.org'
|
|
23
23
|
|
|
24
|
-
// Embedded pattern library —
|
|
24
|
+
// Embedded pattern library — Sherlock audit-competition findings, updated 2026-07-21.
|
|
25
|
+
// Methodology: only contests where Sherlock's own judging-repo result structure could be
|
|
26
|
+
// exactly reconciled (accepted + invalid === total submissions) are included — 10 contests,
|
|
27
|
+
// 1,032 findings. A much larger raw corpus exists (27,681 submissions across 105 contests),
|
|
28
|
+
// but acceptance could not be verified for the rest without risking a wrong number, so
|
|
29
|
+
// those are excluded rather than estimated. See BENCHMARK.md for the full methodology note.
|
|
25
30
|
const PATTERN_LIBRARY = {
|
|
26
|
-
'
|
|
27
|
-
'
|
|
28
|
-
'trusted-actor': { acceptance_rate: '0.
|
|
29
|
-
'fee-miscalculation': { acceptance_rate: '0.
|
|
30
|
-
'
|
|
31
|
-
'
|
|
32
|
-
'access-control': { acceptance_rate: '0.
|
|
33
|
-
'
|
|
34
|
-
'
|
|
35
|
-
'
|
|
36
|
-
'
|
|
37
|
-
'liquidation': { acceptance_rate: '0.
|
|
31
|
+
'reentrancy': { acceptance_rate: '0.78', total: 51, accepted: 40, examples_accepted: ['m4k2 - Reentrancy in GoatPairV1::burn if token is a non-standard ERC20', 'Tonchi - Reentrancy in burn function, as it does not update the state of liquidity', 'C1rdan - hacker can steal fee from LPs'] },
|
|
32
|
+
'overflow': { acceptance_rate: '0.58', total: 45, accepted: 26, examples_accepted: ['Fassi_Security - When an order is exactly matched, a buyer can end up paying more', 'mstpr-brainbot - Pairs with MAX_FEE can revert due to rounding inconsistencies', 'irresponsible - Partition rounds up which can cause orders to be unfillable'] },
|
|
33
|
+
'trusted-actor': { acceptance_rate: '0.51', total: 311, accepted: 159, examples_accepted: ['LTDingZhen - Users can grief fillers by set malicious ValidationContract', 'mstpr-brainbot - Pairs with MAX_FEE can revert due to rounding inconsistencies', 'HSP - Attacker can submit malicious order and user may lose funds'] },
|
|
34
|
+
'fee-miscalculation': { acceptance_rate: '0.50', total: 109, accepted: 55, examples_accepted: ['WangAudit - [H] RubiconFeeController::getFeeOutputs incorrectly creates feeOut', 'turvec - The feeController differs from its specification', 'hals - IG contract can be DoS-ed from minting or burning options'] },
|
|
35
|
+
'staleness': { acceptance_rate: '0.49', total: 174, accepted: 86, examples_accepted: ['LTDingZhen - Users can grief fillers by set malicious ValidationContract', 'HSP - Attacker can submit malicious order and user may lose funds', 'pkqs90 - Orders with equal decayStartTime and decayEndTime benefit the filler'] },
|
|
36
|
+
'mev-slippage': { acceptance_rate: '0.49', total: 89, accepted: 44, examples_accepted: ['ge6a - DOS of IG mint/burn because of _deltaHedgePosition() revert', 'ge6a - Dos through large deposit', 'ge6a - Permanent Dos through trackVaultFee()'] },
|
|
37
|
+
'access-control': { acceptance_rate: '0.47', total: 74, accepted: 35, examples_accepted: ['skatas192 - The owner address has never been set which will cause the auth modifier', 'bearonbike - DVP _mint/_burn function could be DoS-ed by FeeManager', 'KingNFT - Attack on FeeManager.trackVaultFee() to make the IG contract'] },
|
|
38
|
+
'flash-loan': { acceptance_rate: '0.46', total: 13, accepted: 6, examples_accepted: ['cawfree - The invariant maxDeposit for a Vault can be exceeded', 'mgf15 - Use of slot0 to get sqrtPriceX96 can lead to price manipulation', 'bughuntoor - Any user can do a AAVE flashloan on behalf of FlashLoanAggregator'] },
|
|
39
|
+
'dos-griefing': { acceptance_rate: '0.41', total: 156, accepted: 64, examples_accepted: ['ni8mare - execute transactions can be reverted by a malicious user', 'LTDingZhen - Users can grief fillers by set malicious ValidationContract', 'mstpr-brainbot - Pairs with MAX_FEE can revert due to rounding inconsistencies'] },
|
|
40
|
+
'rounding': { acceptance_rate: '0.40', total: 63, accepted: 25, examples_accepted: ['mstpr-brainbot - Pairs with MAX_FEE can revert due to rounding inconsistencies', 'KingNFT - Execution of orders would revert unexpectedly', 'blutorque - PartialFillLib::partition() unexpectedly reverts'] },
|
|
41
|
+
'oracle-manipulation':{ acceptance_rate: '0.36', total: 131, accepted: 47, examples_accepted: ['ge6a - DOS of IG mint/burn because of _deltaHedgePosition() revert', 'ge6a - Dos through large deposit', 'jasonxiale - Vault._state.liquidity.totalDeposit can avoid being decreased'] },
|
|
42
|
+
'liquidation': { acceptance_rate: '0.29', total: 14, accepted: 4, examples_accepted: ['M3azad - No updation of _state.liquidity.pendingWithdrawals may lead to abnormal', 'bughuntoor - Liquidation bonus scales exponentially instead of linearly', 'cawfree - Calls to addToBlackList(address,address[]) can be frontrun'] },
|
|
38
43
|
}
|
|
39
44
|
|
|
40
45
|
// ── helpers ──────────────────────────────────────────────────────────────────
|
|
@@ -124,7 +129,8 @@ server.setRequestHandler(ListToolsRequestSchema, async () => ({
|
|
|
124
129
|
{
|
|
125
130
|
name: 'list_vulnerability_patterns',
|
|
126
131
|
description: [
|
|
127
|
-
'Returns historical vulnerability patterns and
|
|
132
|
+
'Returns historical vulnerability patterns and acceptance rates from Sherlock audit-competition findings.',
|
|
133
|
+
'Numbers are limited to contests where results could be exactly reconciled against Sherlock\'s own published outcomes (1,032 findings across 10 contests) — no estimated or unverifiable figures.',
|
|
128
134
|
'Useful for understanding what types of bugs are most likely to be valid in a given protocol type.',
|
|
129
135
|
'This tool is FREE — no payment required.',
|
|
130
136
|
].join(' '),
|
|
@@ -239,7 +245,7 @@ server.setRequestHandler(CallToolRequestSchema, async (req) => {
|
|
|
239
245
|
}
|
|
240
246
|
}
|
|
241
247
|
|
|
242
|
-
const lines = [`Vulnerability patterns for ${protocol_type} protocols (from
|
|
248
|
+
const lines = [`Vulnerability patterns for ${protocol_type} protocols (from Sherlock audit findings, exact-reconciled contests only):`, '']
|
|
243
249
|
for (const p of patterns) {
|
|
244
250
|
const pct = Math.round(parseFloat(p.acceptance_rate) * 100)
|
|
245
251
|
lines.push(`${p.pattern.padEnd(24)} ${pct}% accepted (${p.total_cases} cases)`)
|
package/package.json
CHANGED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "bug-bounty-intelligence-mcp",
|
|
3
|
-
"version": "1.0.
|
|
4
|
-
"description": "MCP server for AI-powered smart contract security analysis. $5 USDC/scan via x402.
|
|
3
|
+
"version": "1.0.2",
|
|
4
|
+
"description": "MCP server for AI-powered smart contract security analysis. $5 USDC/scan via x402. Drawn from 27,681 Sherlock/Code4rena submissions, verified acceptance rates from 1,032 exact-reconciled findings.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "index.mjs",
|
|
7
7
|
"bin": { "bug-bounty-intelligence-mcp": "index.mjs" },
|
|
8
|
-
"files": ["index.mjs", "README.md", "BENCHMARK.md"],
|
|
8
|
+
"files": ["index.mjs", "README.md", "BENCHMARK.md", "METHODOLOGY.md"],
|
|
9
9
|
"repository": { "type": "git", "url": "git+https://github.com/holistis/bug-bounty-intelligence-mcp.git" },
|
|
10
10
|
"homepage": "https://github.com/holistis/bug-bounty-intelligence-mcp#readme",
|
|
11
11
|
"keywords": ["mcp", "smart-contract", "security", "audit", "solidity", "defi", "x402", "bug-bounty"],
|