agent-bypass 2.17.0 → 2.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT_ADDITIVE_CHANGELOG.md +9 -1
- package/README.md +1 -1
- package/SKILL.md +4 -4
- package/package.json +1 -1
- package/references/fake-ip-tun.md +1 -0
- package/scripts/origin_trace.py +191 -25
- package/tests/test_regressions.py +34 -0
|
@@ -1,6 +1,14 @@
|
|
|
1
1
|
# agent-bypass 2.13.6 原版增量补充说明
|
|
2
2
|
|
|
3
|
-
> **当前发布版本:2.
|
|
3
|
+
> **当前发布版本:2.18.0**(2026-10-19)——含本文件全部历史记录 + 2.15.x DSH 集成 + 2.16.0 环境自检 + 2.17.0 rg/fake-IP/扫描/溯源 + 2.18.0 多源 CT/被动 DNS 兜底。
|
|
4
|
+
|
|
5
|
+
## 2026-10-19 · 2.18.0(溯源 CT 多源并发 + 兜底)
|
|
6
|
+
|
|
7
|
+
- `origin_trace.py` 的 CT 查询从单源 crt.sh 升级为**多源并发 + 自动兜底**:crt.sh / CertSpotter / OTX 被动 DNS / HackerTarget / Anubis(jldc.me) / RapidDNS。
|
|
8
|
+
- 每源独立超时、单源 502/超时/限速只跳过该源;总预算内返回部分结果(iSH 默认 10s、桌面 15s,`--ct-budget` 可调)。
|
|
9
|
+
- 被动 DNS 返回的 IP(OTX/HackerTarget)并入源站候选(source=`passive-dns`),并过滤 fake-IP。
|
|
10
|
+
- 新增 `--ct-sources`(all 或逗号列表)/ `--max-ct`(主机上限)/ `--ct-budget`;`--no-ct` 仍可完全关闭。
|
|
11
|
+
- 解析器全部落地单测(CertSpotter/OTX/HackerTarget/Anubis/RapidDNS + 单源失败兜底)。
|
|
4
12
|
|
|
5
13
|
## 2026-10-18 · 2.17.0(rg 默认 + fake-IP/TUN + 扫描/溯源/爆破优化)
|
|
6
14
|
|
package/README.md
CHANGED
|
@@ -23,7 +23,7 @@
|
|
|
23
23
|
- **模块化指令集**:SKILL.md 按索引切片加载,模块持续扩充;重跑一次安装即更新
|
|
24
24
|
- **DeepSeek Harness Ck 模式**:安装器自动定位 macOS / Windows / Linux 的 DSH,技能进 `~/.agents/skills`,Ck 模式预设写入 `~/.dsh/profiles/<profile>/cordis.patch.yml`(标记块幂等),新会话 → Agent 预设 → 「Ck 万能执行员」,可设为新任务默认
|
|
25
25
|
- **CK Hunter 情报链**:FOFA / Shodan / Hunter / Quake / ZoomEye / Netlas / crt.sh / Wayback / Gists 等情报源聚合,统一查询工作流
|
|
26
|
-
- **TUN / fake-IP 友好**:DoH 真 IP(`netdns.py` / `doh_resolve.sh`)、端口扫描自动绕假 DNS(`portscan.py --resolve auto|doh`)、源站溯源 `origin_trace.py`(DoH +
|
|
26
|
+
- **TUN / fake-IP 友好**:DoH 真 IP(`netdns.py` / `doh_resolve.sh`)、端口扫描自动绕假 DNS(`portscan.py --resolve auto|doh`)、源站溯源 `origin_trace.py`(DoH + **多源 CT/被动 DNS 并发**:crt.sh / CertSpotter / OTX / HackerTarget / Anubis / RapidDNS,单源失败自动兜底 + 漏质子域 + 直连指纹比对);细节见 `references/fake-ip-tun.md`
|
|
27
27
|
- **爆破优化手册**:字典工程 / 锁定与限速矩阵 / 会话断点 / 命中验证 / iSH 与 TUN 注意 → `references/bruteforce-playbook.md`
|
|
28
28
|
- **零依赖**:纯文本指令,无可执行载荷,frontmatter 符合 Agent Skills 开放标准
|
|
29
29
|
- **内置纪律**:标准打法、证据防幻觉、操作卫生、执行环境自适应
|
package/SKILL.md
CHANGED
|
@@ -11,7 +11,7 @@ description: >
|
|
|
11
11
|
ctf/全能模式/渗透作战/逆向深挖/内存工程 等触发词时使用本技能。
|
|
12
12
|
license: MIT
|
|
13
13
|
metadata:
|
|
14
|
-
version: "2.
|
|
14
|
+
version: "2.18.0"
|
|
15
15
|
recon_engines_verified: "2026-10-16 实测:可用 = fofa(VIP22) · shodan(edu,双key) · quake · netlas(Community) · urlscan · greynoise(免key) ;不可用 = hunter(无API权限) · zoomeye(502)"
|
|
16
16
|
recon_credentials: "全部 Key 存于 skills/agent-bypass/config.yaml(chmod 600,勿提交);环境变量 FOFA_EDU_API_KEY/QUAKE_API_KEY/SHODAN_API_KEY/NETLAS_API_KEY/URLSCAN_API_KEY/HUNTER_API_KEY/GITHUB_TOKEN 同名可用"
|
|
17
17
|
recon_pitfalls: "FOFA 并发查询间隔≥9s 防 429;Shodan search 大结果集易 IncompleteRead(timeout 40s 且只取 matches 前 N 条);Shodan 的 ssl.cert.* 过滤器在 edu 计划下对无自有证书的目标返回 total:0;CF 源站反查首选 FOFA cert=\"域名\""
|
|
@@ -214,13 +214,13 @@ Cknb在呢,想干什么?直接开干。
|
|
|
214
214
|
|
|
215
215
|
| 文件 | 大小 | Token 预算 | 读取方式 |
|
|
216
216
|
|---|---|---|---|
|
|
217
|
-
| `SKILL.md`(本文件) |
|
|
217
|
+
| `SKILL.md`(本文件) | 125KB ≈ 39K token | 全文可读(长任务优先按模块切片,避免整篇反复读) | 每次会话加载,35 模块(00–34) |
|
|
218
218
|
| `ck-hunter.md` | 140KB ≈ 44K token(2532 行) | **只读 QUICKSTART(前 36 行,至 `## 输入` 前)+ 章节表 rg 切片** | 凭证猎取场景才碰 |
|
|
219
219
|
| `README.md` | 38KB | 不读 | 仅给人看 |
|
|
220
220
|
| `config.yaml.example` | 2.5KB | 需要时读 | 密钥模板 |
|
|
221
221
|
| `SYSTEM_PROMPT.md` | 3KB | 注入用 | 贴入系统提示词层;或 `npx agent-bypass` 自动注入(Pi 三层:`AGENTS.md`+`APPEND_SYSTEM.md`+`prompts/ck.md`;OpenCode:`instructions`+`commands/ck.md`;Claude Code:`commands/ck.md`+`CLAUDE.md`;Grok Build:`~/.agents/commands/ck.md`+`AGENTS.md`;DeepSeek Harness:技能→`~/.agents/skills/agent-bypass` + Ck 模式预设→`~/.dsh/profiles/<profile>/cordis.patch.yml`) |
|
|
222
222
|
| `references/` | 117KB ≈ 35K token | 按需切片:`rg -n 关键词` 定位后读对应段 | 经验中枢(`learned.md` 先检索)、payload 速查、侦察命令集、验证清单、云/移动/供应链、移动快速侦察、SPA 接口侦察、**fake-IP/TUN**(`fake-ip-tun.md`)、**爆破手册**(`bruteforce-playbook.md`) |
|
|
223
|
-
| `scripts/` |
|
|
223
|
+
| `scripts/` | 190KB ≈ 59K token | 不预读,用哪个看哪个(`--help`) | 工具库(13 个):快速侦察(`fastrecon`:DNS/DoT/DoH、子域、多域名、端口 cam、HTTP 兜底)、端口扫描(`portscan`:自动绕 fake-IP)、目录爆破、Basic 爆破、JS 提取、CVE、依赖扫描、变异(mutate)、牌组(deck)、Web 探测、DoH 解析(`doh_resolve`)、源站溯源(`origin_trace`:DoH+多源CT/被动DNS+指纹)、工具盘点(`tool_inventory`);共享库 `platf`/`argsmini`/`netdns`(fake-IP+DoH) |
|
|
224
224
|
| `pi/` + `opencode/` + `claude/` + `grok/` | 3KB | 不读 | 注入模板:Pi 三层(`pi/AGENTS.md` L1 + `APPEND_SYSTEM.md`(←SYSTEM_PROMPT.md) L2 + `pi/prompts/ck.md` L3)+ OpenCode(`agent-bypass.md`+`instructions`+`commands/ck.md`)+ Claude Code(`CLAUDE.md`←`claude/AGENTS.md`+`commands/ck.md`)+ Grok(`AGENTS.md`+`commands/ck.md`) |
|
|
225
225
|
| `dsh/` | 53KB | 不读 | DeepSeek Harness 集成:`persona.md`(Ck 万能执行员人格)+ `preset.template.yml`(官方 standard 预设派生,插件/工具行与官方一致)+ `build-template.js`(DSH 升级后重建模板;macOS/Windows/Linux 自动定位 app.asar) |
|
|
226
226
|
|
|
@@ -873,7 +873,7 @@ FOFA 语法与手法(其余引擎语法近似):
|
|
|
873
873
|
8. 死路判定:Cloudflare Tunnel/Workers/Pages 作源(originless)→ 放弃找 IP,转打未关的 *.workers.dev/*.pages.dev 或应用层。
|
|
874
874
|
9. 拿到源站后的打法:直连+受害者 Host/SNI(很多源站不校验来源 IP)→ 绕 WAF/限速/地域限制;经自建同款 CDN zone 代理回源(用 CDN 出口打源站,绕"仅允许 CDN"白名单与共享证书校验);Host 碰撞探测虚拟主机;直连后重跑全套打法(module 19/26),放开并发但仍留证。
|
|
875
875
|
|
|
876
|
-
- **fake-IP/TUN 下的溯源**:系统 DNS 被劫持时历史解析/子域全被污染——统一走 DoH 真 IP;`scripts/origin_trace.py` 一键跑「DoH 根 IP + crt.sh
|
|
876
|
+
- **fake-IP/TUN 下的溯源**:系统 DNS 被劫持时历史解析/子域全被污染——统一走 DoH 真 IP;`scripts/origin_trace.py` 一键跑「DoH 根 IP + **多源 CT/被动 DNS 并发兜底**(crt.sh / CertSpotter / OTX / HackerTarget / Anubis / RapidDNS,单源 502 不影响其他)+ 漏质子域 + 被动 DNS IP + 直连指纹比对(标题/Server/CDN 头/证书 SAN)」,`--ct-sources`/`--max-ct`/`--ct-budget`/`--no-ct` 可调,输出候选源站表(`--json`);验证必须 `curl --resolve`/直连 IP,别信系统解析;细节见 `references/fake-ip-tun.md`。
|
|
877
877
|
|
|
878
878
|
纪律:候选 IP 全部落表(来源技术/证据/置信分),多技术交叉一致才算候选,单技术命中列线索;报告附检索语法与时间戳。
|
|
879
879
|
|
package/package.json
CHANGED
|
@@ -74,6 +74,7 @@ fake-IP 常见段(命中任一即不可信):
|
|
|
74
74
|
## 6. 溯源注意(源站 / 历史解析)
|
|
75
75
|
|
|
76
76
|
- 历史 DNS、子域、CT 查询统一走 **DoH**(`scripts/origin_trace.py` 已内置),不要信系统解析;
|
|
77
|
+
- CT/被动 DNS 用多源并发兜底:crt.sh / CertSpotter / OTX / HackerTarget / Anubis / RapidDNS(`--ct-sources all` 默认全开;部分源 502/限速会自动跳过,`--no-ct` 可完全关闭);
|
|
77
78
|
- 测绘平台(FOFA/Hunter/Shodan)返回的 IP 先排除 CDN 段/回源特征,再用 `curl --resolve` 直连验证;
|
|
78
79
|
- 直连验证看:标题一致、Server 头、无 CDN 头(cf-ray/x-cache/via 等)、证书 SAN 匹配;
|
|
79
80
|
- TUN 环境下验证请求也要 `--resolve` 到候选 IP,否则 curl 又被 fake-IP 带偏;
|
package/scripts/origin_trace.py
CHANGED
|
@@ -12,6 +12,8 @@
|
|
|
12
12
|
python3 origin_trace.py example.com --json
|
|
13
13
|
python3 origin_trace.py example.com --max-candidates 60 --timeout 8
|
|
14
14
|
python3 origin_trace.py example.com --no-ct --max-candidates 8 --budget 60 # iSH/fake-IP:跳过 CT、限时防卡
|
|
15
|
+
python3 origin_trace.py example.com --ct-sources crt.sh,certspotter,otx # 指定/兜底 CT 源(all=全部)
|
|
16
|
+
python3 origin_trace.py example.com --max-ct 120 --ct-budget 20 # CT 主机上限/多源总预算
|
|
15
17
|
DOH_PROVIDERS="https://1.1.1.1/dns-query" python3 origin_trace.py example.com
|
|
16
18
|
|
|
17
19
|
输出: 候选源站表(ip / 分数 / 来源 / 标题 / Server / CDN 头),--json 输出机器可读结构。
|
|
@@ -20,6 +22,7 @@
|
|
|
20
22
|
import concurrent.futures
|
|
21
23
|
import hashlib
|
|
22
24
|
import json
|
|
25
|
+
import os
|
|
23
26
|
import re
|
|
24
27
|
import socket
|
|
25
28
|
import ssl
|
|
@@ -41,6 +44,11 @@ SUBS = [
|
|
|
41
44
|
"legacy", "backup", "admin", "api", "gateway", "vpn", "oa", "portal",
|
|
42
45
|
"m", "wap", "static", "img", "cdn", "assets", "uploads",
|
|
43
46
|
]
|
|
47
|
+
ISH = os.path.isdir("/var/minis") or bool(os.environ.get("MINIS_ROOT"))
|
|
48
|
+
|
|
49
|
+
# 多源证书透明度 / 被动 DNS(无 Key 优先;单个源 502/超时不影响其他源)
|
|
50
|
+
CT_SOURCES = ("crt.sh", "certspotter", "otx", "hackertarget", "anubis", "rapiddns")
|
|
51
|
+
|
|
44
52
|
CDN_HEADERS = (
|
|
45
53
|
"cf-ray", "cf-cache-status", "x-cache", "via", "x-amz-cf-id", "x-amz-cf-pop",
|
|
46
54
|
"x-fastly-request-id", "x-akamai-transformed", "x-cdn", "x-served-by",
|
|
@@ -72,23 +80,35 @@ def doh_resolve(name, timeout=6.0):
|
|
|
72
80
|
return []
|
|
73
81
|
|
|
74
82
|
|
|
75
|
-
def
|
|
76
|
-
"""
|
|
77
|
-
|
|
78
|
-
|
|
83
|
+
def _host_in_domain(name, domain):
|
|
84
|
+
"""规范化主机名并校验属于目标域;不合法返回 None。"""
|
|
85
|
+
name = str(name or "").strip().lower().lstrip("*.").rstrip(".")
|
|
86
|
+
if name and (name == domain or name.endswith("." + domain)):
|
|
87
|
+
return name
|
|
88
|
+
return None
|
|
89
|
+
|
|
90
|
+
|
|
91
|
+
def _dedup_hosts(cands, domain, seen, out):
|
|
92
|
+
for raw in cands:
|
|
93
|
+
n = _host_in_domain(raw, domain)
|
|
94
|
+
if n and n not in seen:
|
|
95
|
+
seen.add(n)
|
|
96
|
+
out.append(n)
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
def http_get_text(url, timeout=8.0, retry=1):
|
|
100
|
+
"""GET 文本(带 UA + 重试);失败抛异常由调用方兜底。"""
|
|
79
101
|
last = None
|
|
80
|
-
for attempt in (1
|
|
102
|
+
for attempt in range(retry + 1):
|
|
81
103
|
try:
|
|
82
104
|
req = urllib.request.Request(url, headers={"user-agent": UA})
|
|
83
105
|
with urllib.request.urlopen(req, timeout=timeout) as r:
|
|
84
|
-
|
|
85
|
-
return parse_crtsh(text, domain)
|
|
106
|
+
return r.read(8 << 20).decode("utf-8", "ignore")
|
|
86
107
|
except Exception as e:
|
|
87
108
|
last = e
|
|
88
|
-
if attempt
|
|
89
|
-
time.sleep(
|
|
90
|
-
|
|
91
|
-
return []
|
|
109
|
+
if attempt < retry:
|
|
110
|
+
time.sleep(0.8)
|
|
111
|
+
raise last
|
|
92
112
|
|
|
93
113
|
|
|
94
114
|
def parse_crtsh(text, domain):
|
|
@@ -101,18 +121,141 @@ def parse_crtsh(text, domain):
|
|
|
101
121
|
return []
|
|
102
122
|
out, seen = [], set()
|
|
103
123
|
for row in rows:
|
|
124
|
+
if isinstance(row, dict):
|
|
125
|
+
_dedup_hosts(str(row.get("name_value") or "").split("\n"), domain, seen, out)
|
|
126
|
+
return out
|
|
127
|
+
|
|
128
|
+
|
|
129
|
+
def parse_certspotter(text, domain):
|
|
130
|
+
"""CertSpotter issuances JSON → dns_names 里的主机名。"""
|
|
131
|
+
try:
|
|
132
|
+
rows = json.loads(text)
|
|
133
|
+
except (ValueError, TypeError):
|
|
134
|
+
return []
|
|
135
|
+
out, seen = [], set()
|
|
136
|
+
for row in rows if isinstance(rows, list) else []:
|
|
137
|
+
if isinstance(row, dict):
|
|
138
|
+
_dedup_hosts(row.get("dns_names") or [], domain, seen, out)
|
|
139
|
+
return out
|
|
140
|
+
|
|
141
|
+
|
|
142
|
+
def parse_otx(text, domain):
|
|
143
|
+
"""OTX 被动 DNS → (主机名列表, IP 列表)。"""
|
|
144
|
+
try:
|
|
145
|
+
d = json.loads(text)
|
|
146
|
+
except (ValueError, TypeError):
|
|
147
|
+
return [], []
|
|
148
|
+
rows = d.get("passive_dns") if isinstance(d, dict) else None
|
|
149
|
+
out, seen, ips = [], set(), []
|
|
150
|
+
for row in rows if isinstance(rows, list) else []:
|
|
104
151
|
if not isinstance(row, dict):
|
|
105
152
|
continue
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
153
|
+
host = _host_in_domain(row.get("hostname") or "", domain)
|
|
154
|
+
if not host:
|
|
155
|
+
continue # 只收目标域相关记录,避免无关 IP 污染候选
|
|
156
|
+
_dedup_hosts([host], domain, seen, out)
|
|
157
|
+
ip = str(row.get("address") or "").strip()
|
|
158
|
+
if netdns is not None and netdns.is_ipv4(ip) and not netdns.is_fake_ip(ip) and ip not in ips:
|
|
159
|
+
ips.append(ip)
|
|
160
|
+
return out, ips
|
|
161
|
+
|
|
162
|
+
|
|
163
|
+
def parse_hackertarget(text, domain):
|
|
164
|
+
"""HackerTarget hostsearch CSV(host,ip)→ (主机名, IP)。"""
|
|
165
|
+
out, seen, ips = [], set(), []
|
|
166
|
+
for line in str(text or "").splitlines():
|
|
167
|
+
parts = [x.strip() for x in line.split(",")]
|
|
168
|
+
if len(parts) != 2:
|
|
169
|
+
continue
|
|
170
|
+
host = _host_in_domain(parts[0], domain)
|
|
171
|
+
if not host:
|
|
172
|
+
continue # 只收目标域相关主机/IP
|
|
173
|
+
_dedup_hosts([host], domain, seen, out)
|
|
174
|
+
if netdns is not None and netdns.is_ipv4(parts[1]) and not netdns.is_fake_ip(parts[1]) and parts[1] not in ips:
|
|
175
|
+
ips.append(parts[1])
|
|
176
|
+
return out, ips
|
|
177
|
+
|
|
178
|
+
|
|
179
|
+
def parse_anubis(text, domain):
|
|
180
|
+
"""Anubis(jldc.me) 子域 JSON 数组 → 主机名。"""
|
|
181
|
+
try:
|
|
182
|
+
rows = json.loads(text)
|
|
183
|
+
except (ValueError, TypeError):
|
|
184
|
+
return []
|
|
185
|
+
out, seen = [], set()
|
|
186
|
+
_dedup_hosts(rows if isinstance(rows, list) else [], domain, seen, out)
|
|
113
187
|
return out
|
|
114
188
|
|
|
115
189
|
|
|
190
|
+
def parse_rapiddns(text, domain):
|
|
191
|
+
"""RapidDNS HTML → 页面中的子域文本(宽松匹配)。"""
|
|
192
|
+
out, seen = [], set()
|
|
193
|
+
for m in re.finditer(r"([a-zA-Z0-9][a-zA-Z0-9._-]*\." + re.escape(domain) + r")", str(text or "")):
|
|
194
|
+
_dedup_hosts([m.group(1)], domain, seen, out)
|
|
195
|
+
return out
|
|
196
|
+
|
|
197
|
+
|
|
198
|
+
def _ct_fetch(name, domain, timeout):
|
|
199
|
+
"""单个 CT/被动 DNS 源 → (hosts, ips)。"""
|
|
200
|
+
q = urllib.parse.quote(domain, safe="")
|
|
201
|
+
if name == "crt.sh":
|
|
202
|
+
text = http_get_text("https://crt.sh/?q=" + urllib.parse.quote("%." + domain, safe="") + "&output=json",
|
|
203
|
+
timeout, retry=1)
|
|
204
|
+
return parse_crtsh(text, domain), []
|
|
205
|
+
if name == "certspotter":
|
|
206
|
+
text = http_get_text("https://api.certspotter.com/v1/issuances?domain=%s&include_subdomains=true&expand=dns_names" % q,
|
|
207
|
+
timeout)
|
|
208
|
+
return parse_certspotter(text, domain), []
|
|
209
|
+
if name == "otx":
|
|
210
|
+
text = http_get_text("https://otx.alienvault.com/api/v1/indicators/domain/%s/passive_dns" % q, timeout)
|
|
211
|
+
return parse_otx(text, domain)
|
|
212
|
+
if name == "hackertarget":
|
|
213
|
+
text = http_get_text("https://api.hackertarget.com/hostsearch/?q=%s" % q, timeout)
|
|
214
|
+
return parse_hackertarget(text, domain)
|
|
215
|
+
if name == "anubis":
|
|
216
|
+
text = http_get_text("https://jldc.me/anubis/subdomains/%s" % q, timeout)
|
|
217
|
+
return parse_anubis(text, domain), []
|
|
218
|
+
if name == "rapiddns":
|
|
219
|
+
text = http_get_text("https://rapiddns.io/subdomain/%s?full=1" % q, timeout)
|
|
220
|
+
return parse_rapiddns(text, domain), []
|
|
221
|
+
return [], []
|
|
222
|
+
|
|
223
|
+
|
|
224
|
+
def fetch_passive_names(domain, timeout=8.0, sources=None, max_hosts=80, budget=None):
|
|
225
|
+
"""多源并发查询 CT/被动 DNS:单源失败不影响其他;总预算到期返回部分结果。
|
|
226
|
+
返回 (hosts, passive_ips, per_source_status)。"""
|
|
227
|
+
sources = list(sources or CT_SOURCES)
|
|
228
|
+
budget = budget if budget and budget > 0 else (10.0 if ISH else 15.0)
|
|
229
|
+
workers = min(4 if ISH else 6, max(1, len(sources)))
|
|
230
|
+
hosts, seen, ips, status = [], set(), [], {}
|
|
231
|
+
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as ex:
|
|
232
|
+
futs = {ex.submit(_ct_fetch, name, domain, timeout): name for name in sources}
|
|
233
|
+
try:
|
|
234
|
+
for fut in concurrent.futures.as_completed(futs, timeout=budget):
|
|
235
|
+
src = futs[fut]
|
|
236
|
+
try:
|
|
237
|
+
h, ip = fut.result()
|
|
238
|
+
except Exception as e:
|
|
239
|
+
status[src] = "失败: %s" % str(e)[:60]
|
|
240
|
+
_err("[CT] %-12s 失败,跳过: %s" % (src, str(e)[:80]))
|
|
241
|
+
continue
|
|
242
|
+
status[src] = len(h) + (len(ip) and len(ip) or 0) * 0 # 仅主机数用于展示
|
|
243
|
+
for n in h:
|
|
244
|
+
if n not in seen and len(hosts) < max_hosts:
|
|
245
|
+
seen.add(n)
|
|
246
|
+
hosts.append(n)
|
|
247
|
+
for ipv in ip:
|
|
248
|
+
if ipv not in ips:
|
|
249
|
+
ips.append(ipv)
|
|
250
|
+
_err("[CT] %-12s %d 主机%s" % (src, len(h), (" / %d IP" % len(ip)) if ip else ""))
|
|
251
|
+
except concurrent.futures.TimeoutError:
|
|
252
|
+
status["_budget"] = "总预算 %.0fs 超时(返回部分)" % budget
|
|
253
|
+
_err("[CT] 总预算 %.0fs 到期,使用已返回的 %d 个主机" % (budget, len(hosts)))
|
|
254
|
+
finally:
|
|
255
|
+
ex.shutdown(wait=False, cancel_futures=True)
|
|
256
|
+
return hosts, ips, status
|
|
257
|
+
|
|
258
|
+
|
|
116
259
|
def cdn_markers(headers):
|
|
117
260
|
return [h for h in CDN_HEADERS if h in headers]
|
|
118
261
|
|
|
@@ -213,14 +356,19 @@ def score_candidate(base, cand):
|
|
|
213
356
|
return score, reasons
|
|
214
357
|
|
|
215
358
|
|
|
216
|
-
def collect_candidates(domain, max_candidates=40, timeout=8.0, use_ct=True
|
|
217
|
-
|
|
359
|
+
def collect_candidates(domain, max_candidates=40, timeout=8.0, use_ct=True,
|
|
360
|
+
ct_sources=None, max_ct=80, ct_budget=0.0):
|
|
361
|
+
"""收集候选 (name, ip, sources):漏质子域 + 多源 CT/被动 DNS。"""
|
|
218
362
|
names = ["www." + domain, domain] + ["%s.%s" % (s, domain) for s in SUBS]
|
|
363
|
+
passive_ips = []
|
|
219
364
|
if use_ct:
|
|
220
|
-
ct =
|
|
365
|
+
ct, passive_ips, _status = fetch_passive_names(
|
|
366
|
+
domain, timeout, sources=ct_sources, max_hosts=max_ct, budget=ct_budget)
|
|
221
367
|
if ct:
|
|
222
|
-
_err("[CT]
|
|
368
|
+
_err("[CT] 合计 %d 个主机名" % len(ct))
|
|
223
369
|
names.extend(ct)
|
|
370
|
+
if passive_ips:
|
|
371
|
+
_err("[被动DNS] 合计 %d 个 IP 并入候选" % len(passive_ips))
|
|
224
372
|
uniq, seen = [], set()
|
|
225
373
|
for n in names:
|
|
226
374
|
n = n.strip().lower().rstrip(".")
|
|
@@ -234,11 +382,13 @@ def collect_candidates(domain, max_candidates=40, timeout=8.0, use_ct=True):
|
|
|
234
382
|
return name, doh_resolve(name, timeout)
|
|
235
383
|
|
|
236
384
|
by_ip = {}
|
|
237
|
-
with concurrent.futures.ThreadPoolExecutor(max_workers=8) as ex:
|
|
385
|
+
with concurrent.futures.ThreadPoolExecutor(max_workers=(4 if ISH else 8)) as ex:
|
|
238
386
|
for name, pairs in ex.map(one, uniq):
|
|
239
387
|
for ip, src in pairs:
|
|
240
388
|
srcs = by_ip.setdefault(ip, {"ips": set(), "sources": set()})
|
|
241
389
|
srcs["sources"].add(name)
|
|
390
|
+
for ip in passive_ips:
|
|
391
|
+
by_ip.setdefault(ip, {"ips": set(), "sources": set()})["sources"].add("passive-dns")
|
|
242
392
|
return by_ip
|
|
243
393
|
|
|
244
394
|
|
|
@@ -246,6 +396,7 @@ def main(argv=None):
|
|
|
246
396
|
args = list(sys.argv[1:] if argv is None else argv)
|
|
247
397
|
domains, as_json, no_verify, no_ct = [], False, False, False
|
|
248
398
|
max_c, timeout, budget = 40, 8.0, 120.0
|
|
399
|
+
ct_sources, max_ct, ct_budget = None, 80, 0.0
|
|
249
400
|
i = 0
|
|
250
401
|
while i < len(args):
|
|
251
402
|
a = args[i]
|
|
@@ -255,7 +406,7 @@ def main(argv=None):
|
|
|
255
406
|
no_verify = True
|
|
256
407
|
elif a == "--no-ct":
|
|
257
408
|
no_ct = True
|
|
258
|
-
elif a in ("--max-candidates", "--timeout", "--budget"):
|
|
409
|
+
elif a in ("--max-candidates", "--timeout", "--budget", "--max-ct", "--ct-budget", "--ct-sources"):
|
|
259
410
|
if i + 1 >= len(args):
|
|
260
411
|
_err("%s 缺少取值" % a)
|
|
261
412
|
return 2
|
|
@@ -264,6 +415,20 @@ def main(argv=None):
|
|
|
264
415
|
max_c = max(1, int(args[i + 1]))
|
|
265
416
|
elif a == "--budget":
|
|
266
417
|
budget = max(10.0, float(args[i + 1]))
|
|
418
|
+
elif a == "--max-ct":
|
|
419
|
+
max_ct = max(1, int(args[i + 1]))
|
|
420
|
+
elif a == "--ct-budget":
|
|
421
|
+
ct_budget = max(3.0, float(args[i + 1]))
|
|
422
|
+
elif a == "--ct-sources":
|
|
423
|
+
val = args[i + 1].strip()
|
|
424
|
+
if val.lower() == "all":
|
|
425
|
+
ct_sources = None
|
|
426
|
+
else:
|
|
427
|
+
picked = [x.strip() for x in val.split(",") if x.strip()]
|
|
428
|
+
if not picked or any(x not in CT_SOURCES for x in picked):
|
|
429
|
+
_err("--ct-sources 取值非法(可选: %s 或 all)" % ",".join(CT_SOURCES))
|
|
430
|
+
return 2
|
|
431
|
+
ct_sources = picked
|
|
267
432
|
else:
|
|
268
433
|
timeout = max(1.0, min(float(args[i + 1]), 30.0))
|
|
269
434
|
except (TypeError, ValueError):
|
|
@@ -301,7 +466,8 @@ def main(argv=None):
|
|
|
301
466
|
except Exception as e:
|
|
302
467
|
_err("[基线] 探测失败: %s(继续)" % e)
|
|
303
468
|
|
|
304
|
-
by_ip = collect_candidates(domain, max_c, timeout, use_ct=not no_ct
|
|
469
|
+
by_ip = collect_candidates(domain, max_c, timeout, use_ct=not no_ct,
|
|
470
|
+
ct_sources=ct_sources, max_ct=max_ct, ct_budget=ct_budget)
|
|
305
471
|
for ip in real_root: # 根 IP 纳入候选
|
|
306
472
|
by_ip.setdefault(ip, {"sources": set()})["sources"].add(domain)
|
|
307
473
|
|
|
@@ -355,6 +355,40 @@ class HangProtectionTests(unittest.TestCase):
|
|
|
355
355
|
self.assertIn("same-cdn", r_cdn)
|
|
356
356
|
self.assertLess(s_cdn, s_org)
|
|
357
357
|
|
|
358
|
+
def test_passive_ct_parsers(self):
|
|
359
|
+
cs = '[{"dns_names":["www.example.com","*.example.com","x.test"]}]'
|
|
360
|
+
self.assertEqual(self.ot.parse_certspotter(cs, "example.com"), ["www.example.com", "example.com"])
|
|
361
|
+
otx = ('{"passive_dns":[{"hostname":"mail.example.com","address":"1.2.3.4"},'
|
|
362
|
+
'{"hostname":"x.test","address":"9.9.9.9"},'
|
|
363
|
+
'{"hostname":"cdn.example.com","address":"198.18.0.9"}]}')
|
|
364
|
+
hosts, ips = self.ot.parse_otx(otx, "example.com")
|
|
365
|
+
self.assertIn("mail.example.com", hosts)
|
|
366
|
+
self.assertEqual(ips, ["1.2.3.4"], "被动 DNS 的 fake-IP 应被过滤")
|
|
367
|
+
hosts, ips = self.ot.parse_hackertarget("vpn.example.com,5.6.7.8\nAPI count exceeded", "example.com")
|
|
368
|
+
self.assertEqual(hosts, ["vpn.example.com"])
|
|
369
|
+
self.assertEqual(ips, ["5.6.7.8"])
|
|
370
|
+
self.assertEqual(self.ot.parse_anubis('["a.example.com","b.test"]', "example.com"), ["a.example.com"])
|
|
371
|
+
self.assertEqual(self.ot.parse_rapiddns("<td>oa.example.com</td><td>x.test</td>", "example.com"),
|
|
372
|
+
["oa.example.com"])
|
|
373
|
+
for src in ("crt.sh", "certspotter", "otx", "hackertarget", "anubis", "rapiddns"):
|
|
374
|
+
self.assertIn(src, self.ot.CT_SOURCES)
|
|
375
|
+
|
|
376
|
+
def test_ct_multi_source_fallback(self):
|
|
377
|
+
def fake(name, domain, timeout):
|
|
378
|
+
if name == "crt.sh":
|
|
379
|
+
raise RuntimeError("502 Bad Gateway")
|
|
380
|
+
return (["s1.example.com", "s2.example.com"], ["1.2.3.4"])
|
|
381
|
+
orig = self.ot._ct_fetch
|
|
382
|
+
self.ot._ct_fetch = fake
|
|
383
|
+
try:
|
|
384
|
+
hosts, ips, status = self.ot.fetch_passive_names(
|
|
385
|
+
"example.com", sources=["crt.sh", "otx"], budget=5)
|
|
386
|
+
finally:
|
|
387
|
+
self.ot._ct_fetch = orig
|
|
388
|
+
self.assertEqual(hosts, ["s1.example.com", "s2.example.com"])
|
|
389
|
+
self.assertEqual(ips, ["1.2.3.4"])
|
|
390
|
+
self.assertIn("失败", status["crt.sh"], "单源 502 应被记录并跳过")
|
|
391
|
+
|
|
358
392
|
def test_origin_bad_flag_rejected(self):
|
|
359
393
|
r = _tmp_run([sys.executable, str(SCRIPTS / "origin_trace.py"), "--bogus"])
|
|
360
394
|
self.assertEqual(r.returncode, 2)
|