wimotek-chromium-sync 1.3.10 → 1.3.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +151 -198
- package/package.json +1 -1
- package/wimotek_chromium_sync/core.py +152 -165
package/README.md
CHANGED
|
@@ -1,263 +1,216 @@
|
|
|
1
1
|
# wimotek-chromium-sync
|
|
2
2
|
|
|
3
3
|
[](https://pypi.org/project/wimotek-chromium-sync/)
|
|
4
|
-
[](https://www.npmjs.com/package/wimotek-chromium-sync)
|
|
5
4
|
[](LICENSE)
|
|
6
5
|
|
|
7
|
-
专为 Chromium / Chrome 设计的生产级会话与 Cookie
|
|
6
|
+
专为 Chromium / Chrome 设计的生产级会话与 Cookie 集群同步调度引擎。
|
|
7
|
+
**专注于多节点凭据同步、差量指纹比对、域名清洗与 CDP Hub 4 种连接模式调度**。SSH 隧道逻辑已彻底解耦，委托由通用的独立模块 [`wimotek-ssh-tunnel`](https://github.com/wimotek/wimotek-ssh-tunnel) 统一承载。
|
|
8
8
|
|
|
9
9
|
---
|
|
10
10
|
|
|
11
11
|
## 🌟 核心工程特性
|
|
12
12
|
|
|
13
|
-
1.
|
|
14
|
-
*
|
|
15
|
-
*
|
|
16
|
-
2.
|
|
17
|
-
*
|
|
18
|
-
*
|
|
19
|
-
|
|
13
|
+
1. **全面对接 CDP Hub 4 种连接方式 (Hub Matrix)**：
|
|
14
|
+
* 原生兼容 **1. 直连 (Direct)**、**2. wss直连 (WSS Direct)**、**3. wss反穿 (WSS Reverse)** 与 **4. ssh穿透 (SSH Tunnel)**。
|
|
15
|
+
* 对于 SSH 穿透模式，无缝对接独立通用模块 `wimotek-ssh-tunnel`，本模块纯粹专注于同步业务，架构清晰解耦。
|
|
16
|
+
2. **高容错跳过与强化日志输出 (Resilient Skip & Enhanced Logging)**：
|
|
17
|
+
* 单个节点离线、握手超时或异常时，**绝不抛出未捕获异常崩溃中断流程**。
|
|
18
|
+
* 自动输出标准结构化告警日志：`⚠️ [SKIP] 节点 [节点名] 离线或连接异常 (URL)，自动跳过: <原因>`。
|
|
19
|
+
* 执行结果包含 `nodes_synced`、`nodes_skipped` 与 `skips` 详细审计诊断列表。
|
|
20
|
+
* **源节点离线明确通知并终止 (`⚠️ [SOURCE_OFFLINE]`)**：在单向 `sync` 广播模式下，若指定的数据源节点宕机或异常，引擎立即输出明确通知日志并直接终止流程（源节点下线无需继续后续同步），返回完整的跳过审计信息。
|
|
21
|
+
3. **域名白名单 / 黑名单过滤机制 (Domain Filter)**：
|
|
22
|
+
* **`include_domains`**：仅提取与同步关键业务站点 Cookie（如 `['.google.com', '.github.com', '.twitter.com']`），剔除海量无关数据。
|
|
23
|
+
* **`exclude_domains`**：过滤广告、埋点分析类垃圾 Cookie（如 `['doubleclick.net', 'google-analytics.com']`），保护从属端隐私与纯净度。
|
|
24
|
+
4. **会话级连接池复用 (Connection Pooling)**：
|
|
20
25
|
* 原生支持异步与同步上下文管理器：`async with ChromiumSync(...) as syncer:`。
|
|
21
|
-
*
|
|
22
|
-
|
|
23
|
-
* 默认开启 `diff=True
|
|
24
|
-
*
|
|
25
|
-
|
|
26
|
-
* `
|
|
27
|
-
* `ssh_password`: 直接支持 SSH 密码认证，支持集群级默认凭证与单节点自定义。
|
|
28
|
-
6. **结构化执行指标返回 (Structured Return Dict)**：
|
|
29
|
-
* 执行后返回包含 `status`、`transferred_count`、`diff_skipped_count`、`duration` 等度量指标的标准字典，Web 控制台可直接消费展示。
|
|
30
|
-
7. **全量纯函数工具方法开箱即用**：
|
|
31
|
-
* 模块顶级导出 `merge_cookies`、`filter_cookies`、`compute_cookie_delta` 等独立纯函数工具，无需 CDP 环境亦可直接处理和计算 Cookie 差量。
|
|
26
|
+
* 在高频定时同步场景下，复用底层常驻 WebSocket CDP 会话，避免重复握手损耗。
|
|
27
|
+
5. **Cookie 指纹变更差量同步 (Diff / Delta Sync)**：
|
|
28
|
+
* 默认开启 `diff=True`：比对目标端与待同步凭据的核心指纹 `(name, domain, path, value)`。
|
|
29
|
+
* 仅对实质变更的 Cookie 执行落盘与热重载，避免无效 IO 读写，**大幅延长从属端 SSD 寿命**。
|
|
30
|
+
6. **全量纯函数工具方法开箱即用**：
|
|
31
|
+
* 模块顶级导出 `merge_cookies`、`filter_cookies`、`compute_cookie_delta` 等独立纯函数工具。
|
|
32
32
|
|
|
33
33
|
---
|
|
34
34
|
|
|
35
|
-
##
|
|
35
|
+
## 🌐 与 CDP Hub 端的对接规范 (4 种连接方式)
|
|
36
36
|
|
|
37
|
-
|
|
38
|
-
```bash
|
|
39
|
-
pip install wimotek-chromium-sync
|
|
40
|
-
playwright install chromium
|
|
41
|
-
```
|
|
42
|
-
|
|
43
|
-
### Node.js (npm)
|
|
44
|
-
```bash
|
|
45
|
-
npm install -g wimotek-chromium-sync
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
---
|
|
37
|
+
CDP Hub 在管理大规模分布式浏览器节点时，节点所处的网络拓扑各不相同。`wimotek-chromium-sync` 原生支持所有 4 种连接模式：
|
|
49
38
|
|
|
50
|
-
|
|
39
|
+
| 模式 | 连接方式 | 传输协议 | Hub 拓扑与工作机制 | 是否需要 SSH 隧道？ |
|
|
40
|
+
| :--- | :--- | :--- | :--- | :--- |
|
|
41
|
+
| **1. 直连** | Direct CDP | HTTP / TCP (`ws://`) | Hub 直接请求目标节点的 CDP 端口，如 `http://192.168.1.50:9222/json/list`。 | ❌ **不需要**（内网或直连端口互通） |
|
|
42
|
+
| **2. wss直连** | WSS Direct | WSS over TLS | Hub 经由公网反向代理或安全网关（如 `:8443`），支持传入鉴权 Header（如 `Bearer Token`）建立加密直连。 | ❌ **不需要**（公网 TLS 网关直通） |
|
|
43
|
+
| **3. wss反穿** | WSS Reverse | Reverse WebSocket | 边缘设备（如 NAT 后的移动端或无公网机器）主动出站向 Hub 的反向监听端点建立连接，Hub 从内存会话池中直接消费。 | ❌ **不需要**（端侧出站建立连接） |
|
|
44
|
+
| **4. ssh穿透** | **SSH Tunnel** | **Encrypted SSH (Port 22)** | 远程机器 Chrome 严格监听在 `127.0.0.1:9222` 且受安全组阻断。**委托独立通用库 `wimotek-ssh-tunnel`** 建立本地端口转发（`127.0.0.1:19222 -> remote:9222`）供同步引擎调用。 | ✅ **由独立库 `wimotek-ssh-tunnel` 承载** |
|
|
51
45
|
|
|
52
|
-
###
|
|
53
|
-
使用 `async with` 上下文管理器常驻复用 CDP 连接与隧道，杜绝频繁断连与握手损耗：
|
|
54
|
-
```python
|
|
55
|
-
import asyncio
|
|
56
|
-
from chromium_sync import ChromiumSync
|
|
46
|
+
### Hub 端配置示例 (`nodes.json`)
|
|
57
47
|
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
48
|
+
```json
|
|
49
|
+
[
|
|
50
|
+
{
|
|
51
|
+
"id": "node-direct",
|
|
52
|
+
"name": "本地开发/内网容器",
|
|
53
|
+
"cdp_host": "192.168.1.100",
|
|
54
|
+
"cdp_port": 9222
|
|
55
|
+
},
|
|
56
|
+
{
|
|
57
|
+
"id": "node-wss-direct",
|
|
58
|
+
"name": "海外云网关节点 (WSS直连)",
|
|
59
|
+
"endpoint_url": "wss://gateway.cluster.internal:8443/devtools/browser/node-us-1",
|
|
60
|
+
"headers": {
|
|
61
|
+
"Authorization": "Bearer wimotek-secret-token"
|
|
62
|
+
}
|
|
63
|
+
},
|
|
64
|
+
{
|
|
65
|
+
"id": "node-wss-reverse",
|
|
66
|
+
"name": "移动边缘设备 (WSS反穿)",
|
|
67
|
+
"endpoint_url": "ws://127.0.0.1:9000/reverse/dev-phone-01"
|
|
68
|
+
},
|
|
69
|
+
{
|
|
70
|
+
"id": "node-ssh-tunnel",
|
|
71
|
+
"name": "生产私有主机 (SSH穿透)",
|
|
72
|
+
"ssh_host": "10.200.1.50",
|
|
73
|
+
"ssh_user": "root",
|
|
74
|
+
"cdp_port": 9222,
|
|
75
|
+
"tunnel_port": 19225
|
|
76
|
+
}
|
|
77
|
+
]
|
|
76
78
|
```
|
|
77
79
|
|
|
78
80
|
---
|
|
79
81
|
|
|
80
|
-
|
|
81
|
-
```python
|
|
82
|
-
from chromium_sync import ChromiumSync
|
|
82
|
+
## 🛡️ 强化日志输出与异常跳过机制
|
|
83
83
|
|
|
84
|
-
|
|
85
|
-
nodes=[
|
|
86
|
-
{"id": "local", "cdp_port": 9223},
|
|
87
|
-
{"id": "kr2", "ssh_host": "kr2.wimotek.com", "ssh_user": "root", "ssh_key": "~/.ssh/id_rsa", "cdp_port": 9222},
|
|
88
|
-
{"id": "sg1", "ssh_host": "sg1.wimotek.com", "ssh_user": "ubuntu", "ssh_password": "YourPassword", "cdp_port": 9222}
|
|
89
|
-
],
|
|
90
|
-
mode="merge",
|
|
91
|
-
# 白名单：仅同步以下业务站点的 Cookie
|
|
92
|
-
include_domains=["google.com", "twitter.com", "x.com", "github.com"],
|
|
93
|
-
# 黑名单：过滤广告与追踪类 Cookie
|
|
94
|
-
exclude_domains=["doubleclick.net", "google-analytics.com"],
|
|
95
|
-
# 差量比对 (默认 True)
|
|
96
|
-
diff=True
|
|
97
|
-
)
|
|
84
|
+
在生产级分布式批量同步时，网络抖动或单台节点死机会常态发生。`wimotek-chromium-sync` 具备强化的日志输出机制与跳过策略：
|
|
98
85
|
|
|
99
|
-
|
|
100
|
-
|
|
86
|
+
### 1. 节点离线自动跳过日志
|
|
87
|
+
当任何节点无法连通、握手超时或鉴权失败时，输出带有明显标志的警告日志并记录上下文：
|
|
88
|
+
```text
|
|
89
|
+
2026-11-07 07:46:11 [WARNING] ⚠️ [SKIP] 节点 [Node-US-West] 离线或连接异常 (http://10.200.1.20:19222)，自动跳过: Connection refused: 127.0.0.1:19222
|
|
90
|
+
2026-11-07 07:46:12 [WARNING] ⚠️ [SKIP] 节点 [Node-SG-Edge] 离线或连接异常 (wss://gateway:8443/...), 自动跳过: HTTP 401 Unauthorized
|
|
101
91
|
```
|
|
102
|
-
|
|
92
|
+
|
|
93
|
+
### 2. 单向同步源节点离线终止通知 (`⚠️ [SOURCE_OFFLINE]`)
|
|
94
|
+
在单向 `sync` 广播模式下，数据源节点是凭据基准。若指定的源节点离线或异常，没必要继续后续流程，调度器会打印明确错误通知并立即终止：
|
|
95
|
+
```text
|
|
96
|
+
2026-11-07 07:49:10 [ERROR] ⚠️ [SOURCE_OFFLINE] 单向同步指定的源节点 [Master-Node] (node-master) 离线或连接异常，无法提取基准凭据，同步流程终止。
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
### 3. 返回结构化审计指标
|
|
100
|
+
无论执行成功还是遇到故障，同步方法均返回结构化字典，供 Hub 控制台实时消费与告警统计：
|
|
103
101
|
```python
|
|
104
102
|
{
|
|
105
|
-
"status": "success",
|
|
106
|
-
"mode": "merge",
|
|
107
|
-
"transferred_count":
|
|
108
|
-
"diff_skipped_count":
|
|
109
|
-
"source_total":
|
|
110
|
-
"target_total":
|
|
111
|
-
"nodes_synced": ["
|
|
112
|
-
"
|
|
113
|
-
"
|
|
103
|
+
"status": "success", # "success" (部分或全部成功) | "failed" (全部离线)
|
|
104
|
+
"mode": "merge", # "merge" | "sync"
|
|
105
|
+
"transferred_count": 14, # 本次真正产生指纹变更并落盘的凭证数
|
|
106
|
+
"diff_skipped_count": 86, # 指纹一致、跳过重复写入保护磁盘的凭证数
|
|
107
|
+
"source_total": 100, # 过滤后的业务总凭证数
|
|
108
|
+
"target_total": 100,
|
|
109
|
+
"nodes_synced": ["node-direct", "node-wss-direct"], # 成功同步的节点
|
|
110
|
+
"nodes_skipped": ["node-ssh-tunnel"], # 自动跳过的异常节点 ID
|
|
111
|
+
"skips": [ # 详细跳过事件记录
|
|
112
|
+
{
|
|
113
|
+
"id": "node-ssh-tunnel",
|
|
114
|
+
"name": "生产私有主机 (SSH穿透)",
|
|
115
|
+
"mode": "ssh_tunnel",
|
|
116
|
+
"cdp_url": "http://127.0.0.1:19225",
|
|
117
|
+
"reason": "Connection refused / ConnectTimeout"
|
|
118
|
+
}
|
|
119
|
+
],
|
|
120
|
+
"duration": 1.12, # 耗时（秒）
|
|
121
|
+
"details": "全节点 Merge 成功: 写入 14 条凭证 (跳过异常节点: ['生产私有主机 (SSH穿透)'])",
|
|
114
122
|
"error": None
|
|
115
123
|
}
|
|
116
124
|
```
|
|
117
125
|
|
|
118
126
|
---
|
|
119
127
|
|
|
120
|
-
|
|
121
|
-
单点探测某个节点时，直接 1 行静态调用：
|
|
122
|
-
```python
|
|
123
|
-
from chromium_sync import ChromiumSync
|
|
124
|
-
|
|
125
|
-
# 1. 单点探测节点在线状态与页面信息
|
|
126
|
-
info = ChromiumSync.probe(port=9222)
|
|
127
|
-
print(info)
|
|
128
|
-
# {'online': True, 'cookie_count': 42, 'page_title': 'GitHub', 'page_url': 'https://github.com', ...}
|
|
129
|
-
|
|
130
|
-
# 2. 单点直接提取 Cookie (支持域名过滤)
|
|
131
|
-
cookies = ChromiumSync.get_cookies(port=9222, include_domains=["github.com"])
|
|
132
|
-
|
|
133
|
-
# 3. 单点写入 Cookie 并启用差量检测
|
|
134
|
-
ChromiumSync.set_cookies(cookies, port=9223, reload=True, diff=True)
|
|
135
|
-
```
|
|
136
|
-
|
|
137
|
-
---
|
|
128
|
+
## 🚀 极简开发示例
|
|
138
129
|
|
|
139
|
-
###
|
|
140
|
-
即使没有运行中的 CDP 浏览器，亦可使用顶级导出的纯函数对 Cookie 结构做离线处理：
|
|
130
|
+
### 1. Hub 多模式混合集群融合 (全节点最新凭证合并)
|
|
141
131
|
```python
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
132
|
+
import asyncio
|
|
133
|
+
from wimotek_chromium_sync import ChromiumSync
|
|
134
|
+
|
|
135
|
+
nodes = [
|
|
136
|
+
# 模式 1: 直连
|
|
137
|
+
{"id": "local", "name": "本机容器", "cdp_port": 9222},
|
|
138
|
+
# 模式 2: WSS 直连 (带 Token)
|
|
139
|
+
{
|
|
140
|
+
"id": "hub-remote",
|
|
141
|
+
"name": "海外网关",
|
|
142
|
+
"endpoint_url": "wss://hub.domain.internal:8443/devtools/browser/abc",
|
|
143
|
+
"headers": {"Authorization": "Bearer secret_token"}
|
|
144
|
+
},
|
|
145
|
+
# 模式 3: WSS 反穿
|
|
146
|
+
{"id": "mobile-edge", "name": "反向移动端", "endpoint_url": "ws://127.0.0.1:9000/reverse/phone-1"},
|
|
147
|
+
# 模式 4: SSH 穿透 (委托 wimotek-ssh-tunnel 管理端口映射)
|
|
148
|
+
{"id": "private-srv", "name": "专网主机", "cdp_port": 19225}
|
|
147
149
|
]
|
|
148
150
|
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
merged = merge_cookies([[c1, c2], [c3]])
|
|
157
|
-
```
|
|
158
|
-
|
|
159
|
-
---
|
|
160
|
-
|
|
161
|
-
### 5. 命令行 (CLI) 快速执行
|
|
151
|
+
syncer = ChromiumSync(
|
|
152
|
+
nodes=nodes,
|
|
153
|
+
mode="merge",
|
|
154
|
+
include_domains=["google.com", "github.com"], # 仅同步核心业务站点
|
|
155
|
+
exclude_domains=["doubleclick.net"], # 剔除垃圾广告追踪
|
|
156
|
+
diff=True # 开启差量指纹比对保护磁盘
|
|
157
|
+
)
|
|
162
158
|
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
--include-domains google.com github.com twitter.com \
|
|
168
|
-
--exclude-domains doubleclick.net google-analytics.com \
|
|
169
|
-
--mode merge
|
|
159
|
+
report = syncer.run()
|
|
160
|
+
print(f"同步完成状态: {report['status']}")
|
|
161
|
+
print(f"成功同步节点: {report['nodes_synced']}")
|
|
162
|
+
print(f"自动跳过异常节点: {report['nodes_skipped']}")
|
|
170
163
|
```
|
|
171
164
|
|
|
172
|
-
|
|
173
|
-
```
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
--include-domains google.com github.com \
|
|
177
|
-
--status
|
|
178
|
-
```
|
|
179
|
-
输出样例：
|
|
180
|
-
```text
|
|
181
|
-
============================================================================================
|
|
182
|
-
节点 ID | 节点名称 | 类型 | CDP端口/映射 | Cookie有效数 | 当前页面状态
|
|
183
|
-
--------------------------------------------------------------------------------------------
|
|
184
|
-
local | 本地容器 | 本地直连 | 9223 | 18 条 | [GitHub] https://github.com
|
|
185
|
-
kr2 | 韩国生产服 | SSH远程 | 19222 | 16 条 | [Google] https://google.com
|
|
186
|
-
--------------------------------------------------------------------------------------------
|
|
187
|
-
过滤规则: [白名单: google.com, github.com]
|
|
188
|
-
============================================================================================
|
|
189
|
-
```
|
|
165
|
+
### 2. 结合独立 `wimotek-ssh-tunnel` 管理模式 4 隧道
|
|
166
|
+
```python
|
|
167
|
+
from wimotek_ssh_tunnel import SSHTunnelManager, TunnelConfig
|
|
168
|
+
from wimotek_chromium_sync import ChromiumSync
|
|
190
169
|
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
170
|
+
# 1. 由独立的 wimotek-ssh-tunnel 批量拉起 SSH 隧道映射
|
|
171
|
+
with SSHTunnelManager(start_port=19222) as tm:
|
|
172
|
+
tm.add_tunnel(TunnelConfig(id="srv1", ssh_host="10.200.1.10", ssh_user="root"))
|
|
173
|
+
active_ports = tm.start_all(skip_on_failure=True)
|
|
174
|
+
# active_ports => {"srv1": 19222}
|
|
195
175
|
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
176
|
+
# 2. 将映射后的端口送入 wimotek-chromium-sync 执行多端凭据同步
|
|
177
|
+
syncer = ChromiumSync(
|
|
178
|
+
nodes=[
|
|
179
|
+
{"id": "local", "cdp_port": 9222},
|
|
180
|
+
{"id": "srv1", "cdp_port": active_ports["srv1"]}
|
|
181
|
+
],
|
|
182
|
+
mode="sync",
|
|
183
|
+
source="local"
|
|
184
|
+
)
|
|
185
|
+
res = syncer.run()
|
|
186
|
+
print("同步指标:", res)
|
|
203
187
|
```
|
|
204
188
|
|
|
205
189
|
---
|
|
206
190
|
|
|
207
191
|
## ⚙️ 核心参数与配置规范
|
|
208
192
|
|
|
209
|
-
### `cluster.json` 配置文件结构
|
|
210
|
-
```json
|
|
211
|
-
[
|
|
212
|
-
{
|
|
213
|
-
"id": "local",
|
|
214
|
-
"name": "本机容器",
|
|
215
|
-
"cdp_port": 9222
|
|
216
|
-
},
|
|
217
|
-
{
|
|
218
|
-
"id": "kr2",
|
|
219
|
-
"name": "韩国服",
|
|
220
|
-
"ssh_host": "kr2.wimotek.com",
|
|
221
|
-
"ssh_port": 22,
|
|
222
|
-
"ssh_user": "root",
|
|
223
|
-
"ssh_key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----",
|
|
224
|
-
"cdp_port": 9222
|
|
225
|
-
},
|
|
226
|
-
{
|
|
227
|
-
"id": "sg1",
|
|
228
|
-
"name": "新加坡服",
|
|
229
|
-
"ssh_host": "sg1.wimotek.com",
|
|
230
|
-
"ssh_user": "ubuntu",
|
|
231
|
-
"ssh_password": "Password123",
|
|
232
|
-
"cdp_port": 9222
|
|
233
|
-
}
|
|
234
|
-
]
|
|
235
|
-
```
|
|
236
|
-
|
|
237
193
|
### `ChromiumSync` 初始化参数
|
|
238
194
|
|
|
239
195
|
| 参数字段 | 类型 | 默认值 | 详细说明 |
|
|
240
196
|
| :--- | :--- | :--- | :--- |
|
|
241
|
-
| `nodes` | list | 必填 | 节点配置列表 (支持 `NodeConfig` 或 dict) |
|
|
197
|
+
| `nodes` | list | 必填 | 节点配置列表 (支持 `NodeConfig` 或 dict，适配 4 种连接模式) |
|
|
242
198
|
| `mode` | str | `'sync'` | 同步策略: `'merge'`(全端融合最新凭证) 或 `'sync'`(单源广播) |
|
|
243
|
-
| `include_domains` | list | `None` |
|
|
199
|
+
| `include_domains` | list | `None` | **域名白名单**：仅同步关注的业务站点 Cookie (如 `['google.com']`) |
|
|
244
200
|
| `exclude_domains` | list | `None` | **域名黑名单**：排除广告/统计类 Cookie (如 `['doubleclick.net']`) |
|
|
245
|
-
| `diff` | bool | `True` |
|
|
246
|
-
| `source` | str | 第一个节点 | `'sync'`
|
|
201
|
+
| `diff` | bool | `True` | **指纹差量同步**：相同凭证跳过写入，避免频繁刷盘保护 SSD 寿命 |
|
|
202
|
+
| `source` | str | 第一个节点 | `'sync'` 模式下的基准源节点 ID (离线时打印错误通知并直接终止同步) |
|
|
247
203
|
| `targets` | list | 其余节点 | `'sync'` 模式下的目标节点 ID 列表 |
|
|
248
|
-
| `
|
|
249
|
-
| `ssh_password` | str | `None` | 集群默认 SSH 密码认证 |
|
|
250
|
-
| `auto_close_tunnel` | bool | `False` | 任务完成后是否自动关闭创建的 SSH 隧道 |
|
|
251
|
-
| `verbose` | bool | `True` | 是否输出详细调度日志 |
|
|
204
|
+
| `verbose` | bool | `True` | 是否输出详细强化日志 (`⚠️ [SKIP]`, `⚠️ [SOURCE_OFFLINE]`) |
|
|
252
205
|
|
|
253
206
|
---
|
|
254
207
|
|
|
255
208
|
## 📚 模块导出接口一览 (`__all__`)
|
|
256
209
|
|
|
257
210
|
```python
|
|
258
|
-
from
|
|
211
|
+
from wimotek_chromium_sync import (
|
|
259
212
|
ChromiumSync, # 主集群调度器与连接池上下文
|
|
260
|
-
NodeConfig, #
|
|
213
|
+
NodeConfig, # 统一节点配置数据模型 (支持 4 种 CDP 连接模式)
|
|
261
214
|
merge_cookies, # 多节点 Cookie 智能融合纯函数
|
|
262
215
|
filter_cookies, # 域名白名单/黑名单过滤纯函数
|
|
263
216
|
compute_cookie_delta # Cookie 指纹变更差量计算纯函数
|
package/package.json
CHANGED
|
@@ -125,15 +125,17 @@ def compute_cookie_delta(
|
|
|
125
125
|
@dataclass
|
|
126
126
|
class NodeConfig:
|
|
127
127
|
"""
|
|
128
|
-
统一节点配置
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
* ssh_password: SSH 密码认证
|
|
128
|
+
统一节点配置 (支持 CDP Hub 全部 4 种连接方式与离线弹性跳过)
|
|
129
|
+
1. 直连 (direct): 本地或内网直接访问 cdp_port，如 http://192.168.1.100:9222
|
|
130
|
+
2. wss直连 (wss_direct): TLS 网关加密直连，如 wss://hub.internal:8443/devtools/...
|
|
131
|
+
3. wss反穿 (wss_reverse): 边缘端向 Hub 反向长连接的会话映射
|
|
132
|
+
4. ssh穿透 (ssh_tunnel): 通过 wimotek-ssh-tunnel 模块建立加密端口转发 (如 19222 -> 9222)
|
|
134
133
|
"""
|
|
135
134
|
id: str
|
|
136
135
|
name: str = ""
|
|
136
|
+
mode: str = "direct" # "direct" | "wss_direct" | "wss_reverse" | "ssh_tunnel"
|
|
137
|
+
endpoint_url: Optional[str] = None # 显式指定完整 CDP 调试 URL (如 wss://... 或 ws://...)
|
|
138
|
+
headers: Optional[Dict[str, str]] = None # 可选 WebSocket/HTTP 鉴权请求头 (如 Bearer token)
|
|
137
139
|
is_local: bool = False
|
|
138
140
|
cdp_host: str = "127.0.0.1"
|
|
139
141
|
cdp_port: int = 9222
|
|
@@ -150,32 +152,52 @@ class NodeConfig:
|
|
|
150
152
|
if not self.name:
|
|
151
153
|
self.name = self.id
|
|
152
154
|
|
|
153
|
-
#
|
|
154
|
-
if
|
|
155
|
+
# 智能判定连接模式
|
|
156
|
+
if self.endpoint_url:
|
|
157
|
+
raw_url = self.endpoint_url.strip()
|
|
158
|
+
if raw_url.startswith("wss://"):
|
|
159
|
+
self.mode = "wss_direct"
|
|
160
|
+
elif raw_url.startswith("ws://") and ("reverse" in raw_url or "agent" in raw_url):
|
|
161
|
+
self.mode = "wss_reverse"
|
|
162
|
+
elif raw_url.startswith("http://") or raw_url.startswith("ws://"):
|
|
163
|
+
self.mode = "direct"
|
|
164
|
+
self.is_local = False
|
|
165
|
+
elif self.ssh_host and self.ssh_host.strip() not in ("127.0.0.1", "localhost", "::1", ""):
|
|
166
|
+
self.mode = "ssh_tunnel"
|
|
167
|
+
self.is_local = False
|
|
168
|
+
else:
|
|
169
|
+
self.mode = "direct"
|
|
155
170
|
self.is_local = True
|
|
156
171
|
|
|
157
|
-
#
|
|
158
|
-
if self.is_local:
|
|
172
|
+
# 本地免密直连无需隧道映射
|
|
173
|
+
if self.is_local and not self.tunnel_port:
|
|
159
174
|
self.tunnel_port = self.cdp_port
|
|
160
175
|
|
|
161
|
-
# 智能解析 ssh_key
|
|
176
|
+
# 智能解析 ssh_key
|
|
162
177
|
key_input = self.ssh_key or self.ssh_key_path or self.ssh_key_data
|
|
163
178
|
if key_input:
|
|
164
179
|
key_str = key_input.strip()
|
|
165
180
|
if "-----BEGIN" in key_str or "\n" in key_str:
|
|
166
181
|
self.ssh_key_data = key_str
|
|
167
182
|
else:
|
|
168
|
-
|
|
169
|
-
if os.path.exists(expanded):
|
|
170
|
-
self.ssh_key_path = expanded
|
|
171
|
-
else:
|
|
172
|
-
self.ssh_key_path = expanded
|
|
183
|
+
self.ssh_key_path = os.path.expanduser(key_str)
|
|
173
184
|
|
|
174
185
|
@property
|
|
175
186
|
def cdp_url(self) -> str:
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
187
|
+
# 1. 显式指定端点 URL (支持 wss://, ws://, http://)
|
|
188
|
+
if self.endpoint_url:
|
|
189
|
+
return self.endpoint_url.strip()
|
|
190
|
+
|
|
191
|
+
# 2. SSH 穿透模式 (通过本地映射端口访问)
|
|
192
|
+
if self.mode == "ssh_tunnel":
|
|
193
|
+
return f"http://127.0.0.1:{self.tunnel_port or 19222}"
|
|
194
|
+
|
|
195
|
+
# 3. WSS 直连模式
|
|
196
|
+
if self.mode == "wss_direct":
|
|
197
|
+
return f"wss://{self.cdp_host}:{self.cdp_port}"
|
|
198
|
+
|
|
199
|
+
# 4. 直连模式 (默认)
|
|
200
|
+
return f"http://{self.cdp_host}:{self.cdp_port}"
|
|
179
201
|
|
|
180
202
|
|
|
181
203
|
def merge_cookies(
|
|
@@ -225,140 +247,26 @@ def merge_cookies(
|
|
|
225
247
|
return list(merged_map.values())
|
|
226
248
|
|
|
227
249
|
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
- 支持密钥文本 (PEM String) 自动写入安全受限临时文件
|
|
232
|
-
- 支持密钥路径与密码认证
|
|
233
|
-
- 支持端口复用检测与防僵尸连接自愈
|
|
234
|
-
"""
|
|
235
|
-
def __init__(self, node: NodeConfig):
|
|
236
|
-
self.node = node
|
|
237
|
-
self.proc: Optional[subprocess.Popen] = None
|
|
238
|
-
self.is_reused: bool = False
|
|
239
|
-
self.temp_key_file: Optional[str] = None
|
|
240
|
-
self.askpass_script: Optional[str] = None
|
|
241
|
-
|
|
242
|
-
def _is_port_in_use(self, port: int) -> bool:
|
|
243
|
-
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
|
|
244
|
-
s.settimeout(0.3)
|
|
245
|
-
return s.connect_ex(("127.0.0.1", port)) == 0
|
|
246
|
-
|
|
247
|
-
def _is_cdp_ready(self, url: str) -> bool:
|
|
248
|
-
try:
|
|
249
|
-
proxy_handler = urllib.request.ProxyHandler({})
|
|
250
|
-
opener = urllib.request.build_opener(proxy_handler)
|
|
251
|
-
resp = opener.open(f"{url}/json/version", timeout=1.5)
|
|
252
|
-
return resp.getcode() == 200
|
|
253
|
-
except Exception:
|
|
254
|
-
return False
|
|
255
|
-
|
|
256
|
-
def start(self, auto_close: bool = False):
|
|
257
|
-
if self.node.is_local:
|
|
258
|
-
return
|
|
259
|
-
|
|
260
|
-
target_local_port = self.node.tunnel_port
|
|
261
|
-
target_url = f"http://127.0.0.1:{target_local_port}"
|
|
262
|
-
|
|
263
|
-
if self._is_port_in_use(target_local_port):
|
|
264
|
-
if self._is_cdp_ready(target_url):
|
|
265
|
-
logging.info(f"[*] 节点 [{self.node.name}] 本地端口 {target_local_port} 已通畅，复用现有连接。")
|
|
266
|
-
self.is_reused = True
|
|
267
|
-
return
|
|
268
|
-
|
|
269
|
-
logging.info(
|
|
270
|
-
f"[*] 正在拉起 SSH 隧道: {self.node.name} -> {self.node.ssh_user}@{self.node.ssh_host}:{self.node.ssh_port} "
|
|
271
|
-
f"(本地 127.0.0.1:{target_local_port} -> 远程 127.0.0.1:{self.node.cdp_port}) ..."
|
|
272
|
-
)
|
|
273
|
-
|
|
274
|
-
ssh_cmd = [
|
|
275
|
-
"ssh", "-N",
|
|
276
|
-
"-p", str(self.node.ssh_port),
|
|
277
|
-
"-o", "ServerAliveInterval=15",
|
|
278
|
-
"-o", "ServerAliveCountMax=3",
|
|
279
|
-
"-o", "TCPKeepAlive=yes",
|
|
280
|
-
"-o", "StrictHostKeyChecking=accept-new",
|
|
281
|
-
"-o", "ExitOnForwardFailure=yes",
|
|
282
|
-
"-L", f"{target_local_port}:127.0.0.1:{self.node.cdp_port}"
|
|
283
|
-
]
|
|
284
|
-
|
|
285
|
-
env = os.environ.copy()
|
|
286
|
-
|
|
287
|
-
if self.node.ssh_key_data:
|
|
288
|
-
fd, tmp_path = tempfile.mkstemp(prefix="ssh_key_")
|
|
289
|
-
with open(tmp_path, "w") as f:
|
|
290
|
-
f.write(self.node.ssh_key_data.strip() + "\n")
|
|
291
|
-
os.close(fd)
|
|
292
|
-
os.chmod(tmp_path, 0o600)
|
|
293
|
-
self.temp_key_file = tmp_path
|
|
294
|
-
ssh_cmd.extend(["-i", tmp_path])
|
|
295
|
-
elif self.node.ssh_key_path:
|
|
296
|
-
expanded_key = os.path.expanduser(self.node.ssh_key_path)
|
|
297
|
-
if os.path.exists(expanded_key):
|
|
298
|
-
ssh_cmd.extend(["-i", expanded_key])
|
|
299
|
-
|
|
300
|
-
if self.node.ssh_password:
|
|
301
|
-
if shutil.which("sshpass"):
|
|
302
|
-
ssh_cmd = ["sshpass", "-p", self.node.ssh_password] + ssh_cmd
|
|
303
|
-
else:
|
|
304
|
-
fd, askpass_path = tempfile.mkstemp(prefix="askpass_", suffix=".sh")
|
|
305
|
-
escaped_pw = self.node.ssh_password.replace("'", "'\\''")
|
|
306
|
-
with open(askpass_path, "w") as f:
|
|
307
|
-
f.write(f"#!/bin/sh\necho '{escaped_pw}'\n")
|
|
308
|
-
os.close(fd)
|
|
309
|
-
os.chmod(askpass_path, 0o700)
|
|
310
|
-
self.askpass_script = askpass_path
|
|
311
|
-
env["SSH_ASKPASS"] = askpass_path
|
|
312
|
-
env["SSH_ASKPASS_REQUIRE"] = "force"
|
|
313
|
-
env["DISPLAY"] = ":0"
|
|
314
|
-
ssh_cmd.extend(["-o", "BatchMode=no"])
|
|
315
|
-
|
|
316
|
-
ssh_cmd.append(f"{self.node.ssh_user}@{self.node.ssh_host}")
|
|
317
|
-
|
|
318
|
-
try:
|
|
319
|
-
self.proc = subprocess.Popen(
|
|
320
|
-
ssh_cmd,
|
|
321
|
-
env=env,
|
|
322
|
-
stdout=subprocess.DEVNULL,
|
|
323
|
-
stderr=subprocess.DEVNULL,
|
|
324
|
-
start_new_session=(not auto_close)
|
|
325
|
-
)
|
|
326
|
-
except FileNotFoundError:
|
|
327
|
-
raise RuntimeError(f"[-] 系统未找到 ssh 可执行文件，无法为远程节点 [{self.node.name}] 建立隧道！")
|
|
328
|
-
|
|
329
|
-
for _ in range(25):
|
|
330
|
-
time.sleep(0.4)
|
|
331
|
-
if self._is_port_in_use(target_local_port) and self._is_cdp_ready(target_url):
|
|
332
|
-
logging.info(f"[+] 节点 [{self.node.name}] SSH 隧道建立成功 (端口: {target_local_port})。")
|
|
333
|
-
return
|
|
334
|
-
if self.proc.poll() is not None:
|
|
335
|
-
break
|
|
336
|
-
|
|
337
|
-
self.stop(force=True)
|
|
338
|
-
raise RuntimeError(f"[-] 建立 SSH 隧道失败 [{self.node.name}]，请检查网络或认证凭据！")
|
|
250
|
+
# =============================================================================
|
|
251
|
+
# SSH 隧道管理 (已解耦剥离至独立通用模块 wimotek-ssh-tunnel)
|
|
252
|
+
# =============================================================================
|
|
339
253
|
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
pass
|
|
357
|
-
if self.askpass_script and os.path.exists(self.askpass_script):
|
|
358
|
-
try:
|
|
359
|
-
os.remove(self.askpass_script)
|
|
360
|
-
except Exception:
|
|
361
|
-
pass
|
|
254
|
+
try:
|
|
255
|
+
from wimotek_ssh_tunnel import SSHTunnel, SSHTunnelManager, SshTunnelSession, TunnelConfig, TunnelException
|
|
256
|
+
HAS_SSH_TUNNEL = True
|
|
257
|
+
except ImportError:
|
|
258
|
+
try:
|
|
259
|
+
import sys
|
|
260
|
+
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "wimotek-ssh-tunnel")))
|
|
261
|
+
from wimotek_ssh_tunnel import SSHTunnel, SSHTunnelManager, SshTunnelSession, TunnelConfig, TunnelException
|
|
262
|
+
HAS_SSH_TUNNEL = True
|
|
263
|
+
except Exception:
|
|
264
|
+
HAS_SSH_TUNNEL = False
|
|
265
|
+
SSHTunnel = None
|
|
266
|
+
SSHTunnelManager = None
|
|
267
|
+
SshTunnelSession = None
|
|
268
|
+
TunnelConfig = None
|
|
269
|
+
TunnelException = Exception
|
|
362
270
|
|
|
363
271
|
|
|
364
272
|
class ChromiumSync:
|
|
@@ -652,11 +560,35 @@ class ChromiumSync:
|
|
|
652
560
|
# =========================================================================
|
|
653
561
|
|
|
654
562
|
def _start_all_tunnels(self):
|
|
563
|
+
"""
|
|
564
|
+
拉起所有需要 SSH 穿透的节点 (模式 4)
|
|
565
|
+
直接委托给通用解耦模块 wimotek-ssh-tunnel
|
|
566
|
+
"""
|
|
655
567
|
for node in self.nodes:
|
|
656
|
-
if not node.is_local:
|
|
657
|
-
|
|
658
|
-
|
|
659
|
-
|
|
568
|
+
if getattr(node, "mode", "direct") == "ssh_tunnel" and not node.is_local:
|
|
569
|
+
if not HAS_SSH_TUNNEL or not SshTunnelSession:
|
|
570
|
+
raise RuntimeError(
|
|
571
|
+
f"[-] 节点 [{node.name}] 处于 ssh_tunnel 模式，但未安装独立模块 wimotek-ssh-tunnel！\n"
|
|
572
|
+
f"请先运行: pip install wimotek-ssh-tunnel"
|
|
573
|
+
)
|
|
574
|
+
cfg = TunnelConfig(
|
|
575
|
+
id=node.id,
|
|
576
|
+
name=node.name,
|
|
577
|
+
ssh_host=node.ssh_host,
|
|
578
|
+
ssh_port=node.ssh_port,
|
|
579
|
+
ssh_user=node.ssh_user,
|
|
580
|
+
remote_host=node.cdp_host,
|
|
581
|
+
remote_port=node.cdp_port,
|
|
582
|
+
local_port=node.tunnel_port or 19222,
|
|
583
|
+
ssh_key=node.ssh_key_data or node.ssh_key_path or node.ssh_key,
|
|
584
|
+
ssh_password=node.ssh_password,
|
|
585
|
+
)
|
|
586
|
+
try:
|
|
587
|
+
tunnel = SshTunnelSession(cfg)
|
|
588
|
+
tunnel.start(auto_close=self.auto_close_tunnel)
|
|
589
|
+
self.tunnels[node.id] = tunnel
|
|
590
|
+
except Exception as e:
|
|
591
|
+
logging.warning(f"⚠️ [SKIP] 节点 [{node.name}] SSH 穿透拉起失败，标记异常跳过: {e}")
|
|
660
592
|
|
|
661
593
|
def _stop_all_tunnels(self, force: bool = False):
|
|
662
594
|
for tunnel in self.tunnels.values():
|
|
@@ -914,6 +846,8 @@ class ChromiumSync:
|
|
|
914
846
|
"source_total": 0,
|
|
915
847
|
"target_total": 0,
|
|
916
848
|
"nodes_synced": [],
|
|
849
|
+
"nodes_skipped": [],
|
|
850
|
+
"skips": [],
|
|
917
851
|
"duration": elapsed,
|
|
918
852
|
"details": "未安装 playwright 模块，请先运行: pip install playwright && playwright install chromium",
|
|
919
853
|
"error": "playwright not installed"
|
|
@@ -928,23 +862,44 @@ class ChromiumSync:
|
|
|
928
862
|
contexts: Dict[str, BrowserContext] = {}
|
|
929
863
|
nodes_data: Dict[str, dict] = {}
|
|
930
864
|
|
|
865
|
+
skipped_nodes: List[Dict[str, Any]] = []
|
|
866
|
+
|
|
931
867
|
if is_pooled:
|
|
932
868
|
contexts = self._pooled_contexts
|
|
933
869
|
for node in self.nodes:
|
|
934
870
|
if node.id in contexts:
|
|
935
871
|
nodes_data[node.id] = await self._fetch_node_data(contexts[node.id], node)
|
|
872
|
+
else:
|
|
873
|
+
logging.warning(f"⚠️ [SKIP] 连接池中节点 [{node.name}] 不可用，自动跳过。")
|
|
874
|
+
skipped_nodes.append({
|
|
875
|
+
"id": node.id,
|
|
876
|
+
"name": node.name,
|
|
877
|
+
"mode": getattr(node, "mode", "direct"),
|
|
878
|
+
"cdp_url": node.cdp_url,
|
|
879
|
+
"reason": "Not connected in pool"
|
|
880
|
+
})
|
|
936
881
|
else:
|
|
937
882
|
async with async_playwright() as p:
|
|
938
883
|
for node in self.nodes:
|
|
939
884
|
cdp_url = node.cdp_url
|
|
940
885
|
try:
|
|
941
|
-
|
|
886
|
+
connect_kwargs = {}
|
|
887
|
+
if getattr(node, "headers", None):
|
|
888
|
+
connect_kwargs["headers"] = node.headers
|
|
889
|
+
b = await p.chromium.connect_over_cdp(cdp_url, **connect_kwargs)
|
|
942
890
|
ctx = b.contexts[0] if b.contexts else await b.new_context()
|
|
943
891
|
browsers[node.id] = b
|
|
944
892
|
contexts[node.id] = ctx
|
|
945
893
|
nodes_data[node.id] = await self._fetch_node_data(ctx, node)
|
|
946
894
|
except Exception as e:
|
|
947
|
-
logging.
|
|
895
|
+
logging.warning(f"⚠️ [SKIP] 节点 [{node.name}] 离线或连接异常 ({cdp_url})，自动跳过: {e}")
|
|
896
|
+
skipped_nodes.append({
|
|
897
|
+
"id": node.id,
|
|
898
|
+
"name": node.name,
|
|
899
|
+
"mode": getattr(node, "mode", "direct"),
|
|
900
|
+
"cdp_url": cdp_url,
|
|
901
|
+
"reason": str(e)
|
|
902
|
+
})
|
|
948
903
|
|
|
949
904
|
if not contexts:
|
|
950
905
|
elapsed = round(time.time() - start_time, 2)
|
|
@@ -956,13 +911,15 @@ class ChromiumSync:
|
|
|
956
911
|
"source_total": 0,
|
|
957
912
|
"target_total": 0,
|
|
958
913
|
"nodes_synced": [],
|
|
914
|
+
"nodes_skipped": [n["id"] for n in skipped_nodes],
|
|
915
|
+
"skips": skipped_nodes,
|
|
959
916
|
"duration": elapsed,
|
|
960
|
-
"details": "
|
|
917
|
+
"details": "全部节点离线或异常，无可用在线节点，同步终止",
|
|
961
918
|
"error": "No available online nodes"
|
|
962
919
|
}
|
|
963
920
|
|
|
964
921
|
res = await self._execute_sync_flow(
|
|
965
|
-
active_mode, active_source, active_targets, contexts, nodes_data, start_time
|
|
922
|
+
active_mode, active_source, active_targets, contexts, nodes_data, start_time, skipped_nodes
|
|
966
923
|
)
|
|
967
924
|
for b in browsers.values():
|
|
968
925
|
await b.close()
|
|
@@ -979,13 +936,15 @@ class ChromiumSync:
|
|
|
979
936
|
"source_total": 0,
|
|
980
937
|
"target_total": 0,
|
|
981
938
|
"nodes_synced": [],
|
|
939
|
+
"nodes_skipped": [n["id"] for n in skipped_nodes],
|
|
940
|
+
"skips": skipped_nodes,
|
|
982
941
|
"duration": elapsed,
|
|
983
942
|
"details": "连接池中无可用在线节点",
|
|
984
943
|
"error": "No available online nodes in pool"
|
|
985
944
|
}
|
|
986
945
|
|
|
987
946
|
return await self._execute_sync_flow(
|
|
988
|
-
active_mode, active_source, active_targets, contexts, nodes_data, start_time
|
|
947
|
+
active_mode, active_source, active_targets, contexts, nodes_data, start_time, skipped_nodes
|
|
989
948
|
)
|
|
990
949
|
|
|
991
950
|
except Exception as e:
|
|
@@ -999,6 +958,8 @@ class ChromiumSync:
|
|
|
999
958
|
"source_total": 0,
|
|
1000
959
|
"target_total": 0,
|
|
1001
960
|
"nodes_synced": [],
|
|
961
|
+
"nodes_skipped": [n["id"] for n in skipped_nodes] if 'skipped_nodes' in locals() else [],
|
|
962
|
+
"skips": skipped_nodes if 'skipped_nodes' in locals() else [],
|
|
1002
963
|
"duration": elapsed,
|
|
1003
964
|
"details": f"同步发生异常: {e}",
|
|
1004
965
|
"error": str(e)
|
|
@@ -1014,12 +975,14 @@ class ChromiumSync:
|
|
|
1014
975
|
active_targets: Optional[List[str]],
|
|
1015
976
|
contexts: Dict[str, BrowserContext],
|
|
1016
977
|
nodes_data: Dict[str, dict],
|
|
1017
|
-
start_time: float
|
|
978
|
+
start_time: float,
|
|
979
|
+
skipped_nodes: Optional[List[Dict[str, Any]]] = None
|
|
1018
980
|
) -> Dict[str, Any]:
|
|
1019
981
|
"""内部执行流"""
|
|
1020
982
|
print(self.get_summary_table(nodes_data))
|
|
1021
983
|
total_transferred = 0
|
|
1022
984
|
total_skipped = 0
|
|
985
|
+
skips_list = skipped_nodes or []
|
|
1023
986
|
|
|
1024
987
|
if active_mode == "merge":
|
|
1025
988
|
logging.info("[*] 正在执行全节点 Cookie 智能 Merge (合并最新有效凭据)...")
|
|
@@ -1063,16 +1026,38 @@ class ChromiumSync:
|
|
|
1063
1026
|
"source_total": source_total,
|
|
1064
1027
|
"target_total": len(merged),
|
|
1065
1028
|
"nodes_synced": [n.id for n in dest_nodes],
|
|
1029
|
+
"nodes_skipped": [n["id"] for n in skips_list],
|
|
1030
|
+
"skips": skips_list,
|
|
1066
1031
|
"duration": elapsed,
|
|
1067
|
-
"details": f"全节点 Merge 成功:
|
|
1032
|
+
"details": f"全节点 Merge 成功: 向 {[n.name for n in dest_nodes]} 写入 {total_transferred} 条凭证 (跳过异常节点: {[n['name'] for n in skips_list] if skips_list else '无'})",
|
|
1068
1033
|
"error": None
|
|
1069
1034
|
}
|
|
1070
1035
|
|
|
1071
1036
|
elif active_mode == "sync":
|
|
1072
1037
|
if not active_source:
|
|
1073
1038
|
active_source = self.nodes[0].id
|
|
1039
|
+
|
|
1040
|
+
# 源节点离线检查：如果指定源节点离线，打印明确通知日志并立即终止同步流程
|
|
1074
1041
|
if active_source not in contexts:
|
|
1075
|
-
|
|
1042
|
+
source_node_obj = next((n for n in self.nodes if n.id == active_source), None)
|
|
1043
|
+
source_name = source_node_obj.name if source_node_obj else active_source
|
|
1044
|
+
elapsed = round(time.time() - start_time, 2)
|
|
1045
|
+
msg = f"⚠️ [SOURCE_OFFLINE] 单向同步指定的源节点 [{source_name}] ({active_source}) 离线或连接异常，无法提取基准凭据，同步流程终止。"
|
|
1046
|
+
logging.error(msg)
|
|
1047
|
+
return {
|
|
1048
|
+
"status": "failed",
|
|
1049
|
+
"mode": "sync",
|
|
1050
|
+
"transferred_count": 0,
|
|
1051
|
+
"diff_skipped_count": 0,
|
|
1052
|
+
"source_total": 0,
|
|
1053
|
+
"target_total": 0,
|
|
1054
|
+
"nodes_synced": [],
|
|
1055
|
+
"nodes_skipped": [n["id"] for n in skips_list],
|
|
1056
|
+
"skips": skips_list,
|
|
1057
|
+
"duration": elapsed,
|
|
1058
|
+
"details": msg,
|
|
1059
|
+
"error": f"Source node '{active_source}' is offline"
|
|
1060
|
+
}
|
|
1076
1061
|
|
|
1077
1062
|
src_data = nodes_data[active_source]
|
|
1078
1063
|
src_cookies = src_data["cookies"]
|
|
@@ -1105,8 +1090,10 @@ class ChromiumSync:
|
|
|
1105
1090
|
"source_total": len(src_cookies),
|
|
1106
1091
|
"target_total": len(src_cookies),
|
|
1107
1092
|
"nodes_synced": [n.id for n in dest_nodes],
|
|
1093
|
+
"nodes_skipped": [n["id"] for n in skips_list],
|
|
1094
|
+
"skips": skips_list,
|
|
1108
1095
|
"duration": elapsed,
|
|
1109
|
-
"details": f"单源同步成功: [{src_data['name']}] 写入 {total_transferred} 条凭据至 {[n.name for n in dest_nodes]} (
|
|
1096
|
+
"details": f"单源同步成功: [{src_data['name']}] 写入 {total_transferred} 条凭据至 {[n.name for n in dest_nodes]} (跳过异常节点: {[n['name'] for n in skips_list] if skips_list else '无'})",
|
|
1110
1097
|
"error": None
|
|
1111
1098
|
}
|
|
1112
1099
|
|