modelscope-multi-proxy-download 0.1.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,185 @@
1
+ Metadata-Version: 2.4
2
+ Name: modelscope-multi-proxy-download
3
+ Version: 0.1.1
4
+ Summary: Multi-proxy downloader for ModelScope model repos
5
+ Requires-Python: >=3.9
6
+ Description-Content-Type: text/markdown
7
+ Requires-Dist: modelscope
8
+ Requires-Dist: requests
9
+
10
+ # modelscope-multi-proxy-download
11
+
12
+ 多代理并发下载 ModelScope 模型的命令行工具。
13
+
14
+ ## 安装
15
+
16
+ ```bash
17
+ python3 -m venv .venv
18
+ .venv/bin/python -m pip install -e .
19
+ ```
20
+
21
+ 安装后命令为:
22
+
23
+ ```bash
24
+ .venv/bin/modelscope_multi_proxy_download --help
25
+ ```
26
+
27
+ ## 快速使用指南
28
+
29
+ 由于不同机器通常都可能遇到网络限速,推荐配合代理使用多路下载。
30
+
31
+ ```bash
32
+ .venv/bin/modelscope_multi_proxy_download \
33
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
34
+ --local_dir ./Qwen3-VL-30B-A3B-Instruct-FP8 \
35
+ --proxy http://127.0.0.1:8888
36
+ ```
37
+
38
+ 仅检查代理链路可用性:
39
+
40
+ ```bash
41
+ .venv/bin/modelscope_multi_proxy_download \
42
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
43
+ --proxy http://127.0.0.1:8888 \
44
+ --check-only
45
+ ```
46
+
47
+ ## Windows 场景快速指南
48
+
49
+ ### 场景 A:Linux 服务器要下载模型,Windows 主动连接 Linux 并共享本地代理
50
+
51
+ 目标:让 Linux 上的 `modelscope-multi-download` 通过 Windows 代理出网。
52
+
53
+ 1. Windows 安装并启动 `proxy.py`(监听 8888)
54
+
55
+ ```powershell
56
+ py -m pip install proxy.py
57
+ py -m proxy --hostname 0.0.0.0 --port 8888
58
+ ```
59
+
60
+ 2. Windows 主动连接 Linux,建立反向端口转发(把 Linux 上的 `127.0.0.1:8888` 映射到 Windows 的 `127.0.0.1:8888`)
61
+
62
+ ```powershell
63
+ ssh -N -R 8888:127.0.0.1:8888 linux_user@linux_ip
64
+ ```
65
+
66
+ 3. Linux 下载模型(访问 Linux 本地 `127.0.0.1:8888`,实际走 Windows 代理)
67
+
68
+ ```bash
69
+ .venv/bin/modelscope_multi_proxy_download \
70
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
71
+ --proxy http://127.0.0.1:8888
72
+ ```
73
+
74
+ ### 场景 B:Windows 是要下载模型的机器,Linux 提供代理
75
+
76
+ 目标:让 Windows 上的下载进程通过 Linux 代理出网。
77
+
78
+ 1. Linux 启动 `proxy.py`(监听 8888)
79
+
80
+ ```bash
81
+ python3 -m pip install proxy.py
82
+ python3 -m proxy --hostname 0.0.0.0 --port 8888
83
+ ```
84
+
85
+ 2. Windows 建立本地端口转发(把 Windows 的 `127.0.0.1:8888` 映射到 Linux `127.0.0.1:8888`)
86
+
87
+ ```powershell
88
+ ssh -N -L 8888:127.0.0.1:8888 linux_user@linux_ip
89
+ ```
90
+
91
+ 3. 在 Windows 上运行下载命令(PowerShell 或 WSL),代理填本地 8888
92
+
93
+ ```powershell
94
+ modelscope_multi_proxy_download Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 --proxy http://127.0.0.1:8888
95
+ ```
96
+
97
+ ## 详细使用步骤
98
+
99
+ 兼容 `modelscope download MODEL_ID --local_dir ...` 的核心用法,并新增:
100
+
101
+ - `--proxy`(可重复传入,值可为代理地址或代理列表文件)
102
+ - 默认自动包含 `DIRECT` 直连
103
+ - `--local_dir` 不传时默认取 `model_id` 最后一段
104
+ - `--no-direct` 一般不需要;仅在你明确希望“禁止直连、只允许代理”时再启用
105
+
106
+ ### 示例 1:多个代理并发下载
107
+
108
+ ```bash
109
+ .venv/bin/modelscope_multi_proxy_download \
110
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
111
+ --local_dir Qwen3-VL-30B-A3B-Instruct-FP8 \
112
+ --proxy http://127.0.0.1:10090 \
113
+ --proxy http://127.0.0.1:10091
114
+ ```
115
+
116
+ ### 示例 2:省略 `--local_dir`
117
+
118
+ ```bash
119
+ .venv/bin/modelscope_multi_proxy_download \
120
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
121
+ --proxy http://127.0.0.1:10090
122
+ ```
123
+
124
+ ### 示例 3:只检测路由可达性
125
+
126
+ ```bash
127
+ .venv/bin/modelscope_multi_proxy_download \
128
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
129
+ --proxy http://127.0.0.1:10090 \
130
+ --check-only
131
+ ```
132
+
133
+ ### 示例 4:禁用直连(只走代理)
134
+
135
+ ```bash
136
+ .venv/bin/modelscope_multi_proxy_download \
137
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
138
+ --proxy http://127.0.0.1:10090 \
139
+ --proxy http://127.0.0.1:10091 \
140
+ --no-direct
141
+ ```
142
+
143
+ ### 示例 5:使用代理列表文件
144
+
145
+ ```bash
146
+ .venv/bin/modelscope_multi_proxy_download \
147
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
148
+ --proxy ./proxies.txt
149
+ ```
150
+
151
+ `proxies.txt` 示例:
152
+
153
+ ```txt
154
+ http://127.0.0.1:10090
155
+ http://127.0.0.1:10091
156
+ DIRECT
157
+ ```
158
+
159
+ ### 参数说明
160
+
161
+ - `model_id`: 模型 ID,例如 `Qwen/Qwen3-VL-30B-A3B-Instruct-FP8`
162
+ - `--local_dir <dir>`: 下载目录,默认是 `model_id` 最后一段
163
+ - `--revision <rev>`: 分支、标签或提交
164
+ - `--cache_dir <dir>`: ModelScope 缓存目录
165
+ - `--token <token>`: 私有模型 token
166
+ - `--allow-pattern <glob>`: 仅下载匹配文件(可重复)
167
+ - `--ignore-pattern <glob>`: 忽略匹配文件(可重复)
168
+ - `--proxy <proxy_or_file>`: 代理地址或代理列表文件(可重复)
169
+ - `--max-workers <n>`: 并发 worker 数,默认等于可用路由数
170
+ - `--no-direct`: 排除直连路由
171
+ - `--dry-run`: 仅列出将下载文件
172
+ - `--no-worker-progress`: 关闭主进程实时任务进度区
173
+ - `--no-route-fallback`: 失败后不主动切换其他路由
174
+ - `--check-proxies`: 下载前检测每条路由
175
+ - `--check-only`: 仅检测并退出
176
+ - `--check-url <url>`: 路由检测目标 URL(默认 `https://www.modelscope.cn`)
177
+ - `--check-timeout <sec>`: 路由检测超时时间,默认 `8`
178
+
179
+ ### 行为说明
180
+
181
+ - 每个 worker 进程固定绑定一个路由,按文件粒度并发下载。
182
+ - `DIRECT`/`NONE`/`NO_PROXY`/`LOCAL` 会被识别为直连。
183
+ - 若某任务速度连续 60 秒低于 `500KB/s`,优先切换空闲代理,否则最多尝试 1 次断开重连。
184
+ - 失败文件最多自动重试 3 次。
185
+ - 有失败文件时最终返回非 0 退出码。
@@ -0,0 +1,176 @@
1
+ # modelscope-multi-proxy-download
2
+
3
+ 多代理并发下载 ModelScope 模型的命令行工具。
4
+
5
+ ## 安装
6
+
7
+ ```bash
8
+ python3 -m venv .venv
9
+ .venv/bin/python -m pip install -e .
10
+ ```
11
+
12
+ 安装后命令为:
13
+
14
+ ```bash
15
+ .venv/bin/modelscope_multi_proxy_download --help
16
+ ```
17
+
18
+ ## 快速使用指南
19
+
20
+ 由于不同机器通常都可能遇到网络限速,推荐配合代理使用多路下载。
21
+
22
+ ```bash
23
+ .venv/bin/modelscope_multi_proxy_download \
24
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
25
+ --local_dir ./Qwen3-VL-30B-A3B-Instruct-FP8 \
26
+ --proxy http://127.0.0.1:8888
27
+ ```
28
+
29
+ 仅检查代理链路可用性:
30
+
31
+ ```bash
32
+ .venv/bin/modelscope_multi_proxy_download \
33
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
34
+ --proxy http://127.0.0.1:8888 \
35
+ --check-only
36
+ ```
37
+
38
+ ## Windows 场景快速指南
39
+
40
+ ### 场景 A:Linux 服务器要下载模型,Windows 主动连接 Linux 并共享本地代理
41
+
42
+ 目标:让 Linux 上的 `modelscope-multi-download` 通过 Windows 代理出网。
43
+
44
+ 1. Windows 安装并启动 `proxy.py`(监听 8888)
45
+
46
+ ```powershell
47
+ py -m pip install proxy.py
48
+ py -m proxy --hostname 0.0.0.0 --port 8888
49
+ ```
50
+
51
+ 2. Windows 主动连接 Linux,建立反向端口转发(把 Linux 上的 `127.0.0.1:8888` 映射到 Windows 的 `127.0.0.1:8888`)
52
+
53
+ ```powershell
54
+ ssh -N -R 8888:127.0.0.1:8888 linux_user@linux_ip
55
+ ```
56
+
57
+ 3. Linux 下载模型(访问 Linux 本地 `127.0.0.1:8888`,实际走 Windows 代理)
58
+
59
+ ```bash
60
+ .venv/bin/modelscope_multi_proxy_download \
61
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
62
+ --proxy http://127.0.0.1:8888
63
+ ```
64
+
65
+ ### 场景 B:Windows 是要下载模型的机器,Linux 提供代理
66
+
67
+ 目标:让 Windows 上的下载进程通过 Linux 代理出网。
68
+
69
+ 1. Linux 启动 `proxy.py`(监听 8888)
70
+
71
+ ```bash
72
+ python3 -m pip install proxy.py
73
+ python3 -m proxy --hostname 0.0.0.0 --port 8888
74
+ ```
75
+
76
+ 2. Windows 建立本地端口转发(把 Windows 的 `127.0.0.1:8888` 映射到 Linux `127.0.0.1:8888`)
77
+
78
+ ```powershell
79
+ ssh -N -L 8888:127.0.0.1:8888 linux_user@linux_ip
80
+ ```
81
+
82
+ 3. 在 Windows 上运行下载命令(PowerShell 或 WSL),代理填本地 8888
83
+
84
+ ```powershell
85
+ modelscope_multi_proxy_download Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 --proxy http://127.0.0.1:8888
86
+ ```
87
+
88
+ ## 详细使用步骤
89
+
90
+ 兼容 `modelscope download MODEL_ID --local_dir ...` 的核心用法,并新增:
91
+
92
+ - `--proxy`(可重复传入,值可为代理地址或代理列表文件)
93
+ - 默认自动包含 `DIRECT` 直连
94
+ - `--local_dir` 不传时默认取 `model_id` 最后一段
95
+ - `--no-direct` 一般不需要;仅在你明确希望“禁止直连、只允许代理”时再启用
96
+
97
+ ### 示例 1:多个代理并发下载
98
+
99
+ ```bash
100
+ .venv/bin/modelscope_multi_proxy_download \
101
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
102
+ --local_dir Qwen3-VL-30B-A3B-Instruct-FP8 \
103
+ --proxy http://127.0.0.1:10090 \
104
+ --proxy http://127.0.0.1:10091
105
+ ```
106
+
107
+ ### 示例 2:省略 `--local_dir`
108
+
109
+ ```bash
110
+ .venv/bin/modelscope_multi_proxy_download \
111
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
112
+ --proxy http://127.0.0.1:10090
113
+ ```
114
+
115
+ ### 示例 3:只检测路由可达性
116
+
117
+ ```bash
118
+ .venv/bin/modelscope_multi_proxy_download \
119
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
120
+ --proxy http://127.0.0.1:10090 \
121
+ --check-only
122
+ ```
123
+
124
+ ### 示例 4:禁用直连(只走代理)
125
+
126
+ ```bash
127
+ .venv/bin/modelscope_multi_proxy_download \
128
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
129
+ --proxy http://127.0.0.1:10090 \
130
+ --proxy http://127.0.0.1:10091 \
131
+ --no-direct
132
+ ```
133
+
134
+ ### 示例 5:使用代理列表文件
135
+
136
+ ```bash
137
+ .venv/bin/modelscope_multi_proxy_download \
138
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
139
+ --proxy ./proxies.txt
140
+ ```
141
+
142
+ `proxies.txt` 示例:
143
+
144
+ ```txt
145
+ http://127.0.0.1:10090
146
+ http://127.0.0.1:10091
147
+ DIRECT
148
+ ```
149
+
150
+ ### 参数说明
151
+
152
+ - `model_id`: 模型 ID,例如 `Qwen/Qwen3-VL-30B-A3B-Instruct-FP8`
153
+ - `--local_dir <dir>`: 下载目录,默认是 `model_id` 最后一段
154
+ - `--revision <rev>`: 分支、标签或提交
155
+ - `--cache_dir <dir>`: ModelScope 缓存目录
156
+ - `--token <token>`: 私有模型 token
157
+ - `--allow-pattern <glob>`: 仅下载匹配文件(可重复)
158
+ - `--ignore-pattern <glob>`: 忽略匹配文件(可重复)
159
+ - `--proxy <proxy_or_file>`: 代理地址或代理列表文件(可重复)
160
+ - `--max-workers <n>`: 并发 worker 数,默认等于可用路由数
161
+ - `--no-direct`: 排除直连路由
162
+ - `--dry-run`: 仅列出将下载文件
163
+ - `--no-worker-progress`: 关闭主进程实时任务进度区
164
+ - `--no-route-fallback`: 失败后不主动切换其他路由
165
+ - `--check-proxies`: 下载前检测每条路由
166
+ - `--check-only`: 仅检测并退出
167
+ - `--check-url <url>`: 路由检测目标 URL(默认 `https://www.modelscope.cn`)
168
+ - `--check-timeout <sec>`: 路由检测超时时间,默认 `8`
169
+
170
+ ### 行为说明
171
+
172
+ - 每个 worker 进程固定绑定一个路由,按文件粒度并发下载。
173
+ - `DIRECT`/`NONE`/`NO_PROXY`/`LOCAL` 会被识别为直连。
174
+ - 若某任务速度连续 60 秒低于 `500KB/s`,优先切换空闲代理,否则最多尝试 1 次断开重连。
175
+ - 失败文件最多自动重试 3 次。
176
+ - 有失败文件时最终返回非 0 退出码。
@@ -0,0 +1,185 @@
1
+ Metadata-Version: 2.4
2
+ Name: modelscope-multi-proxy-download
3
+ Version: 0.1.1
4
+ Summary: Multi-proxy downloader for ModelScope model repos
5
+ Requires-Python: >=3.9
6
+ Description-Content-Type: text/markdown
7
+ Requires-Dist: modelscope
8
+ Requires-Dist: requests
9
+
10
+ # modelscope-multi-proxy-download
11
+
12
+ 多代理并发下载 ModelScope 模型的命令行工具。
13
+
14
+ ## 安装
15
+
16
+ ```bash
17
+ python3 -m venv .venv
18
+ .venv/bin/python -m pip install -e .
19
+ ```
20
+
21
+ 安装后命令为:
22
+
23
+ ```bash
24
+ .venv/bin/modelscope_multi_proxy_download --help
25
+ ```
26
+
27
+ ## 快速使用指南
28
+
29
+ 由于不同机器通常都可能遇到网络限速,推荐配合代理使用多路下载。
30
+
31
+ ```bash
32
+ .venv/bin/modelscope_multi_proxy_download \
33
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
34
+ --local_dir ./Qwen3-VL-30B-A3B-Instruct-FP8 \
35
+ --proxy http://127.0.0.1:8888
36
+ ```
37
+
38
+ 仅检查代理链路可用性:
39
+
40
+ ```bash
41
+ .venv/bin/modelscope_multi_proxy_download \
42
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
43
+ --proxy http://127.0.0.1:8888 \
44
+ --check-only
45
+ ```
46
+
47
+ ## Windows 场景快速指南
48
+
49
+ ### 场景 A:Linux 服务器要下载模型,Windows 主动连接 Linux 并共享本地代理
50
+
51
+ 目标:让 Linux 上的 `modelscope-multi-download` 通过 Windows 代理出网。
52
+
53
+ 1. Windows 安装并启动 `proxy.py`(监听 8888)
54
+
55
+ ```powershell
56
+ py -m pip install proxy.py
57
+ py -m proxy --hostname 0.0.0.0 --port 8888
58
+ ```
59
+
60
+ 2. Windows 主动连接 Linux,建立反向端口转发(把 Linux 上的 `127.0.0.1:8888` 映射到 Windows 的 `127.0.0.1:8888`)
61
+
62
+ ```powershell
63
+ ssh -N -R 8888:127.0.0.1:8888 linux_user@linux_ip
64
+ ```
65
+
66
+ 3. Linux 下载模型(访问 Linux 本地 `127.0.0.1:8888`,实际走 Windows 代理)
67
+
68
+ ```bash
69
+ .venv/bin/modelscope_multi_proxy_download \
70
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
71
+ --proxy http://127.0.0.1:8888
72
+ ```
73
+
74
+ ### 场景 B:Windows 是要下载模型的机器,Linux 提供代理
75
+
76
+ 目标:让 Windows 上的下载进程通过 Linux 代理出网。
77
+
78
+ 1. Linux 启动 `proxy.py`(监听 8888)
79
+
80
+ ```bash
81
+ python3 -m pip install proxy.py
82
+ python3 -m proxy --hostname 0.0.0.0 --port 8888
83
+ ```
84
+
85
+ 2. Windows 建立本地端口转发(把 Windows 的 `127.0.0.1:8888` 映射到 Linux `127.0.0.1:8888`)
86
+
87
+ ```powershell
88
+ ssh -N -L 8888:127.0.0.1:8888 linux_user@linux_ip
89
+ ```
90
+
91
+ 3. 在 Windows 上运行下载命令(PowerShell 或 WSL),代理填本地 8888
92
+
93
+ ```powershell
94
+ modelscope_multi_proxy_download Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 --proxy http://127.0.0.1:8888
95
+ ```
96
+
97
+ ## 详细使用步骤
98
+
99
+ 兼容 `modelscope download MODEL_ID --local_dir ...` 的核心用法,并新增:
100
+
101
+ - `--proxy`(可重复传入,值可为代理地址或代理列表文件)
102
+ - 默认自动包含 `DIRECT` 直连
103
+ - `--local_dir` 不传时默认取 `model_id` 最后一段
104
+ - `--no-direct` 一般不需要;仅在你明确希望“禁止直连、只允许代理”时再启用
105
+
106
+ ### 示例 1:多个代理并发下载
107
+
108
+ ```bash
109
+ .venv/bin/modelscope_multi_proxy_download \
110
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
111
+ --local_dir Qwen3-VL-30B-A3B-Instruct-FP8 \
112
+ --proxy http://127.0.0.1:10090 \
113
+ --proxy http://127.0.0.1:10091
114
+ ```
115
+
116
+ ### 示例 2:省略 `--local_dir`
117
+
118
+ ```bash
119
+ .venv/bin/modelscope_multi_proxy_download \
120
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
121
+ --proxy http://127.0.0.1:10090
122
+ ```
123
+
124
+ ### 示例 3:只检测路由可达性
125
+
126
+ ```bash
127
+ .venv/bin/modelscope_multi_proxy_download \
128
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
129
+ --proxy http://127.0.0.1:10090 \
130
+ --check-only
131
+ ```
132
+
133
+ ### 示例 4:禁用直连(只走代理)
134
+
135
+ ```bash
136
+ .venv/bin/modelscope_multi_proxy_download \
137
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
138
+ --proxy http://127.0.0.1:10090 \
139
+ --proxy http://127.0.0.1:10091 \
140
+ --no-direct
141
+ ```
142
+
143
+ ### 示例 5:使用代理列表文件
144
+
145
+ ```bash
146
+ .venv/bin/modelscope_multi_proxy_download \
147
+ Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
148
+ --proxy ./proxies.txt
149
+ ```
150
+
151
+ `proxies.txt` 示例:
152
+
153
+ ```txt
154
+ http://127.0.0.1:10090
155
+ http://127.0.0.1:10091
156
+ DIRECT
157
+ ```
158
+
159
+ ### 参数说明
160
+
161
+ - `model_id`: 模型 ID,例如 `Qwen/Qwen3-VL-30B-A3B-Instruct-FP8`
162
+ - `--local_dir <dir>`: 下载目录,默认是 `model_id` 最后一段
163
+ - `--revision <rev>`: 分支、标签或提交
164
+ - `--cache_dir <dir>`: ModelScope 缓存目录
165
+ - `--token <token>`: 私有模型 token
166
+ - `--allow-pattern <glob>`: 仅下载匹配文件(可重复)
167
+ - `--ignore-pattern <glob>`: 忽略匹配文件(可重复)
168
+ - `--proxy <proxy_or_file>`: 代理地址或代理列表文件(可重复)
169
+ - `--max-workers <n>`: 并发 worker 数,默认等于可用路由数
170
+ - `--no-direct`: 排除直连路由
171
+ - `--dry-run`: 仅列出将下载文件
172
+ - `--no-worker-progress`: 关闭主进程实时任务进度区
173
+ - `--no-route-fallback`: 失败后不主动切换其他路由
174
+ - `--check-proxies`: 下载前检测每条路由
175
+ - `--check-only`: 仅检测并退出
176
+ - `--check-url <url>`: 路由检测目标 URL(默认 `https://www.modelscope.cn`)
177
+ - `--check-timeout <sec>`: 路由检测超时时间,默认 `8`
178
+
179
+ ### 行为说明
180
+
181
+ - 每个 worker 进程固定绑定一个路由,按文件粒度并发下载。
182
+ - `DIRECT`/`NONE`/`NO_PROXY`/`LOCAL` 会被识别为直连。
183
+ - 若某任务速度连续 60 秒低于 `500KB/s`,优先切换空闲代理,否则最多尝试 1 次断开重连。
184
+ - 失败文件最多自动重试 3 次。
185
+ - 有失败文件时最终返回非 0 退出码。
@@ -0,0 +1,9 @@
1
+ README.md
2
+ modelscope_multi_proxy_download.py
3
+ pyproject.toml
4
+ modelscope_multi_proxy_download.egg-info/PKG-INFO
5
+ modelscope_multi_proxy_download.egg-info/SOURCES.txt
6
+ modelscope_multi_proxy_download.egg-info/dependency_links.txt
7
+ modelscope_multi_proxy_download.egg-info/entry_points.txt
8
+ modelscope_multi_proxy_download.egg-info/requires.txt
9
+ modelscope_multi_proxy_download.egg-info/top_level.txt
@@ -0,0 +1,2 @@
1
+ [console_scripts]
2
+ modelscope_multi_proxy_download = modelscope_multi_proxy_download:main
@@ -0,0 +1,761 @@
1
+ #!/usr/bin/env python3
2
+ """Multi-proxy downloader for ModelScope model repos."""
3
+
4
+ from __future__ import annotations
5
+
6
+ import argparse
7
+ import fnmatch
8
+ import multiprocessing
9
+ import os
10
+ import queue
11
+ import sys
12
+ import threading
13
+ import time
14
+ from concurrent.futures import ProcessPoolExecutor, TimeoutError, as_completed
15
+ from pathlib import Path
16
+ from typing import Iterable
17
+
18
+
19
+ _PROXY_ENV_KEYS = (
20
+ "http_proxy",
21
+ "https_proxy",
22
+ "HTTP_PROXY",
23
+ "HTTPS_PROXY",
24
+ "all_proxy",
25
+ "ALL_PROXY",
26
+ )
27
+ _DIRECT_ROUTE = "DIRECT"
28
+ _DIRECT_TOKENS = {"DIRECT", "NONE", "NO_PROXY", "LOCAL"}
29
+ _WORKER_PROGRESS_QUEUE = None
30
+ _WORKER_SLOT = -1
31
+ _WORKER_ABORT_FLAGS = None
32
+ _SLOW_ABORT_SENTINEL = "__SLOW_ABORT__"
33
+ _LOW_SPEED_THRESHOLD_BPS = 500 * 1024
34
+ _LOW_SPEED_WINDOW_SEC = 60
35
+ _MAX_FILE_RETRIES = 3
36
+
37
+
38
+ def _read_proxy_file(path: str) -> list[str]:
39
+ proxies: list[str] = []
40
+ with open(path, "r", encoding="utf-8") as f:
41
+ for raw in f:
42
+ line = raw.strip()
43
+ if not line or line.startswith("#"):
44
+ continue
45
+ proxies.append(line)
46
+ return proxies
47
+
48
+
49
+ def _expand_proxy_args(proxy_args: Iterable[str]) -> list[str]:
50
+ expanded: list[str] = []
51
+ for item in proxy_args:
52
+ value = item.strip()
53
+ if not value:
54
+ continue
55
+ proxy_file = Path(value)
56
+ if proxy_file.is_file():
57
+ expanded.extend(_read_proxy_file(str(proxy_file)))
58
+ else:
59
+ expanded.append(value)
60
+ return expanded
61
+
62
+
63
+ def _normalize_proxy_list(raw_proxies: Iterable[str]) -> list[str]:
64
+ deduped: list[str] = []
65
+ seen: set[str] = set()
66
+ for item in raw_proxies:
67
+ proxy = item.strip()
68
+ if not proxy:
69
+ continue
70
+ if proxy.upper() in _DIRECT_TOKENS:
71
+ proxy = _DIRECT_ROUTE
72
+ elif "://" not in proxy:
73
+ proxy = f"http://{proxy}"
74
+ if proxy in seen:
75
+ continue
76
+ seen.add(proxy)
77
+ deduped.append(proxy)
78
+ return deduped
79
+
80
+
81
+ def _set_proxy_env(proxy: str) -> dict[str, str | None]:
82
+ old: dict[str, str | None] = {k: os.environ.get(k) for k in _PROXY_ENV_KEYS}
83
+ if proxy == _DIRECT_ROUTE:
84
+ for key in _PROXY_ENV_KEYS:
85
+ os.environ.pop(key, None)
86
+ else:
87
+ for key in _PROXY_ENV_KEYS:
88
+ os.environ[key] = proxy
89
+ return old
90
+
91
+
92
+ def _restore_proxy_env(old: dict[str, str | None]) -> None:
93
+ for key, value in old.items():
94
+ if value is None:
95
+ os.environ.pop(key, None)
96
+ else:
97
+ os.environ[key] = value
98
+
99
+
100
+ def _list_model_files(
101
+ model_id: str,
102
+ revision: str | None,
103
+ token: str | None,
104
+ proxies: list[str],
105
+ ) -> tuple[str, list[str], str]:
106
+ from modelscope.hub.api import HubApi
107
+
108
+ last_error = "unknown error"
109
+ for proxy in proxies:
110
+ old_env = _set_proxy_env(proxy)
111
+ try:
112
+ api = HubApi(token=token)
113
+ endpoint = api.get_endpoint_for_read(
114
+ repo_id=model_id, repo_type="model", token=token
115
+ )
116
+ rev = api.get_valid_revision(
117
+ model_id, revision=revision, cookies=api.get_cookies(), endpoint=endpoint
118
+ )
119
+ files = api.get_model_files(
120
+ model_id=model_id,
121
+ revision=rev,
122
+ recursive=True,
123
+ use_cookies=False if api.get_cookies() is None else api.get_cookies(),
124
+ endpoint=endpoint,
125
+ )
126
+ paths = [f["Path"] for f in files if f.get("Type") != "tree" and "Path" in f]
127
+ if not paths:
128
+ raise RuntimeError(f"No downloadable files found in {model_id}@{rev}")
129
+ return rev, paths, proxy
130
+ except Exception as exc: # pragma: no cover - runtime/network dependent
131
+ last_error = f"{type(exc).__name__}: {exc}"
132
+ finally:
133
+ _restore_proxy_env(old_env)
134
+ raise RuntimeError(f"Failed to list files via all proxies: {last_error}")
135
+
136
+
137
+ def _match_patterns(path: str, allow: list[str], ignore: list[str]) -> bool:
138
+ if allow and not any(fnmatch.fnmatch(path, p) for p in allow):
139
+ return False
140
+ if ignore and any(fnmatch.fnmatch(path, p) for p in ignore):
141
+ return False
142
+ return True
143
+
144
+
145
+ def _format_bytes(num: float) -> str:
146
+ units = ["B", "KB", "MB", "GB", "TB"]
147
+ value = float(max(0.0, num))
148
+ for unit in units:
149
+ if value < 1024.0 or unit == units[-1]:
150
+ if unit == "B":
151
+ return f"{int(value)}{unit}"
152
+ return f"{value:.1f}{unit}"
153
+ value /= 1024.0
154
+ return f"{value:.1f}TB"
155
+
156
+
157
+ class _QueueTqdm:
158
+ """Replacement for worker tqdm; emits progress events to parent process."""
159
+
160
+ def __init__(self, *args, **kwargs):
161
+ self.desc = kwargs.get("desc", "")
162
+ self.total = kwargs.get("total", 0) or 0
163
+ self.n = kwargs.get("initial", 0) or 0
164
+ self._last_emit = 0.0
165
+ self._emit(force=True)
166
+
167
+ def _emit(self, force: bool = False) -> None:
168
+ if _WORKER_PROGRESS_QUEUE is None:
169
+ return
170
+ now = time.time()
171
+ if not force and now - self._last_emit < 0.2:
172
+ return
173
+ self._last_emit = now
174
+ _WORKER_PROGRESS_QUEUE.put(("progress", _WORKER_SLOT, self.desc, self.n, self.total))
175
+
176
+ def update(self, n: int = 1) -> None:
177
+ if _WORKER_ABORT_FLAGS is not None and _WORKER_ABORT_FLAGS.get(_WORKER_SLOT, False):
178
+ raise RuntimeError(_SLOW_ABORT_SENTINEL)
179
+ self.n += n
180
+ self._emit()
181
+
182
+ def close(self) -> None:
183
+ self._emit(force=True)
184
+ if _WORKER_PROGRESS_QUEUE is not None:
185
+ _WORKER_PROGRESS_QUEUE.put(
186
+ ("progress_done", _WORKER_SLOT, self.desc, self.n, self.total)
187
+ )
188
+
189
+
190
+ class _ProgressLayout:
191
+ def __init__(self, workers: int):
192
+ self.workers = workers
193
+ self.status = [f"[Task {i + 1}] idle" for i in range(workers)]
194
+ self.footer = "[Total] speed: 0B/s"
195
+ self.enabled = sys.stdout.isatty()
196
+ self._printed = False
197
+ self._lock = threading.Lock()
198
+
199
+ def _render(self) -> None:
200
+ if not self.enabled:
201
+ return
202
+ if not self._printed:
203
+ sys.stdout.write("\n" * (self.workers + 1))
204
+ self._printed = True
205
+ sys.stdout.write(f"\x1b[{self.workers + 1}F")
206
+ for line in self.status:
207
+ sys.stdout.write(f"\r\x1b[2K{line[:200]}\n")
208
+ sys.stdout.write(f"\r\x1b[2K{self.footer[:200]}\n")
209
+ sys.stdout.flush()
210
+
211
+ def update(self, slot: int, line: str) -> None:
212
+ if slot < 0 or slot >= self.workers:
213
+ return
214
+ with self._lock:
215
+ self.status[slot] = line
216
+ self._render()
217
+
218
+ def set_footer(self, line: str) -> None:
219
+ with self._lock:
220
+ self.footer = line
221
+ self._render()
222
+
223
+ def log(self, line: str) -> None:
224
+ with self._lock:
225
+ if not self.enabled or not self._printed:
226
+ print(line)
227
+ return
228
+ sys.stdout.write(f"\x1b[{self.workers + 1}F")
229
+ sys.stdout.write(f"\r\x1b[2K{line}\n")
230
+ for status_line in self.status:
231
+ sys.stdout.write(f"\r\x1b[2K{status_line[:200]}\n")
232
+ sys.stdout.write(f"\r\x1b[2K{self.footer[:200]}\n")
233
+ sys.stdout.flush()
234
+
235
+
236
+ class _SpeedState:
237
+ def __init__(self, workers: int):
238
+ self._lock = threading.Lock()
239
+ self._speed = [0.0] * workers
240
+ self._desc = [""] * workers
241
+ self._done = [0.0] * workers
242
+ self._ts = [0.0] * workers
243
+ self._low_since = [None] * workers
244
+
245
+ def update(self, slot: int, desc: str, done: float, speed: float, now: float) -> None:
246
+ with self._lock:
247
+ self._speed[slot] = speed
248
+ if desc != self._desc[slot]:
249
+ self._low_since[slot] = None
250
+ self._desc[slot] = desc
251
+ self._done[slot] = done
252
+ self._ts[slot] = now
253
+ if done > 0 and speed < _LOW_SPEED_THRESHOLD_BPS and desc:
254
+ if self._low_since[slot] is None:
255
+ self._low_since[slot] = now
256
+ else:
257
+ self._low_since[slot] = None
258
+
259
+ def snapshot_slot(self, slot: int) -> dict:
260
+ with self._lock:
261
+ return {
262
+ "speed": self._speed[slot],
263
+ "desc": self._desc[slot],
264
+ "done": self._done[slot],
265
+ "ts": self._ts[slot],
266
+ "low_since": self._low_since[slot],
267
+ }
268
+
269
+
270
+ def _worker_init(
271
+ proxy: str,
272
+ enable_progress_layout: bool = False,
273
+ progress_queue=None,
274
+ worker_slot: int = -1,
275
+ abort_flags=None,
276
+ ) -> None:
277
+ global _WORKER_PROGRESS_QUEUE, _WORKER_SLOT, _WORKER_ABORT_FLAGS
278
+ _set_proxy_env(proxy)
279
+ _WORKER_PROGRESS_QUEUE = progress_queue
280
+ _WORKER_SLOT = worker_slot
281
+ _WORKER_ABORT_FLAGS = abort_flags
282
+ if enable_progress_layout and progress_queue is not None:
283
+ import modelscope.hub.callback as callback_mod
284
+ import modelscope.hub.file_download as file_download_mod
285
+
286
+ callback_mod.tqdm = _QueueTqdm
287
+ file_download_mod.tqdm = _QueueTqdm
288
+
289
+
290
+ def _progress_consumer(
291
+ progress_queue, renderer: _ProgressLayout, speed_state: _SpeedState, stop_event: threading.Event
292
+ ) -> None:
293
+ last_done = [0.0] * renderer.workers
294
+ last_ts = [0.0] * renderer.workers
295
+ speed = [0.0] * renderer.workers
296
+ last_desc = [""] * renderer.workers
297
+ while not stop_event.is_set():
298
+ try:
299
+ event = progress_queue.get(timeout=0.2)
300
+ except queue.Empty:
301
+ continue
302
+ except Exception:
303
+ break
304
+
305
+ _, slot, desc, done, total = event
306
+ now = time.time()
307
+ if slot < 0 or slot >= renderer.workers:
308
+ continue
309
+
310
+ if desc != last_desc[slot] or done < last_done[slot]:
311
+ last_done[slot] = float(done)
312
+ last_ts[slot] = now
313
+ speed[slot] = 0.0
314
+ last_desc[slot] = desc
315
+ else:
316
+ dt = now - last_ts[slot]
317
+ delta = float(done) - last_done[slot]
318
+ if dt > 0 and delta >= 0:
319
+ inst = delta / dt
320
+ speed[slot] = inst if speed[slot] <= 0 else (speed[slot] * 0.7 + inst * 0.3)
321
+ last_done[slot] = float(done)
322
+ last_ts[slot] = now
323
+
324
+ if total and total > 0:
325
+ pct = f"{(done / total) * 100:5.1f}%"
326
+ progress = f"{_format_bytes(done)}/{_format_bytes(total)}"
327
+ else:
328
+ pct = " ?.?%"
329
+ progress = f"{_format_bytes(done)}"
330
+ renderer.update(
331
+ slot,
332
+ f"[Task {slot + 1}] {pct} {progress} {_format_bytes(speed[slot])}/s {desc}",
333
+ )
334
+ renderer.set_footer(f"[Total] speed: {_format_bytes(sum(speed))}/s")
335
+ speed_state.update(slot=slot, desc=desc, done=float(done), speed=float(speed[slot]), now=now)
336
+
337
+
338
+ def _check_proxy_reachable(proxy: str, timeout: float, test_url: str) -> tuple[bool, str]:
339
+ import requests
340
+
341
+ old_env = _set_proxy_env(proxy)
342
+ started = time.time()
343
+ try:
344
+ resp = requests.get(test_url, timeout=timeout)
345
+ elapsed = (time.time() - started) * 1000.0
346
+ return True, f"HTTP {resp.status_code}, {elapsed:.0f}ms"
347
+ except Exception as exc: # pragma: no cover - runtime/network dependent
348
+ return False, f"{type(exc).__name__}: {exc}"
349
+ finally:
350
+ _restore_proxy_env(old_env)
351
+
352
+
353
+ def _download_one(task: dict) -> tuple[str, bool, str]:
354
+ from modelscope.hub.file_download import _repo_file_download
355
+
356
+ path = task["file_path"]
357
+ try:
358
+ _repo_file_download(
359
+ repo_id=task["model_id"],
360
+ file_path=path,
361
+ repo_type="model",
362
+ revision=task["revision"],
363
+ cache_dir=task["cache_dir"],
364
+ local_dir=task["local_dir"],
365
+ disable_tqdm=task["disable_worker_tqdm"],
366
+ token=task["token"],
367
+ )
368
+ return path, True, ""
369
+ except Exception as exc: # pragma: no cover - runtime/network dependent
370
+ return path, False, f"{type(exc).__name__}: {exc}"
371
+
372
+
373
+ def _download_one_via_route(task: dict, route: str) -> tuple[bool, str]:
374
+ """Try downloading one file via a specific route in current process."""
375
+ from modelscope.hub.file_download import _repo_file_download
376
+
377
+ old_env = _set_proxy_env(route)
378
+ try:
379
+ _repo_file_download(
380
+ repo_id=task["model_id"],
381
+ file_path=task["file_path"],
382
+ repo_type="model",
383
+ revision=task["revision"],
384
+ cache_dir=task["cache_dir"],
385
+ local_dir=task["local_dir"],
386
+ disable_tqdm=task["disable_worker_tqdm"],
387
+ token=task["token"],
388
+ )
389
+ return True, ""
390
+ except Exception as exc: # pragma: no cover - runtime/network dependent
391
+ return False, f"{type(exc).__name__}: {exc}"
392
+ finally:
393
+ _restore_proxy_env(old_env)
394
+
395
+
396
+ def build_parser() -> argparse.ArgumentParser:
397
+ parser = argparse.ArgumentParser(
398
+ description=(
399
+ "Download a ModelScope model with multiple proxies in parallel. "
400
+ "Compatible with `modelscope download MODEL_ID --local_dir ...` core args."
401
+ )
402
+ )
403
+ parser.add_argument("model_id", help="Model id, e.g. Qwen/Qwen3-VL-30B-A3B-Instruct-FP8")
404
+ parser.add_argument(
405
+ "--local_dir",
406
+ default=None,
407
+ help="Target directory for downloaded files (default: last path segment of model_id)",
408
+ )
409
+ parser.add_argument("--revision", default=None, help="Branch/tag/commit revision")
410
+ parser.add_argument("--cache_dir", default=None, help="ModelScope cache directory")
411
+ parser.add_argument("--token", default=None, help="ModelScope token for private models")
412
+ parser.add_argument("--allow-pattern", dest="allow_patterns", action="append", default=[])
413
+ parser.add_argument("--ignore-pattern", dest="ignore_patterns", action="append", default=[])
414
+ parser.add_argument(
415
+ "--proxy",
416
+ action="append",
417
+ default=[],
418
+ help=(
419
+ "Proxy endpoint or proxy-list file path. Can be passed multiple times, "
420
+ "e.g. --proxy http://127.0.0.1:7890 or --proxy client/proxies.txt"
421
+ ),
422
+ )
423
+ parser.add_argument(
424
+ "--max-workers",
425
+ type=int,
426
+ default=None,
427
+ help="Override worker count (default: number of proxies)",
428
+ )
429
+ parser.add_argument(
430
+ "--no-direct",
431
+ action="store_true",
432
+ help="Exclude DIRECT route from route pool",
433
+ )
434
+ parser.add_argument("--dry-run", action="store_true", help="Only list matched files")
435
+ parser.add_argument(
436
+ "--no-worker-progress",
437
+ action="store_true",
438
+ help="Disable managed real-time worker progress lines in parent process",
439
+ )
440
+ parser.add_argument(
441
+ "--no-route-fallback",
442
+ action="store_true",
443
+ help="Disable retrying failed files on other routes",
444
+ )
445
+ parser.add_argument(
446
+ "--check-proxies",
447
+ action="store_true",
448
+ help="Check whether each proxy route is reachable before download",
449
+ )
450
+ parser.add_argument(
451
+ "--check-only",
452
+ action="store_true",
453
+ help="Only check route reachability and exit without listing/downloading model files",
454
+ )
455
+ parser.add_argument(
456
+ "--check-url",
457
+ default="https://www.modelscope.cn",
458
+ help="URL used for route reachability checks",
459
+ )
460
+ parser.add_argument(
461
+ "--check-timeout",
462
+ type=float,
463
+ default=8.0,
464
+ help="Timeout seconds for each route reachability check (default: 8)",
465
+ )
466
+ return parser
467
+
468
+
469
+ def main() -> int:
470
+ args = build_parser().parse_args()
471
+ if not args.local_dir:
472
+ model_id_clean = args.model_id.strip().strip("/")
473
+ if not model_id_clean:
474
+ print("Error: invalid model_id for deriving local_dir", file=sys.stderr)
475
+ return 2
476
+ args.local_dir = model_id_clean.split("/")[-1]
477
+
478
+ proxy_inputs = _expand_proxy_args(args.proxy)
479
+ proxies = _normalize_proxy_list(proxy_inputs)
480
+ if not args.no_direct and _DIRECT_ROUTE not in proxies:
481
+ proxies.insert(0, _DIRECT_ROUTE)
482
+ if args.no_direct:
483
+ proxies = [p for p in proxies if p != _DIRECT_ROUTE]
484
+ if not proxies:
485
+ print(
486
+ "Error: no available routes. Provide --proxy values or remove --no-direct.",
487
+ file=sys.stderr,
488
+ )
489
+ return 2
490
+
491
+ if args.check_proxies or args.check_only:
492
+ print(f"Checking routes via: {args.check_url}")
493
+ ok_count = 0
494
+ for proxy in proxies:
495
+ ok, detail = _check_proxy_reachable(
496
+ proxy=proxy, timeout=args.check_timeout, test_url=args.check_url
497
+ )
498
+ label = "OK " if ok else "FAIL"
499
+ print(f"[{label}] {proxy} -> {detail}")
500
+ if ok:
501
+ ok_count += 1
502
+ print(f"Reachable routes: {ok_count}/{len(proxies)}")
503
+ if args.check_only:
504
+ return 0 if ok_count > 0 else 1
505
+
506
+ Path(args.local_dir).mkdir(parents=True, exist_ok=True)
507
+ revision, all_files, list_proxy = _list_model_files(
508
+ model_id=args.model_id,
509
+ revision=args.revision,
510
+ token=args.token,
511
+ proxies=proxies,
512
+ )
513
+ matched_files = [
514
+ f for f in all_files if _match_patterns(f, args.allow_patterns, args.ignore_patterns)
515
+ ]
516
+ if not matched_files:
517
+ print("No files matched allow/ignore patterns; nothing to download.")
518
+ return 0
519
+
520
+ print(f"Model: {args.model_id}")
521
+ print(f"Revision: {revision}")
522
+ print(f"Listed via proxy: {list_proxy}")
523
+ print(f"Matched files: {len(matched_files)} / Total files: {len(all_files)}")
524
+ print(f"Proxies: {len(proxies)}")
525
+
526
+ if args.dry_run:
527
+ for path in matched_files:
528
+ print(path)
529
+ return 0
530
+
531
+ workers = args.max_workers if args.max_workers is not None else len(proxies)
532
+ workers = max(1, min(workers, len(proxies)))
533
+ worker_proxies = proxies[:workers]
534
+ print(f"Workers: {workers}")
535
+
536
+ show_worker_progress = not args.no_worker_progress
537
+ manager = multiprocessing.Manager()
538
+ progress_queue = manager.Queue() if manager is not None else None
539
+ progress_renderer = _ProgressLayout(workers) if show_worker_progress else None
540
+ speed_state = _SpeedState(workers)
541
+ progress_stop_event = threading.Event() if show_worker_progress else None
542
+ abort_flags = manager.dict({slot: False for slot in range(workers)})
543
+ progress_thread = None
544
+ if show_worker_progress:
545
+ progress_thread = threading.Thread(
546
+ target=_progress_consumer,
547
+ args=(progress_queue, progress_renderer, speed_state, progress_stop_event),
548
+ daemon=True,
549
+ )
550
+ progress_thread.start()
551
+
552
+ executors = []
553
+ for slot, proxy in enumerate(worker_proxies):
554
+ executors.append(
555
+ ProcessPoolExecutor(
556
+ max_workers=1,
557
+ initializer=_worker_init,
558
+ initargs=(proxy, show_worker_progress, progress_queue, slot, abort_flags),
559
+ )
560
+ )
561
+
562
+ pending = {}
563
+ running_by_slot: dict[int, object] = {}
564
+ task_queue: list[dict] = [
565
+ {
566
+ "file_path": fp,
567
+ "preferred_proxy": None,
568
+ "reconnect_used": False,
569
+ "slow_abort_count": 0,
570
+ }
571
+ for fp in matched_files
572
+ ]
573
+ file_fail_retries: dict[str, int] = {fp: 0 for fp in matched_files}
574
+ failed: list[tuple[str, str]] = []
575
+
576
+ def _log(msg: str, err: bool = False) -> None:
577
+ if progress_renderer is not None:
578
+ progress_renderer.log(msg)
579
+ elif err:
580
+ print(msg, file=sys.stderr)
581
+ else:
582
+ print(msg)
583
+
584
+ def _submit_task(slot: int, task_item: dict) -> None:
585
+ file_path = task_item["file_path"]
586
+ task = {
587
+ "model_id": args.model_id,
588
+ "file_path": file_path,
589
+ "revision": revision,
590
+ "cache_dir": args.cache_dir,
591
+ "local_dir": args.local_dir,
592
+ "disable_worker_tqdm": not show_worker_progress,
593
+ "token": args.token,
594
+ }
595
+ abort_flags[slot] = False
596
+ future = executors[slot].submit(_download_one, task)
597
+ pending[future] = {
598
+ "slot": slot,
599
+ "proxy": worker_proxies[slot],
600
+ "file_path": file_path,
601
+ "preferred_proxy": task_item.get("preferred_proxy"),
602
+ "reconnect_used": bool(task_item.get("reconnect_used", False)),
603
+ "slow_abort_count": int(task_item.get("slow_abort_count", 0)),
604
+ "slow_action_taken": False,
605
+ "resubmit_proxy": None,
606
+ }
607
+ running_by_slot[slot] = future
608
+
609
+ def _pick_idle_slot(preferred_proxy: str | None) -> int | None:
610
+ idle_slots = [slot for slot in range(workers) if slot not in running_by_slot]
611
+ if not idle_slots:
612
+ return None
613
+ if preferred_proxy:
614
+ for slot in idle_slots:
615
+ if worker_proxies[slot] == preferred_proxy:
616
+ return slot
617
+ return idle_slots[0]
618
+
619
+ try:
620
+ completed = 0
621
+ total = len(matched_files)
622
+ while completed < total:
623
+ # Fill idle workers
624
+ while task_queue:
625
+ next_item = task_queue[0]
626
+ slot = _pick_idle_slot(next_item.get("preferred_proxy"))
627
+ if slot is None:
628
+ break
629
+ task_queue.pop(0)
630
+ _submit_task(slot, next_item)
631
+
632
+ # Check low-speed switching/reconnect policy
633
+ now = time.time()
634
+ for slot, future in list(running_by_slot.items()):
635
+ meta = pending.get(future)
636
+ if meta is None:
637
+ continue
638
+ snap = speed_state.snapshot_slot(slot)
639
+ low_since = snap["low_since"]
640
+ if low_since is None or now - low_since < _LOW_SPEED_WINDOW_SEC:
641
+ continue
642
+ if meta.get("slow_action_taken", False):
643
+ continue
644
+
645
+ idle_slots = [s for s in range(workers) if s not in running_by_slot and s != slot]
646
+ if idle_slots:
647
+ target_slot = idle_slots[0]
648
+ meta["resubmit_proxy"] = worker_proxies[target_slot]
649
+ meta["slow_action_taken"] = True
650
+ abort_flags[slot] = True
651
+ _log(
652
+ f"[SLOW] {meta['file_path']} <500KB/s for 60s, switch to proxy "
653
+ f"{meta['resubmit_proxy']}"
654
+ )
655
+ elif not meta["reconnect_used"]:
656
+ meta["resubmit_proxy"] = worker_proxies[slot]
657
+ meta["reconnect_used"] = True
658
+ meta["slow_action_taken"] = True
659
+ abort_flags[slot] = True
660
+ _log(
661
+ f"[SLOW] {meta['file_path']} <500KB/s for 60s, reconnect once via "
662
+ f"{meta['resubmit_proxy']}"
663
+ )
664
+
665
+ if not pending:
666
+ continue
667
+
668
+ done_futures = []
669
+ try:
670
+ for future in as_completed(list(pending.keys()), timeout=1):
671
+ done_futures.append(future)
672
+ except TimeoutError:
673
+ done_futures = []
674
+
675
+ for future in done_futures:
676
+ meta = pending.pop(future, None)
677
+ if meta is None:
678
+ continue
679
+ slot = meta["slot"]
680
+ running_by_slot.pop(slot, None)
681
+ abort_flags[slot] = False
682
+ file_path = meta["file_path"]
683
+
684
+ try:
685
+ result_path, ok, err = future.result()
686
+ except Exception as exc: # pragma: no cover
687
+ ok = False
688
+ result_path = file_path
689
+ err = f"{type(exc).__name__}: {exc}"
690
+
691
+ if ok:
692
+ completed += 1
693
+ _log(f"[{completed}/{total}] OK {result_path}")
694
+ continue
695
+
696
+ if _SLOW_ABORT_SENTINEL in err:
697
+ slow_abort_count = meta["slow_abort_count"] + 1
698
+ if slow_abort_count > (_MAX_FILE_RETRIES + 1):
699
+ completed += 1
700
+ fail_msg = f"{err} (exceeded slow-abort retries)"
701
+ _log(f"[{completed}/{total}] FAIL {file_path} -> {fail_msg}", err=True)
702
+ failed.append((file_path, fail_msg))
703
+ continue
704
+ task_queue.append(
705
+ {
706
+ "file_path": file_path,
707
+ "preferred_proxy": meta.get("resubmit_proxy"),
708
+ "reconnect_used": meta.get("reconnect_used", False),
709
+ "slow_abort_count": slow_abort_count,
710
+ }
711
+ )
712
+ _log(
713
+ f"[REQUEUE] {file_path} due to slow speed, retry via "
714
+ f"{meta.get('resubmit_proxy') or 'next available proxy'}"
715
+ )
716
+ continue
717
+
718
+ file_fail_retries[file_path] += 1
719
+ retry_count = file_fail_retries[file_path]
720
+ if retry_count <= _MAX_FILE_RETRIES:
721
+ preferred = worker_proxies[slot] if args.no_route_fallback else None
722
+ task_queue.append(
723
+ {
724
+ "file_path": file_path,
725
+ "preferred_proxy": preferred,
726
+ "reconnect_used": meta.get("reconnect_used", False),
727
+ "slow_abort_count": meta.get("slow_abort_count", 0),
728
+ }
729
+ )
730
+ _log(
731
+ f"[RETRY {retry_count}/{_MAX_FILE_RETRIES}] {file_path} -> {err}",
732
+ err=True,
733
+ )
734
+ else:
735
+ completed += 1
736
+ _log(f"[{completed}/{total}] FAIL {file_path} -> {err}", err=True)
737
+ failed.append((file_path, err))
738
+ finally:
739
+ for executor in executors:
740
+ executor.shutdown(wait=True, cancel_futures=False)
741
+ if progress_stop_event is not None:
742
+ progress_stop_event.set()
743
+ if progress_thread is not None:
744
+ progress_thread.join(timeout=1.0)
745
+ if manager is not None:
746
+ manager.shutdown()
747
+
748
+ if failed:
749
+ print(f"\nFailed files: {len(failed)}", file=sys.stderr)
750
+ for path, err in failed[:20]:
751
+ print(f"- {path}: {err}", file=sys.stderr)
752
+ if len(failed) > 20:
753
+ print(f"... and {len(failed) - 20} more", file=sys.stderr)
754
+ return 1
755
+
756
+ print("\nAll files downloaded successfully.")
757
+ return 0
758
+
759
+
760
+ if __name__ == "__main__":
761
+ raise SystemExit(main())
@@ -0,0 +1,20 @@
1
+ [build-system]
2
+ requires = ["setuptools>=68", "wheel"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "modelscope-multi-proxy-download"
7
+ version = "0.1.1"
8
+ description = "Multi-proxy downloader for ModelScope model repos"
9
+ readme = "README.md"
10
+ requires-python = ">=3.9"
11
+ dependencies = [
12
+ "modelscope",
13
+ "requests",
14
+ ]
15
+
16
+ [project.scripts]
17
+ modelscope_multi_proxy_download = "modelscope_multi_proxy_download:main"
18
+
19
+ [tool.setuptools]
20
+ py-modules = ["modelscope_multi_proxy_download"]
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+