modelscope-multi-proxy-download 0.1.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- modelscope_multi_proxy_download-0.1.1/PKG-INFO +185 -0
- modelscope_multi_proxy_download-0.1.1/README.md +176 -0
- modelscope_multi_proxy_download-0.1.1/modelscope_multi_proxy_download.egg-info/PKG-INFO +185 -0
- modelscope_multi_proxy_download-0.1.1/modelscope_multi_proxy_download.egg-info/SOURCES.txt +9 -0
- modelscope_multi_proxy_download-0.1.1/modelscope_multi_proxy_download.egg-info/dependency_links.txt +1 -0
- modelscope_multi_proxy_download-0.1.1/modelscope_multi_proxy_download.egg-info/entry_points.txt +2 -0
- modelscope_multi_proxy_download-0.1.1/modelscope_multi_proxy_download.egg-info/requires.txt +2 -0
- modelscope_multi_proxy_download-0.1.1/modelscope_multi_proxy_download.egg-info/top_level.txt +1 -0
- modelscope_multi_proxy_download-0.1.1/modelscope_multi_proxy_download.py +761 -0
- modelscope_multi_proxy_download-0.1.1/pyproject.toml +20 -0
- modelscope_multi_proxy_download-0.1.1/setup.cfg +4 -0
|
@@ -0,0 +1,185 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: modelscope-multi-proxy-download
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: Multi-proxy downloader for ModelScope model repos
|
|
5
|
+
Requires-Python: >=3.9
|
|
6
|
+
Description-Content-Type: text/markdown
|
|
7
|
+
Requires-Dist: modelscope
|
|
8
|
+
Requires-Dist: requests
|
|
9
|
+
|
|
10
|
+
# modelscope-multi-proxy-download
|
|
11
|
+
|
|
12
|
+
多代理并发下载 ModelScope 模型的命令行工具。
|
|
13
|
+
|
|
14
|
+
## 安装
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
python3 -m venv .venv
|
|
18
|
+
.venv/bin/python -m pip install -e .
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
安装后命令为:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
.venv/bin/modelscope_multi_proxy_download --help
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## 快速使用指南
|
|
28
|
+
|
|
29
|
+
由于不同机器通常都可能遇到网络限速,推荐配合代理使用多路下载。
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
33
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
34
|
+
--local_dir ./Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
35
|
+
--proxy http://127.0.0.1:8888
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
仅检查代理链路可用性:
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
42
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
43
|
+
--proxy http://127.0.0.1:8888 \
|
|
44
|
+
--check-only
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Windows 场景快速指南
|
|
48
|
+
|
|
49
|
+
### 场景 A:Linux 服务器要下载模型,Windows 主动连接 Linux 并共享本地代理
|
|
50
|
+
|
|
51
|
+
目标:让 Linux 上的 `modelscope-multi-download` 通过 Windows 代理出网。
|
|
52
|
+
|
|
53
|
+
1. Windows 安装并启动 `proxy.py`(监听 8888)
|
|
54
|
+
|
|
55
|
+
```powershell
|
|
56
|
+
py -m pip install proxy.py
|
|
57
|
+
py -m proxy --hostname 0.0.0.0 --port 8888
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
2. Windows 主动连接 Linux,建立反向端口转发(把 Linux 上的 `127.0.0.1:8888` 映射到 Windows 的 `127.0.0.1:8888`)
|
|
61
|
+
|
|
62
|
+
```powershell
|
|
63
|
+
ssh -N -R 8888:127.0.0.1:8888 linux_user@linux_ip
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
3. Linux 下载模型(访问 Linux 本地 `127.0.0.1:8888`,实际走 Windows 代理)
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
70
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
71
|
+
--proxy http://127.0.0.1:8888
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
### 场景 B:Windows 是要下载模型的机器,Linux 提供代理
|
|
75
|
+
|
|
76
|
+
目标:让 Windows 上的下载进程通过 Linux 代理出网。
|
|
77
|
+
|
|
78
|
+
1. Linux 启动 `proxy.py`(监听 8888)
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
python3 -m pip install proxy.py
|
|
82
|
+
python3 -m proxy --hostname 0.0.0.0 --port 8888
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
2. Windows 建立本地端口转发(把 Windows 的 `127.0.0.1:8888` 映射到 Linux `127.0.0.1:8888`)
|
|
86
|
+
|
|
87
|
+
```powershell
|
|
88
|
+
ssh -N -L 8888:127.0.0.1:8888 linux_user@linux_ip
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
3. 在 Windows 上运行下载命令(PowerShell 或 WSL),代理填本地 8888
|
|
92
|
+
|
|
93
|
+
```powershell
|
|
94
|
+
modelscope_multi_proxy_download Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 --proxy http://127.0.0.1:8888
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## 详细使用步骤
|
|
98
|
+
|
|
99
|
+
兼容 `modelscope download MODEL_ID --local_dir ...` 的核心用法,并新增:
|
|
100
|
+
|
|
101
|
+
- `--proxy`(可重复传入,值可为代理地址或代理列表文件)
|
|
102
|
+
- 默认自动包含 `DIRECT` 直连
|
|
103
|
+
- `--local_dir` 不传时默认取 `model_id` 最后一段
|
|
104
|
+
- `--no-direct` 一般不需要;仅在你明确希望“禁止直连、只允许代理”时再启用
|
|
105
|
+
|
|
106
|
+
### 示例 1:多个代理并发下载
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
110
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
111
|
+
--local_dir Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
112
|
+
--proxy http://127.0.0.1:10090 \
|
|
113
|
+
--proxy http://127.0.0.1:10091
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
### 示例 2:省略 `--local_dir`
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
120
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
121
|
+
--proxy http://127.0.0.1:10090
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### 示例 3:只检测路由可达性
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
128
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
129
|
+
--proxy http://127.0.0.1:10090 \
|
|
130
|
+
--check-only
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
### 示例 4:禁用直连(只走代理)
|
|
134
|
+
|
|
135
|
+
```bash
|
|
136
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
137
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
138
|
+
--proxy http://127.0.0.1:10090 \
|
|
139
|
+
--proxy http://127.0.0.1:10091 \
|
|
140
|
+
--no-direct
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
### 示例 5:使用代理列表文件
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
147
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
148
|
+
--proxy ./proxies.txt
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
`proxies.txt` 示例:
|
|
152
|
+
|
|
153
|
+
```txt
|
|
154
|
+
http://127.0.0.1:10090
|
|
155
|
+
http://127.0.0.1:10091
|
|
156
|
+
DIRECT
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
### 参数说明
|
|
160
|
+
|
|
161
|
+
- `model_id`: 模型 ID,例如 `Qwen/Qwen3-VL-30B-A3B-Instruct-FP8`
|
|
162
|
+
- `--local_dir <dir>`: 下载目录,默认是 `model_id` 最后一段
|
|
163
|
+
- `--revision <rev>`: 分支、标签或提交
|
|
164
|
+
- `--cache_dir <dir>`: ModelScope 缓存目录
|
|
165
|
+
- `--token <token>`: 私有模型 token
|
|
166
|
+
- `--allow-pattern <glob>`: 仅下载匹配文件(可重复)
|
|
167
|
+
- `--ignore-pattern <glob>`: 忽略匹配文件(可重复)
|
|
168
|
+
- `--proxy <proxy_or_file>`: 代理地址或代理列表文件(可重复)
|
|
169
|
+
- `--max-workers <n>`: 并发 worker 数,默认等于可用路由数
|
|
170
|
+
- `--no-direct`: 排除直连路由
|
|
171
|
+
- `--dry-run`: 仅列出将下载文件
|
|
172
|
+
- `--no-worker-progress`: 关闭主进程实时任务进度区
|
|
173
|
+
- `--no-route-fallback`: 失败后不主动切换其他路由
|
|
174
|
+
- `--check-proxies`: 下载前检测每条路由
|
|
175
|
+
- `--check-only`: 仅检测并退出
|
|
176
|
+
- `--check-url <url>`: 路由检测目标 URL(默认 `https://www.modelscope.cn`)
|
|
177
|
+
- `--check-timeout <sec>`: 路由检测超时时间,默认 `8`
|
|
178
|
+
|
|
179
|
+
### 行为说明
|
|
180
|
+
|
|
181
|
+
- 每个 worker 进程固定绑定一个路由,按文件粒度并发下载。
|
|
182
|
+
- `DIRECT`/`NONE`/`NO_PROXY`/`LOCAL` 会被识别为直连。
|
|
183
|
+
- 若某任务速度连续 60 秒低于 `500KB/s`,优先切换空闲代理,否则最多尝试 1 次断开重连。
|
|
184
|
+
- 失败文件最多自动重试 3 次。
|
|
185
|
+
- 有失败文件时最终返回非 0 退出码。
|
|
@@ -0,0 +1,176 @@
|
|
|
1
|
+
# modelscope-multi-proxy-download
|
|
2
|
+
|
|
3
|
+
多代理并发下载 ModelScope 模型的命令行工具。
|
|
4
|
+
|
|
5
|
+
## 安装
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
python3 -m venv .venv
|
|
9
|
+
.venv/bin/python -m pip install -e .
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
安装后命令为:
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
.venv/bin/modelscope_multi_proxy_download --help
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## 快速使用指南
|
|
19
|
+
|
|
20
|
+
由于不同机器通常都可能遇到网络限速,推荐配合代理使用多路下载。
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
24
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
25
|
+
--local_dir ./Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
26
|
+
--proxy http://127.0.0.1:8888
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
仅检查代理链路可用性:
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
33
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
34
|
+
--proxy http://127.0.0.1:8888 \
|
|
35
|
+
--check-only
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Windows 场景快速指南
|
|
39
|
+
|
|
40
|
+
### 场景 A:Linux 服务器要下载模型,Windows 主动连接 Linux 并共享本地代理
|
|
41
|
+
|
|
42
|
+
目标:让 Linux 上的 `modelscope-multi-download` 通过 Windows 代理出网。
|
|
43
|
+
|
|
44
|
+
1. Windows 安装并启动 `proxy.py`(监听 8888)
|
|
45
|
+
|
|
46
|
+
```powershell
|
|
47
|
+
py -m pip install proxy.py
|
|
48
|
+
py -m proxy --hostname 0.0.0.0 --port 8888
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
2. Windows 主动连接 Linux,建立反向端口转发(把 Linux 上的 `127.0.0.1:8888` 映射到 Windows 的 `127.0.0.1:8888`)
|
|
52
|
+
|
|
53
|
+
```powershell
|
|
54
|
+
ssh -N -R 8888:127.0.0.1:8888 linux_user@linux_ip
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
3. Linux 下载模型(访问 Linux 本地 `127.0.0.1:8888`,实际走 Windows 代理)
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
61
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
62
|
+
--proxy http://127.0.0.1:8888
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
### 场景 B:Windows 是要下载模型的机器,Linux 提供代理
|
|
66
|
+
|
|
67
|
+
目标:让 Windows 上的下载进程通过 Linux 代理出网。
|
|
68
|
+
|
|
69
|
+
1. Linux 启动 `proxy.py`(监听 8888)
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
python3 -m pip install proxy.py
|
|
73
|
+
python3 -m proxy --hostname 0.0.0.0 --port 8888
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
2. Windows 建立本地端口转发(把 Windows 的 `127.0.0.1:8888` 映射到 Linux `127.0.0.1:8888`)
|
|
77
|
+
|
|
78
|
+
```powershell
|
|
79
|
+
ssh -N -L 8888:127.0.0.1:8888 linux_user@linux_ip
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
3. 在 Windows 上运行下载命令(PowerShell 或 WSL),代理填本地 8888
|
|
83
|
+
|
|
84
|
+
```powershell
|
|
85
|
+
modelscope_multi_proxy_download Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 --proxy http://127.0.0.1:8888
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
## 详细使用步骤
|
|
89
|
+
|
|
90
|
+
兼容 `modelscope download MODEL_ID --local_dir ...` 的核心用法,并新增:
|
|
91
|
+
|
|
92
|
+
- `--proxy`(可重复传入,值可为代理地址或代理列表文件)
|
|
93
|
+
- 默认自动包含 `DIRECT` 直连
|
|
94
|
+
- `--local_dir` 不传时默认取 `model_id` 最后一段
|
|
95
|
+
- `--no-direct` 一般不需要;仅在你明确希望“禁止直连、只允许代理”时再启用
|
|
96
|
+
|
|
97
|
+
### 示例 1:多个代理并发下载
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
101
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
102
|
+
--local_dir Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
103
|
+
--proxy http://127.0.0.1:10090 \
|
|
104
|
+
--proxy http://127.0.0.1:10091
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
### 示例 2:省略 `--local_dir`
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
111
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
112
|
+
--proxy http://127.0.0.1:10090
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### 示例 3:只检测路由可达性
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
119
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
120
|
+
--proxy http://127.0.0.1:10090 \
|
|
121
|
+
--check-only
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### 示例 4:禁用直连(只走代理)
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
128
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
129
|
+
--proxy http://127.0.0.1:10090 \
|
|
130
|
+
--proxy http://127.0.0.1:10091 \
|
|
131
|
+
--no-direct
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
### 示例 5:使用代理列表文件
|
|
135
|
+
|
|
136
|
+
```bash
|
|
137
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
138
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
139
|
+
--proxy ./proxies.txt
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
`proxies.txt` 示例:
|
|
143
|
+
|
|
144
|
+
```txt
|
|
145
|
+
http://127.0.0.1:10090
|
|
146
|
+
http://127.0.0.1:10091
|
|
147
|
+
DIRECT
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
### 参数说明
|
|
151
|
+
|
|
152
|
+
- `model_id`: 模型 ID,例如 `Qwen/Qwen3-VL-30B-A3B-Instruct-FP8`
|
|
153
|
+
- `--local_dir <dir>`: 下载目录,默认是 `model_id` 最后一段
|
|
154
|
+
- `--revision <rev>`: 分支、标签或提交
|
|
155
|
+
- `--cache_dir <dir>`: ModelScope 缓存目录
|
|
156
|
+
- `--token <token>`: 私有模型 token
|
|
157
|
+
- `--allow-pattern <glob>`: 仅下载匹配文件(可重复)
|
|
158
|
+
- `--ignore-pattern <glob>`: 忽略匹配文件(可重复)
|
|
159
|
+
- `--proxy <proxy_or_file>`: 代理地址或代理列表文件(可重复)
|
|
160
|
+
- `--max-workers <n>`: 并发 worker 数,默认等于可用路由数
|
|
161
|
+
- `--no-direct`: 排除直连路由
|
|
162
|
+
- `--dry-run`: 仅列出将下载文件
|
|
163
|
+
- `--no-worker-progress`: 关闭主进程实时任务进度区
|
|
164
|
+
- `--no-route-fallback`: 失败后不主动切换其他路由
|
|
165
|
+
- `--check-proxies`: 下载前检测每条路由
|
|
166
|
+
- `--check-only`: 仅检测并退出
|
|
167
|
+
- `--check-url <url>`: 路由检测目标 URL(默认 `https://www.modelscope.cn`)
|
|
168
|
+
- `--check-timeout <sec>`: 路由检测超时时间,默认 `8`
|
|
169
|
+
|
|
170
|
+
### 行为说明
|
|
171
|
+
|
|
172
|
+
- 每个 worker 进程固定绑定一个路由,按文件粒度并发下载。
|
|
173
|
+
- `DIRECT`/`NONE`/`NO_PROXY`/`LOCAL` 会被识别为直连。
|
|
174
|
+
- 若某任务速度连续 60 秒低于 `500KB/s`,优先切换空闲代理,否则最多尝试 1 次断开重连。
|
|
175
|
+
- 失败文件最多自动重试 3 次。
|
|
176
|
+
- 有失败文件时最终返回非 0 退出码。
|
|
@@ -0,0 +1,185 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: modelscope-multi-proxy-download
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: Multi-proxy downloader for ModelScope model repos
|
|
5
|
+
Requires-Python: >=3.9
|
|
6
|
+
Description-Content-Type: text/markdown
|
|
7
|
+
Requires-Dist: modelscope
|
|
8
|
+
Requires-Dist: requests
|
|
9
|
+
|
|
10
|
+
# modelscope-multi-proxy-download
|
|
11
|
+
|
|
12
|
+
多代理并发下载 ModelScope 模型的命令行工具。
|
|
13
|
+
|
|
14
|
+
## 安装
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
python3 -m venv .venv
|
|
18
|
+
.venv/bin/python -m pip install -e .
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
安装后命令为:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
.venv/bin/modelscope_multi_proxy_download --help
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## 快速使用指南
|
|
28
|
+
|
|
29
|
+
由于不同机器通常都可能遇到网络限速,推荐配合代理使用多路下载。
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
33
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
34
|
+
--local_dir ./Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
35
|
+
--proxy http://127.0.0.1:8888
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
仅检查代理链路可用性:
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
42
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
43
|
+
--proxy http://127.0.0.1:8888 \
|
|
44
|
+
--check-only
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Windows 场景快速指南
|
|
48
|
+
|
|
49
|
+
### 场景 A:Linux 服务器要下载模型,Windows 主动连接 Linux 并共享本地代理
|
|
50
|
+
|
|
51
|
+
目标:让 Linux 上的 `modelscope-multi-download` 通过 Windows 代理出网。
|
|
52
|
+
|
|
53
|
+
1. Windows 安装并启动 `proxy.py`(监听 8888)
|
|
54
|
+
|
|
55
|
+
```powershell
|
|
56
|
+
py -m pip install proxy.py
|
|
57
|
+
py -m proxy --hostname 0.0.0.0 --port 8888
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
2. Windows 主动连接 Linux,建立反向端口转发(把 Linux 上的 `127.0.0.1:8888` 映射到 Windows 的 `127.0.0.1:8888`)
|
|
61
|
+
|
|
62
|
+
```powershell
|
|
63
|
+
ssh -N -R 8888:127.0.0.1:8888 linux_user@linux_ip
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
3. Linux 下载模型(访问 Linux 本地 `127.0.0.1:8888`,实际走 Windows 代理)
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
70
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
71
|
+
--proxy http://127.0.0.1:8888
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
### 场景 B:Windows 是要下载模型的机器,Linux 提供代理
|
|
75
|
+
|
|
76
|
+
目标:让 Windows 上的下载进程通过 Linux 代理出网。
|
|
77
|
+
|
|
78
|
+
1. Linux 启动 `proxy.py`(监听 8888)
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
python3 -m pip install proxy.py
|
|
82
|
+
python3 -m proxy --hostname 0.0.0.0 --port 8888
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
2. Windows 建立本地端口转发(把 Windows 的 `127.0.0.1:8888` 映射到 Linux `127.0.0.1:8888`)
|
|
86
|
+
|
|
87
|
+
```powershell
|
|
88
|
+
ssh -N -L 8888:127.0.0.1:8888 linux_user@linux_ip
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
3. 在 Windows 上运行下载命令(PowerShell 或 WSL),代理填本地 8888
|
|
92
|
+
|
|
93
|
+
```powershell
|
|
94
|
+
modelscope_multi_proxy_download Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 --proxy http://127.0.0.1:8888
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## 详细使用步骤
|
|
98
|
+
|
|
99
|
+
兼容 `modelscope download MODEL_ID --local_dir ...` 的核心用法,并新增:
|
|
100
|
+
|
|
101
|
+
- `--proxy`(可重复传入,值可为代理地址或代理列表文件)
|
|
102
|
+
- 默认自动包含 `DIRECT` 直连
|
|
103
|
+
- `--local_dir` 不传时默认取 `model_id` 最后一段
|
|
104
|
+
- `--no-direct` 一般不需要;仅在你明确希望“禁止直连、只允许代理”时再启用
|
|
105
|
+
|
|
106
|
+
### 示例 1:多个代理并发下载
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
110
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
111
|
+
--local_dir Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
112
|
+
--proxy http://127.0.0.1:10090 \
|
|
113
|
+
--proxy http://127.0.0.1:10091
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
### 示例 2:省略 `--local_dir`
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
120
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
121
|
+
--proxy http://127.0.0.1:10090
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### 示例 3:只检测路由可达性
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
128
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
129
|
+
--proxy http://127.0.0.1:10090 \
|
|
130
|
+
--check-only
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
### 示例 4:禁用直连(只走代理)
|
|
134
|
+
|
|
135
|
+
```bash
|
|
136
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
137
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
138
|
+
--proxy http://127.0.0.1:10090 \
|
|
139
|
+
--proxy http://127.0.0.1:10091 \
|
|
140
|
+
--no-direct
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
### 示例 5:使用代理列表文件
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
.venv/bin/modelscope_multi_proxy_download \
|
|
147
|
+
Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
|
|
148
|
+
--proxy ./proxies.txt
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
`proxies.txt` 示例:
|
|
152
|
+
|
|
153
|
+
```txt
|
|
154
|
+
http://127.0.0.1:10090
|
|
155
|
+
http://127.0.0.1:10091
|
|
156
|
+
DIRECT
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
### 参数说明
|
|
160
|
+
|
|
161
|
+
- `model_id`: 模型 ID,例如 `Qwen/Qwen3-VL-30B-A3B-Instruct-FP8`
|
|
162
|
+
- `--local_dir <dir>`: 下载目录,默认是 `model_id` 最后一段
|
|
163
|
+
- `--revision <rev>`: 分支、标签或提交
|
|
164
|
+
- `--cache_dir <dir>`: ModelScope 缓存目录
|
|
165
|
+
- `--token <token>`: 私有模型 token
|
|
166
|
+
- `--allow-pattern <glob>`: 仅下载匹配文件(可重复)
|
|
167
|
+
- `--ignore-pattern <glob>`: 忽略匹配文件(可重复)
|
|
168
|
+
- `--proxy <proxy_or_file>`: 代理地址或代理列表文件(可重复)
|
|
169
|
+
- `--max-workers <n>`: 并发 worker 数,默认等于可用路由数
|
|
170
|
+
- `--no-direct`: 排除直连路由
|
|
171
|
+
- `--dry-run`: 仅列出将下载文件
|
|
172
|
+
- `--no-worker-progress`: 关闭主进程实时任务进度区
|
|
173
|
+
- `--no-route-fallback`: 失败后不主动切换其他路由
|
|
174
|
+
- `--check-proxies`: 下载前检测每条路由
|
|
175
|
+
- `--check-only`: 仅检测并退出
|
|
176
|
+
- `--check-url <url>`: 路由检测目标 URL(默认 `https://www.modelscope.cn`)
|
|
177
|
+
- `--check-timeout <sec>`: 路由检测超时时间,默认 `8`
|
|
178
|
+
|
|
179
|
+
### 行为说明
|
|
180
|
+
|
|
181
|
+
- 每个 worker 进程固定绑定一个路由,按文件粒度并发下载。
|
|
182
|
+
- `DIRECT`/`NONE`/`NO_PROXY`/`LOCAL` 会被识别为直连。
|
|
183
|
+
- 若某任务速度连续 60 秒低于 `500KB/s`,优先切换空闲代理,否则最多尝试 1 次断开重连。
|
|
184
|
+
- 失败文件最多自动重试 3 次。
|
|
185
|
+
- 有失败文件时最终返回非 0 退出码。
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
README.md
|
|
2
|
+
modelscope_multi_proxy_download.py
|
|
3
|
+
pyproject.toml
|
|
4
|
+
modelscope_multi_proxy_download.egg-info/PKG-INFO
|
|
5
|
+
modelscope_multi_proxy_download.egg-info/SOURCES.txt
|
|
6
|
+
modelscope_multi_proxy_download.egg-info/dependency_links.txt
|
|
7
|
+
modelscope_multi_proxy_download.egg-info/entry_points.txt
|
|
8
|
+
modelscope_multi_proxy_download.egg-info/requires.txt
|
|
9
|
+
modelscope_multi_proxy_download.egg-info/top_level.txt
|
modelscope_multi_proxy_download-0.1.1/modelscope_multi_proxy_download.egg-info/dependency_links.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
modelscope_multi_proxy_download
|
|
@@ -0,0 +1,761 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Multi-proxy downloader for ModelScope model repos."""
|
|
3
|
+
|
|
4
|
+
from __future__ import annotations
|
|
5
|
+
|
|
6
|
+
import argparse
|
|
7
|
+
import fnmatch
|
|
8
|
+
import multiprocessing
|
|
9
|
+
import os
|
|
10
|
+
import queue
|
|
11
|
+
import sys
|
|
12
|
+
import threading
|
|
13
|
+
import time
|
|
14
|
+
from concurrent.futures import ProcessPoolExecutor, TimeoutError, as_completed
|
|
15
|
+
from pathlib import Path
|
|
16
|
+
from typing import Iterable
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
_PROXY_ENV_KEYS = (
|
|
20
|
+
"http_proxy",
|
|
21
|
+
"https_proxy",
|
|
22
|
+
"HTTP_PROXY",
|
|
23
|
+
"HTTPS_PROXY",
|
|
24
|
+
"all_proxy",
|
|
25
|
+
"ALL_PROXY",
|
|
26
|
+
)
|
|
27
|
+
_DIRECT_ROUTE = "DIRECT"
|
|
28
|
+
_DIRECT_TOKENS = {"DIRECT", "NONE", "NO_PROXY", "LOCAL"}
|
|
29
|
+
_WORKER_PROGRESS_QUEUE = None
|
|
30
|
+
_WORKER_SLOT = -1
|
|
31
|
+
_WORKER_ABORT_FLAGS = None
|
|
32
|
+
_SLOW_ABORT_SENTINEL = "__SLOW_ABORT__"
|
|
33
|
+
_LOW_SPEED_THRESHOLD_BPS = 500 * 1024
|
|
34
|
+
_LOW_SPEED_WINDOW_SEC = 60
|
|
35
|
+
_MAX_FILE_RETRIES = 3
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
def _read_proxy_file(path: str) -> list[str]:
|
|
39
|
+
proxies: list[str] = []
|
|
40
|
+
with open(path, "r", encoding="utf-8") as f:
|
|
41
|
+
for raw in f:
|
|
42
|
+
line = raw.strip()
|
|
43
|
+
if not line or line.startswith("#"):
|
|
44
|
+
continue
|
|
45
|
+
proxies.append(line)
|
|
46
|
+
return proxies
|
|
47
|
+
|
|
48
|
+
|
|
49
|
+
def _expand_proxy_args(proxy_args: Iterable[str]) -> list[str]:
|
|
50
|
+
expanded: list[str] = []
|
|
51
|
+
for item in proxy_args:
|
|
52
|
+
value = item.strip()
|
|
53
|
+
if not value:
|
|
54
|
+
continue
|
|
55
|
+
proxy_file = Path(value)
|
|
56
|
+
if proxy_file.is_file():
|
|
57
|
+
expanded.extend(_read_proxy_file(str(proxy_file)))
|
|
58
|
+
else:
|
|
59
|
+
expanded.append(value)
|
|
60
|
+
return expanded
|
|
61
|
+
|
|
62
|
+
|
|
63
|
+
def _normalize_proxy_list(raw_proxies: Iterable[str]) -> list[str]:
|
|
64
|
+
deduped: list[str] = []
|
|
65
|
+
seen: set[str] = set()
|
|
66
|
+
for item in raw_proxies:
|
|
67
|
+
proxy = item.strip()
|
|
68
|
+
if not proxy:
|
|
69
|
+
continue
|
|
70
|
+
if proxy.upper() in _DIRECT_TOKENS:
|
|
71
|
+
proxy = _DIRECT_ROUTE
|
|
72
|
+
elif "://" not in proxy:
|
|
73
|
+
proxy = f"http://{proxy}"
|
|
74
|
+
if proxy in seen:
|
|
75
|
+
continue
|
|
76
|
+
seen.add(proxy)
|
|
77
|
+
deduped.append(proxy)
|
|
78
|
+
return deduped
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
def _set_proxy_env(proxy: str) -> dict[str, str | None]:
|
|
82
|
+
old: dict[str, str | None] = {k: os.environ.get(k) for k in _PROXY_ENV_KEYS}
|
|
83
|
+
if proxy == _DIRECT_ROUTE:
|
|
84
|
+
for key in _PROXY_ENV_KEYS:
|
|
85
|
+
os.environ.pop(key, None)
|
|
86
|
+
else:
|
|
87
|
+
for key in _PROXY_ENV_KEYS:
|
|
88
|
+
os.environ[key] = proxy
|
|
89
|
+
return old
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
def _restore_proxy_env(old: dict[str, str | None]) -> None:
|
|
93
|
+
for key, value in old.items():
|
|
94
|
+
if value is None:
|
|
95
|
+
os.environ.pop(key, None)
|
|
96
|
+
else:
|
|
97
|
+
os.environ[key] = value
|
|
98
|
+
|
|
99
|
+
|
|
100
|
+
def _list_model_files(
|
|
101
|
+
model_id: str,
|
|
102
|
+
revision: str | None,
|
|
103
|
+
token: str | None,
|
|
104
|
+
proxies: list[str],
|
|
105
|
+
) -> tuple[str, list[str], str]:
|
|
106
|
+
from modelscope.hub.api import HubApi
|
|
107
|
+
|
|
108
|
+
last_error = "unknown error"
|
|
109
|
+
for proxy in proxies:
|
|
110
|
+
old_env = _set_proxy_env(proxy)
|
|
111
|
+
try:
|
|
112
|
+
api = HubApi(token=token)
|
|
113
|
+
endpoint = api.get_endpoint_for_read(
|
|
114
|
+
repo_id=model_id, repo_type="model", token=token
|
|
115
|
+
)
|
|
116
|
+
rev = api.get_valid_revision(
|
|
117
|
+
model_id, revision=revision, cookies=api.get_cookies(), endpoint=endpoint
|
|
118
|
+
)
|
|
119
|
+
files = api.get_model_files(
|
|
120
|
+
model_id=model_id,
|
|
121
|
+
revision=rev,
|
|
122
|
+
recursive=True,
|
|
123
|
+
use_cookies=False if api.get_cookies() is None else api.get_cookies(),
|
|
124
|
+
endpoint=endpoint,
|
|
125
|
+
)
|
|
126
|
+
paths = [f["Path"] for f in files if f.get("Type") != "tree" and "Path" in f]
|
|
127
|
+
if not paths:
|
|
128
|
+
raise RuntimeError(f"No downloadable files found in {model_id}@{rev}")
|
|
129
|
+
return rev, paths, proxy
|
|
130
|
+
except Exception as exc: # pragma: no cover - runtime/network dependent
|
|
131
|
+
last_error = f"{type(exc).__name__}: {exc}"
|
|
132
|
+
finally:
|
|
133
|
+
_restore_proxy_env(old_env)
|
|
134
|
+
raise RuntimeError(f"Failed to list files via all proxies: {last_error}")
|
|
135
|
+
|
|
136
|
+
|
|
137
|
+
def _match_patterns(path: str, allow: list[str], ignore: list[str]) -> bool:
|
|
138
|
+
if allow and not any(fnmatch.fnmatch(path, p) for p in allow):
|
|
139
|
+
return False
|
|
140
|
+
if ignore and any(fnmatch.fnmatch(path, p) for p in ignore):
|
|
141
|
+
return False
|
|
142
|
+
return True
|
|
143
|
+
|
|
144
|
+
|
|
145
|
+
def _format_bytes(num: float) -> str:
|
|
146
|
+
units = ["B", "KB", "MB", "GB", "TB"]
|
|
147
|
+
value = float(max(0.0, num))
|
|
148
|
+
for unit in units:
|
|
149
|
+
if value < 1024.0 or unit == units[-1]:
|
|
150
|
+
if unit == "B":
|
|
151
|
+
return f"{int(value)}{unit}"
|
|
152
|
+
return f"{value:.1f}{unit}"
|
|
153
|
+
value /= 1024.0
|
|
154
|
+
return f"{value:.1f}TB"
|
|
155
|
+
|
|
156
|
+
|
|
157
|
+
class _QueueTqdm:
|
|
158
|
+
"""Replacement for worker tqdm; emits progress events to parent process."""
|
|
159
|
+
|
|
160
|
+
def __init__(self, *args, **kwargs):
|
|
161
|
+
self.desc = kwargs.get("desc", "")
|
|
162
|
+
self.total = kwargs.get("total", 0) or 0
|
|
163
|
+
self.n = kwargs.get("initial", 0) or 0
|
|
164
|
+
self._last_emit = 0.0
|
|
165
|
+
self._emit(force=True)
|
|
166
|
+
|
|
167
|
+
def _emit(self, force: bool = False) -> None:
|
|
168
|
+
if _WORKER_PROGRESS_QUEUE is None:
|
|
169
|
+
return
|
|
170
|
+
now = time.time()
|
|
171
|
+
if not force and now - self._last_emit < 0.2:
|
|
172
|
+
return
|
|
173
|
+
self._last_emit = now
|
|
174
|
+
_WORKER_PROGRESS_QUEUE.put(("progress", _WORKER_SLOT, self.desc, self.n, self.total))
|
|
175
|
+
|
|
176
|
+
def update(self, n: int = 1) -> None:
|
|
177
|
+
if _WORKER_ABORT_FLAGS is not None and _WORKER_ABORT_FLAGS.get(_WORKER_SLOT, False):
|
|
178
|
+
raise RuntimeError(_SLOW_ABORT_SENTINEL)
|
|
179
|
+
self.n += n
|
|
180
|
+
self._emit()
|
|
181
|
+
|
|
182
|
+
def close(self) -> None:
|
|
183
|
+
self._emit(force=True)
|
|
184
|
+
if _WORKER_PROGRESS_QUEUE is not None:
|
|
185
|
+
_WORKER_PROGRESS_QUEUE.put(
|
|
186
|
+
("progress_done", _WORKER_SLOT, self.desc, self.n, self.total)
|
|
187
|
+
)
|
|
188
|
+
|
|
189
|
+
|
|
190
|
+
class _ProgressLayout:
|
|
191
|
+
def __init__(self, workers: int):
|
|
192
|
+
self.workers = workers
|
|
193
|
+
self.status = [f"[Task {i + 1}] idle" for i in range(workers)]
|
|
194
|
+
self.footer = "[Total] speed: 0B/s"
|
|
195
|
+
self.enabled = sys.stdout.isatty()
|
|
196
|
+
self._printed = False
|
|
197
|
+
self._lock = threading.Lock()
|
|
198
|
+
|
|
199
|
+
def _render(self) -> None:
|
|
200
|
+
if not self.enabled:
|
|
201
|
+
return
|
|
202
|
+
if not self._printed:
|
|
203
|
+
sys.stdout.write("\n" * (self.workers + 1))
|
|
204
|
+
self._printed = True
|
|
205
|
+
sys.stdout.write(f"\x1b[{self.workers + 1}F")
|
|
206
|
+
for line in self.status:
|
|
207
|
+
sys.stdout.write(f"\r\x1b[2K{line[:200]}\n")
|
|
208
|
+
sys.stdout.write(f"\r\x1b[2K{self.footer[:200]}\n")
|
|
209
|
+
sys.stdout.flush()
|
|
210
|
+
|
|
211
|
+
def update(self, slot: int, line: str) -> None:
|
|
212
|
+
if slot < 0 or slot >= self.workers:
|
|
213
|
+
return
|
|
214
|
+
with self._lock:
|
|
215
|
+
self.status[slot] = line
|
|
216
|
+
self._render()
|
|
217
|
+
|
|
218
|
+
def set_footer(self, line: str) -> None:
|
|
219
|
+
with self._lock:
|
|
220
|
+
self.footer = line
|
|
221
|
+
self._render()
|
|
222
|
+
|
|
223
|
+
def log(self, line: str) -> None:
|
|
224
|
+
with self._lock:
|
|
225
|
+
if not self.enabled or not self._printed:
|
|
226
|
+
print(line)
|
|
227
|
+
return
|
|
228
|
+
sys.stdout.write(f"\x1b[{self.workers + 1}F")
|
|
229
|
+
sys.stdout.write(f"\r\x1b[2K{line}\n")
|
|
230
|
+
for status_line in self.status:
|
|
231
|
+
sys.stdout.write(f"\r\x1b[2K{status_line[:200]}\n")
|
|
232
|
+
sys.stdout.write(f"\r\x1b[2K{self.footer[:200]}\n")
|
|
233
|
+
sys.stdout.flush()
|
|
234
|
+
|
|
235
|
+
|
|
236
|
+
class _SpeedState:
|
|
237
|
+
def __init__(self, workers: int):
|
|
238
|
+
self._lock = threading.Lock()
|
|
239
|
+
self._speed = [0.0] * workers
|
|
240
|
+
self._desc = [""] * workers
|
|
241
|
+
self._done = [0.0] * workers
|
|
242
|
+
self._ts = [0.0] * workers
|
|
243
|
+
self._low_since = [None] * workers
|
|
244
|
+
|
|
245
|
+
def update(self, slot: int, desc: str, done: float, speed: float, now: float) -> None:
|
|
246
|
+
with self._lock:
|
|
247
|
+
self._speed[slot] = speed
|
|
248
|
+
if desc != self._desc[slot]:
|
|
249
|
+
self._low_since[slot] = None
|
|
250
|
+
self._desc[slot] = desc
|
|
251
|
+
self._done[slot] = done
|
|
252
|
+
self._ts[slot] = now
|
|
253
|
+
if done > 0 and speed < _LOW_SPEED_THRESHOLD_BPS and desc:
|
|
254
|
+
if self._low_since[slot] is None:
|
|
255
|
+
self._low_since[slot] = now
|
|
256
|
+
else:
|
|
257
|
+
self._low_since[slot] = None
|
|
258
|
+
|
|
259
|
+
def snapshot_slot(self, slot: int) -> dict:
|
|
260
|
+
with self._lock:
|
|
261
|
+
return {
|
|
262
|
+
"speed": self._speed[slot],
|
|
263
|
+
"desc": self._desc[slot],
|
|
264
|
+
"done": self._done[slot],
|
|
265
|
+
"ts": self._ts[slot],
|
|
266
|
+
"low_since": self._low_since[slot],
|
|
267
|
+
}
|
|
268
|
+
|
|
269
|
+
|
|
270
|
+
def _worker_init(
|
|
271
|
+
proxy: str,
|
|
272
|
+
enable_progress_layout: bool = False,
|
|
273
|
+
progress_queue=None,
|
|
274
|
+
worker_slot: int = -1,
|
|
275
|
+
abort_flags=None,
|
|
276
|
+
) -> None:
|
|
277
|
+
global _WORKER_PROGRESS_QUEUE, _WORKER_SLOT, _WORKER_ABORT_FLAGS
|
|
278
|
+
_set_proxy_env(proxy)
|
|
279
|
+
_WORKER_PROGRESS_QUEUE = progress_queue
|
|
280
|
+
_WORKER_SLOT = worker_slot
|
|
281
|
+
_WORKER_ABORT_FLAGS = abort_flags
|
|
282
|
+
if enable_progress_layout and progress_queue is not None:
|
|
283
|
+
import modelscope.hub.callback as callback_mod
|
|
284
|
+
import modelscope.hub.file_download as file_download_mod
|
|
285
|
+
|
|
286
|
+
callback_mod.tqdm = _QueueTqdm
|
|
287
|
+
file_download_mod.tqdm = _QueueTqdm
|
|
288
|
+
|
|
289
|
+
|
|
290
|
+
def _progress_consumer(
|
|
291
|
+
progress_queue, renderer: _ProgressLayout, speed_state: _SpeedState, stop_event: threading.Event
|
|
292
|
+
) -> None:
|
|
293
|
+
last_done = [0.0] * renderer.workers
|
|
294
|
+
last_ts = [0.0] * renderer.workers
|
|
295
|
+
speed = [0.0] * renderer.workers
|
|
296
|
+
last_desc = [""] * renderer.workers
|
|
297
|
+
while not stop_event.is_set():
|
|
298
|
+
try:
|
|
299
|
+
event = progress_queue.get(timeout=0.2)
|
|
300
|
+
except queue.Empty:
|
|
301
|
+
continue
|
|
302
|
+
except Exception:
|
|
303
|
+
break
|
|
304
|
+
|
|
305
|
+
_, slot, desc, done, total = event
|
|
306
|
+
now = time.time()
|
|
307
|
+
if slot < 0 or slot >= renderer.workers:
|
|
308
|
+
continue
|
|
309
|
+
|
|
310
|
+
if desc != last_desc[slot] or done < last_done[slot]:
|
|
311
|
+
last_done[slot] = float(done)
|
|
312
|
+
last_ts[slot] = now
|
|
313
|
+
speed[slot] = 0.0
|
|
314
|
+
last_desc[slot] = desc
|
|
315
|
+
else:
|
|
316
|
+
dt = now - last_ts[slot]
|
|
317
|
+
delta = float(done) - last_done[slot]
|
|
318
|
+
if dt > 0 and delta >= 0:
|
|
319
|
+
inst = delta / dt
|
|
320
|
+
speed[slot] = inst if speed[slot] <= 0 else (speed[slot] * 0.7 + inst * 0.3)
|
|
321
|
+
last_done[slot] = float(done)
|
|
322
|
+
last_ts[slot] = now
|
|
323
|
+
|
|
324
|
+
if total and total > 0:
|
|
325
|
+
pct = f"{(done / total) * 100:5.1f}%"
|
|
326
|
+
progress = f"{_format_bytes(done)}/{_format_bytes(total)}"
|
|
327
|
+
else:
|
|
328
|
+
pct = " ?.?%"
|
|
329
|
+
progress = f"{_format_bytes(done)}"
|
|
330
|
+
renderer.update(
|
|
331
|
+
slot,
|
|
332
|
+
f"[Task {slot + 1}] {pct} {progress} {_format_bytes(speed[slot])}/s {desc}",
|
|
333
|
+
)
|
|
334
|
+
renderer.set_footer(f"[Total] speed: {_format_bytes(sum(speed))}/s")
|
|
335
|
+
speed_state.update(slot=slot, desc=desc, done=float(done), speed=float(speed[slot]), now=now)
|
|
336
|
+
|
|
337
|
+
|
|
338
|
+
def _check_proxy_reachable(proxy: str, timeout: float, test_url: str) -> tuple[bool, str]:
|
|
339
|
+
import requests
|
|
340
|
+
|
|
341
|
+
old_env = _set_proxy_env(proxy)
|
|
342
|
+
started = time.time()
|
|
343
|
+
try:
|
|
344
|
+
resp = requests.get(test_url, timeout=timeout)
|
|
345
|
+
elapsed = (time.time() - started) * 1000.0
|
|
346
|
+
return True, f"HTTP {resp.status_code}, {elapsed:.0f}ms"
|
|
347
|
+
except Exception as exc: # pragma: no cover - runtime/network dependent
|
|
348
|
+
return False, f"{type(exc).__name__}: {exc}"
|
|
349
|
+
finally:
|
|
350
|
+
_restore_proxy_env(old_env)
|
|
351
|
+
|
|
352
|
+
|
|
353
|
+
def _download_one(task: dict) -> tuple[str, bool, str]:
|
|
354
|
+
from modelscope.hub.file_download import _repo_file_download
|
|
355
|
+
|
|
356
|
+
path = task["file_path"]
|
|
357
|
+
try:
|
|
358
|
+
_repo_file_download(
|
|
359
|
+
repo_id=task["model_id"],
|
|
360
|
+
file_path=path,
|
|
361
|
+
repo_type="model",
|
|
362
|
+
revision=task["revision"],
|
|
363
|
+
cache_dir=task["cache_dir"],
|
|
364
|
+
local_dir=task["local_dir"],
|
|
365
|
+
disable_tqdm=task["disable_worker_tqdm"],
|
|
366
|
+
token=task["token"],
|
|
367
|
+
)
|
|
368
|
+
return path, True, ""
|
|
369
|
+
except Exception as exc: # pragma: no cover - runtime/network dependent
|
|
370
|
+
return path, False, f"{type(exc).__name__}: {exc}"
|
|
371
|
+
|
|
372
|
+
|
|
373
|
+
def _download_one_via_route(task: dict, route: str) -> tuple[bool, str]:
|
|
374
|
+
"""Try downloading one file via a specific route in current process."""
|
|
375
|
+
from modelscope.hub.file_download import _repo_file_download
|
|
376
|
+
|
|
377
|
+
old_env = _set_proxy_env(route)
|
|
378
|
+
try:
|
|
379
|
+
_repo_file_download(
|
|
380
|
+
repo_id=task["model_id"],
|
|
381
|
+
file_path=task["file_path"],
|
|
382
|
+
repo_type="model",
|
|
383
|
+
revision=task["revision"],
|
|
384
|
+
cache_dir=task["cache_dir"],
|
|
385
|
+
local_dir=task["local_dir"],
|
|
386
|
+
disable_tqdm=task["disable_worker_tqdm"],
|
|
387
|
+
token=task["token"],
|
|
388
|
+
)
|
|
389
|
+
return True, ""
|
|
390
|
+
except Exception as exc: # pragma: no cover - runtime/network dependent
|
|
391
|
+
return False, f"{type(exc).__name__}: {exc}"
|
|
392
|
+
finally:
|
|
393
|
+
_restore_proxy_env(old_env)
|
|
394
|
+
|
|
395
|
+
|
|
396
|
+
def build_parser() -> argparse.ArgumentParser:
|
|
397
|
+
parser = argparse.ArgumentParser(
|
|
398
|
+
description=(
|
|
399
|
+
"Download a ModelScope model with multiple proxies in parallel. "
|
|
400
|
+
"Compatible with `modelscope download MODEL_ID --local_dir ...` core args."
|
|
401
|
+
)
|
|
402
|
+
)
|
|
403
|
+
parser.add_argument("model_id", help="Model id, e.g. Qwen/Qwen3-VL-30B-A3B-Instruct-FP8")
|
|
404
|
+
parser.add_argument(
|
|
405
|
+
"--local_dir",
|
|
406
|
+
default=None,
|
|
407
|
+
help="Target directory for downloaded files (default: last path segment of model_id)",
|
|
408
|
+
)
|
|
409
|
+
parser.add_argument("--revision", default=None, help="Branch/tag/commit revision")
|
|
410
|
+
parser.add_argument("--cache_dir", default=None, help="ModelScope cache directory")
|
|
411
|
+
parser.add_argument("--token", default=None, help="ModelScope token for private models")
|
|
412
|
+
parser.add_argument("--allow-pattern", dest="allow_patterns", action="append", default=[])
|
|
413
|
+
parser.add_argument("--ignore-pattern", dest="ignore_patterns", action="append", default=[])
|
|
414
|
+
parser.add_argument(
|
|
415
|
+
"--proxy",
|
|
416
|
+
action="append",
|
|
417
|
+
default=[],
|
|
418
|
+
help=(
|
|
419
|
+
"Proxy endpoint or proxy-list file path. Can be passed multiple times, "
|
|
420
|
+
"e.g. --proxy http://127.0.0.1:7890 or --proxy client/proxies.txt"
|
|
421
|
+
),
|
|
422
|
+
)
|
|
423
|
+
parser.add_argument(
|
|
424
|
+
"--max-workers",
|
|
425
|
+
type=int,
|
|
426
|
+
default=None,
|
|
427
|
+
help="Override worker count (default: number of proxies)",
|
|
428
|
+
)
|
|
429
|
+
parser.add_argument(
|
|
430
|
+
"--no-direct",
|
|
431
|
+
action="store_true",
|
|
432
|
+
help="Exclude DIRECT route from route pool",
|
|
433
|
+
)
|
|
434
|
+
parser.add_argument("--dry-run", action="store_true", help="Only list matched files")
|
|
435
|
+
parser.add_argument(
|
|
436
|
+
"--no-worker-progress",
|
|
437
|
+
action="store_true",
|
|
438
|
+
help="Disable managed real-time worker progress lines in parent process",
|
|
439
|
+
)
|
|
440
|
+
parser.add_argument(
|
|
441
|
+
"--no-route-fallback",
|
|
442
|
+
action="store_true",
|
|
443
|
+
help="Disable retrying failed files on other routes",
|
|
444
|
+
)
|
|
445
|
+
parser.add_argument(
|
|
446
|
+
"--check-proxies",
|
|
447
|
+
action="store_true",
|
|
448
|
+
help="Check whether each proxy route is reachable before download",
|
|
449
|
+
)
|
|
450
|
+
parser.add_argument(
|
|
451
|
+
"--check-only",
|
|
452
|
+
action="store_true",
|
|
453
|
+
help="Only check route reachability and exit without listing/downloading model files",
|
|
454
|
+
)
|
|
455
|
+
parser.add_argument(
|
|
456
|
+
"--check-url",
|
|
457
|
+
default="https://www.modelscope.cn",
|
|
458
|
+
help="URL used for route reachability checks",
|
|
459
|
+
)
|
|
460
|
+
parser.add_argument(
|
|
461
|
+
"--check-timeout",
|
|
462
|
+
type=float,
|
|
463
|
+
default=8.0,
|
|
464
|
+
help="Timeout seconds for each route reachability check (default: 8)",
|
|
465
|
+
)
|
|
466
|
+
return parser
|
|
467
|
+
|
|
468
|
+
|
|
469
|
+
def main() -> int:
|
|
470
|
+
args = build_parser().parse_args()
|
|
471
|
+
if not args.local_dir:
|
|
472
|
+
model_id_clean = args.model_id.strip().strip("/")
|
|
473
|
+
if not model_id_clean:
|
|
474
|
+
print("Error: invalid model_id for deriving local_dir", file=sys.stderr)
|
|
475
|
+
return 2
|
|
476
|
+
args.local_dir = model_id_clean.split("/")[-1]
|
|
477
|
+
|
|
478
|
+
proxy_inputs = _expand_proxy_args(args.proxy)
|
|
479
|
+
proxies = _normalize_proxy_list(proxy_inputs)
|
|
480
|
+
if not args.no_direct and _DIRECT_ROUTE not in proxies:
|
|
481
|
+
proxies.insert(0, _DIRECT_ROUTE)
|
|
482
|
+
if args.no_direct:
|
|
483
|
+
proxies = [p for p in proxies if p != _DIRECT_ROUTE]
|
|
484
|
+
if not proxies:
|
|
485
|
+
print(
|
|
486
|
+
"Error: no available routes. Provide --proxy values or remove --no-direct.",
|
|
487
|
+
file=sys.stderr,
|
|
488
|
+
)
|
|
489
|
+
return 2
|
|
490
|
+
|
|
491
|
+
if args.check_proxies or args.check_only:
|
|
492
|
+
print(f"Checking routes via: {args.check_url}")
|
|
493
|
+
ok_count = 0
|
|
494
|
+
for proxy in proxies:
|
|
495
|
+
ok, detail = _check_proxy_reachable(
|
|
496
|
+
proxy=proxy, timeout=args.check_timeout, test_url=args.check_url
|
|
497
|
+
)
|
|
498
|
+
label = "OK " if ok else "FAIL"
|
|
499
|
+
print(f"[{label}] {proxy} -> {detail}")
|
|
500
|
+
if ok:
|
|
501
|
+
ok_count += 1
|
|
502
|
+
print(f"Reachable routes: {ok_count}/{len(proxies)}")
|
|
503
|
+
if args.check_only:
|
|
504
|
+
return 0 if ok_count > 0 else 1
|
|
505
|
+
|
|
506
|
+
Path(args.local_dir).mkdir(parents=True, exist_ok=True)
|
|
507
|
+
revision, all_files, list_proxy = _list_model_files(
|
|
508
|
+
model_id=args.model_id,
|
|
509
|
+
revision=args.revision,
|
|
510
|
+
token=args.token,
|
|
511
|
+
proxies=proxies,
|
|
512
|
+
)
|
|
513
|
+
matched_files = [
|
|
514
|
+
f for f in all_files if _match_patterns(f, args.allow_patterns, args.ignore_patterns)
|
|
515
|
+
]
|
|
516
|
+
if not matched_files:
|
|
517
|
+
print("No files matched allow/ignore patterns; nothing to download.")
|
|
518
|
+
return 0
|
|
519
|
+
|
|
520
|
+
print(f"Model: {args.model_id}")
|
|
521
|
+
print(f"Revision: {revision}")
|
|
522
|
+
print(f"Listed via proxy: {list_proxy}")
|
|
523
|
+
print(f"Matched files: {len(matched_files)} / Total files: {len(all_files)}")
|
|
524
|
+
print(f"Proxies: {len(proxies)}")
|
|
525
|
+
|
|
526
|
+
if args.dry_run:
|
|
527
|
+
for path in matched_files:
|
|
528
|
+
print(path)
|
|
529
|
+
return 0
|
|
530
|
+
|
|
531
|
+
workers = args.max_workers if args.max_workers is not None else len(proxies)
|
|
532
|
+
workers = max(1, min(workers, len(proxies)))
|
|
533
|
+
worker_proxies = proxies[:workers]
|
|
534
|
+
print(f"Workers: {workers}")
|
|
535
|
+
|
|
536
|
+
show_worker_progress = not args.no_worker_progress
|
|
537
|
+
manager = multiprocessing.Manager()
|
|
538
|
+
progress_queue = manager.Queue() if manager is not None else None
|
|
539
|
+
progress_renderer = _ProgressLayout(workers) if show_worker_progress else None
|
|
540
|
+
speed_state = _SpeedState(workers)
|
|
541
|
+
progress_stop_event = threading.Event() if show_worker_progress else None
|
|
542
|
+
abort_flags = manager.dict({slot: False for slot in range(workers)})
|
|
543
|
+
progress_thread = None
|
|
544
|
+
if show_worker_progress:
|
|
545
|
+
progress_thread = threading.Thread(
|
|
546
|
+
target=_progress_consumer,
|
|
547
|
+
args=(progress_queue, progress_renderer, speed_state, progress_stop_event),
|
|
548
|
+
daemon=True,
|
|
549
|
+
)
|
|
550
|
+
progress_thread.start()
|
|
551
|
+
|
|
552
|
+
executors = []
|
|
553
|
+
for slot, proxy in enumerate(worker_proxies):
|
|
554
|
+
executors.append(
|
|
555
|
+
ProcessPoolExecutor(
|
|
556
|
+
max_workers=1,
|
|
557
|
+
initializer=_worker_init,
|
|
558
|
+
initargs=(proxy, show_worker_progress, progress_queue, slot, abort_flags),
|
|
559
|
+
)
|
|
560
|
+
)
|
|
561
|
+
|
|
562
|
+
pending = {}
|
|
563
|
+
running_by_slot: dict[int, object] = {}
|
|
564
|
+
task_queue: list[dict] = [
|
|
565
|
+
{
|
|
566
|
+
"file_path": fp,
|
|
567
|
+
"preferred_proxy": None,
|
|
568
|
+
"reconnect_used": False,
|
|
569
|
+
"slow_abort_count": 0,
|
|
570
|
+
}
|
|
571
|
+
for fp in matched_files
|
|
572
|
+
]
|
|
573
|
+
file_fail_retries: dict[str, int] = {fp: 0 for fp in matched_files}
|
|
574
|
+
failed: list[tuple[str, str]] = []
|
|
575
|
+
|
|
576
|
+
def _log(msg: str, err: bool = False) -> None:
|
|
577
|
+
if progress_renderer is not None:
|
|
578
|
+
progress_renderer.log(msg)
|
|
579
|
+
elif err:
|
|
580
|
+
print(msg, file=sys.stderr)
|
|
581
|
+
else:
|
|
582
|
+
print(msg)
|
|
583
|
+
|
|
584
|
+
def _submit_task(slot: int, task_item: dict) -> None:
|
|
585
|
+
file_path = task_item["file_path"]
|
|
586
|
+
task = {
|
|
587
|
+
"model_id": args.model_id,
|
|
588
|
+
"file_path": file_path,
|
|
589
|
+
"revision": revision,
|
|
590
|
+
"cache_dir": args.cache_dir,
|
|
591
|
+
"local_dir": args.local_dir,
|
|
592
|
+
"disable_worker_tqdm": not show_worker_progress,
|
|
593
|
+
"token": args.token,
|
|
594
|
+
}
|
|
595
|
+
abort_flags[slot] = False
|
|
596
|
+
future = executors[slot].submit(_download_one, task)
|
|
597
|
+
pending[future] = {
|
|
598
|
+
"slot": slot,
|
|
599
|
+
"proxy": worker_proxies[slot],
|
|
600
|
+
"file_path": file_path,
|
|
601
|
+
"preferred_proxy": task_item.get("preferred_proxy"),
|
|
602
|
+
"reconnect_used": bool(task_item.get("reconnect_used", False)),
|
|
603
|
+
"slow_abort_count": int(task_item.get("slow_abort_count", 0)),
|
|
604
|
+
"slow_action_taken": False,
|
|
605
|
+
"resubmit_proxy": None,
|
|
606
|
+
}
|
|
607
|
+
running_by_slot[slot] = future
|
|
608
|
+
|
|
609
|
+
def _pick_idle_slot(preferred_proxy: str | None) -> int | None:
|
|
610
|
+
idle_slots = [slot for slot in range(workers) if slot not in running_by_slot]
|
|
611
|
+
if not idle_slots:
|
|
612
|
+
return None
|
|
613
|
+
if preferred_proxy:
|
|
614
|
+
for slot in idle_slots:
|
|
615
|
+
if worker_proxies[slot] == preferred_proxy:
|
|
616
|
+
return slot
|
|
617
|
+
return idle_slots[0]
|
|
618
|
+
|
|
619
|
+
try:
|
|
620
|
+
completed = 0
|
|
621
|
+
total = len(matched_files)
|
|
622
|
+
while completed < total:
|
|
623
|
+
# Fill idle workers
|
|
624
|
+
while task_queue:
|
|
625
|
+
next_item = task_queue[0]
|
|
626
|
+
slot = _pick_idle_slot(next_item.get("preferred_proxy"))
|
|
627
|
+
if slot is None:
|
|
628
|
+
break
|
|
629
|
+
task_queue.pop(0)
|
|
630
|
+
_submit_task(slot, next_item)
|
|
631
|
+
|
|
632
|
+
# Check low-speed switching/reconnect policy
|
|
633
|
+
now = time.time()
|
|
634
|
+
for slot, future in list(running_by_slot.items()):
|
|
635
|
+
meta = pending.get(future)
|
|
636
|
+
if meta is None:
|
|
637
|
+
continue
|
|
638
|
+
snap = speed_state.snapshot_slot(slot)
|
|
639
|
+
low_since = snap["low_since"]
|
|
640
|
+
if low_since is None or now - low_since < _LOW_SPEED_WINDOW_SEC:
|
|
641
|
+
continue
|
|
642
|
+
if meta.get("slow_action_taken", False):
|
|
643
|
+
continue
|
|
644
|
+
|
|
645
|
+
idle_slots = [s for s in range(workers) if s not in running_by_slot and s != slot]
|
|
646
|
+
if idle_slots:
|
|
647
|
+
target_slot = idle_slots[0]
|
|
648
|
+
meta["resubmit_proxy"] = worker_proxies[target_slot]
|
|
649
|
+
meta["slow_action_taken"] = True
|
|
650
|
+
abort_flags[slot] = True
|
|
651
|
+
_log(
|
|
652
|
+
f"[SLOW] {meta['file_path']} <500KB/s for 60s, switch to proxy "
|
|
653
|
+
f"{meta['resubmit_proxy']}"
|
|
654
|
+
)
|
|
655
|
+
elif not meta["reconnect_used"]:
|
|
656
|
+
meta["resubmit_proxy"] = worker_proxies[slot]
|
|
657
|
+
meta["reconnect_used"] = True
|
|
658
|
+
meta["slow_action_taken"] = True
|
|
659
|
+
abort_flags[slot] = True
|
|
660
|
+
_log(
|
|
661
|
+
f"[SLOW] {meta['file_path']} <500KB/s for 60s, reconnect once via "
|
|
662
|
+
f"{meta['resubmit_proxy']}"
|
|
663
|
+
)
|
|
664
|
+
|
|
665
|
+
if not pending:
|
|
666
|
+
continue
|
|
667
|
+
|
|
668
|
+
done_futures = []
|
|
669
|
+
try:
|
|
670
|
+
for future in as_completed(list(pending.keys()), timeout=1):
|
|
671
|
+
done_futures.append(future)
|
|
672
|
+
except TimeoutError:
|
|
673
|
+
done_futures = []
|
|
674
|
+
|
|
675
|
+
for future in done_futures:
|
|
676
|
+
meta = pending.pop(future, None)
|
|
677
|
+
if meta is None:
|
|
678
|
+
continue
|
|
679
|
+
slot = meta["slot"]
|
|
680
|
+
running_by_slot.pop(slot, None)
|
|
681
|
+
abort_flags[slot] = False
|
|
682
|
+
file_path = meta["file_path"]
|
|
683
|
+
|
|
684
|
+
try:
|
|
685
|
+
result_path, ok, err = future.result()
|
|
686
|
+
except Exception as exc: # pragma: no cover
|
|
687
|
+
ok = False
|
|
688
|
+
result_path = file_path
|
|
689
|
+
err = f"{type(exc).__name__}: {exc}"
|
|
690
|
+
|
|
691
|
+
if ok:
|
|
692
|
+
completed += 1
|
|
693
|
+
_log(f"[{completed}/{total}] OK {result_path}")
|
|
694
|
+
continue
|
|
695
|
+
|
|
696
|
+
if _SLOW_ABORT_SENTINEL in err:
|
|
697
|
+
slow_abort_count = meta["slow_abort_count"] + 1
|
|
698
|
+
if slow_abort_count > (_MAX_FILE_RETRIES + 1):
|
|
699
|
+
completed += 1
|
|
700
|
+
fail_msg = f"{err} (exceeded slow-abort retries)"
|
|
701
|
+
_log(f"[{completed}/{total}] FAIL {file_path} -> {fail_msg}", err=True)
|
|
702
|
+
failed.append((file_path, fail_msg))
|
|
703
|
+
continue
|
|
704
|
+
task_queue.append(
|
|
705
|
+
{
|
|
706
|
+
"file_path": file_path,
|
|
707
|
+
"preferred_proxy": meta.get("resubmit_proxy"),
|
|
708
|
+
"reconnect_used": meta.get("reconnect_used", False),
|
|
709
|
+
"slow_abort_count": slow_abort_count,
|
|
710
|
+
}
|
|
711
|
+
)
|
|
712
|
+
_log(
|
|
713
|
+
f"[REQUEUE] {file_path} due to slow speed, retry via "
|
|
714
|
+
f"{meta.get('resubmit_proxy') or 'next available proxy'}"
|
|
715
|
+
)
|
|
716
|
+
continue
|
|
717
|
+
|
|
718
|
+
file_fail_retries[file_path] += 1
|
|
719
|
+
retry_count = file_fail_retries[file_path]
|
|
720
|
+
if retry_count <= _MAX_FILE_RETRIES:
|
|
721
|
+
preferred = worker_proxies[slot] if args.no_route_fallback else None
|
|
722
|
+
task_queue.append(
|
|
723
|
+
{
|
|
724
|
+
"file_path": file_path,
|
|
725
|
+
"preferred_proxy": preferred,
|
|
726
|
+
"reconnect_used": meta.get("reconnect_used", False),
|
|
727
|
+
"slow_abort_count": meta.get("slow_abort_count", 0),
|
|
728
|
+
}
|
|
729
|
+
)
|
|
730
|
+
_log(
|
|
731
|
+
f"[RETRY {retry_count}/{_MAX_FILE_RETRIES}] {file_path} -> {err}",
|
|
732
|
+
err=True,
|
|
733
|
+
)
|
|
734
|
+
else:
|
|
735
|
+
completed += 1
|
|
736
|
+
_log(f"[{completed}/{total}] FAIL {file_path} -> {err}", err=True)
|
|
737
|
+
failed.append((file_path, err))
|
|
738
|
+
finally:
|
|
739
|
+
for executor in executors:
|
|
740
|
+
executor.shutdown(wait=True, cancel_futures=False)
|
|
741
|
+
if progress_stop_event is not None:
|
|
742
|
+
progress_stop_event.set()
|
|
743
|
+
if progress_thread is not None:
|
|
744
|
+
progress_thread.join(timeout=1.0)
|
|
745
|
+
if manager is not None:
|
|
746
|
+
manager.shutdown()
|
|
747
|
+
|
|
748
|
+
if failed:
|
|
749
|
+
print(f"\nFailed files: {len(failed)}", file=sys.stderr)
|
|
750
|
+
for path, err in failed[:20]:
|
|
751
|
+
print(f"- {path}: {err}", file=sys.stderr)
|
|
752
|
+
if len(failed) > 20:
|
|
753
|
+
print(f"... and {len(failed) - 20} more", file=sys.stderr)
|
|
754
|
+
return 1
|
|
755
|
+
|
|
756
|
+
print("\nAll files downloaded successfully.")
|
|
757
|
+
return 0
|
|
758
|
+
|
|
759
|
+
|
|
760
|
+
if __name__ == "__main__":
|
|
761
|
+
raise SystemExit(main())
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=68", "wheel"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "modelscope-multi-proxy-download"
|
|
7
|
+
version = "0.1.1"
|
|
8
|
+
description = "Multi-proxy downloader for ModelScope model repos"
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.9"
|
|
11
|
+
dependencies = [
|
|
12
|
+
"modelscope",
|
|
13
|
+
"requests",
|
|
14
|
+
]
|
|
15
|
+
|
|
16
|
+
[project.scripts]
|
|
17
|
+
modelscope_multi_proxy_download = "modelscope_multi_proxy_download:main"
|
|
18
|
+
|
|
19
|
+
[tool.setuptools]
|
|
20
|
+
py-modules = ["modelscope_multi_proxy_download"]
|