dsh-scnet 0.4.0 → 0.4.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +1 -1
- package/README.md +70 -66
- package/README_CN.md +107 -0
- package/cordis.patch.yml +8 -0
- package/docs/installation.md +60 -0
- package/docs/installation.zh-CN.md +73 -0
- package/docs/testing.md +63 -0
- package/docs/testing.zh-CN.md +62 -0
- package/package.json +13 -8
- package/skills/scnet-hpc/SKILL.md +88 -246
- package/skills/scnet-hpc/clusters/zzeshell.conf +1 -1
- package/skills/scnet-hpc/references/adding-cluster.md +13 -16
- package/skills/scnet-hpc/references/environment.md +15 -15
- package/skills/scnet-hpc/references/quickstart-en.md +4 -0
- package/skills/scnet-hpc/references/setup.md +9 -11
- package/skills/scnet-hpc/references/software-compatibility.md +6 -6
- package/skills/scnet-hpc/references/troubleshooting.md +15 -16
- package/skills/scnet-hpc/scripts/new-job.sh +5 -5
- package/skills/scnet-hpc/scripts/refresh-cluster.sh +1 -1
|
@@ -1,269 +1,111 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: scnet-hpc
|
|
3
|
-
description:
|
|
3
|
+
description: Configure and operate SCNet HPC clusters through SSH and Slurm, including profile-based resource requests, job submission and diagnosis, environment preparation, and accelerator compatibility checks. Use for SCNet access, Slurm CPU/DCU jobs, Hygon DCU/DTK, offline compute nodes, or adding and verifying cluster profiles. Treat other accelerator platforms as unverified until tested on their target compute nodes.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
#
|
|
6
|
+
# SCNet HPC
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
8
|
+
Operate SCNet clusters from repository profiles and target-cluster evidence. The included
|
|
9
|
+
accelerator guidance is validated primarily for Hygon DCU/DTK and `gfx9xx`; do not transfer
|
|
10
|
+
those capability conclusions to other accelerator stacks. The repository scripts support Linux
|
|
11
|
+
and macOS natively. On Windows, use WSL2; native Windows execution is not supported.
|
|
11
12
|
|
|
12
|
-
##
|
|
13
|
+
## 中文说明
|
|
13
14
|
|
|
14
|
-
|
|
15
|
-
ls clusters/*.conf # 有哪些集群
|
|
16
|
-
cat clusters/<集群短名>.conf # 看具体参数
|
|
17
|
-
```
|
|
18
|
-
|
|
19
|
-
回答用户关于内存上限、分区名、硬件规格的问题时,**读 profile,不要凭记忆**。
|
|
20
|
-
不同集群的 `DEF_MEM_PER_CPU`、`PARTITION`、`MIN_GRES` 都不一样。
|
|
21
|
-
|
|
22
|
-
| 文件 | 用途 |
|
|
23
|
-
|---|---|
|
|
24
|
-
| `clusters/*.conf` | 每个集群一个 profile(连接、调度器、硬件、已知问题)|
|
|
25
|
-
| `clusters/_template.conf` | 新增集群的模板 |
|
|
26
|
-
| `scripts/setup-ssh.sh` | 新电脑上配连接(macOS/Linux,幂等)|
|
|
27
|
-
| `scripts/probe-cluster.sh` | 探测新集群,自动生成 profile |
|
|
28
|
-
| `scripts/refresh-cluster.sh` | 动态刷新已有集群的规则缓存 |
|
|
29
|
-
| `scripts/run-compute-probe.sh` | 在计算节点运行最小能力探针 |
|
|
30
|
-
| `scripts/new-job.sh` | 按 profile 生成合规的作业脚本 |
|
|
31
|
-
| `references/setup.md` | 连接配置详解、平台差异、故障排查 |
|
|
32
|
-
| `references/environment.md` | 环境栈、装依赖、目录约定、迁移注意 |
|
|
33
|
-
| `references/troubleshooting.md` | 排查手册、调试策略 |
|
|
34
|
-
| `references/adding-cluster.md` | 新增集群的完整步骤 |
|
|
35
|
-
| `references/hygon-dcu-development.md` | 海光 DCU 开发流程、DTK 工具库与官方资源索引 |
|
|
36
|
-
| `references/software-compatibility.md` | 国产超算上的软件兼容性排查、验证分层与公开报告脱敏 |
|
|
37
|
-
| `references/quickstart-en.md` | English quick start for SSH, Slurm, CPU/DCU jobs, and troubleshooting |
|
|
38
|
-
|
|
39
|
-
用户使用英文提问或需要英文操作说明时,读取 `references/quickstart-en.md`;集群具体
|
|
40
|
-
参数仍以 `clusters/*.conf` 为准,不从英文示例猜测。
|
|
41
|
-
|
|
42
|
-
## 换机器
|
|
43
|
-
|
|
44
|
-
本插件已随 DeepSeek Harness 安装,换机器时用 `dsh plugin --profile web add dsh-scnet` 重装即可。
|
|
45
|
-
配置集群连接:
|
|
46
|
-
|
|
47
|
-
```bash
|
|
48
|
-
./scripts/setup-ssh.sh --cluster <集群短名> <私钥文件> <用户名>
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
只有一个 profile 时可省略 `--cluster`;公开 profile 里的 `KEY_NAME_MARKER` 是占位符,所以需要显式传用户名。
|
|
52
|
-
## 新增集群
|
|
53
|
-
|
|
54
|
-
```bash
|
|
55
|
-
# 1. 手工确认能 ssh 上去(用平台给的私钥)
|
|
56
|
-
# 2. 探测,自动填大部分字段
|
|
57
|
-
./scripts/probe-cluster.sh <ssh别名> <集群短名> > clusters/<集群短名>.conf
|
|
58
|
-
# 3. 补齐探测不到的项(加速器型号、module 名等),profile 里有提示
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
完整步骤见 `references/adding-cluster.md`。
|
|
62
|
-
|
|
63
|
-
## 动态刷新已有集群规则
|
|
64
|
-
|
|
65
|
-
profile 里的 `SSH_HOST` / `SSH_PORT` 是固定连接参数,应写死在
|
|
66
|
-
`clusters/<集群短名>.conf`。其余容易随平台调整的规则
|
|
67
|
-
(`DEF_MEM_PER_CPU`、`MIN_GRES`、`PARTITION`、`NET_OK`、登录节点缺库等)
|
|
68
|
-
可以动态探测:
|
|
69
|
-
|
|
70
|
-
```bash
|
|
71
|
-
./scripts/refresh-cluster.sh --cluster <集群短名> # 只做登录节点只读探测
|
|
72
|
-
./scripts/refresh-cluster.sh --cluster <集群短名> --compute # 再提交一个 10 分钟小作业
|
|
73
|
-
./scripts/refresh-cluster.sh --cluster <集群短名> --dry-run # 只看结果,不写缓存
|
|
74
|
-
```
|
|
75
|
-
|
|
76
|
-
结果写到 `clusters/.cache/<集群短名>.auto.conf`,后续脚本会自动加载并覆盖
|
|
77
|
-
profile 中的同名字段。删除该缓存文件即可回退到纯 profile。生成作业时可用
|
|
78
|
-
`./scripts/new-job.sh --refresh ...` 先刷新再生成,或用 `--no-auto` 忽略缓存。
|
|
79
|
-
|
|
80
|
-
默认只做登录节点探测,不提交作业;`--compute` 才会到计算节点实测加速器架构、
|
|
81
|
-
FP8、Triton、bitsandbytes 和外网状态。
|
|
82
|
-
|
|
83
|
-
分区节点数 `NODE_COUNT` 是例外:每次加载集群时都会实时查询当前
|
|
84
|
-
`PARTITION` 的 `TotalNodes`,查不到才回退到 cache/profile。需要关闭时设
|
|
85
|
-
`SCNET_HPC_LIVE_NODE_COUNT=no`。
|
|
86
|
-
|
|
87
|
-
## 通用硬约束
|
|
88
|
-
|
|
89
|
-
以下几条在国产超算上普遍存在,**具体数值查 profile**。
|
|
90
|
-
|
|
91
|
-
### 1. 可能强制申请加速器
|
|
92
|
-
|
|
93
|
-
若 profile 里 `MIN_GRES` 非空,该集群的 QOS 强制每个作业至少申请那么多加速器,
|
|
94
|
-
**纯 CPU 作业会被拒**:
|
|
95
|
-
|
|
96
|
-
```
|
|
97
|
-
sbatch: error: min tres(gres/dcu) request 0 exceeds per-job max tres limit 1
|
|
98
|
-
sbatch: error: QOSMinGRES
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
哪怕作业根本不用 GPU(比如只读文件的探针),也要写 `--gres=<GRES_TYPE>:1`。
|
|
102
|
-
|
|
103
|
-
### 2. 内存上限 = `cpus-per-task × DEF_MEM_PER_CPU`
|
|
104
|
-
|
|
105
|
-
超了直接被拒(`too much memory was requested relative to the number of CPUs`)。
|
|
106
|
-
**要更多内存只能加 CPU 核数。**
|
|
107
|
-
|
|
108
|
-
用 `scripts/new-job.sh` 生成脚本可避免手算。以 `DEF_MEM_PER_CPU=3888` 为例:
|
|
109
|
-
|
|
110
|
-
| `--cpus-per-task` | 上限 | 安全写法 |
|
|
111
|
-
|---|---|---|
|
|
112
|
-
| 8 | 30.38 GB | `--mem=30gb` |
|
|
113
|
-
| 16 | 60.75 GB | `--mem=60gb` |
|
|
114
|
-
| 32 | 121.50 GB | `--mem=121gb` |
|
|
115
|
-
|
|
116
|
-
注意 `--mem` 只是请求量,cgroup 实际按 `cpus × DEF_MEM_PER_CPU` 给
|
|
117
|
-
——所以 `MaxRSS` 可能超过你写的 `--mem` 值。
|
|
118
|
-
|
|
119
|
-
### 3. 登录节点可能无法 import 深度学习框架
|
|
120
|
-
|
|
121
|
-
若 profile 的 `LOGIN_NODE_MISSING_LIBS` 非空,登录节点缺那些库,
|
|
122
|
-
即使 `module load` 之后也 import 不了:
|
|
123
|
-
|
|
124
|
-
```
|
|
125
|
-
ImportError: libmsgpackc.so.2: cannot open shared object file
|
|
126
|
-
```
|
|
127
|
-
|
|
128
|
-
**不要在登录节点试 torch 代码然后困惑为什么报错。** 哪怕只想看一眼 tensor 的
|
|
129
|
-
dtype,也要走 `srun` 或 `sbatch`。
|
|
130
|
-
|
|
131
|
-
登录节点通常用于 `git`、文件管理、`sacct`/`squeue` 和准备依赖;能否访问软件源、
|
|
132
|
-
下载模型或执行 `pip install` 必须查看 profile 的网络字段并现场确认。
|
|
15
|
+
本技能用于通过 SSH 和 Slurm 操作 SCNet 超算集群,覆盖基于 profile 的资源申请、作业生成与诊断、环境准备,以及加速器兼容性验证。Linux 和 macOS 可直接运行仓库脚本;Windows 不提供原生脚本入口,建议使用 WSL2。官方 SCNet 客户端支持 Windows 10 及以上、macOS 12 Monterey 及以上(ARM/x86),下载地址见 [SCNet 客户端下载页](https://www.scnet.cn/ui/mall/client/download)。
|
|
133
16
|
|
|
134
|
-
|
|
17
|
+
## Source of truth
|
|
135
18
|
|
|
136
|
-
|
|
137
|
-
|
|
19
|
+
- Read `clusters/<cluster>.conf` before reporting connection, partition, memory, hardware,
|
|
20
|
+
module, network, or known-limit information.
|
|
21
|
+
- Treat `clusters/.cache/<cluster>.auto.conf` as optional, time-sensitive probe data that
|
|
22
|
+
overrides matching profile fields.
|
|
23
|
+
- Verify dynamic scheduler state on the target cluster when the task depends on current
|
|
24
|
+
availability, limits, or node count.
|
|
25
|
+
- Do not infer one cluster's parameters from another cluster.
|
|
138
26
|
|
|
139
|
-
|
|
140
|
-
`kernels` 机制(从 HF Hub 下载预编译 kernel)。作业脚本里先声明离线,
|
|
141
|
-
让失败快速且信息明确:
|
|
27
|
+
List available profiles with:
|
|
142
28
|
|
|
143
29
|
```bash
|
|
144
|
-
|
|
145
|
-
export TRANSFORMERS_OFFLINE=1
|
|
30
|
+
ls clusters/*.conf
|
|
146
31
|
```
|
|
147
32
|
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
针对任何集群提交测试或探针任务时,**不要使用 `/tmp`,也不要从根目录下创建临时
|
|
151
|
-
目录**。统一在共享家目录中创建,例如 `$HOME/.scnet-hpc/tmp/$SLURM_JOB_ID`,并在
|
|
152
|
-
作业中导出 `TMPDIR`、`TMP`、`TEMP` 指向该目录。这样登录节点、计算节点和后续作业
|
|
153
|
-
都能访问同一份临时文件。任务结束后只清理本次作业自己的子目录。
|
|
154
|
-
|
|
155
|
-
`TMP_SHARED=no` 表示系统 `/tmp` 不作为跨节点共享路径使用;即使某个集群实测 `/tmp`
|
|
156
|
-
可见,测试脚本仍默认遵循上述家目录规则。
|
|
157
|
-
|
|
158
|
-
### 6. 先验证二进制兼容性,再做完整安装
|
|
159
|
-
|
|
160
|
-
遇到 Python/C++/GPU 软件安装问题时,先确认系统 ABI、manylinux 标签、编译器/DTK 版本和运行节点网络条件,再选择版本。不要只根据 pip/conda 能否解析依赖判断可用性;必须在目标计算节点完成最小 import、kernel、功能和端到端验证。
|
|
161
|
-
|
|
162
|
-
## 提交作业
|
|
163
|
-
|
|
164
|
-
```bash
|
|
165
|
-
./scripts/new-job.sh <作业名> [加速器数] [cpu数] [时长]
|
|
166
|
-
./scripts/new-job.sh probe # 1 卡, 8 核, 20 分钟
|
|
167
|
-
./scripts/new-job.sh infer 8 32 01:30:00 # 8 卡, 32 核, 1.5 小时
|
|
168
|
-
./scripts/new-job.sh --cpu-only build 0 32 01:00:00
|
|
169
|
-
```
|
|
33
|
+
If more than one profile exists, require an explicit cluster selection before generating a
|
|
34
|
+
job, changing SSH configuration, submitting work, or installing software.
|
|
170
35
|
|
|
171
|
-
|
|
172
|
-
临时指定其他分区时使用 `--partition <名称>`。
|
|
173
|
-
|
|
174
|
-
上传后先验证配额而不真排队:
|
|
175
|
-
|
|
176
|
-
```bash
|
|
177
|
-
ssh <集群> 'sbatch --test-only <脚本路径>'
|
|
178
|
-
```
|
|
179
|
-
|
|
180
|
-
返回 `Job N to start at <时间>` 表示配额合法;`Requested node configuration is
|
|
181
|
-
not available` 是当前无空闲节点(配额没问题);`too much memory` 才是超配额。
|
|
182
|
-
|
|
183
|
-
### ★ 作业脚本末尾必须写 `exit $rc`
|
|
184
|
-
|
|
185
|
-
**这是最容易被骗的地方。** bash 里 Python 崩了之后 bash 继续执行后面的行,
|
|
186
|
-
作业以最后一条命令的退出码结束:
|
|
187
|
-
|
|
188
|
-
```
|
|
189
|
-
$ sacct -j <JOB_ID>
|
|
190
|
-
<JOB_ID> | example-job | COMPLETED | 0:0 | 00:34:32 ← 可能是假的!
|
|
191
|
-
```
|
|
36
|
+
## Route the task
|
|
192
37
|
|
|
193
|
-
|
|
38
|
+
Read only the references required for the current operation:
|
|
194
39
|
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
`
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
1. **先读日志,不要相信 `sacct` 的 State**
|
|
245
|
-
2. `.err` 被 `Loading weights: ...%` 进度条刷屏时,`grep -v` 过滤掉再看
|
|
246
|
-
3. 按报错分类:
|
|
247
|
-
|
|
248
|
-
| 报错 | 原因 |
|
|
40
|
+
| Operation | Reference |
|
|
41
|
+
|---|---|
|
|
42
|
+
| Configure or repair SSH access | `references/setup.md` |
|
|
43
|
+
| Prepare Python, modules, virtual environments, or offline dependencies | `references/environment.md` |
|
|
44
|
+
| Diagnose Slurm, runtime, library, network, or accelerator failures | `references/troubleshooting.md` |
|
|
45
|
+
| Add or refresh a cluster profile | `references/adding-cluster.md` |
|
|
46
|
+
| Work with Hygon DCU, DTK, HIP libraries, profiling, migration, or containers | `references/hygon-dcu-development.md` |
|
|
47
|
+
| Evaluate software compatibility or prepare a public compatibility report | `references/software-compatibility.md` |
|
|
48
|
+
| Provide English operating instructions | `references/quickstart-en.md` |
|
|
49
|
+
|
|
50
|
+
Cluster parameters always come from the selected profile, including when an example in a
|
|
51
|
+
reference uses different values.
|
|
52
|
+
|
|
53
|
+
## Execution boundaries
|
|
54
|
+
|
|
55
|
+
- Read-only inspection and local script generation do not authorize SSH configuration changes,
|
|
56
|
+
dependency installation, job submission, job cancellation, or remote file deletion.
|
|
57
|
+
- Before a remote mutation, resolve the target cluster, command, paths, and expected effect.
|
|
58
|
+
- A compute-node probe consumes scheduler resources. Run it only when the user requests
|
|
59
|
+
submission or verification that requires compute-node evidence.
|
|
60
|
+
- Preserve existing SSH configuration. `scripts/setup-ssh.sh` adds a missing host entry and
|
|
61
|
+
backs up a conflicting destination key; it does not rewrite an existing host block.
|
|
62
|
+
- Do not place credentials, usernames, internal hosts, node names, job IDs, private mirrors, or
|
|
63
|
+
absolute home paths in committed profiles or public reports.
|
|
64
|
+
|
|
65
|
+
## Operational invariants
|
|
66
|
+
|
|
67
|
+
1. **Scheduler constraints are profile-specific.** If `MIN_GRES` is non-empty, request at least
|
|
68
|
+
that accelerator resource unless an independent CPU partition is selected.
|
|
69
|
+
2. **Memory requests follow the partition limit.** When `DEF_MEM_PER_CPU` is defined, keep
|
|
70
|
+
requested memory within `cpus-per-task × DEF_MEM_PER_CPU`. Use `scripts/new-job.sh` to
|
|
71
|
+
calculate a conservative value.
|
|
72
|
+
3. **Framework validation belongs on compute nodes.** A login node may lack accelerator
|
|
73
|
+
libraries even after modules are loaded. Use `srun`, `sbatch`, or the compute probe for
|
|
74
|
+
imports, kernels, collectives, and accelerator capabilities.
|
|
75
|
+
4. **Offline nodes require staged dependencies.** When `COMPUTE_NODE_OFFLINE=yes`, download or
|
|
76
|
+
build dependencies on a networked node and disable runtime downloads in the job.
|
|
77
|
+
5. **Test files use shared storage.** Create per-job temporary directories under
|
|
78
|
+
`$HOME/.scnet-hpc/tmp/$SLURM_JOB_ID`; do not depend on `/tmp` being shared across nodes.
|
|
79
|
+
6. **Job exit status must propagate.** Capture the workload return code and terminate the batch
|
|
80
|
+
script with `exit "$rc"`. Evaluate both accounting state and job logs.
|
|
81
|
+
7. **Compatibility claims require target-node evidence.** Package resolution or import success
|
|
82
|
+
alone is insufficient for native extensions and accelerator software. Validate the relevant
|
|
83
|
+
ABI, import, kernel, functional, and end-to-end layers.
|
|
84
|
+
|
|
85
|
+
## Repository tools
|
|
86
|
+
|
|
87
|
+
| Tool | Purpose |
|
|
249
88
|
|---|---|
|
|
250
|
-
| `
|
|
251
|
-
| `
|
|
252
|
-
| `
|
|
253
|
-
| `
|
|
254
|
-
| `
|
|
255
|
-
| `libgcvm.so.17git: cannot open shared object file` | Triton 在当前 DTK/DCU 栈不可用 |
|
|
256
|
-
| `python3: command not found` | 昆山计算节点需 `module load python/3.8.10` |
|
|
257
|
-
| `sacct` COMPLETED 但结果不对 | 脚本缺 `exit $rc` |
|
|
89
|
+
| `scripts/setup-ssh.sh` | Install a supplied key and add an SSH host entry |
|
|
90
|
+
| `scripts/new-job.sh` | Generate a profile-aware Slurm script |
|
|
91
|
+
| `scripts/probe-cluster.sh` | Produce an initial profile from read-only login-node probes |
|
|
92
|
+
| `scripts/refresh-cluster.sh` | Refresh dynamic profile fields; compute probing is opt-in |
|
|
93
|
+
| `scripts/run-compute-probe.sh` | Submit a minimal accelerator capability probe |
|
|
258
94
|
|
|
259
|
-
|
|
95
|
+
Use `scripts/new-job.sh --cluster <name> ...` for routine job generation. Validate the generated
|
|
96
|
+
script with `sbatch --test-only` before submission when the target cluster supports it.
|
|
260
97
|
|
|
261
|
-
##
|
|
98
|
+
## Evidence and reporting
|
|
262
99
|
|
|
263
|
-
|
|
100
|
+
Distinguish:
|
|
264
101
|
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
102
|
+
- profile values;
|
|
103
|
+
- live scheduler observations;
|
|
104
|
+
- login-node observations;
|
|
105
|
+
- compute-node observations;
|
|
106
|
+
- conclusions inferred from those observations.
|
|
268
107
|
|
|
269
|
-
|
|
108
|
+
For failures, report the failing command, relevant log excerpt, scheduler state and exit code,
|
|
109
|
+
target profile, and the next bounded diagnostic action. For compatibility results, state the
|
|
110
|
+
software and toolchain versions, accelerator architecture, validation layer reached, and
|
|
111
|
+
untested boundaries.
|
|
@@ -66,7 +66,7 @@ TMP_SHARED="no" # /tmp 是否跨节点共享
|
|
|
66
66
|
# 登录节点缺失的库(导致无法在登录节点 import 深度学习框架)
|
|
67
67
|
LOGIN_NODE_MISSING_LIBS="libgalaxyhip.so.5"
|
|
68
68
|
|
|
69
|
-
#
|
|
69
|
+
# 已验证的加速器能力限制
|
|
70
70
|
KNOWN_LIMITATIONS="Triton 不可用(实测 libgcvm.so.17git 或 libamdhip64.so 缺 hipDrvLaunchKernelEx);\
|
|
71
71
|
FP8 无硬件计算支持(torch._scaled_mm 拒绝 gfx936);\
|
|
72
72
|
bitsandbytes 8-bit 不可用(hipBLASLt 无 gfx936 INT8 algo);\
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
# 新增一个集群
|
|
2
2
|
|
|
3
3
|
这套 skill 的集群参数都在 `clusters/*.conf`,脚本从 profile 读取。新增集群
|
|
4
|
-
|
|
4
|
+
通常只需增加一个 profile;若调度器或字段语义不同,仍需扩展脚本。
|
|
5
5
|
|
|
6
|
-
##
|
|
6
|
+
## 配置流程
|
|
7
7
|
|
|
8
|
-
### 1.
|
|
8
|
+
### 1. 验证基础连接
|
|
9
9
|
|
|
10
10
|
用平台控制台下载的私钥,临时连一次:
|
|
11
11
|
|
|
@@ -14,9 +14,9 @@ chmod 600 <私钥>
|
|
|
14
14
|
ssh -i <私钥> -p <端口> <用户名>@<主机名> 'hostname; whoami'
|
|
15
15
|
```
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
连接失败时检查:私钥是否过期、本机 IP 是否在平台白名单、端口是否可达。
|
|
18
18
|
|
|
19
|
-
### 2.
|
|
19
|
+
### 2. 添加临时 SSH 别名
|
|
20
20
|
|
|
21
21
|
```bash
|
|
22
22
|
cat >> ~/.ssh/config <<'EOF'
|
|
@@ -30,7 +30,7 @@ Host newcluster-tmp
|
|
|
30
30
|
EOF
|
|
31
31
|
```
|
|
32
32
|
|
|
33
|
-
### 3.
|
|
33
|
+
### 3. 生成初始 profile
|
|
34
34
|
|
|
35
35
|
```bash
|
|
36
36
|
./scripts/probe-cluster.sh newcluster-tmp <集群短名> > clusters/<集群短名>.conf
|
|
@@ -39,7 +39,7 @@ EOF
|
|
|
39
39
|
脚本会自动填:SSH 参数、调度器类型、分区、`DEF_MEM_PER_CPU`、QOS 的 `MIN_GRES`、
|
|
40
40
|
节点规格、网络可达性、登录节点缺失的库、家目录容量。
|
|
41
41
|
|
|
42
|
-
|
|
42
|
+
**探测不到的项会留空并在文件末尾列出**,不填入推测值。
|
|
43
43
|
|
|
44
44
|
### 4. 补齐手工项
|
|
45
45
|
|
|
@@ -57,12 +57,12 @@ srun -p <分区> --gres=<类型>:1 --time=00:05:00 bash -lc 'module load <模块
|
|
|
57
57
|
srun -p <分区> --gres=gpu:1 --time=00:05:00 nvidia-smi --query-gpu=name --format=csv
|
|
58
58
|
```
|
|
59
59
|
|
|
60
|
-
**module 名** —— `ssh <集群> 'module avail 2>&1'`
|
|
60
|
+
**module 名** —— `ssh <集群> 'module avail 2>&1'` 中选择与目标环境匹配的编译器和加速器 SDK。
|
|
61
61
|
|
|
62
62
|
**`NEEDS_UPDATE_HOSTKEYS_NO`** —— 如果连接时看到
|
|
63
63
|
`client_global_hostkeys_prove_confirm: server gave bad signature`,设成 `yes`。
|
|
64
64
|
|
|
65
|
-
**`KEY_NAME_MARKER`** ——
|
|
65
|
+
**`KEY_NAME_MARKER`** —— 根据私钥文件名格式设置。若形如
|
|
66
66
|
`zhangsan_xxx.scnet.cn_RsaKeyExpireTime_2027-01-01.txt`,则设
|
|
67
67
|
`KEY_NAME_MARKER="_<集群主机>_"`,脚本就能自动提取用户名。
|
|
68
68
|
|
|
@@ -77,10 +77,9 @@ srun -p <分区> --gres=gpu:1 --time=00:05:00 nvidia-smi --query-gpu=name --form
|
|
|
77
77
|
公开 profile 使用占位符时,`<用户名>` 必须显式传入;如果已在本地私有 profile
|
|
78
78
|
里填了真实 `KEY_NAME_MARKER`,可以省略。
|
|
79
79
|
|
|
80
|
-
### 6.
|
|
80
|
+
### 6. 验证加速器能力并记录结论
|
|
81
81
|
|
|
82
|
-
|
|
83
|
-
花 5 分钟测清楚,能省掉后面几天的困惑。
|
|
82
|
+
加速器能力必须以目标计算节点的探针结果为准,不能从上游 ROCm、CUDA 或其他集群的结果推断。
|
|
84
83
|
|
|
85
84
|
所有探针和测试任务的临时文件必须放在共享家目录中,例如
|
|
86
85
|
`$HOME/.scnet-hpc/tmp/$SLURM_JOB_ID`。不要使用 `/tmp` 或从根目录创建临时目录;作业
|
|
@@ -127,15 +126,13 @@ except Exception as e:
|
|
|
127
126
|
|
|
128
127
|
把结论写进 profile 的 `KNOWN_LIMITATIONS`,格式参考已有的 profile。
|
|
129
128
|
|
|
130
|
-
Triton
|
|
131
|
-
vLLM 的 ROCm MoE 后端、transformers 的 finegrained-fp8)。提前知道能避免在框架
|
|
132
|
-
适配上白费时间。
|
|
129
|
+
Triton 是多种推理框架的运行前提;不可用时,对应的 Triton 执行路径也不可用(SGLang 的 FP8 路径、vLLM 的 ROCm MoE 后端、transformers 的 finegrained-fp8)。该结论应在框架适配前确认。
|
|
133
130
|
|
|
134
131
|
已有集群后续不必重跑完整流程,可用
|
|
135
132
|
`./scripts/refresh-cluster.sh --cluster <集群短名> --compute` 动态刷新调度器、
|
|
136
133
|
内存、分区、网络和计算节点能力,结果写到 `clusters/.cache/<集群短名>.auto.conf`。
|
|
137
134
|
|
|
138
|
-
## profile
|
|
135
|
+
## profile 字段参考
|
|
139
136
|
|
|
140
137
|
必填三项:`CLUSTER_ID`、`SSH_HOST`、`SSH_PORT`。其余留空则脚本跳过、文档显示为未探测。
|
|
141
138
|
|
|
@@ -2,13 +2,13 @@
|
|
|
2
2
|
|
|
3
3
|
具体参数在 `clusters/<集群短名>.conf`,本文讲通用方法。
|
|
4
4
|
|
|
5
|
-
##
|
|
5
|
+
## 读取和核验集群参数
|
|
6
6
|
|
|
7
7
|
```bash
|
|
8
8
|
cat clusters/<集群短名>.conf
|
|
9
9
|
```
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
任务依赖当前状态时,执行现场核验:
|
|
12
12
|
|
|
13
13
|
```bash
|
|
14
14
|
ssh <集群> 'scontrol show partition <分区>' # DefMemPerCPU / MaxTime / TotalNodes
|
|
@@ -16,7 +16,7 @@ ssh <集群> 'sinfo -p <分区> -o "%n %c %m %G %t"' # 每节点 CPU/内存/GRE
|
|
|
16
16
|
ssh <集群> 'sacctmgr show qos where name=<QOS名> format=Name,MinTRES%30,MaxTRES%30 -P'
|
|
17
17
|
```
|
|
18
18
|
|
|
19
|
-
|
|
19
|
+
也可使用动态刷新脚本,登录节点只读探测不会提交作业:
|
|
20
20
|
|
|
21
21
|
```bash
|
|
22
22
|
./scripts/refresh-cluster.sh --cluster <集群短名> --dry-run
|
|
@@ -42,7 +42,7 @@ module load compiler/gcc/9.3.0 compiler/dtk/26.04
|
|
|
42
42
|
DTK 是海光对 ROCm 的发行版,含 HIP、rocBLAS、hipBLASLt、MIOpen、RCCL 等,
|
|
43
43
|
把 ROCm 的部分库改了名(如 `libgalaxyhip.so` 对应 `libamdhip64.so`)。
|
|
44
44
|
|
|
45
|
-
|
|
45
|
+
列出可用模块:
|
|
46
46
|
|
|
47
47
|
```bash
|
|
48
48
|
ssh <集群> 'module avail 2>&1'
|
|
@@ -51,14 +51,14 @@ ssh <集群> 'module avail 2>&1'
|
|
|
51
51
|
## Python 环境
|
|
52
52
|
|
|
53
53
|
超算通常没有系统级的 torch,用 venv 或 conda env。profile 的 `DEFAULT_VENV`
|
|
54
|
-
|
|
54
|
+
记录默认环境路径(相对家目录)。
|
|
55
55
|
|
|
56
56
|
```bash
|
|
57
57
|
module load <MODULE_LOADS> # 必须先 load
|
|
58
58
|
source ~/<DEFAULT_VENV>/bin/activate
|
|
59
59
|
```
|
|
60
60
|
|
|
61
|
-
|
|
61
|
+
必须先加载工具链模块,再激活 Python 环境;否则,`import torch` 会报缺共享库:
|
|
62
62
|
|
|
63
63
|
```
|
|
64
64
|
ImportError: libgalaxyhip.so.5: cannot open shared object file
|
|
@@ -72,7 +72,7 @@ module load python/3.8.10
|
|
|
72
72
|
|
|
73
73
|
## 厂商预编译的 wheel
|
|
74
74
|
|
|
75
|
-
国产加速器的 torch
|
|
75
|
+
国产加速器的 torch 等框架需要厂商适配版,文件名带厂商标记。海光示例:
|
|
76
76
|
|
|
77
77
|
```
|
|
78
78
|
torch-2.7.1+das.opt1.dtk2604-cp312-...whl
|
|
@@ -85,16 +85,16 @@ pip 想升级 torch,加 `--no-deps` 阻止。
|
|
|
85
85
|
|
|
86
86
|
## 装依赖
|
|
87
87
|
|
|
88
|
-
|
|
88
|
+
当计算节点无外网时,在具备网络访问的登录节点准备依赖:
|
|
89
89
|
|
|
90
90
|
```bash
|
|
91
91
|
source ~/<venv>/bin/activate
|
|
92
92
|
pip install -i <PIP_INDEX> <包名>
|
|
93
93
|
```
|
|
94
94
|
|
|
95
|
-
`PIP_INDEX` 在 profile
|
|
95
|
+
`PIP_INDEX` 在 profile 里。镜像选择以 profile 和现场可达性为准。
|
|
96
96
|
|
|
97
|
-
|
|
97
|
+
安装后在目标计算节点验证(登录节点可能 import 不了框架):
|
|
98
98
|
|
|
99
99
|
```bash
|
|
100
100
|
srun -p <分区> --gres=<类型>:1 --cpus-per-task=8 --mem=30gb --time=00:05:00 \
|
|
@@ -115,14 +115,14 @@ snapshot_download('<repo>', cache_dir='/public/home/$USER/models/_ms_cache')
|
|
|
115
115
|
"
|
|
116
116
|
```
|
|
117
117
|
|
|
118
|
-
|
|
118
|
+
若 ModelScope 首次下载结果不完整或只保留 `.lock` 文件,可使用
|
|
119
119
|
`allow_patterns` 单独补:
|
|
120
120
|
|
|
121
121
|
```python
|
|
122
122
|
snapshot_download(repo, cache_dir=..., allow_patterns=["inference/*", "*.json"])
|
|
123
123
|
```
|
|
124
124
|
|
|
125
|
-
|
|
125
|
+
下载后核验所需文件、数量和校验信息。
|
|
126
126
|
|
|
127
127
|
## 家目录布局约定
|
|
128
128
|
|
|
@@ -140,7 +140,7 @@ softwares/ 代码、venv、编译产物
|
|
|
140
140
|
|
|
141
141
|
让所有 `#SBATCH --output` 指向 `scripts/logs/`,否则日志会散落在提交目录。
|
|
142
142
|
|
|
143
|
-
##
|
|
143
|
+
## 迁移环境目录
|
|
144
144
|
|
|
145
145
|
**移动含 venv / conda env 的目录前,检查 `bin/` 下的 shebang。** conda 把解释器
|
|
146
146
|
绝对路径写死在 `pip`、`torchrun` 等入口点第一行,直接移动会报 bad interpreter:
|
|
@@ -153,7 +153,7 @@ for f in *; do
|
|
|
153
153
|
done
|
|
154
154
|
```
|
|
155
155
|
|
|
156
|
-
|
|
156
|
+
迁移后验证入口脚本和环境前缀:
|
|
157
157
|
|
|
158
158
|
```bash
|
|
159
159
|
<新路径>/bin/python -c "import sys; print(sys.prefix)"
|
|
@@ -170,7 +170,7 @@ df -h ~ # 文件系统使用情况
|
|
|
170
170
|
du -sh */ | sort -h # 各目录体积(大目录很慢,考虑放后台)
|
|
171
171
|
```
|
|
172
172
|
|
|
173
|
-
|
|
173
|
+
`df` 显示的是**整个文件系统**的容量,不一定是你的个人配额。查个人配额要用
|
|
174
174
|
平台特定的命令(如 `lfs quota`)或看控制台。
|
|
175
175
|
|
|
176
176
|
模型权重是主要占用,几个大模型就是几百 GB。
|
|
@@ -4,6 +4,10 @@ This guide covers the common workflow for SCNet clusters. Current repository pro
|
|
|
4
4
|
mainly on Hygon DCU/DTK systems. Always read the target profile before using a partition,
|
|
5
5
|
memory value, accelerator type, module, or home-directory path.
|
|
6
6
|
|
|
7
|
+
The repository scripts support Linux and macOS natively. On Windows, use WSL2; native Windows
|
|
8
|
+
execution and Git Bash are not fully validated. This is separate from the official SCNet desktop
|
|
9
|
+
client, which supports Windows 10+ and macOS 12 Monterey+.
|
|
10
|
+
|
|
7
11
|
## 1. Inspect the available profiles
|
|
8
12
|
|
|
9
13
|
```bash
|