dsh-scnet 0.6.2 → 0.6.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +13 -3
- package/README_CN.md +11 -2
- package/docs/testing.md +3 -0
- package/docs/testing.zh-CN.md +3 -0
- package/index.mjs +115 -13
- package/package.json +1 -1
- package/skills/scnet-hpc/SKILL.md +4 -3
- package/skills/scnet-hpc/VERSION +1 -1
- package/skills/scnet-hpc/clusters/_template.conf +1 -1
- package/skills/scnet-hpc/references/adding-cluster.md +1 -1
- package/skills/scnet-hpc/references/backends.md +3 -3
- package/skills/scnet-hpc/references/openapi.md +14 -0
- package/skills/scnet-hpc/references/quickstart-en.md +2 -1
- package/skills/scnet-hpc/scripts/scnet.py +86 -7
- package/skills/scnet-hpc/scripts/scnet_backends/openapi.py +33 -0
- package/skills/scnet-hpc/scripts/scnet_backends/profile.py +1 -1
- package/skills/scnet-hpc/scripts/setup.sh +4 -4
package/README.md
CHANGED
|
@@ -4,11 +4,18 @@
|
|
|
4
4
|
|
|
5
5
|
DSH-SCNet (`dsh-scnet` on npm) is a community-maintained DSH bundle for operating Supercomputing Network (SCNet). It packages the source [`scnet-hpc`](https://github.com/lql341/scnet-hpc) Skill, profile-aware utilities, and bounded tools for SSH, Slurm, OpenAPI files, regional resources, and Notebook discovery.
|
|
6
6
|
|
|
7
|
-
Current release: **0.6.
|
|
7
|
+
Current release: **0.6.4**
|
|
8
8
|
|
|
9
9
|
> This is an independent community project. It is compatible with DeepSeek Harness but is not an official DeepSeek product and does not imply endorsement, partnership, or authorization by DeepSeek.
|
|
10
10
|
|
|
11
|
-
## What's new in 0.6.
|
|
11
|
+
## What's new in 0.6.3
|
|
12
|
+
|
|
13
|
+
- Added read-only account and resource summaries plus bounded `scnet_job_wait`.
|
|
14
|
+
- Submission queue preflight now fails closed and never silently selects a queue.
|
|
15
|
+
- Download paths can default to the remote filename; ordinary and chunked uploads return the same
|
|
16
|
+
directory/filename result shape.
|
|
17
|
+
|
|
18
|
+
## 0.6.2 highlights
|
|
12
19
|
|
|
13
20
|
- Completed jobs now resolve through the fast filtered history-list endpoint rather than timing
|
|
14
21
|
out on the empty history-detail response.
|
|
@@ -67,11 +74,14 @@ See [Installation](./docs/installation.md) for npm, GitHub, and local-checkout w
|
|
|
67
74
|
| `scnet_run_compute_probe` | Submit a minimal compute-node capability probe |
|
|
68
75
|
| `scnet_status` | Read the saved backend, SSH profile, and OpenAPI configuration |
|
|
69
76
|
| `scnet_openapi_regions` | List authorized OpenAPI regions without exposing tokens |
|
|
77
|
+
| `scnet_account_summary` | Read account status and balance |
|
|
78
|
+
| `scnet_resource_summary` | Summarize regional queues and resource limits |
|
|
70
79
|
| `scnet_job_queues` | Query regional Slurm queues and live resource availability |
|
|
71
80
|
| `scnet_job_list` | List active or historical jobs with compact normalized records |
|
|
72
81
|
| `scnet_limits` | Read user and scheduler resource limits |
|
|
73
82
|
| `scnet_submit_job` | Submit a job through OpenAPI or SSH; `dry_run=true` only previews the request |
|
|
74
83
|
| `scnet_job_show` | Read one job's state, resources, and log paths |
|
|
84
|
+
| `scnet_job_wait` | Wait for a job to reach a terminal state with a bounded timeout |
|
|
75
85
|
| `scnet_job_logs` | Read job logs by explicit path, or by `job_id` + `work_dir` |
|
|
76
86
|
| `scnet_job_cancel` | Cancel a job; `dry_run=true` only previews the request |
|
|
77
87
|
| `scnet_file_list` | List shared-storage files through the common file API |
|
|
@@ -120,7 +130,7 @@ synchronizes the skill and version into `dsh-scnet` and `codex-scnet-hpc`. Publi
|
|
|
120
130
|
`dsh-scnet` npm package only after that downstream change is reviewed.
|
|
121
131
|
|
|
122
132
|
For a synchronized release, run the package validation on `main`, then trigger the repository's
|
|
123
|
-
`Publish DSH-SCNet` workflow with the exact value from `package.json` (currently `0.6.
|
|
133
|
+
`Publish DSH-SCNet` workflow with the exact value from `package.json` (currently `0.6.3`). The
|
|
124
134
|
workflow verifies the package/Skill version match, runs validation, creates the tarball, publishes
|
|
125
135
|
stable releases with the `latest` tag, and creates the matching Git tag. Do not bump only
|
|
126
136
|
`package.json`: `skills/scnet-hpc/VERSION`, the package version, and both README release markers
|
package/README_CN.md
CHANGED
|
@@ -4,10 +4,16 @@
|
|
|
4
4
|
|
|
5
5
|
<p align="center">在 DeepSeek Harness 里操作超算互联网(SCNet):配置 SSH、管理 Slurm 作业、查询 OpenAPI 区域和文件、检查 Notebook 资源与实例,并保留海光 DCU/DTK 诊断能力。</p>
|
|
6
6
|
|
|
7
|
-
当前版本:**0.6.
|
|
7
|
+
当前版本:**0.6.4**
|
|
8
8
|
|
|
9
9
|
> 本项目是独立维护的社区项目,与 DeepSeek Harness 兼容,但不是 DeepSeek 官方产品,也不表示 DeepSeek 对本项目提供背书、合作或授权。
|
|
10
10
|
|
|
11
|
+
## 0.6.3 更新
|
|
12
|
+
|
|
13
|
+
- 新增只读账户摘要、资源摘要和有界等待作业工具 `scnet_job_wait`。
|
|
14
|
+
- 提交前队列预检失败会停止提交,不再静默选择队列。
|
|
15
|
+
- 下载时可省略本地路径自动使用远端文件名;普通上传和分片上传返回统一结果结构。
|
|
16
|
+
|
|
11
17
|
## 0.6.2 更新
|
|
12
18
|
|
|
13
19
|
- 已结束作业改用快速的历史列表过滤查询,不再卡在返回空数据的历史详情接口。
|
|
@@ -71,11 +77,14 @@ git 源、本地目录安装以及验证步骤见[安装说明](./docs/installat
|
|
|
71
77
|
| scnet_run_compute_probe | 在计算节点运行最小能力探针 |
|
|
72
78
|
| scnet_status | 只读查看 backend、SSH profile 和 OpenAPI 配置 |
|
|
73
79
|
| scnet_openapi_regions | 列出授权 OpenAPI 区域,不暴露 token |
|
|
80
|
+
| scnet_account_summary | 查询账户状态和余额 |
|
|
81
|
+
| scnet_resource_summary | 汇总区域队列和资源限制 |
|
|
74
82
|
| scnet_job_queues | 查询区域 Slurm 队列和实时空闲资源 |
|
|
75
83
|
| scnet_job_list | 紧凑列出当前或历史作业 |
|
|
76
84
|
| scnet_limits | 查询用户和调度器资源限制 |
|
|
77
85
|
| scnet_submit_job | 通过 OpenAPI 或 SSH 提交作业;dry_run=true 只预览请求 |
|
|
78
86
|
| scnet_job_show | 查询单个作业的状态、资源与日志路径 |
|
|
87
|
+
| scnet_job_wait | 在限定时间内等待作业进入终态 |
|
|
79
88
|
| scnet_job_logs | 按显式 path,或按 job_id + work_dir 读取作业日志 |
|
|
80
89
|
| scnet_job_cancel | 取消作业;dry_run=true 只预览请求 |
|
|
81
90
|
| scnet_file_list | 通过通用文件 API 列出共享存储 |
|
|
@@ -132,7 +141,7 @@ git 源、本地目录安装以及验证步骤见[安装说明](./docs/installat
|
|
|
132
141
|
才能发布对应版本的 `dsh-scnet` npm 包。
|
|
133
142
|
|
|
134
143
|
下游同步合并到 `main` 后,先运行本地校验,再手动触发仓库的 `Publish DSH-SCNet` workflow,
|
|
135
|
-
输入 `package.json` 中的精确版本(当前为 `0.6.
|
|
144
|
+
输入 `package.json` 中的精确版本(当前为 `0.6.3`)。workflow 会校验 package 与 Skill
|
|
136
145
|
版本一致,执行验证、生成 tarball、以稳定版 `latest` 标签发布,并创建对应 Git tag。
|
|
137
146
|
不要只修改 `package.json`;`skills/scnet-hpc/VERSION`、package 版本和两份 README 的版本
|
|
138
147
|
标记必须保持一致。
|
package/docs/testing.md
CHANGED
|
@@ -39,11 +39,14 @@ Confirm that the composed configuration includes:
|
|
|
39
39
|
| `scnet_generate_job` | Writes a profile-aware Slurm script in the current directory |
|
|
40
40
|
| `scnet_status` | Returns redacted local SCNet configuration |
|
|
41
41
|
| `scnet_openapi_regions` | Returns authorized regions without tokens |
|
|
42
|
+
| `scnet_account_summary` | Returns account status and balance |
|
|
43
|
+
| `scnet_resource_summary` | Returns queues and resource limits for a region |
|
|
42
44
|
| `scnet_job_queues` | Returns live regional Slurm queues |
|
|
43
45
|
| `scnet_job_list` | Lists active or historical jobs |
|
|
44
46
|
| `scnet_limits` | Returns user and scheduler resource limits |
|
|
45
47
|
| `scnet_submit_job` | Submits a job through OpenAPI or SSH; `dry_run=true` returns the pending request only |
|
|
46
48
|
| `scnet_job_show` | Returns one job's state, resources, and log paths |
|
|
49
|
+
| `scnet_job_wait` | Waits for a job to reach a terminal state with a bounded timeout |
|
|
47
50
|
| `scnet_job_logs` | Reads job logs by explicit path, or by `job_id` + `work_dir` |
|
|
48
51
|
| `scnet_job_cancel` | Cancels one job; `dry_run=true` returns the pending request only |
|
|
49
52
|
| `scnet_file_list` | Lists shared-storage files without modification |
|
package/docs/testing.zh-CN.md
CHANGED
|
@@ -60,12 +60,15 @@ DEEPSEEK_API_KEY: sk-你的key
|
|
|
60
60
|
| 7 | 用 scnet_run_compute_probe 跑计算节点探针 | 返回 PROBE_ 结果 | 高(提交作业并等待) |
|
|
61
61
|
| 8 | 用 scnet_status 查看当前配置 | 返回脱敏配置 | 只读 |
|
|
62
62
|
| 9 | 用 scnet_openapi_regions 查看授权区域 | 返回区域且不含 token | 只读 |
|
|
63
|
+
| 9a | 用 scnet_account_summary 查看账户摘要 | 返回账户状态和余额 | 只读 |
|
|
64
|
+
| 9b | 用 scnet_resource_summary 查看资源摘要 | 返回队列和资源限制 | 只读 |
|
|
63
65
|
| 10 | 用 scnet_job_queues 查看昆山队列 | 返回实时队列和空闲资源 | 只读 |
|
|
64
66
|
| 11 | 用 scnet_job_list 列出当前或历史作业 | 返回紧凑作业记录 | 只读 |
|
|
65
67
|
| 12 | 用 scnet_limits 查询资源限制 | 返回用户和调度器限制 | 只读 |
|
|
66
68
|
| 13 | 用 scnet_submit_job(dry_run=true)预览一次提交 | 返回将要发送的请求,不真正提交 | 只读(dry_run) |
|
|
67
69
|
| 14 | 用 scnet_submit_job 提交一个 5 秒小作业 | 返回 job_id、作业目录与日志路径 | 高(会消耗配额) |
|
|
68
70
|
| 15 | 用 scnet_job_show 查该 job_id | 返回状态与 stdout/stderr 路径 | 只读 |
|
|
71
|
+
| 15a | 用 scnet_job_wait 等待作业终态 | 最多等待限定时间并返回状态 | 只读 |
|
|
69
72
|
| 16 | 用 scnet_job_logs 读该作业日志 | 返回日志内容(同时验证 path 与 job_id+work_dir 两种模式结果一致) | 只读 |
|
|
70
73
|
| 17 | 用 scnet_job_cancel 取消一个作业 | 建议先用 dry_run=true 预览 | 高(变更) |
|
|
71
74
|
| 18 | 用 scnet_file_list 查看默认目录 | 返回共享存储文件 | 只读 |
|
package/index.mjs
CHANGED
|
@@ -182,6 +182,16 @@ function errorTextFrom(res) {
|
|
|
182
182
|
return matched ? matched[1].trim() : raw
|
|
183
183
|
}
|
|
184
184
|
|
|
185
|
+
function errorCodeFrom(res) {
|
|
186
|
+
const raw = String(res.stdout || '').trim() || String(res.stderr || '').trim()
|
|
187
|
+
try {
|
|
188
|
+
const parsed = JSON.parse(raw)
|
|
189
|
+
return parsed && typeof parsed.error_code === 'string' ? parsed.error_code : ''
|
|
190
|
+
} catch {
|
|
191
|
+
return ''
|
|
192
|
+
}
|
|
193
|
+
}
|
|
194
|
+
|
|
185
195
|
// 一次只给一条可执行的提示(不展示整套参数清单)
|
|
186
196
|
function translateScnetError(text) {
|
|
187
197
|
const value = String(text || '')
|
|
@@ -203,14 +213,23 @@ function translateScnetError(text) {
|
|
|
203
213
|
return value
|
|
204
214
|
}
|
|
205
215
|
|
|
216
|
+
function translateScnetFailure(res) {
|
|
217
|
+
const code = errorCodeFrom(res)
|
|
218
|
+
if (code === 'AUTHENTICATION_FAILED') return 'SCNet 认证失败:请检查 OpenAPI 配置或重新运行 setup。'
|
|
219
|
+
if (code === 'QUEUE_QUERY_FAILED') return '队列查询失败,已停止提交;请先检查区域、调度器和凭据。'
|
|
220
|
+
if (code === 'NO_QUEUE_AVAILABLE') return '当前资源条件下没有可用队列,请调整资源或选择其他区域。'
|
|
221
|
+
if (code === 'NETWORK_TIMEOUT') return 'SCNet 请求超时:请稍后重试;提交/取消类操作不要盲目重试。'
|
|
222
|
+
return translateScnetError(errorTextFrom(res))
|
|
223
|
+
}
|
|
224
|
+
|
|
206
225
|
function scnetJobResult(res, action) {
|
|
207
226
|
if (res.ok) return res.stdout.trim() || '{}'
|
|
208
|
-
return `${action}失败(exit ${res.code}):\n${
|
|
227
|
+
return `${action}失败(exit ${res.code}):\n${translateScnetFailure(res)}`
|
|
209
228
|
}
|
|
210
229
|
|
|
211
230
|
async function runScnetEnvelope(argv) {
|
|
212
231
|
const res = await runScnet(argv)
|
|
213
|
-
if (!res.ok) return { ok: false, error:
|
|
232
|
+
if (!res.ok) return { ok: false, error: translateScnetFailure(res) }
|
|
214
233
|
try {
|
|
215
234
|
const parsed = JSON.parse(res.stdout)
|
|
216
235
|
return { ok: true, envelope: parsed, data: parsed?.data }
|
|
@@ -236,22 +255,18 @@ function queueCandidates(queues, wantsAccelerator) {
|
|
|
236
255
|
.sort((a, b) => Number(b?.free_nodes ?? 0) - Number(a?.free_nodes ?? 0))
|
|
237
256
|
}
|
|
238
257
|
|
|
239
|
-
//
|
|
258
|
+
// 提交前队列预检:查询失败或空结果都必须阻止提交,不能把失败当成可提交。
|
|
240
259
|
async function precheckQueue(target, args) {
|
|
241
260
|
const check = await runScnetEnvelope([...target.argv, 'queues'])
|
|
242
|
-
if (!check.ok) return {
|
|
261
|
+
if (!check.ok) return { error: `队列查询失败,已停止提交:${check.error}` }
|
|
243
262
|
const queues = Array.isArray(check.data) ? check.data : []
|
|
244
|
-
if (queues.length === 0) return {
|
|
263
|
+
if (queues.length === 0) return { error: '队列查询成功但没有可用队列,已停止提交。' }
|
|
245
264
|
|
|
246
265
|
const queue = String(args.queue ?? '').trim()
|
|
247
266
|
const dcus = Number(String(args.dcus ?? '').trim() || 0)
|
|
248
267
|
|
|
249
268
|
if (!queue) {
|
|
250
269
|
const candidates = queueCandidates(queues, dcus > 0)
|
|
251
|
-
if (candidates.length === 1) {
|
|
252
|
-
const only = candidates[0]
|
|
253
|
-
return { queue: String(only.partition), note: `已自动选择队列 ${only.partition}` }
|
|
254
|
-
}
|
|
255
270
|
if (candidates.length > 1) {
|
|
256
271
|
const best = candidates[0]
|
|
257
272
|
return {
|
|
@@ -260,7 +275,10 @@ async function precheckQueue(target, args) {
|
|
|
260
275
|
`按你要的资源建议 ${best.partition}(空闲 ${best.free_nodes} 节点)。`,
|
|
261
276
|
}
|
|
262
277
|
}
|
|
263
|
-
|
|
278
|
+
if (candidates.length === 1) {
|
|
279
|
+
return { error: `请确认 queue:当前唯一可用队列是 ${candidates[0].partition}(不会自动选择计费队列)。` }
|
|
280
|
+
}
|
|
281
|
+
return { error: '没有找到匹配当前资源需求的可用队列。' }
|
|
264
282
|
}
|
|
265
283
|
|
|
266
284
|
const hit = queues.find((item) => String(item?.partition) === queue)
|
|
@@ -348,6 +366,51 @@ export function apply(ctx) {
|
|
|
348
366
|
}),
|
|
349
367
|
)
|
|
350
368
|
|
|
369
|
+
ctx.tools.register(
|
|
370
|
+
defineTool({
|
|
371
|
+
name: 'scnet_account_summary',
|
|
372
|
+
description:
|
|
373
|
+
'只读查询当前 SCNet 账户摘要,包括用户名、账户状态、余额、默认区域和区域信息;不需要额外参数。',
|
|
374
|
+
parameters: {},
|
|
375
|
+
output: {
|
|
376
|
+
schema: { type: 'string' },
|
|
377
|
+
render: (_args, value) => textBlock(value),
|
|
378
|
+
},
|
|
379
|
+
async execute() {
|
|
380
|
+
return scnetResult(
|
|
381
|
+
await runScnet(['--backend', 'openapi', 'account']),
|
|
382
|
+
'查询账户摘要',
|
|
383
|
+
)
|
|
384
|
+
},
|
|
385
|
+
}),
|
|
386
|
+
)
|
|
387
|
+
|
|
388
|
+
ctx.tools.register(
|
|
389
|
+
defineTool({
|
|
390
|
+
name: 'scnet_resource_summary',
|
|
391
|
+
description:
|
|
392
|
+
'只读汇总目标区域的队列、空闲资源和用户资源限制;region 可省略,后端会使用默认区域。',
|
|
393
|
+
parameters: {
|
|
394
|
+
region: { type: 'string', description: '区域名称或 ID;省略时使用默认区域' },
|
|
395
|
+
scheduler_id: { type: 'string', description: '调度器 ID;多调度器时填写' },
|
|
396
|
+
},
|
|
397
|
+
output: {
|
|
398
|
+
schema: { type: 'string' },
|
|
399
|
+
render: (_args, value) => textBlock(value),
|
|
400
|
+
},
|
|
401
|
+
async execute(args) {
|
|
402
|
+
const argv = [
|
|
403
|
+
'--backend',
|
|
404
|
+
'openapi',
|
|
405
|
+
...regionArgs(args.region),
|
|
406
|
+
...(args.scheduler_id ? ['--scheduler-id', String(args.scheduler_id)] : []),
|
|
407
|
+
'resource-summary',
|
|
408
|
+
]
|
|
409
|
+
return scnetResult(await runScnet(argv), '查询资源摘要')
|
|
410
|
+
},
|
|
411
|
+
}),
|
|
412
|
+
)
|
|
413
|
+
|
|
351
414
|
ctx.tools.register(
|
|
352
415
|
defineTool({
|
|
353
416
|
name: 'scnet_job_queues',
|
|
@@ -602,6 +665,43 @@ export function apply(ctx) {
|
|
|
602
665
|
}),
|
|
603
666
|
)
|
|
604
667
|
|
|
668
|
+
ctx.tools.register(
|
|
669
|
+
defineTool({
|
|
670
|
+
name: 'scnet_job_wait',
|
|
671
|
+
description:
|
|
672
|
+
'只读等待一个作业进入终态。默认最多等待 300 秒、每 10 秒查询一次;不会提交、取消或自动重试作业。',
|
|
673
|
+
parameters: {
|
|
674
|
+
job_id: { type: 'string', required: true, description: '作业号' },
|
|
675
|
+
backend: { type: 'string', description: 'openapi(默认)或 ssh' },
|
|
676
|
+
region: { type: 'string', description: 'OpenAPI 区域;省略时使用默认区域' },
|
|
677
|
+
scheduler_id: { type: 'string', description: 'OpenAPI 调度器 ID;多调度器时填写' },
|
|
678
|
+
cluster: { type: 'string', description: 'SSH 集群短名;多个 profile 时必须指定' },
|
|
679
|
+
wait_timeout: { type: 'number', description: '最长等待秒数,默认 300,最大 600' },
|
|
680
|
+
interval: { type: 'number', description: '查询间隔秒数,默认 10,范围 1-60' },
|
|
681
|
+
},
|
|
682
|
+
timeoutMs: 660000,
|
|
683
|
+
output: {
|
|
684
|
+
schema: { type: 'string' },
|
|
685
|
+
render: (_args, value) => textBlock(value),
|
|
686
|
+
},
|
|
687
|
+
async execute(args) {
|
|
688
|
+
const jobId = String(args.job_id ?? '').trim()
|
|
689
|
+
if (!jobId) return MISSING_HINTS.job_id
|
|
690
|
+
const target = await resolveTarget(args)
|
|
691
|
+
if (target.error) return target.error
|
|
692
|
+
const argv = [...target.argv, 'wait', jobId]
|
|
693
|
+
for (const [flag, key] of [['--wait-timeout', 'wait_timeout'], ['--interval', 'interval']]) {
|
|
694
|
+
if (args[key] !== undefined) {
|
|
695
|
+
const checked = positiveInt(args[key], key)
|
|
696
|
+
if (checked?.error) return checked.error
|
|
697
|
+
argv.push(flag, checked.value)
|
|
698
|
+
}
|
|
699
|
+
}
|
|
700
|
+
return scnetJobResult(await runScnet(argv, 660000), '等待作业')
|
|
701
|
+
},
|
|
702
|
+
}),
|
|
703
|
+
)
|
|
704
|
+
|
|
605
705
|
ctx.tools.register(
|
|
606
706
|
defineTool({
|
|
607
707
|
name: 'scnet_job_logs',
|
|
@@ -706,7 +806,7 @@ export function apply(ctx) {
|
|
|
706
806
|
'在本地与 SCNet 区域共享存储之间传输一个文件。direction=upload 或 download;upload 的 remote_dir 是目录而不是文件名。默认不覆盖,执行前应确认区域和路径,dry_run=true 只预览。',
|
|
707
807
|
parameters: {
|
|
708
808
|
direction: { type: 'string', required: true, description: 'upload 或 download' },
|
|
709
|
-
local_path: { type: 'string',
|
|
809
|
+
local_path: { type: 'string', description: '本地文件路径;download 省略时使用远端文件名' },
|
|
710
810
|
remote_dir: { type: 'string', description: 'upload 时必填:区域共享存储中的绝对目录;文件名由 local_path 保留' },
|
|
711
811
|
remote_path: { type: 'string', description: 'download:区域共享存储中的绝对文件路径' },
|
|
712
812
|
backend: { type: 'string', description: 'openapi(默认)或 ssh' },
|
|
@@ -727,7 +827,7 @@ export function apply(ctx) {
|
|
|
727
827
|
const localPath = String(args.local_path ?? '').trim()
|
|
728
828
|
const remoteDir = String(args.remote_dir ?? '').trim()
|
|
729
829
|
const remotePath = String(args.remote_path ?? '').trim()
|
|
730
|
-
if (!localPath) return '缺少必填参数 local_path。'
|
|
830
|
+
if (direction === 'upload' && !localPath) return '缺少必填参数 local_path。'
|
|
731
831
|
if (direction === 'upload' && !remoteDir) return '缺少必填参数 remote_dir(目录,不是文件名)。'
|
|
732
832
|
if (direction === 'download' && !remotePath) return '缺少必填参数 remote_path。'
|
|
733
833
|
const target = await resolveTarget(args)
|
|
@@ -737,7 +837,8 @@ export function apply(ctx) {
|
|
|
737
837
|
if (direction === 'upload') {
|
|
738
838
|
argv.push('upload', localPath, remoteDir)
|
|
739
839
|
} else {
|
|
740
|
-
argv.push('download', remotePath
|
|
840
|
+
argv.push('download', remotePath)
|
|
841
|
+
if (localPath) argv.push(localPath)
|
|
741
842
|
}
|
|
742
843
|
if (args.overwrite === true) argv.push('--cover')
|
|
743
844
|
return scnetResult(
|
|
@@ -1067,6 +1168,7 @@ export const __testables = {
|
|
|
1067
1168
|
MISSING_HINTS,
|
|
1068
1169
|
SUBMIT_REQUIRED_ORDER,
|
|
1069
1170
|
errorTextFrom,
|
|
1171
|
+
errorCodeFrom,
|
|
1070
1172
|
firstMissingRequired,
|
|
1071
1173
|
parseWalltimeSeconds,
|
|
1072
1174
|
pickBackend,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-scnet",
|
|
3
|
-
"version": "0.6.
|
|
3
|
+
"version": "0.6.4",
|
|
4
4
|
"description": "Community DeepSeek Harness bundle for SCNet SSH and OpenAPI jobs, files, Notebook resources, Slurm diagnostics, and Hygon DCU/DTK guidance.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./index.mjs",
|
|
@@ -5,8 +5,9 @@ description: Configure or operate SCNet HPC through SSH, OpenAPI, and pluggable
|
|
|
5
5
|
|
|
6
6
|
# SCNet HPC
|
|
7
7
|
|
|
8
|
-
Use the
|
|
9
|
-
|
|
8
|
+
Use OpenAPI AK/SK as the preferred backend for structured cluster operations. Keep SSH as the
|
|
9
|
+
secondary backend for environment setup, compilation, interactive diagnosis, and operations not
|
|
10
|
+
available through the API. Never silently switch backends.
|
|
10
11
|
|
|
11
12
|
## First use and selection
|
|
12
13
|
|
|
@@ -14,7 +15,7 @@ OpenAPI is the structured control plane. Never silently switch backends.
|
|
|
14
15
|
- Advanced OpenAPI discovery: `python3 scripts/scnet.py setup`.
|
|
15
16
|
- Read-only preflight: `python3 scripts/scnet.py doctor`.
|
|
16
17
|
- Backend precedence: `--backend`, `SCNET_HPC_BACKEND`, saved setup choice, profile
|
|
17
|
-
`DEFAULT_BACKEND`, then `
|
|
18
|
+
`DEFAULT_BACKEND`, then `openapi`.
|
|
18
19
|
- Read `clusters/<cluster>.conf` before reporting static facts. Treat
|
|
19
20
|
`clusters/.cache/<cluster>.auto.conf` as optional, time-sensitive SSH probe data.
|
|
20
21
|
- With multiple profiles, require an explicit cluster before access changes, installation,
|
package/skills/scnet-hpc/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.6.
|
|
1
|
+
0.6.4
|
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
# ---- 通用与 SSH 连接 ----
|
|
13
13
|
CLUSTER_ID="" # 短名,同时用作 ssh 别名,如 "zzeshell"
|
|
14
14
|
CLUSTER_DESC="" # 人类可读描述,如 "海光 DCU(郑州)"
|
|
15
|
-
DEFAULT_BACKEND="
|
|
15
|
+
DEFAULT_BACKEND="openapi" # openapi(推荐)/ ssh / 已注册的外部 backend
|
|
16
16
|
SSH_HOST="" # 如 "xxx.scnet.cn"
|
|
17
17
|
SSH_PORT="22"
|
|
18
18
|
HOME_BASE="/public/home" # 家目录父路径
|
|
@@ -142,7 +142,7 @@ Triton 是多种推理框架的运行前提;不可用时,对应的 Triton
|
|
|
142
142
|
|
|
143
143
|
| 字段 | 谁用 | 留空的后果 |
|
|
144
144
|
|---|---|---|
|
|
145
|
-
| `DEFAULT_BACKEND` | `scnet.py` | 默认使用 `
|
|
145
|
+
| `DEFAULT_BACKEND` | `scnet.py` | 默认使用 `openapi` |
|
|
146
146
|
| `OPENAPI_REGION_ID` | OpenAPI backend | 多区域账户需要每次显式传 `--region` |
|
|
147
147
|
| `OPENAPI_SCHEDULER_ID` | OpenAPI backend | 多调度器区域需要每次显式传 `--scheduler-id` |
|
|
148
148
|
| `MIN_GRES` | 文档提示 | 不提示 QOS 强制申请加速器 |
|
|
@@ -12,7 +12,7 @@ Backend precedence:
|
|
|
12
12
|
2. `SCNET_HPC_BACKEND`;
|
|
13
13
|
3. saved setup-panel `default_backend`;
|
|
14
14
|
4. selected profile `DEFAULT_BACKEND`;
|
|
15
|
-
5. `
|
|
15
|
+
5. `openapi`.
|
|
16
16
|
|
|
17
17
|
Do not silently fall back between backends. A failed OpenAPI request must not become an SSH
|
|
18
18
|
submission, and a failed SSH command must not be replayed through OpenAPI. Replaying mutations
|
|
@@ -39,7 +39,7 @@ Built-in capability summary:
|
|
|
39
39
|
| Modules, builds, interactive diagnosis | yes | no |
|
|
40
40
|
| Notebook read-only discovery | no | yes |
|
|
41
41
|
|
|
42
|
-
Use OpenAPI
|
|
42
|
+
Use OpenAPI as the preferred structured control-plane backend. Use SSH for environment setup, compilation,
|
|
43
43
|
interactive inspection, full scheduler tooling, and operations not represented by a structured
|
|
44
44
|
backend.
|
|
45
45
|
|
|
@@ -168,7 +168,7 @@ until the official interface is available and verified.
|
|
|
168
168
|
Profiles may contain non-secret backend hints:
|
|
169
169
|
|
|
170
170
|
```bash
|
|
171
|
-
DEFAULT_BACKEND="
|
|
171
|
+
DEFAULT_BACKEND="openapi"
|
|
172
172
|
OPENAPI_REGION_ID=""
|
|
173
173
|
OPENAPI_SCHEDULER_ID=""
|
|
174
174
|
OPENAPI_USERNAME=""
|
|
@@ -128,6 +128,20 @@ python3 scripts/scnet.py --backend openapi --region <id> \
|
|
|
128
128
|
`job <job-id>` checks the realtime endpoint first and automatically falls back to the filtered
|
|
129
129
|
history-list endpoint. The caller does not need to provide `acctTime`.
|
|
130
130
|
|
|
131
|
+
Account and resource summaries are read-only:
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
python3 scripts/scnet.py --backend openapi --region <id> account
|
|
135
|
+
python3 scripts/scnet.py --backend openapi --region <id> resource-summary
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
Wait for a job with a bounded timeout:
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
142
|
+
wait <job-id> --wait-timeout 300 --interval 10
|
|
143
|
+
```
|
|
144
|
+
|
|
131
145
|
Region tokens are cached under `${XDG_CACHE_HOME:-~/.cache}/scnet-hpc` when that location is
|
|
132
146
|
writable. Cache locking or writes are best-effort; an unavailable cache never blocks an OpenAPI
|
|
133
147
|
operation.
|
|
@@ -32,7 +32,8 @@ List available access backends:
|
|
|
32
32
|
python3 scripts/scnet.py backends
|
|
33
33
|
```
|
|
34
34
|
|
|
35
|
-
|
|
35
|
+
OpenAPI with AK/SK is the preferred default for structured queue, limit, job, log, and file
|
|
36
|
+
operations. Use SSH for environment setup and interactive shell work:
|
|
36
37
|
|
|
37
38
|
```bash
|
|
38
39
|
python3 scripts/scnet.py --backend openapi clusters
|
|
@@ -15,6 +15,7 @@ import shlex
|
|
|
15
15
|
import shutil
|
|
16
16
|
import subprocess
|
|
17
17
|
import sys
|
|
18
|
+
import time
|
|
18
19
|
from pathlib import Path
|
|
19
20
|
from typing import Any, Mapping
|
|
20
21
|
|
|
@@ -59,7 +60,7 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
59
60
|
"--backend",
|
|
60
61
|
help=(
|
|
61
62
|
"backend name; defaults to SCNET_HPC_BACKEND, saved setup choice, "
|
|
62
|
-
"profile DEFAULT_BACKEND, then
|
|
63
|
+
"profile DEFAULT_BACKEND, then openapi"
|
|
63
64
|
),
|
|
64
65
|
)
|
|
65
66
|
parser.add_argument("--cluster", help="local cluster profile name")
|
|
@@ -120,9 +121,19 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
120
121
|
limits = subparsers.add_parser("limits", help="show scheduler or user resource limits")
|
|
121
122
|
limits.add_argument("--partition")
|
|
122
123
|
|
|
124
|
+
account = subparsers.add_parser("account", help="show account summary")
|
|
125
|
+
resource_summary = subparsers.add_parser(
|
|
126
|
+
"resource-summary", help="show region resources and limits"
|
|
127
|
+
)
|
|
128
|
+
|
|
123
129
|
job = subparsers.add_parser("job", help="show one job")
|
|
124
130
|
job.add_argument("job_id")
|
|
125
131
|
|
|
132
|
+
wait = subparsers.add_parser("wait", help="wait for one job to reach a terminal state")
|
|
133
|
+
wait.add_argument("job_id")
|
|
134
|
+
wait.add_argument("--wait-timeout", type=int, default=300)
|
|
135
|
+
wait.add_argument("--interval", type=int, default=10)
|
|
136
|
+
|
|
126
137
|
jobs = subparsers.add_parser("jobs", help="list active or historical jobs")
|
|
127
138
|
jobs.add_argument("--scope", choices=("active", "history"), default="active")
|
|
128
139
|
jobs.add_argument("--limit", type=int, default=20)
|
|
@@ -192,7 +203,7 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
192
203
|
|
|
193
204
|
download = subparsers.add_parser("download", help="download one file or folder")
|
|
194
205
|
download.add_argument("remote_path")
|
|
195
|
-
download.add_argument("local_path")
|
|
206
|
+
download.add_argument("local_path", nargs="?")
|
|
196
207
|
download.add_argument("--cover", action="store_true")
|
|
197
208
|
|
|
198
209
|
execute = subparsers.add_parser("exec", help="run an SSH command")
|
|
@@ -209,7 +220,7 @@ def choose_backend(
|
|
|
209
220
|
or os.environ.get("SCNET_HPC_BACKEND")
|
|
210
221
|
or str(user_config.get("default_backend") or "")
|
|
211
222
|
or profile.get("DEFAULT_BACKEND")
|
|
212
|
-
or "
|
|
223
|
+
or "openapi"
|
|
213
224
|
)
|
|
214
225
|
|
|
215
226
|
|
|
@@ -225,6 +236,68 @@ def compact_output(operation: str, data: Any) -> str:
|
|
|
225
236
|
return json.dumps(data, ensure_ascii=False, indent=2, sort_keys=True)
|
|
226
237
|
|
|
227
238
|
|
|
239
|
+
def _job_terminal(state: Any) -> bool:
|
|
240
|
+
value = str(state or "").upper()
|
|
241
|
+
return value in {
|
|
242
|
+
"COMPLETED",
|
|
243
|
+
"FAILED",
|
|
244
|
+
"CANCELLED",
|
|
245
|
+
"TIMEOUT",
|
|
246
|
+
"NODE_ERROR",
|
|
247
|
+
"ABNORMAL",
|
|
248
|
+
"EXITING",
|
|
249
|
+
"EXITED",
|
|
250
|
+
"DONE",
|
|
251
|
+
"OUT_OF_MEMORY",
|
|
252
|
+
"NODE_FAIL",
|
|
253
|
+
}
|
|
254
|
+
|
|
255
|
+
|
|
256
|
+
def wait_for_job(backend, options: Mapping[str, Any]) -> dict[str, Any]:
|
|
257
|
+
timeout = int(options.get("wait_timeout") or 300)
|
|
258
|
+
interval = int(options.get("interval") or 10)
|
|
259
|
+
if timeout < 1 or timeout > 600:
|
|
260
|
+
raise BackendError("wait_timeout must be between 1 and 600 seconds")
|
|
261
|
+
if interval < 1 or interval > 60:
|
|
262
|
+
raise BackendError("interval must be between 1 and 60 seconds")
|
|
263
|
+
started = time.monotonic()
|
|
264
|
+
while True:
|
|
265
|
+
latest = backend.execute("job", options)
|
|
266
|
+
if isinstance(latest, dict) and _job_terminal(latest.get("state")):
|
|
267
|
+
latest["waited_seconds"] = round(time.monotonic() - started, 1)
|
|
268
|
+
latest["timed_out"] = False
|
|
269
|
+
return latest
|
|
270
|
+
elapsed = time.monotonic() - started
|
|
271
|
+
if elapsed >= timeout:
|
|
272
|
+
return {
|
|
273
|
+
"job": latest,
|
|
274
|
+
"waited_seconds": round(elapsed, 1),
|
|
275
|
+
"timed_out": True,
|
|
276
|
+
}
|
|
277
|
+
time.sleep(min(interval, timeout - elapsed))
|
|
278
|
+
|
|
279
|
+
|
|
280
|
+
def error_code_for(text: str) -> str:
|
|
281
|
+
value = str(text).lower()
|
|
282
|
+
if "credentials are not configured" in value or "token" in value and "expired" in value:
|
|
283
|
+
return "AUTHENTICATION_FAILED"
|
|
284
|
+
if "region" in value and ("not available" in value or "missing" in value):
|
|
285
|
+
return "REGION_NOT_FOUND"
|
|
286
|
+
if "scheduler" in value and ("not available" in value or "multiple" in value):
|
|
287
|
+
return "SCHEDULER_NOT_FOUND"
|
|
288
|
+
if "queue" in value and ("not found" in value or "no available" in value):
|
|
289
|
+
return "QUEUE_QUERY_FAILED"
|
|
290
|
+
if "job" in value and ("not found" in value or "no job record" in value):
|
|
291
|
+
return "JOB_NOT_FOUND"
|
|
292
|
+
if "already exists" in value:
|
|
293
|
+
return "FILE_TARGET_EXISTS"
|
|
294
|
+
if "does not exist" in value or "not found" in value and "file" in value:
|
|
295
|
+
return "FILE_NOT_FOUND"
|
|
296
|
+
if "timed out" in value or "timeout" in value:
|
|
297
|
+
return "NETWORK_TIMEOUT"
|
|
298
|
+
return "BACKEND_ERROR"
|
|
299
|
+
|
|
300
|
+
|
|
228
301
|
def _ask(prompt: str, default: str = "") -> str:
|
|
229
302
|
suffix = f" [{default}]" if default else ""
|
|
230
303
|
answer = input(f"{prompt}{suffix}: ").strip()
|
|
@@ -870,14 +943,14 @@ def setup_panel(
|
|
|
870
943
|
print("自动探测区域、调度器和 SSH 元数据;只要求输入必要凭据。\n")
|
|
871
944
|
_print_terminal_help()
|
|
872
945
|
|
|
873
|
-
current_default = str(current.get("default_backend") or "
|
|
946
|
+
current_default = str(current.get("default_backend") or "openapi")
|
|
874
947
|
if mode == "all":
|
|
875
948
|
backend = _choose(
|
|
876
949
|
"选择默认 backend:",
|
|
877
|
-
["
|
|
878
|
-
1 if current_default == "
|
|
950
|
+
["OpenAPI(推荐,结构化控制面)", "SSH(环境配置和深度诊断)"],
|
|
951
|
+
1 if current_default == "openapi" else 2,
|
|
879
952
|
)
|
|
880
|
-
default_backend = "
|
|
953
|
+
default_backend = "openapi" if backend.startswith("OpenAPI") else "ssh"
|
|
881
954
|
configure_ssh = _ask_yes_no(
|
|
882
955
|
"是否添加或更新一个 SSH 区域?", default=default_backend == "ssh"
|
|
883
956
|
)
|
|
@@ -1506,6 +1579,9 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
1506
1579
|
openapi_config = user_config.get("openapi", {})
|
|
1507
1580
|
if isinstance(openapi_config, dict):
|
|
1508
1581
|
apply_openapi_defaults(options, openapi_config)
|
|
1582
|
+
if args.operation == "download" and not options.get("local_path"):
|
|
1583
|
+
remote_path = str(options.get("remote_path") or "").rstrip("/")
|
|
1584
|
+
options["local_path"] = remote_path.rsplit("/", 1)[-1] or "download"
|
|
1509
1585
|
if args.operation == "notebook":
|
|
1510
1586
|
if backend_name != "openapi":
|
|
1511
1587
|
raise BackendError(
|
|
@@ -1532,6 +1608,8 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
1532
1608
|
"name": backend.name,
|
|
1533
1609
|
"capabilities": sorted(backend.capabilities),
|
|
1534
1610
|
}
|
|
1611
|
+
elif args.operation == "wait":
|
|
1612
|
+
data = wait_for_job(backend, options)
|
|
1535
1613
|
elif args.operation == "exec":
|
|
1536
1614
|
options["command"] = " ".join(args.command).strip()
|
|
1537
1615
|
data = backend.execute(args.operation, options)
|
|
@@ -1564,6 +1642,7 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
1564
1642
|
{
|
|
1565
1643
|
"ok": False,
|
|
1566
1644
|
"operation": args.operation,
|
|
1645
|
+
"error_code": error_code_for(str(exc)),
|
|
1567
1646
|
"error": str(exc),
|
|
1568
1647
|
},
|
|
1569
1648
|
ensure_ascii=False,
|
|
@@ -92,6 +92,8 @@ class OpenAPIBackend(Backend):
|
|
|
92
92
|
"clusters",
|
|
93
93
|
"queues",
|
|
94
94
|
"limits",
|
|
95
|
+
"account",
|
|
96
|
+
"resource-summary",
|
|
95
97
|
"job",
|
|
96
98
|
"jobs",
|
|
97
99
|
"logs",
|
|
@@ -351,6 +353,37 @@ class OpenAPIBackend(Backend):
|
|
|
351
353
|
result = {target: data.get(source) for source, target in key_map.items()}
|
|
352
354
|
return {"data": result, "raw": data} if options.get("raw") else result
|
|
353
355
|
|
|
356
|
+
def op_account(self, options: Mapping[str, Any]) -> Any:
|
|
357
|
+
_, token, region = self._center(options)
|
|
358
|
+
url = self._env(
|
|
359
|
+
"SCNET_OPENAPI_USER_URL",
|
|
360
|
+
"https://www.scnet.cn/ac/openapi/v2/user",
|
|
361
|
+
)
|
|
362
|
+
data = self._json_request("GET", url, token=token)
|
|
363
|
+
if not isinstance(data, dict):
|
|
364
|
+
raise BackendError("user endpoint returned an unexpected data shape")
|
|
365
|
+
result = {
|
|
366
|
+
"user": data.get("userName"),
|
|
367
|
+
"account_name": data.get("accountName"),
|
|
368
|
+
"account_status": data.get("accountStatus"),
|
|
369
|
+
"balance": data.get("accountBalance"),
|
|
370
|
+
"computer_center": data.get("computerCenter"),
|
|
371
|
+
"region_id": region.get("clusterId"),
|
|
372
|
+
"region_name": region.get("clusterName"),
|
|
373
|
+
}
|
|
374
|
+
return {"data": result, "raw": data} if options.get("raw") else result
|
|
375
|
+
|
|
376
|
+
def op_resource_summary(self, options: Mapping[str, Any]) -> Any:
|
|
377
|
+
_, _, region = self._center(options)
|
|
378
|
+
queues = self.op_queues(options)
|
|
379
|
+
limits = self.op_limits(options)
|
|
380
|
+
return {
|
|
381
|
+
"region_id": region.get("clusterId"),
|
|
382
|
+
"region_name": region.get("clusterName"),
|
|
383
|
+
"queues": queues,
|
|
384
|
+
"limits": limits,
|
|
385
|
+
}
|
|
386
|
+
|
|
354
387
|
def op_job(self, options: Mapping[str, Any]) -> Any:
|
|
355
388
|
job_id = str(require_option(options, "job_id"))
|
|
356
389
|
hpc_url, token, scheduler_id, _, _ = self._hpc_context(options)
|
|
@@ -58,7 +58,7 @@ def list_profiles(repo_root: Path) -> list[dict[str, str]]:
|
|
|
58
58
|
"name": path.stem,
|
|
59
59
|
"cluster_id": profile.get("CLUSTER_ID", path.stem),
|
|
60
60
|
"description": profile.get("CLUSTER_DESC", ""),
|
|
61
|
-
"default_backend": profile.get("DEFAULT_BACKEND", "
|
|
61
|
+
"default_backend": profile.get("DEFAULT_BACKEND", "openapi"),
|
|
62
62
|
}
|
|
63
63
|
)
|
|
64
64
|
return result
|
|
@@ -345,7 +345,7 @@ case "$(uname -s)" in
|
|
|
345
345
|
&& grep -Eq '^ID=(ubuntu|debian)$' /etc/os-release; then
|
|
346
346
|
printf '%s\n' \
|
|
347
347
|
"Ubuntu/Debian:在 Terminal 中使用 ↑/↓ 和 Enter,也可输入编号;Ctrl+C 可取消。" \
|
|
348
|
-
"
|
|
348
|
+
"本机安全保存 AK/SK 可安装 libsecret-tools(sudo apt install libsecret-tools);使用环境变量时无需安装。" >&2
|
|
349
349
|
else
|
|
350
350
|
printf '%s\n' "Linux:使用 ↑/↓ 和 Enter,也可输入编号;Ctrl+C 可取消。" >&2
|
|
351
351
|
fi
|
|
@@ -382,12 +382,12 @@ fi
|
|
|
382
382
|
|
|
383
383
|
if [ -z "$BACKEND" ]; then
|
|
384
384
|
BACKEND=$(choose "选择要管理的 backend:" \
|
|
385
|
+
"openapi(推荐:平台级 AK/SK 和多区域)" \
|
|
385
386
|
"ssh(区域级用户名和私钥)" \
|
|
386
|
-
"openapi(平台级 AK/SK 和多区域)" \
|
|
387
387
|
"both(同时管理)")
|
|
388
388
|
case "$BACKEND" in
|
|
389
|
-
ssh*) BACKEND=ssh ;;
|
|
390
389
|
openapi*) BACKEND=openapi ;;
|
|
390
|
+
ssh*) BACKEND=ssh ;;
|
|
391
391
|
both*) BACKEND=both ;;
|
|
392
392
|
esac
|
|
393
393
|
fi
|
|
@@ -429,7 +429,7 @@ fi
|
|
|
429
429
|
|
|
430
430
|
DEFAULT_BACKEND="$BACKEND"
|
|
431
431
|
if [ "$BACKEND" = both ]; then
|
|
432
|
-
DEFAULT_BACKEND=$(choose "选择默认 backend:" "
|
|
432
|
+
DEFAULT_BACKEND=$(choose "选择默认 backend:" "openapi" "ssh")
|
|
433
433
|
fi
|
|
434
434
|
|
|
435
435
|
if [ "$BACKEND" = ssh ] || [ "$BACKEND" = both ]; then
|