dsh-scnet 0.5.1 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +37 -1
- package/README_CN.md +32 -1
- package/docs/testing.md +11 -0
- package/docs/testing.zh-CN.md +11 -0
- package/index.mjs +584 -0
- package/package.json +1 -1
- package/skills/scnet-hpc/VERSION +1 -1
- package/skills/scnet-hpc/references/openapi.md +15 -0
- package/skills/scnet-hpc/scripts/scnet.py +7 -2
- package/skills/scnet-hpc/scripts/scnet_backends/openapi.py +153 -17
- package/skills/scnet-hpc/scripts/scnet_backends/ssh.py +4 -1
- package/skills/scnet-hpc/scripts/scnet_sdk/client.py +111 -19
package/README.md
CHANGED
|
@@ -4,10 +4,26 @@
|
|
|
4
4
|
|
|
5
5
|
DSH-SCNet (`dsh-scnet` on npm) is a community-maintained DSH bundle for operating Supercomputing Network (SCNet). It packages the source [`scnet-hpc`](https://github.com/lql341/scnet-hpc) Skill, profile-aware utilities, and bounded tools for SSH, Slurm, OpenAPI files, regional resources, and Notebook discovery.
|
|
6
6
|
|
|
7
|
-
Current release: **0.
|
|
7
|
+
Current release: **0.6.1**
|
|
8
8
|
|
|
9
9
|
> This is an independent community project. It is compatible with DeepSeek Harness but is not an official DeepSeek product and does not imply endorsement, partnership, or authorization by DeepSeek.
|
|
10
10
|
|
|
11
|
+
## What's new in 0.6.1
|
|
12
|
+
|
|
13
|
+
- Job details now fall back to the history endpoint when realtime records expire, with complete
|
|
14
|
+
normalized terminal states.
|
|
15
|
+
- Added compact `scnet_job_list`, `scnet_limits`, and `scnet_file_transfer` tools.
|
|
16
|
+
- File uploads now make the remote-directory contract explicit: the local filename is preserved
|
|
17
|
+
and must not be passed as the remote directory.
|
|
18
|
+
|
|
19
|
+
## 0.6.0 highlights
|
|
20
|
+
|
|
21
|
+
- Added first-class DSH tools for submitting jobs, inspecting job status, reading logs, and
|
|
22
|
+
cancelling jobs through the existing SSH/OpenAPI backends.
|
|
23
|
+
- Added dry-run support and one-missing-decision-at-a-time validation for job lifecycle calls.
|
|
24
|
+
- Job submission reports the authoritative work directory and log paths; OpenAPI log lookup can
|
|
25
|
+
derive `std.out` and `std.err` paths from `job_id` and `work_dir`.
|
|
26
|
+
|
|
11
27
|
## Requirements
|
|
12
28
|
|
|
13
29
|
- Node.js 22.19 or later
|
|
@@ -45,7 +61,14 @@ See [Installation](./docs/installation.md) for npm, GitHub, and local-checkout w
|
|
|
45
61
|
| `scnet_status` | Read the saved backend, SSH profile, and OpenAPI configuration |
|
|
46
62
|
| `scnet_openapi_regions` | List authorized OpenAPI regions without exposing tokens |
|
|
47
63
|
| `scnet_job_queues` | Query regional Slurm queues and live resource availability |
|
|
64
|
+
| `scnet_job_list` | List active or historical jobs with compact normalized records |
|
|
65
|
+
| `scnet_limits` | Read user and scheduler resource limits |
|
|
66
|
+
| `scnet_submit_job` | Submit a job through OpenAPI or SSH; `dry_run=true` only previews the request |
|
|
67
|
+
| `scnet_job_show` | Read one job's state, resources, and log paths |
|
|
68
|
+
| `scnet_job_logs` | Read job logs by explicit path, or by `job_id` + `work_dir` |
|
|
69
|
+
| `scnet_job_cancel` | Cancel a job; `dry_run=true` only previews the request |
|
|
48
70
|
| `scnet_file_list` | List shared-storage files through the common file API |
|
|
71
|
+
| `scnet_file_transfer` | Upload or download one file without overwriting by default |
|
|
49
72
|
| `scnet_notebook_regions` | List Notebook-capable regions |
|
|
50
73
|
| `scnet_notebook_resources` | Query Notebook CPU/GPU/DCU resources |
|
|
51
74
|
| `scnet_notebook_list` | List Notebook instances with sensitive fields redacted |
|
|
@@ -89,6 +112,19 @@ Release order is canonical-to-downstream: update `scnet-hpc/VERSION` first; its
|
|
|
89
112
|
synchronizes the skill and version into `dsh-scnet` and `codex-scnet-hpc`. Publish the matching
|
|
90
113
|
`dsh-scnet` npm package only after that downstream change is reviewed.
|
|
91
114
|
|
|
115
|
+
For a synchronized release, run the package validation on `main`, then trigger the repository's
|
|
116
|
+
`Publish DSH-SCNet` workflow with the exact value from `package.json` (currently `0.6.1`). The
|
|
117
|
+
workflow verifies the package/Skill version match, runs validation, creates the tarball, publishes
|
|
118
|
+
stable releases with the `latest` tag, and creates the matching Git tag. Do not bump only
|
|
119
|
+
`package.json`: `skills/scnet-hpc/VERSION`, the package version, and both README release markers
|
|
120
|
+
must stay aligned.
|
|
121
|
+
|
|
122
|
+
The workflow uses npm Trusted Publishing through GitHub OIDC, so account 2FA/Passkey should remain
|
|
123
|
+
enabled and no long-lived `NPM_TOKEN` is required. Configure `lql341/dsh-scnet` and
|
|
124
|
+
`.github/workflows/publish.yml` as a trusted publisher in the npm package settings before the
|
|
125
|
+
first OIDC release. A granular read/write token with 2FA bypass is a fallback only when Trusted
|
|
126
|
+
Publishing cannot be enabled.
|
|
127
|
+
|
|
92
128
|
## Validation
|
|
93
129
|
|
|
94
130
|
```sh
|
package/README_CN.md
CHANGED
|
@@ -4,10 +4,23 @@
|
|
|
4
4
|
|
|
5
5
|
<p align="center">在 DeepSeek Harness 里操作超算互联网(SCNet):配置 SSH、管理 Slurm 作业、查询 OpenAPI 区域和文件、检查 Notebook 资源与实例,并保留海光 DCU/DTK 诊断能力。</p>
|
|
6
6
|
|
|
7
|
-
当前版本:**0.
|
|
7
|
+
当前版本:**0.6.1**
|
|
8
8
|
|
|
9
9
|
> 本项目是独立维护的社区项目,与 DeepSeek Harness 兼容,但不是 DeepSeek 官方产品,也不表示 DeepSeek 对本项目提供背书、合作或授权。
|
|
10
10
|
|
|
11
|
+
## 0.6.1 更新
|
|
12
|
+
|
|
13
|
+
- 作业实时记录过期时自动回退历史接口,并补全终态映射。
|
|
14
|
+
- 新增紧凑的 `scnet_job_list`、`scnet_limits`、`scnet_file_transfer` 工具。
|
|
15
|
+
- 明确 OpenAPI 上传参数是远端目录,文件名沿用本地文件名,不能把文件名当作远端目录传入。
|
|
16
|
+
|
|
17
|
+
## 0.6.0 更新
|
|
18
|
+
|
|
19
|
+
- 新增作业提交、状态查询、日志读取和取消作业的 DSH 工具,复用现有 SSH/OpenAPI backend。
|
|
20
|
+
- 作业生命周期调用支持 `dry_run`,并按 CLI 顺序一次只提示一个缺失决策。
|
|
21
|
+
- 提交结果返回权威的工作目录和日志路径;OpenAPI 日志查询支持由 `job_id` 和 `work_dir`
|
|
22
|
+
推导 `std.out`、`std.err` 路径。
|
|
23
|
+
|
|
11
24
|
## 这是什么
|
|
12
25
|
|
|
13
26
|
一个 DeepSeek Harness bundle 插件,把 `scnet-hpc` 的 Agent Skill 和 bash 脚本包装成 DSH 可安装包。装好后 DSH 模型在涉及超算集群、Slurm 作业、海光 DCU/DTK 时会加载对应 skill,并可调用工具完成确定性操作。
|
|
@@ -53,7 +66,14 @@ git 源、本地目录安装以及验证步骤见[安装说明](./docs/installat
|
|
|
53
66
|
| scnet_status | 只读查看 backend、SSH profile 和 OpenAPI 配置 |
|
|
54
67
|
| scnet_openapi_regions | 列出授权 OpenAPI 区域,不暴露 token |
|
|
55
68
|
| scnet_job_queues | 查询区域 Slurm 队列和实时空闲资源 |
|
|
69
|
+
| scnet_job_list | 紧凑列出当前或历史作业 |
|
|
70
|
+
| scnet_limits | 查询用户和调度器资源限制 |
|
|
71
|
+
| scnet_submit_job | 通过 OpenAPI 或 SSH 提交作业;dry_run=true 只预览请求 |
|
|
72
|
+
| scnet_job_show | 查询单个作业的状态、资源与日志路径 |
|
|
73
|
+
| scnet_job_logs | 按显式 path,或按 job_id + work_dir 读取作业日志 |
|
|
74
|
+
| scnet_job_cancel | 取消作业;dry_run=true 只预览请求 |
|
|
56
75
|
| scnet_file_list | 通过通用文件 API 列出共享存储 |
|
|
76
|
+
| scnet_file_transfer | 上传或下载单个文件,默认不覆盖 |
|
|
57
77
|
| scnet_notebook_regions | 列出支持 Notebook 的区域 |
|
|
58
78
|
| scnet_notebook_resources | 查询 Notebook CPU/GPU/DCU 资源 |
|
|
59
79
|
| scnet_notebook_list | 列出 Notebook 实例并脱敏敏感字段 |
|
|
@@ -105,6 +125,17 @@ git 源、本地目录安装以及验证步骤见[安装说明](./docs/installat
|
|
|
105
125
|
再把 Skill 内容和版本同步到 `dsh-scnet` 与 `codex-scnet-hpc`。下游变更审核通过后,
|
|
106
126
|
才能发布对应版本的 `dsh-scnet` npm 包。
|
|
107
127
|
|
|
128
|
+
下游同步合并到 `main` 后,先运行本地校验,再手动触发仓库的 `Publish DSH-SCNet` workflow,
|
|
129
|
+
输入 `package.json` 中的精确版本(当前为 `0.6.1`)。workflow 会校验 package 与 Skill
|
|
130
|
+
版本一致,执行验证、生成 tarball、以稳定版 `latest` 标签发布,并创建对应 Git tag。
|
|
131
|
+
不要只修改 `package.json`;`skills/scnet-hpc/VERSION`、package 版本和两份 README 的版本
|
|
132
|
+
标记必须保持一致。
|
|
133
|
+
|
|
134
|
+
该 workflow 使用 GitHub OIDC 的 npm Trusted Publishing,因此应保留账号的 2FA/Passkey,
|
|
135
|
+
不需要长期保存 `NPM_TOKEN`。首次使用 OIDC 发布前,请在 npm package 设置中把
|
|
136
|
+
`lql341/dsh-scnet` 与 `.github/workflows/publish.yml` 配置为 trusted publisher。只有无法
|
|
137
|
+
启用 Trusted Publishing 时,才考虑使用具备 2FA bypass 的 granular read/write token。
|
|
138
|
+
|
|
108
139
|
## 隐私与安全
|
|
109
140
|
|
|
110
141
|
- 仓库不含私钥、token、用户名、密钥指纹。
|
package/docs/testing.md
CHANGED
|
@@ -40,7 +40,14 @@ Confirm that the composed configuration includes:
|
|
|
40
40
|
| `scnet_status` | Returns redacted local SCNet configuration |
|
|
41
41
|
| `scnet_openapi_regions` | Returns authorized regions without tokens |
|
|
42
42
|
| `scnet_job_queues` | Returns live regional Slurm queues |
|
|
43
|
+
| `scnet_job_list` | Lists active or historical jobs |
|
|
44
|
+
| `scnet_limits` | Returns user and scheduler resource limits |
|
|
45
|
+
| `scnet_submit_job` | Submits a job through OpenAPI or SSH; `dry_run=true` returns the pending request only |
|
|
46
|
+
| `scnet_job_show` | Returns one job's state, resources, and log paths |
|
|
47
|
+
| `scnet_job_logs` | Reads job logs by explicit path, or by `job_id` + `work_dir` |
|
|
48
|
+
| `scnet_job_cancel` | Cancels one job; `dry_run=true` returns the pending request only |
|
|
43
49
|
| `scnet_file_list` | Lists shared-storage files without modification |
|
|
50
|
+
| `scnet_file_transfer` | Uploads or downloads one file; defaults to no overwrite |
|
|
44
51
|
| `scnet_notebook_regions` | Lists Notebook-capable regions |
|
|
45
52
|
| `scnet_notebook_resources` | Returns Notebook resource groups |
|
|
46
53
|
| `scnet_notebook_list` | Lists redacted Notebook instances |
|
|
@@ -60,6 +67,10 @@ Check both accelerator and CPU-only job generation. Review the output for the se
|
|
|
60
67
|
|
|
61
68
|
`scnet_run_compute_probe` and refresh operations with compute probing submit Slurm jobs. Confirm the target cluster, account policy, accelerator count, CPU count, and time limit before execution.
|
|
62
69
|
|
|
70
|
+
`scnet_submit_job` and `scnet_job_cancel` are mutating tools and consume or release real scheduler
|
|
71
|
+
resources. Run them with `dry_run=true` first, confirm the resolved target region or cluster in the
|
|
72
|
+
result, and never retry a timed-out submission blindly: query the job state instead.
|
|
73
|
+
|
|
63
74
|
Notebook lifecycle mutations are intentionally not exposed as deterministic DSH tools in this
|
|
64
75
|
release. Test them only through the Skill CLI and start with `--dry-run`.
|
|
65
76
|
|
package/docs/testing.zh-CN.md
CHANGED
|
@@ -61,12 +61,23 @@ DEEPSEEK_API_KEY: sk-你的key
|
|
|
61
61
|
| 8 | 用 scnet_status 查看当前配置 | 返回脱敏配置 | 只读 |
|
|
62
62
|
| 9 | 用 scnet_openapi_regions 查看授权区域 | 返回区域且不含 token | 只读 |
|
|
63
63
|
| 10 | 用 scnet_job_queues 查看昆山队列 | 返回实时队列和空闲资源 | 只读 |
|
|
64
|
+
| 10a | 用 scnet_job_list 列出当前或历史作业 | 返回紧凑作业记录 | 只读 |
|
|
65
|
+
| 10b | 用 scnet_limits 查询资源限制 | 返回用户和调度器限制 | 只读 |
|
|
66
|
+
| 10a | 用 scnet_submit_job(dry_run=true)预览一次提交 | 返回将要发送的请求,不真正提交 | 只读(dry_run) |
|
|
67
|
+
| 10b | 用 scnet_submit_job 提交一个 5 秒小作业 | 返回 job_id、作业目录与日志路径 | 高(会消耗配额) |
|
|
68
|
+
| 10c | 用 scnet_job_show 查该 job_id | 返回状态与 stdout/stderr 路径 | 只读 |
|
|
69
|
+
| 10d | 用 scnet_job_logs 读该作业日志 | 返回日志内容(同时验证 path 与 job_id+work_dir 两种模式结果一致) | 只读 |
|
|
70
|
+
| 10e | 用 scnet_job_cancel 取消一个作业 | 建议先用 dry_run=true 预览 | 高(变更) |
|
|
64
71
|
| 11 | 用 scnet_file_list 查看默认目录 | 返回共享存储文件 | 只读 |
|
|
72
|
+
| 11a | 用 scnet_file_transfer 传输一个文件 | 默认不覆盖目标 | 有副作用 |
|
|
65
73
|
| 12 | 用 scnet_notebook_regions 查看区域 | 返回 Notebook 可用区域 | 只读 |
|
|
66
74
|
| 13 | 用 scnet_notebook_resources 查看昆山资源 | 返回 CPU/GPU/DCU 资源 | 只读 |
|
|
67
75
|
| 14 | 用 scnet_notebook_list 查看实例 | 返回脱敏实例列表 | 只读 |
|
|
68
76
|
|
|
69
77
|
第 4 至 7 步只有在你有私钥和目标集群时才测,测前确认副作用。
|
|
70
78
|
|
|
79
|
+
第 10b、10e 步是变更操作,会真实占用或释放调度器资源:先用 `dry_run=true` 预览,确认返回里的
|
|
80
|
+
目标区域/集群,并且**不要在提交超时后盲目重试**(平台没有幂等键),先查作业状态。
|
|
81
|
+
|
|
71
82
|
Notebook 创建、启动、停止和释放没有作为 DSH deterministic tool 暴露。此类操作通过
|
|
72
83
|
Skill CLI 执行,并先使用 `--dry-run`。
|
package/index.mjs
CHANGED
|
@@ -94,6 +94,198 @@ function positiveInt(value, label) {
|
|
|
94
94
|
return { value: String(value).trim() }
|
|
95
95
|
}
|
|
96
96
|
|
|
97
|
+
// ---- 作业生命周期(submit / job / logs / cancel)辅助 ----
|
|
98
|
+
//
|
|
99
|
+
// 设计约束来自 scnet-hpc Skill 的 SKILL.md:
|
|
100
|
+
// - 变更操作必须解析到一个确定目标(OpenAPI 允许默认区域,SSH 多 profile 时必须显式)
|
|
101
|
+
// - 提交/取消绝不自动重试
|
|
102
|
+
// - 缺失参数一次只提示一个,并给出可执行的下一步
|
|
103
|
+
|
|
104
|
+
const JOB_BACKENDS = new Set(['openapi', 'ssh'])
|
|
105
|
+
|
|
106
|
+
const MISSING_HINTS = {
|
|
107
|
+
name: '缺 name:请给这个作业起一个名字。',
|
|
108
|
+
command: '缺 command:请给出要在计算节点上执行的命令或脚本体。',
|
|
109
|
+
work_dir: '缺 work_dir:请给出一个绝对路径作为作业工作目录(日志默认也落在那里)。',
|
|
110
|
+
queue: '缺 queue:请指定队列;不确定可先看该区域的空闲队列。',
|
|
111
|
+
remote_path: '缺 remote_path:SSH 模式请给出远端已存在的 Slurm 脚本绝对路径。',
|
|
112
|
+
job_id: '缺 job_id:请给出作业号。',
|
|
113
|
+
path: '缺 path:请给出远端日志文件的绝对路径。',
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
// 与 CLI 的必填顺序保持一致(openapi.py 的 _submit_payload:name → command → work_dir → queue)。
|
|
117
|
+
// queue 不在这里:它由队列预检统一处理,可能是自动选择、也可能是"请补一个"。
|
|
118
|
+
const SUBMIT_REQUIRED_ORDER = ['name', 'command', 'work_dir']
|
|
119
|
+
|
|
120
|
+
function firstMissingRequired(args, keys) {
|
|
121
|
+
for (const key of keys) {
|
|
122
|
+
if (!String(args[key] ?? '').trim()) return key
|
|
123
|
+
}
|
|
124
|
+
return ''
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
// backend 默认 openapi,不强制用户填写
|
|
128
|
+
function pickBackend(value) {
|
|
129
|
+
const raw = String(value ?? '').trim().toLowerCase()
|
|
130
|
+
if (!raw) return { backend: 'openapi', argv: ['--backend', 'openapi'] }
|
|
131
|
+
if (!JOB_BACKENDS.has(raw)) return { error: `backend 只支持 openapi 或 ssh,收到 ${value}。` }
|
|
132
|
+
return { backend: raw, argv: ['--backend', raw] }
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
// region / scheduler_id 交给后端自行解析:只有用户显式给了才透传
|
|
136
|
+
function openapiTargetArgs(args) {
|
|
137
|
+
const argv = []
|
|
138
|
+
const region = String(args.region ?? '').trim()
|
|
139
|
+
if (region) argv.push('--region', region)
|
|
140
|
+
const scheduler = String(args.scheduler_id ?? '').trim()
|
|
141
|
+
if (scheduler) argv.push('--scheduler-id', scheduler)
|
|
142
|
+
return argv
|
|
143
|
+
}
|
|
144
|
+
|
|
145
|
+
// SSH:恰好一个 profile 时自动选择;多个才要求用户补 cluster 一个参数
|
|
146
|
+
async function sshClusterArg(args) {
|
|
147
|
+
const cluster = String(args.cluster ?? '').trim()
|
|
148
|
+
if (cluster) return { argv: ['--cluster', cluster], cluster }
|
|
149
|
+
const ids = await listClusterIds()
|
|
150
|
+
if (ids.length === 1) return { argv: ['--cluster', ids[0]], cluster: ids[0] }
|
|
151
|
+
return {
|
|
152
|
+
error: `存在多个集群 profile,请补 cluster 一个参数。可选:${ids.join('、') || '(无)'}`,
|
|
153
|
+
}
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
async function resolveTarget(args) {
|
|
157
|
+
const pick = pickBackend(args.backend)
|
|
158
|
+
if (pick.error) return pick
|
|
159
|
+
const argv = [...pick.argv]
|
|
160
|
+
if (pick.backend === 'openapi') {
|
|
161
|
+
argv.push(...openapiTargetArgs(args))
|
|
162
|
+
return { backend: pick.backend, argv }
|
|
163
|
+
}
|
|
164
|
+
const ssh = await sshClusterArg(args)
|
|
165
|
+
if (ssh.error) return ssh
|
|
166
|
+
argv.push(...ssh.argv)
|
|
167
|
+
return { backend: pick.backend, argv, cluster: ssh.cluster }
|
|
168
|
+
}
|
|
169
|
+
|
|
170
|
+
// 失败时把 --json 错误信封里的 error 取出来
|
|
171
|
+
function errorTextFrom(res) {
|
|
172
|
+
const raw = String(res.stdout || '').trim() || String(res.stderr || '').trim()
|
|
173
|
+
try {
|
|
174
|
+
const parsed = JSON.parse(raw)
|
|
175
|
+
if (parsed && typeof parsed === 'object' && typeof parsed.error === 'string') {
|
|
176
|
+
return parsed.error
|
|
177
|
+
}
|
|
178
|
+
} catch {
|
|
179
|
+
// 非 JSON,退回纯文本解析
|
|
180
|
+
}
|
|
181
|
+
const matched = raw.match(/error:\s*(.+)/)
|
|
182
|
+
return matched ? matched[1].trim() : raw
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
// 一次只给一条可执行的提示(不展示整套参数清单)
|
|
186
|
+
function translateScnetError(text) {
|
|
187
|
+
const value = String(text || '')
|
|
188
|
+
const missing = value.match(/missing required option:\s*([A-Za-z_]+)/)
|
|
189
|
+
if (missing) return MISSING_HINTS[missing[1]] || `缺 ${missing[1]}:这是必填参数。`
|
|
190
|
+
if (/multiple schedulers are available/i.test(value)) {
|
|
191
|
+
const list = value.split('use --scheduler-id:')[1] || ''
|
|
192
|
+
return `该区域有多个调度器,请补 scheduler_id 一个参数。候选:${list.trim()}`
|
|
193
|
+
}
|
|
194
|
+
if (/job endpoint returned an unexpected data shape/i.test(value)) {
|
|
195
|
+
return (
|
|
196
|
+
'平台没有返回这个作业的记录:作业完成一段时间后可能会从该接口消失。' +
|
|
197
|
+
'若只想看日志,请直接给日志文件的 path;若刚提交不久,可稍后重试。'
|
|
198
|
+
)
|
|
199
|
+
}
|
|
200
|
+
return value
|
|
201
|
+
}
|
|
202
|
+
|
|
203
|
+
function scnetJobResult(res, action) {
|
|
204
|
+
if (res.ok) return res.stdout.trim() || '{}'
|
|
205
|
+
return `${action}失败(exit ${res.code}):\n${translateScnetError(errorTextFrom(res))}`
|
|
206
|
+
}
|
|
207
|
+
|
|
208
|
+
async function runScnetEnvelope(argv) {
|
|
209
|
+
const res = await runScnet(argv)
|
|
210
|
+
if (!res.ok) return { ok: false, error: translateScnetError(errorTextFrom(res)) }
|
|
211
|
+
try {
|
|
212
|
+
const parsed = JSON.parse(res.stdout)
|
|
213
|
+
return { ok: true, envelope: parsed, data: parsed?.data }
|
|
214
|
+
} catch {
|
|
215
|
+
return { ok: false, error: '后端返回的不是合法 JSON。' }
|
|
216
|
+
}
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
function parseWalltimeSeconds(value) {
|
|
220
|
+
const matched = String(value || '').trim().match(/^(?:(\d+)-)?(\d{1,2}):(\d{2}):(\d{2})$/)
|
|
221
|
+
if (!matched) return null
|
|
222
|
+
const [, days, hours, minutes, seconds] = matched
|
|
223
|
+
return Number(days || 0) * 86400 + Number(hours) * 3600 + Number(minutes) * 60 + Number(seconds)
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
// 按是否需要加速器筛队列,空闲节点多的优先(只用于"推荐一个",不用于静默多选一)
|
|
227
|
+
function queueCandidates(queues, wantsAccelerator) {
|
|
228
|
+
return queues
|
|
229
|
+
.filter((item) => {
|
|
230
|
+
const maxDcu = Number(item?.max_dcus_per_node ?? 0)
|
|
231
|
+
return wantsAccelerator ? maxDcu > 0 : maxDcu === 0
|
|
232
|
+
})
|
|
233
|
+
.sort((a, b) => Number(b?.free_nodes ?? 0) - Number(a?.free_nodes ?? 0))
|
|
234
|
+
}
|
|
235
|
+
|
|
236
|
+
// best-effort 队列预检:查得到就校验,查不到不阻塞提交
|
|
237
|
+
async function precheckQueue(target, args) {
|
|
238
|
+
const check = await runScnetEnvelope([...target.argv, 'queues'])
|
|
239
|
+
if (!check.ok) return { note: `队列预检已跳过(${check.error})` }
|
|
240
|
+
const queues = Array.isArray(check.data) ? check.data : []
|
|
241
|
+
if (queues.length === 0) return { note: '队列预检已跳过(该区域没有返回队列信息)' }
|
|
242
|
+
|
|
243
|
+
const queue = String(args.queue ?? '').trim()
|
|
244
|
+
const dcus = Number(String(args.dcus ?? '').trim() || 0)
|
|
245
|
+
|
|
246
|
+
if (!queue) {
|
|
247
|
+
const candidates = queueCandidates(queues, dcus > 0)
|
|
248
|
+
if (candidates.length === 1) {
|
|
249
|
+
const only = candidates[0]
|
|
250
|
+
return { queue: String(only.partition), note: `已自动选择队列 ${only.partition}` }
|
|
251
|
+
}
|
|
252
|
+
if (candidates.length > 1) {
|
|
253
|
+
const best = candidates[0]
|
|
254
|
+
return {
|
|
255
|
+
error:
|
|
256
|
+
`请补 queue 一个参数:该区域有 ${candidates.length} 个可用队列,` +
|
|
257
|
+
`按你要的资源建议 ${best.partition}(空闲 ${best.free_nodes} 节点)。`,
|
|
258
|
+
}
|
|
259
|
+
}
|
|
260
|
+
return {}
|
|
261
|
+
}
|
|
262
|
+
|
|
263
|
+
const hit = queues.find((item) => String(item?.partition) === queue)
|
|
264
|
+
if (!hit) {
|
|
265
|
+
const candidates = queueCandidates(queues, dcus > 0)
|
|
266
|
+
const pool = candidates.length ? candidates : queues
|
|
267
|
+
return {
|
|
268
|
+
error: `队列 ${queue} 不存在。可用队列:${pool.map((item) => item.partition).join('、')}`,
|
|
269
|
+
}
|
|
270
|
+
}
|
|
271
|
+
|
|
272
|
+
const walltime = String(args.walltime ?? '').trim()
|
|
273
|
+
if (walltime) {
|
|
274
|
+
const wanted = parseWalltimeSeconds(walltime)
|
|
275
|
+
const limit = Number(hit?.max_walltime ?? 0)
|
|
276
|
+
if (wanted !== null && Number.isFinite(limit) && limit > 0 && wanted > limit) {
|
|
277
|
+
return { error: `walltime ${walltime} 超过队列 ${queue} 的上限(${limit} 秒)。` }
|
|
278
|
+
}
|
|
279
|
+
}
|
|
280
|
+
|
|
281
|
+
const maxDcu = Number(hit?.max_dcus_per_node ?? 0)
|
|
282
|
+
if (dcus > 0 && Number.isFinite(maxDcu) && maxDcu > 0 && dcus > maxDcu) {
|
|
283
|
+
return { error: `dcus ${dcus} 超过队列 ${queue} 单节点上限 ${maxDcu}。` }
|
|
284
|
+
}
|
|
285
|
+
|
|
286
|
+
return { queue, note: `队列预检通过(${queue})` }
|
|
287
|
+
}
|
|
288
|
+
|
|
97
289
|
export function apply(ctx) {
|
|
98
290
|
ctx.tools.register(
|
|
99
291
|
defineTool({
|
|
@@ -173,6 +365,386 @@ export function apply(ctx) {
|
|
|
173
365
|
}),
|
|
174
366
|
)
|
|
175
367
|
|
|
368
|
+
ctx.tools.register(
|
|
369
|
+
defineTool({
|
|
370
|
+
name: 'scnet_job_list',
|
|
371
|
+
description:
|
|
372
|
+
'只读列出 SCNet 作业。默认列出当前区域正在排队或运行的作业;scope=history 时查询最近历史作业。区域、数量和历史天数都可省略,输出保持紧凑。',
|
|
373
|
+
parameters: {
|
|
374
|
+
scope: { type: 'string', description: 'active(默认)或 history' },
|
|
375
|
+
region: { type: 'string', description: '区域名称或 ID;省略时使用已保存的默认区域' },
|
|
376
|
+
limit: { type: 'number', description: '最多返回条数,默认 20,最大 100' },
|
|
377
|
+
days: { type: 'number', description: 'history 模式查询最近天数,默认 30,最大 90' },
|
|
378
|
+
},
|
|
379
|
+
output: {
|
|
380
|
+
schema: { type: 'string' },
|
|
381
|
+
render: (_args, value) => textBlock(value),
|
|
382
|
+
},
|
|
383
|
+
async execute(args) {
|
|
384
|
+
const scope = String(args.scope ?? 'active').trim().toLowerCase()
|
|
385
|
+
if (scope !== 'active' && scope !== 'history') {
|
|
386
|
+
return 'scope 只支持 active 或 history。'
|
|
387
|
+
}
|
|
388
|
+
const argv = ['--backend', 'openapi', ...regionArgs(args.region), 'jobs', '--scope', scope]
|
|
389
|
+
if (args.limit !== undefined) {
|
|
390
|
+
const checked = positiveInt(args.limit, 'limit')
|
|
391
|
+
if (checked?.error) return checked.error
|
|
392
|
+
argv.push('--limit', checked.value)
|
|
393
|
+
}
|
|
394
|
+
if (args.days !== undefined) {
|
|
395
|
+
const checked = positiveInt(args.days, 'days')
|
|
396
|
+
if (checked?.error) return checked.error
|
|
397
|
+
argv.push('--days', checked.value)
|
|
398
|
+
}
|
|
399
|
+
return scnetResult(await runScnet(argv), '查询作业列表')
|
|
400
|
+
},
|
|
401
|
+
}),
|
|
402
|
+
)
|
|
403
|
+
|
|
404
|
+
ctx.tools.register(
|
|
405
|
+
defineTool({
|
|
406
|
+
name: 'scnet_limits',
|
|
407
|
+
description:
|
|
408
|
+
'只读查询当前区域和调度器的用户资源限制。region、scheduler_id 都可省略,后端会自动解析唯一目标。',
|
|
409
|
+
parameters: {
|
|
410
|
+
region: { type: 'string', description: '区域名称或 ID;省略时使用默认区域' },
|
|
411
|
+
scheduler_id: { type: 'string', description: '调度器 ID;只有多个调度器且无法自动选择时填写' },
|
|
412
|
+
},
|
|
413
|
+
output: {
|
|
414
|
+
schema: { type: 'string' },
|
|
415
|
+
render: (_args, value) => textBlock(value),
|
|
416
|
+
},
|
|
417
|
+
async execute(args) {
|
|
418
|
+
const argv = [
|
|
419
|
+
'--backend',
|
|
420
|
+
'openapi',
|
|
421
|
+
...regionArgs(args.region),
|
|
422
|
+
...(args.scheduler_id ? ['--scheduler-id', String(args.scheduler_id)] : []),
|
|
423
|
+
'limits',
|
|
424
|
+
]
|
|
425
|
+
return scnetResult(await runScnet(argv), '查询资源限制')
|
|
426
|
+
},
|
|
427
|
+
}),
|
|
428
|
+
)
|
|
429
|
+
|
|
430
|
+
ctx.tools.register(
|
|
431
|
+
defineTool({
|
|
432
|
+
name: 'scnet_submit_job',
|
|
433
|
+
description:
|
|
434
|
+
'提交一个 SCNet 作业。默认走 OpenAPI;backend=ssh 时提交远端已存在的 Slurm 脚本。这是变更操作,会占用配额,执行前应确认目标、资源与时长。region/cluster/scheduler 能自动解析;dry_run=true 只预览请求不提交。提交成功后返回 job_id 与日志路径,可直接交给 scnet_job_logs。',
|
|
435
|
+
parameters: {
|
|
436
|
+
backend: { type: 'string', description: 'openapi(默认)或 ssh' },
|
|
437
|
+
region: { type: 'string', description: 'OpenAPI 区域;省略时由后端回落已保存的默认区域' },
|
|
438
|
+
scheduler_id: { type: 'string', description: 'OpenAPI 调度器 ID;省略时该区域只有一个调度器则自动选择' },
|
|
439
|
+
cluster: { type: 'string', description: 'SSH 集群短名;只有一个 profile 时自动选择,多个则必须指定' },
|
|
440
|
+
remote_path: { type: 'string', description: 'SSH 模式必填:远端已存在的 Slurm 脚本绝对路径' },
|
|
441
|
+
name: { type: 'string', description: 'OpenAPI 必填:作业名' },
|
|
442
|
+
command: { type: 'string', description: 'OpenAPI 必填:命令或脚本体(可多行)' },
|
|
443
|
+
work_dir: { type: 'string', description: 'OpenAPI 必填:作业工作目录(绝对路径,日志默认落在这里)' },
|
|
444
|
+
queue: { type: 'string', description: 'OpenAPI 必填:队列名;省略时仅在候选唯一时自动选择,否则会问你一个' },
|
|
445
|
+
nodes: { type: 'string', description: '节点数,默认 1' },
|
|
446
|
+
cpus: { type: 'string', description: 'CPU 核数,默认 1' },
|
|
447
|
+
dcus: { type: 'string', description: 'DCU 卡数,默认 0;部分队列的 QOS 要求至少 1(如昆山 kshdnormal)' },
|
|
448
|
+
gpus: { type: 'string', description: 'GPU 卡数,默认 0' },
|
|
449
|
+
memory: { type: 'string', description: '内存,如 12gb;省略时用调度器默认' },
|
|
450
|
+
walltime: { type: 'string', description: '时长,默认 24:00:00,格式 HH:MM:SS 或 D-HH:MM:SS' },
|
|
451
|
+
stdout: { type: 'string', description: '显式覆盖 stdout 路径;会破坏 scnet_job_logs 的便捷模式' },
|
|
452
|
+
stderr: { type: 'string', description: '显式覆盖 stderr 路径;会破坏 scnet_job_logs 的便捷模式' },
|
|
453
|
+
exclusive: { type: 'boolean', description: '独占节点(默认否)' },
|
|
454
|
+
scheduler_options: {
|
|
455
|
+
type: 'array',
|
|
456
|
+
items: { type: 'string' },
|
|
457
|
+
description: '额外的 #SBATCH 选项,逐条透传',
|
|
458
|
+
},
|
|
459
|
+
dry_run: { type: 'boolean', description: '只预览将要发送的请求,不真正提交(默认否)' },
|
|
460
|
+
},
|
|
461
|
+
timeoutMs: 180000,
|
|
462
|
+
output: {
|
|
463
|
+
schema: { type: 'string' },
|
|
464
|
+
render: (_args, value) => textBlock(value),
|
|
465
|
+
},
|
|
466
|
+
async execute(args) {
|
|
467
|
+
const target = await resolveTarget(args)
|
|
468
|
+
if (target.error) return target.error
|
|
469
|
+
|
|
470
|
+
const globals = [...target.argv]
|
|
471
|
+
if (args.dry_run === true) globals.push('--dry-run')
|
|
472
|
+
|
|
473
|
+
if (target.backend === 'ssh') {
|
|
474
|
+
const remotePath = String(args.remote_path ?? '').trim()
|
|
475
|
+
if (!remotePath) return MISSING_HINTS.remote_path
|
|
476
|
+
const res = await runScnet([...globals, 'submit', '--remote-path', remotePath])
|
|
477
|
+
return scnetJobResult(res, '提交作业')
|
|
478
|
+
}
|
|
479
|
+
|
|
480
|
+
for (const key of ['nodes', 'cpus', 'dcus', 'gpus']) {
|
|
481
|
+
const checked = positiveInt(args[key], key)
|
|
482
|
+
if (checked && checked.error) return checked.error
|
|
483
|
+
}
|
|
484
|
+
|
|
485
|
+
// 按 CLI 的必填顺序先做本地检查,一次只提示一个,
|
|
486
|
+
// 并避免在参数不全时就发起队列查询。
|
|
487
|
+
const missing = firstMissingRequired(args, SUBMIT_REQUIRED_ORDER)
|
|
488
|
+
if (missing) return MISSING_HINTS[missing]
|
|
489
|
+
|
|
490
|
+
const precheck = await precheckQueue(target, args)
|
|
491
|
+
if (precheck.error) return precheck.error
|
|
492
|
+
const queue = precheck.queue || String(args.queue ?? '').trim()
|
|
493
|
+
|
|
494
|
+
const argv = [...globals, 'submit']
|
|
495
|
+
const name = String(args.name ?? '').trim()
|
|
496
|
+
const command = String(args.command ?? '')
|
|
497
|
+
const workDir = String(args.work_dir ?? '').trim()
|
|
498
|
+
if (name) argv.push('--name', name)
|
|
499
|
+
if (command.trim()) argv.push('--command', command)
|
|
500
|
+
if (workDir) argv.push('--work-dir', workDir)
|
|
501
|
+
if (queue) argv.push('--queue', queue)
|
|
502
|
+
for (const [flag, key] of [
|
|
503
|
+
['--nodes', 'nodes'],
|
|
504
|
+
['--cpus', 'cpus'],
|
|
505
|
+
['--dcus', 'dcus'],
|
|
506
|
+
['--gpus', 'gpus'],
|
|
507
|
+
]) {
|
|
508
|
+
const checked = positiveInt(args[key], key)
|
|
509
|
+
if (checked && checked.value) argv.push(flag, checked.value)
|
|
510
|
+
}
|
|
511
|
+
for (const [flag, key] of [
|
|
512
|
+
['--memory', 'memory'],
|
|
513
|
+
['--walltime', 'walltime'],
|
|
514
|
+
['--stdout', 'stdout'],
|
|
515
|
+
['--stderr', 'stderr'],
|
|
516
|
+
]) {
|
|
517
|
+
const value = String(args[key] ?? '').trim()
|
|
518
|
+
if (value) argv.push(flag, value)
|
|
519
|
+
}
|
|
520
|
+
if (args.exclusive === true) argv.push('--exclusive')
|
|
521
|
+
const options = Array.isArray(args.scheduler_options) ? args.scheduler_options : []
|
|
522
|
+
for (const option of options) {
|
|
523
|
+
const value = String(option ?? '').trim()
|
|
524
|
+
if (value) argv.push('--scheduler-option', value)
|
|
525
|
+
}
|
|
526
|
+
|
|
527
|
+
const res = await runScnet(argv)
|
|
528
|
+
if (!res.ok) return scnetJobResult(res, '提交作业')
|
|
529
|
+
if (args.dry_run === true) return res.stdout.trim() || '{}'
|
|
530
|
+
|
|
531
|
+
let jobId = ''
|
|
532
|
+
try {
|
|
533
|
+
const parsed = JSON.parse(res.stdout)
|
|
534
|
+
jobId = String(parsed?.data?.job_id ?? parsed?.job_id ?? '')
|
|
535
|
+
} catch {
|
|
536
|
+
// 保底走下面的原始输出分支
|
|
537
|
+
}
|
|
538
|
+
if (!jobId) return `提交请求已发出,但没能解析出 job_id。原始返回:\n${res.stdout.trim()}`
|
|
539
|
+
|
|
540
|
+
// 提交后立刻取一次权威的 work_dir / 日志路径;作业较旧时该接口可能不再返回记录
|
|
541
|
+
const jobRes = await runScnetEnvelope([...target.argv, 'job', jobId])
|
|
542
|
+
let resolvedWorkDir = workDir
|
|
543
|
+
let stdoutPath = String(args.stdout ?? '').trim()
|
|
544
|
+
let stderrPath = String(args.stderr ?? '').trim()
|
|
545
|
+
let source = 'default-rule'
|
|
546
|
+
if (jobRes.ok && jobRes.data) {
|
|
547
|
+
resolvedWorkDir = String(jobRes.data.work_dir || resolvedWorkDir || '')
|
|
548
|
+
stdoutPath = String(jobRes.data.stdout || stdoutPath || '')
|
|
549
|
+
stderrPath = String(jobRes.data.stderr || stderrPath || '')
|
|
550
|
+
source = 'job'
|
|
551
|
+
}
|
|
552
|
+
if (!stdoutPath && resolvedWorkDir) {
|
|
553
|
+
const base = resolvedWorkDir.replace(/\/+$/, '')
|
|
554
|
+
stdoutPath = `${base}/std.out.${jobId}`
|
|
555
|
+
stderrPath = stderrPath || `${base}/std.err.${jobId}`
|
|
556
|
+
}
|
|
557
|
+
|
|
558
|
+
const summary = {
|
|
559
|
+
backend: target.backend,
|
|
560
|
+
region: String(args.region ?? '').trim() || '(由后端默认区域解析)',
|
|
561
|
+
scheduler_id: String(args.scheduler_id ?? '').trim() || undefined,
|
|
562
|
+
job_id: jobId,
|
|
563
|
+
queue: queue || undefined,
|
|
564
|
+
work_dir: resolvedWorkDir || undefined,
|
|
565
|
+
stdout: stdoutPath || undefined,
|
|
566
|
+
stderr: stderrPath || undefined,
|
|
567
|
+
log_paths_source: source,
|
|
568
|
+
precheck: precheck.note || undefined,
|
|
569
|
+
note: '用 scnet_job_logs 读日志:直接传上面的 path,或传 job_id + work_dir。',
|
|
570
|
+
}
|
|
571
|
+
return JSON.stringify(summary, null, 2)
|
|
572
|
+
},
|
|
573
|
+
}),
|
|
574
|
+
)
|
|
575
|
+
|
|
576
|
+
ctx.tools.register(
|
|
577
|
+
defineTool({
|
|
578
|
+
name: 'scnet_job_show',
|
|
579
|
+
description:
|
|
580
|
+
'只读查询一个作业的状态与资源(state / exit_code / work_dir / stdout / stderr)。作业完成较久后平台可能不再返回记录,此时用 scnet_job_logs 的 path 模式读日志即可。',
|
|
581
|
+
parameters: {
|
|
582
|
+
job_id: { type: 'string', required: true, description: '作业号' },
|
|
583
|
+
backend: { type: 'string', description: 'openapi(默认)或 ssh' },
|
|
584
|
+
region: { type: 'string', description: 'OpenAPI 区域;省略时用已保存的默认区域' },
|
|
585
|
+
scheduler_id: { type: 'string', description: 'OpenAPI 调度器 ID;省略时自动' },
|
|
586
|
+
cluster: { type: 'string', description: 'SSH 集群短名;省略时若只有一个 profile 则自动选择' },
|
|
587
|
+
},
|
|
588
|
+
output: {
|
|
589
|
+
schema: { type: 'string' },
|
|
590
|
+
render: (_args, value) => textBlock(value),
|
|
591
|
+
},
|
|
592
|
+
async execute(args) {
|
|
593
|
+
const jobId = String(args.job_id ?? '').trim()
|
|
594
|
+
if (!jobId) return MISSING_HINTS.job_id
|
|
595
|
+
const target = await resolveTarget(args)
|
|
596
|
+
if (target.error) return target.error
|
|
597
|
+
return scnetJobResult(await runScnet([...target.argv, 'job', jobId]), '查询作业')
|
|
598
|
+
},
|
|
599
|
+
}),
|
|
600
|
+
)
|
|
601
|
+
|
|
602
|
+
ctx.tools.register(
|
|
603
|
+
defineTool({
|
|
604
|
+
name: 'scnet_job_logs',
|
|
605
|
+
description:
|
|
606
|
+
'只读读取作业日志。两种模式:显式给 path(远端绝对路径,SSH 下只能用这个);或给 job_id + work_dir + stream,内部按平台默认命名 std.out/std.err.{job_id} 推导。path 优先。',
|
|
607
|
+
parameters: {
|
|
608
|
+
path: { type: 'string', description: '显式模式:远端日志文件的绝对路径' },
|
|
609
|
+
job_id: { type: 'string', description: '便捷模式:作业号' },
|
|
610
|
+
work_dir: { type: 'string', description: '便捷模式:作业工作目录;省略时会尝试查一次作业' },
|
|
611
|
+
stream: { type: 'string', description: '便捷模式:stdout(默认)或 stderr' },
|
|
612
|
+
lines: { type: 'string', description: '读取行数,默认 200(1-10000)' },
|
|
613
|
+
direction: { type: 'string', description: 'head 或 tail(默认 tail)' },
|
|
614
|
+
backend: { type: 'string', description: 'openapi(默认)或 ssh' },
|
|
615
|
+
region: { type: 'string', description: 'OpenAPI 区域;省略时用已保存的默认区域' },
|
|
616
|
+
scheduler_id: { type: 'string', description: 'OpenAPI 调度器 ID;省略时自动' },
|
|
617
|
+
cluster: { type: 'string', description: 'SSH 集群短名;省略时若只有一个 profile 则自动选择' },
|
|
618
|
+
},
|
|
619
|
+
output: {
|
|
620
|
+
schema: { type: 'string' },
|
|
621
|
+
render: (_args, value) => textBlock(value),
|
|
622
|
+
},
|
|
623
|
+
async execute(args) {
|
|
624
|
+
const target = await resolveTarget(args)
|
|
625
|
+
if (target.error) return target.error
|
|
626
|
+
|
|
627
|
+
const lines = positiveInt(args.lines, 'lines')
|
|
628
|
+
if (lines && lines.error) return lines.error
|
|
629
|
+
|
|
630
|
+
let path = String(args.path ?? '').trim()
|
|
631
|
+
const jobId = String(args.job_id ?? '').trim()
|
|
632
|
+
let derived = false
|
|
633
|
+
|
|
634
|
+
if (!path) {
|
|
635
|
+
if (target.backend === 'ssh') {
|
|
636
|
+
return 'SSH 模式的日志路径由作业脚本里的 #SBATCH --output 决定,无法推导;请用 path 给出远端日志文件的绝对路径。'
|
|
637
|
+
}
|
|
638
|
+
if (!jobId) return `${MISSING_HINTS.path}(或给 job_id + work_dir 走便捷模式。)`
|
|
639
|
+
const stream = String(args.stream ?? 'stdout').trim().toLowerCase()
|
|
640
|
+
if (stream !== 'stdout' && stream !== 'stderr') return 'stream 只支持 stdout 或 stderr。'
|
|
641
|
+
let workDir = String(args.work_dir ?? '').trim()
|
|
642
|
+
if (!workDir) {
|
|
643
|
+
const jobRes = await runScnetEnvelope([...target.argv, 'job', jobId])
|
|
644
|
+
if (jobRes.ok && jobRes.data?.work_dir) {
|
|
645
|
+
workDir = String(jobRes.data.work_dir)
|
|
646
|
+
} else {
|
|
647
|
+
return `缺 work_dir:查不到作业 ${jobId} 的工作目录(${jobRes.error || '平台无记录'})。请补 work_dir,或直接用 path。`
|
|
648
|
+
}
|
|
649
|
+
}
|
|
650
|
+
const suffix = stream === 'stderr' ? 'err' : 'out'
|
|
651
|
+
path = `${workDir.replace(/\/+$/, '')}/std.${suffix}.${jobId}`
|
|
652
|
+
derived = true
|
|
653
|
+
}
|
|
654
|
+
|
|
655
|
+
const argv = [...target.argv, 'logs', '--path', path]
|
|
656
|
+
if (lines && lines.value) argv.push('--lines', lines.value)
|
|
657
|
+
const direction = String(args.direction ?? '').trim()
|
|
658
|
+
if (direction) argv.push('--direction', direction)
|
|
659
|
+
|
|
660
|
+
const res = await runScnet(argv)
|
|
661
|
+
if (!res.ok) return scnetJobResult(res, '读取日志')
|
|
662
|
+
const body = res.stdout.trim() || '{}'
|
|
663
|
+
return derived ? `${body}\n\n(日志路径由 job_id 与 work_dir 推导:${path})` : body
|
|
664
|
+
},
|
|
665
|
+
}),
|
|
666
|
+
)
|
|
667
|
+
|
|
668
|
+
ctx.tools.register(
|
|
669
|
+
defineTool({
|
|
670
|
+
name: 'scnet_job_cancel',
|
|
671
|
+
description:
|
|
672
|
+
'取消一个作业。这是变更操作,执行前应确认 job_id 与目标区域;dry_run=true 只预览不发送。超时后不要自动重试,先查状态。',
|
|
673
|
+
parameters: {
|
|
674
|
+
job_id: { type: 'string', required: true, description: '作业号' },
|
|
675
|
+
backend: { type: 'string', description: 'openapi(默认)或 ssh' },
|
|
676
|
+
region: { type: 'string', description: 'OpenAPI 区域;省略时用已保存的默认区域' },
|
|
677
|
+
scheduler_id: { type: 'string', description: 'OpenAPI 调度器 ID;省略时自动' },
|
|
678
|
+
cluster: { type: 'string', description: 'SSH 集群短名;省略时若只有一个 profile 则自动选择' },
|
|
679
|
+
dry_run: { type: 'boolean', description: '只预览,不发送取消请求(默认否)' },
|
|
680
|
+
},
|
|
681
|
+
timeoutMs: 180000,
|
|
682
|
+
output: {
|
|
683
|
+
schema: { type: 'string' },
|
|
684
|
+
render: (_args, value) => textBlock(value),
|
|
685
|
+
},
|
|
686
|
+
async execute(args) {
|
|
687
|
+
const jobId = String(args.job_id ?? '').trim()
|
|
688
|
+
if (!jobId) return MISSING_HINTS.job_id
|
|
689
|
+
const target = await resolveTarget(args)
|
|
690
|
+
if (target.error) return target.error
|
|
691
|
+
const argv = [...target.argv]
|
|
692
|
+
if (args.dry_run === true) argv.push('--dry-run')
|
|
693
|
+
argv.push('cancel', jobId)
|
|
694
|
+
return scnetJobResult(await runScnet(argv), '取消作业')
|
|
695
|
+
},
|
|
696
|
+
}),
|
|
697
|
+
)
|
|
698
|
+
|
|
699
|
+
ctx.tools.register(
|
|
700
|
+
defineTool({
|
|
701
|
+
name: 'scnet_file_transfer',
|
|
702
|
+
description:
|
|
703
|
+
'在本地与 SCNet 区域共享存储之间传输一个文件。direction=upload 或 download;upload 的 remote_dir 是目录而不是文件名。默认不覆盖,执行前应确认区域和路径,dry_run=true 只预览。',
|
|
704
|
+
parameters: {
|
|
705
|
+
direction: { type: 'string', required: true, description: 'upload 或 download' },
|
|
706
|
+
local_path: { type: 'string', required: true, description: '本地文件路径' },
|
|
707
|
+
remote_dir: { type: 'string', description: 'upload 时必填:区域共享存储中的绝对目录;文件名由 local_path 保留' },
|
|
708
|
+
remote_path: { type: 'string', description: 'download:区域共享存储中的绝对文件路径' },
|
|
709
|
+
backend: { type: 'string', description: 'openapi(默认)或 ssh' },
|
|
710
|
+
region: { type: 'string', description: 'OpenAPI 区域;省略时使用默认区域' },
|
|
711
|
+
cluster: { type: 'string', description: 'SSH 集群短名;多个 profile 时必须指定' },
|
|
712
|
+
overwrite: { type: 'boolean', description: '是否覆盖已有目标,默认否' },
|
|
713
|
+
dry_run: { type: 'boolean', description: '只预览,不传输(默认否)' },
|
|
714
|
+
},
|
|
715
|
+
output: {
|
|
716
|
+
schema: { type: 'string' },
|
|
717
|
+
render: (_args, value) => textBlock(value),
|
|
718
|
+
},
|
|
719
|
+
async execute(args) {
|
|
720
|
+
const direction = String(args.direction ?? '').trim().toLowerCase()
|
|
721
|
+
if (direction !== 'upload' && direction !== 'download') {
|
|
722
|
+
return 'direction 只支持 upload 或 download。'
|
|
723
|
+
}
|
|
724
|
+
const localPath = String(args.local_path ?? '').trim()
|
|
725
|
+
const remoteDir = String(args.remote_dir ?? '').trim()
|
|
726
|
+
const remotePath = String(args.remote_path ?? '').trim()
|
|
727
|
+
if (!localPath) return '缺少必填参数 local_path。'
|
|
728
|
+
if (direction === 'upload' && !remoteDir) return '缺少必填参数 remote_dir(目录,不是文件名)。'
|
|
729
|
+
if (direction === 'download' && !remotePath) return '缺少必填参数 remote_path。'
|
|
730
|
+
const target = await resolveTarget(args)
|
|
731
|
+
if (target.error) return target.error
|
|
732
|
+
const argv = [...target.argv]
|
|
733
|
+
if (args.dry_run === true) argv.push('--dry-run')
|
|
734
|
+
if (direction === 'upload') {
|
|
735
|
+
argv.push('upload', localPath, remoteDir)
|
|
736
|
+
} else {
|
|
737
|
+
argv.push('download', remotePath, localPath)
|
|
738
|
+
}
|
|
739
|
+
if (args.overwrite === true) argv.push('--cover')
|
|
740
|
+
return scnetResult(
|
|
741
|
+
await runScnet(argv, 180000),
|
|
742
|
+
direction === 'upload' ? '上传文件' : '下载文件',
|
|
743
|
+
)
|
|
744
|
+
},
|
|
745
|
+
}),
|
|
746
|
+
)
|
|
747
|
+
|
|
176
748
|
ctx.tools.register(
|
|
177
749
|
defineTool({
|
|
178
750
|
name: 'scnet_file_list',
|
|
@@ -486,3 +1058,15 @@ export function apply(ctx) {
|
|
|
486
1058
|
}),
|
|
487
1059
|
)
|
|
488
1060
|
}
|
|
1061
|
+
|
|
1062
|
+
// 仅供 tests/ 使用:把作业生命周期里的纯函数暴露出来做单元测试。
|
|
1063
|
+
export const __testables = {
|
|
1064
|
+
MISSING_HINTS,
|
|
1065
|
+
SUBMIT_REQUIRED_ORDER,
|
|
1066
|
+
errorTextFrom,
|
|
1067
|
+
firstMissingRequired,
|
|
1068
|
+
parseWalltimeSeconds,
|
|
1069
|
+
pickBackend,
|
|
1070
|
+
queueCandidates,
|
|
1071
|
+
translateScnetError,
|
|
1072
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-scnet",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.6.1",
|
|
4
4
|
"description": "Community DeepSeek Harness bundle for SCNet SSH and OpenAPI jobs, files, Notebook resources, Slurm diagnostics, and Hygon DCU/DTK guidance.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./index.mjs",
|
package/skills/scnet-hpc/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.
|
|
1
|
+
0.6.1
|
|
@@ -110,6 +110,21 @@ python3 scripts/scnet.py --backend openapi --region <id> \
|
|
|
110
110
|
Downloads refuse to replace an existing local path unless `--cover` is supplied. Uploads send
|
|
111
111
|
`uncover` by default and require `--cover` for replacement.
|
|
112
112
|
|
|
113
|
+
For uploads, the second argument is always a remote **directory**, not a complete filename.
|
|
114
|
+
The uploaded filename comes from the local file. For example, uploading `mesh.bin` to
|
|
115
|
+
`/public/home/alice/mesh` creates `/public/home/alice/mesh/mesh.bin`; passing
|
|
116
|
+
`/public/home/alice/mesh.bin` as the second argument asks the API to use `mesh.bin` as a
|
|
117
|
+
directory name.
|
|
118
|
+
|
|
119
|
+
List active or historical jobs without supplying a job ID:
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
123
|
+
jobs --scope active --limit 20
|
|
124
|
+
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
125
|
+
jobs --scope history --days 30 --limit 20
|
|
126
|
+
```
|
|
127
|
+
|
|
113
128
|
Uploads larger than 8 MiB automatically use the documented burst/merge flow. Override the
|
|
114
129
|
threshold and chunk size with `--chunk-size <bytes>`; use `--chunk-size 0` to force ordinary
|
|
115
130
|
multipart upload. The current implementation keeps one chunk in memory at a time.
|
|
@@ -123,6 +123,11 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
123
123
|
job = subparsers.add_parser("job", help="show one job")
|
|
124
124
|
job.add_argument("job_id")
|
|
125
125
|
|
|
126
|
+
jobs = subparsers.add_parser("jobs", help="list active or historical jobs")
|
|
127
|
+
jobs.add_argument("--scope", choices=("active", "history"), default="active")
|
|
128
|
+
jobs.add_argument("--limit", type=int, default=20)
|
|
129
|
+
jobs.add_argument("--days", type=int, default=30)
|
|
130
|
+
|
|
126
131
|
logs = subparsers.add_parser("logs", help="read a log file")
|
|
127
132
|
logs.add_argument("--path", required=True)
|
|
128
133
|
logs.add_argument("--lines", type=int, default=200)
|
|
@@ -166,9 +171,9 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
166
171
|
default="name",
|
|
167
172
|
)
|
|
168
173
|
|
|
169
|
-
upload = subparsers.add_parser("upload", help="upload one file")
|
|
174
|
+
upload = subparsers.add_parser("upload", help="upload one file into a remote directory")
|
|
170
175
|
upload.add_argument("local_path")
|
|
171
|
-
upload.add_argument("
|
|
176
|
+
upload.add_argument("remote_dir", help="remote destination directory, not a filename")
|
|
172
177
|
upload.add_argument("--cover", action="store_true")
|
|
173
178
|
upload.add_argument(
|
|
174
179
|
"--chunk-size",
|
|
@@ -6,6 +6,7 @@ import json
|
|
|
6
6
|
import mimetypes
|
|
7
7
|
import re
|
|
8
8
|
import uuid
|
|
9
|
+
from datetime import datetime, timedelta, timezone
|
|
9
10
|
from pathlib import Path
|
|
10
11
|
from typing import Any, Mapping
|
|
11
12
|
from urllib.error import HTTPError, URLError
|
|
@@ -29,6 +30,12 @@ STATUS_MAP = {
|
|
|
29
30
|
"statC": "COMPLETED",
|
|
30
31
|
"statW": "WAITING",
|
|
31
32
|
"statX": "OTHER",
|
|
33
|
+
"statDE": "CANCELLED",
|
|
34
|
+
"statD": "FAILED",
|
|
35
|
+
"statEX": "ABNORMAL",
|
|
36
|
+
"statT": "TIMEOUT",
|
|
37
|
+
"statN": "NODE_ERROR",
|
|
38
|
+
"statRQ": "REQUEUED",
|
|
32
39
|
}
|
|
33
40
|
WALLTIME_RE = re.compile(r"^(?:[0-9]+-)?[0-9]{1,3}:[0-9]{2}:[0-9]{2}$")
|
|
34
41
|
|
|
@@ -44,7 +51,12 @@ def _bounded_int(options: Mapping[str, Any], name: str, minimum: int) -> int:
|
|
|
44
51
|
|
|
45
52
|
|
|
46
53
|
def normalize_job(item: Mapping[str, Any]) -> dict[str, Any]:
|
|
47
|
-
raw_state =
|
|
54
|
+
raw_state = (
|
|
55
|
+
item.get("jobStatus")
|
|
56
|
+
or item.get("jobState")
|
|
57
|
+
or item.get("state")
|
|
58
|
+
or ""
|
|
59
|
+
)
|
|
48
60
|
return {
|
|
49
61
|
"job_id": str(item.get("jobId") or item.get("job_id") or ""),
|
|
50
62
|
"name": item.get("jobName") or item.get("name"),
|
|
@@ -76,6 +88,7 @@ class OpenAPIBackend(Backend):
|
|
|
76
88
|
"queues",
|
|
77
89
|
"limits",
|
|
78
90
|
"job",
|
|
91
|
+
"jobs",
|
|
79
92
|
"logs",
|
|
80
93
|
"submit",
|
|
81
94
|
"cancel",
|
|
@@ -335,18 +348,134 @@ class OpenAPIBackend(Backend):
|
|
|
335
348
|
|
|
336
349
|
def op_job(self, options: Mapping[str, Any]) -> Any:
|
|
337
350
|
job_id = str(require_option(options, "job_id"))
|
|
338
|
-
hpc_url, token,
|
|
351
|
+
hpc_url, token, scheduler_id, _, _ = self._hpc_context(options)
|
|
352
|
+
realtime_error: str | None = None
|
|
353
|
+
data: Any = None
|
|
354
|
+
try:
|
|
355
|
+
data = self._json_request(
|
|
356
|
+
"GET",
|
|
357
|
+
service_endpoint(hpc_url, "hpc", f"/openapi/v2/jobs/{quote(job_id)}"),
|
|
358
|
+
token=token,
|
|
359
|
+
)
|
|
360
|
+
except BackendError as exc:
|
|
361
|
+
if not self._is_missing_job_error(str(exc)):
|
|
362
|
+
raise
|
|
363
|
+
realtime_error = str(exc)
|
|
364
|
+
|
|
365
|
+
if isinstance(data, dict):
|
|
366
|
+
candidate = data
|
|
367
|
+
if candidate.get("jobId") or candidate.get("job_id"):
|
|
368
|
+
result = normalize_job(candidate)
|
|
369
|
+
result["source"] = "realtime"
|
|
370
|
+
if options.get("raw"):
|
|
371
|
+
result["raw"] = candidate
|
|
372
|
+
return result
|
|
373
|
+
|
|
374
|
+
history = self._history_job(
|
|
375
|
+
hpc_url,
|
|
376
|
+
token,
|
|
377
|
+
scheduler_id,
|
|
378
|
+
job_id,
|
|
379
|
+
options,
|
|
380
|
+
)
|
|
381
|
+
if not isinstance(history, dict) or not (
|
|
382
|
+
history.get("jobId") or history.get("job_id")
|
|
383
|
+
):
|
|
384
|
+
detail = "realtime and history endpoints returned no job record"
|
|
385
|
+
if realtime_error:
|
|
386
|
+
detail += f" ({realtime_error})"
|
|
387
|
+
raise BackendError(detail)
|
|
388
|
+
result = normalize_job(history)
|
|
389
|
+
result["source"] = "history"
|
|
390
|
+
if options.get("raw"):
|
|
391
|
+
result["raw"] = history
|
|
392
|
+
return result
|
|
393
|
+
|
|
394
|
+
@staticmethod
|
|
395
|
+
def _is_missing_job_error(text: str) -> bool:
|
|
396
|
+
value = text.lower()
|
|
397
|
+
return any(
|
|
398
|
+
marker in value
|
|
399
|
+
for marker in (
|
|
400
|
+
"unexpected data shape",
|
|
401
|
+
"http 404",
|
|
402
|
+
"not found",
|
|
403
|
+
"does not exist",
|
|
404
|
+
"不存在",
|
|
405
|
+
)
|
|
406
|
+
)
|
|
407
|
+
|
|
408
|
+
def _history_job(
|
|
409
|
+
self,
|
|
410
|
+
hpc_url: str,
|
|
411
|
+
token: str,
|
|
412
|
+
scheduler_id: str,
|
|
413
|
+
job_id: str,
|
|
414
|
+
options: Mapping[str, Any],
|
|
415
|
+
) -> Any:
|
|
416
|
+
if not scheduler_id:
|
|
417
|
+
raise BackendError("history lookup requires a scheduler id")
|
|
418
|
+
path = f"/openapi/v2/historyjobs/{quote(scheduler_id)}/{quote(job_id)}"
|
|
419
|
+
acct_time = str(options.get("acct_time") or "").strip()
|
|
420
|
+
if acct_time:
|
|
421
|
+
path += "?" + urlencode({"acctTime": acct_time})
|
|
422
|
+
return self._json_request(
|
|
423
|
+
"GET",
|
|
424
|
+
service_endpoint(hpc_url, "hpc", path),
|
|
425
|
+
token=token,
|
|
426
|
+
)
|
|
427
|
+
|
|
428
|
+
def op_jobs(self, options: Mapping[str, Any]) -> Any:
|
|
429
|
+
hpc_url, token, scheduler_id, _, _ = self._hpc_context(options)
|
|
430
|
+
scope = str(options.get("scope") or "active").strip().lower()
|
|
431
|
+
if scope not in {"active", "history"}:
|
|
432
|
+
raise BackendError("scope must be active or history")
|
|
433
|
+
limit = _bounded_int(options, "limit", 1)
|
|
434
|
+
if limit > 100:
|
|
435
|
+
raise BackendError("limit must be <= 100")
|
|
436
|
+
if scope == "active":
|
|
437
|
+
path = "/openapi/v2/jobs"
|
|
438
|
+
params = {
|
|
439
|
+
"strClusterIDList": scheduler_id,
|
|
440
|
+
"start": 0,
|
|
441
|
+
"limit": limit,
|
|
442
|
+
}
|
|
443
|
+
else:
|
|
444
|
+
days = _bounded_int(options, "days", 1)
|
|
445
|
+
if days > 90:
|
|
446
|
+
raise BackendError("days must be <= 90")
|
|
447
|
+
end = datetime.now(timezone.utc)
|
|
448
|
+
start = end - timedelta(days=days)
|
|
449
|
+
path = "/openapi/v2/historyjobs"
|
|
450
|
+
params = {
|
|
451
|
+
"strClusterNameList": scheduler_id,
|
|
452
|
+
"timeType": "CUSTOM",
|
|
453
|
+
"isQueryByQueueTime": "false",
|
|
454
|
+
"start": 0,
|
|
455
|
+
"limit": limit,
|
|
456
|
+
"startTime": start.strftime("%Y-%m-%d %H:%M:%S"),
|
|
457
|
+
"endTime": end.strftime("%Y-%m-%d %H:%M:%S"),
|
|
458
|
+
}
|
|
339
459
|
data = self._json_request(
|
|
340
460
|
"GET",
|
|
341
|
-
service_endpoint(hpc_url, "hpc",
|
|
461
|
+
service_endpoint(hpc_url, "hpc", path) + "?" + urlencode(params),
|
|
342
462
|
token=token,
|
|
343
463
|
)
|
|
344
|
-
if
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
464
|
+
if isinstance(data, dict):
|
|
465
|
+
items = data.get("items") or data.get("list") or data.get("rows") or []
|
|
466
|
+
total = data.get("total")
|
|
467
|
+
else:
|
|
468
|
+
items = data or []
|
|
469
|
+
total = None
|
|
470
|
+
if not isinstance(items, list):
|
|
471
|
+
raise BackendError("job list endpoint returned an unexpected data shape")
|
|
472
|
+
result = [normalize_job(item) for item in items if isinstance(item, dict)]
|
|
473
|
+
payload = {
|
|
474
|
+
"scope": scope,
|
|
475
|
+
"items": result,
|
|
476
|
+
"total": total if total is not None else len(result),
|
|
477
|
+
}
|
|
478
|
+
return {"data": payload, "raw": data} if options.get("raw") else payload
|
|
350
479
|
|
|
351
480
|
def op_logs(self, options: Mapping[str, Any]) -> Any:
|
|
352
481
|
path = str(require_option(options, "path"))
|
|
@@ -580,14 +709,17 @@ class OpenAPIBackend(Backend):
|
|
|
580
709
|
|
|
581
710
|
def op_upload(self, options: Mapping[str, Any]) -> Any:
|
|
582
711
|
local_path = Path(str(require_option(options, "local_path"))).expanduser()
|
|
583
|
-
|
|
712
|
+
remote_dir = str(
|
|
713
|
+
options.get("remote_dir")
|
|
714
|
+
or require_option(options, "remote_path")
|
|
715
|
+
)
|
|
584
716
|
if not local_path.is_file():
|
|
585
717
|
raise BackendError(f"local file does not exist: {local_path}")
|
|
586
718
|
if (
|
|
587
|
-
not
|
|
588
|
-
or "\n" in
|
|
589
|
-
or "\r" in
|
|
590
|
-
or "\x00" in
|
|
719
|
+
not remote_dir.startswith("/")
|
|
720
|
+
or "\n" in remote_dir
|
|
721
|
+
or "\r" in remote_dir
|
|
722
|
+
or "\x00" in remote_dir
|
|
591
723
|
):
|
|
592
724
|
raise BackendError("remote upload directory must be absolute")
|
|
593
725
|
chunk_size = int(options.get("chunk_size") or 0)
|
|
@@ -596,7 +728,7 @@ class OpenAPIBackend(Backend):
|
|
|
596
728
|
if chunk_size and chunk_size < 1024 * 1024:
|
|
597
729
|
raise BackendError("chunk size must be at least 1 MiB or 0")
|
|
598
730
|
if chunk_size and local_path.stat().st_size > chunk_size:
|
|
599
|
-
return self._op_chunked_upload(local_path,
|
|
731
|
+
return self._op_chunked_upload(local_path, remote_dir, options, chunk_size)
|
|
600
732
|
efile_url, token, _ = self._efile_context(options)
|
|
601
733
|
content_type = mimetypes.guess_type(local_path.name)[0] or "application/octet-stream"
|
|
602
734
|
self._multipart_request(
|
|
@@ -604,13 +736,17 @@ class OpenAPIBackend(Backend):
|
|
|
604
736
|
token,
|
|
605
737
|
{
|
|
606
738
|
"cover": "cover" if options.get("cover") else "uncover",
|
|
607
|
-
"path":
|
|
739
|
+
"path": remote_dir,
|
|
608
740
|
},
|
|
609
741
|
local_path.name,
|
|
610
742
|
local_path.read_bytes(),
|
|
611
743
|
content_type,
|
|
612
744
|
)
|
|
613
|
-
return {
|
|
745
|
+
return {
|
|
746
|
+
"local_path": str(local_path),
|
|
747
|
+
"remote_dir": remote_dir,
|
|
748
|
+
"remote_filename": local_path.name,
|
|
749
|
+
}
|
|
614
750
|
|
|
615
751
|
def _multipart_request(
|
|
616
752
|
self,
|
|
@@ -220,7 +220,10 @@ class SSHBackend(Backend):
|
|
|
220
220
|
|
|
221
221
|
def op_upload(self, options: Mapping[str, Any]) -> Any:
|
|
222
222
|
local_path = Path(str(require_option(options, "local_path"))).expanduser()
|
|
223
|
-
remote_path = str(
|
|
223
|
+
remote_path = str(
|
|
224
|
+
options.get("remote_dir")
|
|
225
|
+
or require_option(options, "remote_path")
|
|
226
|
+
)
|
|
224
227
|
if not local_path.is_file():
|
|
225
228
|
raise BackendError(f"local file does not exist: {local_path}")
|
|
226
229
|
if not remote_path.startswith("/") or "\n" in remote_path:
|
|
@@ -7,6 +7,7 @@ import hmac
|
|
|
7
7
|
import json
|
|
8
8
|
import os
|
|
9
9
|
import time
|
|
10
|
+
from pathlib import Path
|
|
10
11
|
from typing import Any, Mapping
|
|
11
12
|
from urllib.error import HTTPError, URLError
|
|
12
13
|
from urllib.parse import urlencode, urlsplit, urlunsplit
|
|
@@ -45,6 +46,8 @@ def service_endpoint(base_url: str, service: str, suffix: str) -> str:
|
|
|
45
46
|
class SCNetClient:
|
|
46
47
|
"""One authenticated OpenAPI client shared by service-specific domains."""
|
|
47
48
|
|
|
49
|
+
TOKEN_CACHE_TTL = 4 * 60 * 60
|
|
50
|
+
|
|
48
51
|
def __init__(self, context: BackendContext):
|
|
49
52
|
self.context = context
|
|
50
53
|
self._regions_cache: list[dict[str, Any]] | None = None
|
|
@@ -55,6 +58,79 @@ class SCNetClient:
|
|
|
55
58
|
value = os.environ.get(name)
|
|
56
59
|
return value if value not in (None, "") else default
|
|
57
60
|
|
|
61
|
+
@classmethod
|
|
62
|
+
def _token_cache_paths(cls) -> tuple[Path, Path]:
|
|
63
|
+
root = Path(cls.env("XDG_CACHE_HOME", str(Path.home() / ".cache"))) / "scnet-hpc"
|
|
64
|
+
return root / "openapi-regions.json", root / "openapi-regions.lock"
|
|
65
|
+
|
|
66
|
+
@staticmethod
|
|
67
|
+
def _credential_fingerprint(user: str, access_key: str, secret_key: str) -> str:
|
|
68
|
+
value = "\0".join((user, access_key, secret_key)).encode("utf-8")
|
|
69
|
+
return hashlib.sha256(value).hexdigest()
|
|
70
|
+
|
|
71
|
+
@classmethod
|
|
72
|
+
def _read_token_cache(cls, fingerprint: str) -> list[dict[str, Any]] | None:
|
|
73
|
+
path, _ = cls._token_cache_paths()
|
|
74
|
+
try:
|
|
75
|
+
payload = json.loads(path.read_text(encoding="utf-8"))
|
|
76
|
+
if (
|
|
77
|
+
payload.get("fingerprint") != fingerprint
|
|
78
|
+
or time.time() - float(payload.get("created_at", 0)) > cls.TOKEN_CACHE_TTL
|
|
79
|
+
or not isinstance(payload.get("regions"), list)
|
|
80
|
+
):
|
|
81
|
+
return None
|
|
82
|
+
return [item for item in payload["regions"] if isinstance(item, dict)]
|
|
83
|
+
except (OSError, TypeError, ValueError):
|
|
84
|
+
return None
|
|
85
|
+
|
|
86
|
+
@classmethod
|
|
87
|
+
def _write_token_cache(
|
|
88
|
+
cls, fingerprint: str, regions: list[dict[str, Any]]
|
|
89
|
+
) -> None:
|
|
90
|
+
path, _ = cls._token_cache_paths()
|
|
91
|
+
path.parent.mkdir(parents=True, mode=0o700, exist_ok=True)
|
|
92
|
+
temporary = path.with_name(f".{path.name}.{os.getpid()}.tmp")
|
|
93
|
+
temporary.write_text(
|
|
94
|
+
json.dumps(
|
|
95
|
+
{
|
|
96
|
+
"created_at": time.time(),
|
|
97
|
+
"fingerprint": fingerprint,
|
|
98
|
+
"regions": regions,
|
|
99
|
+
},
|
|
100
|
+
ensure_ascii=False,
|
|
101
|
+
),
|
|
102
|
+
encoding="utf-8",
|
|
103
|
+
)
|
|
104
|
+
os.chmod(temporary, 0o600)
|
|
105
|
+
os.replace(temporary, path)
|
|
106
|
+
|
|
107
|
+
@classmethod
|
|
108
|
+
def _acquire_token_cache_lock(cls) -> Path | None:
|
|
109
|
+
_, lock = cls._token_cache_paths()
|
|
110
|
+
lock.parent.mkdir(parents=True, mode=0o700, exist_ok=True)
|
|
111
|
+
for _ in range(50):
|
|
112
|
+
try:
|
|
113
|
+
fd = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
|
114
|
+
os.close(fd)
|
|
115
|
+
return lock
|
|
116
|
+
except FileExistsError:
|
|
117
|
+
try:
|
|
118
|
+
if time.time() - lock.stat().st_mtime > 30:
|
|
119
|
+
lock.unlink()
|
|
120
|
+
continue
|
|
121
|
+
except OSError:
|
|
122
|
+
pass
|
|
123
|
+
time.sleep(0.2)
|
|
124
|
+
return None
|
|
125
|
+
|
|
126
|
+
@staticmethod
|
|
127
|
+
def _release_token_cache_lock(lock: Path | None) -> None:
|
|
128
|
+
if lock is not None:
|
|
129
|
+
try:
|
|
130
|
+
lock.unlink()
|
|
131
|
+
except FileNotFoundError:
|
|
132
|
+
pass
|
|
133
|
+
|
|
58
134
|
def request(
|
|
59
135
|
self,
|
|
60
136
|
method: str,
|
|
@@ -150,25 +226,41 @@ class SCNetClient:
|
|
|
150
226
|
"OpenAPI credentials are not configured; run setup or set "
|
|
151
227
|
+ ", ".join(missing)
|
|
152
228
|
)
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
229
|
+
fingerprint = self._credential_fingerprint(user, access_key, secret_key)
|
|
230
|
+
if not refresh:
|
|
231
|
+
cached = self._read_token_cache(fingerprint)
|
|
232
|
+
if cached:
|
|
233
|
+
self._regions_cache = cached
|
|
234
|
+
return cached
|
|
235
|
+
lock = self._acquire_token_cache_lock()
|
|
236
|
+
try:
|
|
237
|
+
if not refresh:
|
|
238
|
+
cached = self._read_token_cache(fingerprint)
|
|
239
|
+
if cached:
|
|
240
|
+
self._regions_cache = cached
|
|
241
|
+
return cached
|
|
242
|
+
timestamp = str(int(time.time()))
|
|
243
|
+
signature = canonical_signature(access_key, timestamp, user, secret_key)
|
|
244
|
+
auth_base = self.env("SCNET_OPENAPI_AUTH_BASE", "https://api.scnet.cn")
|
|
245
|
+
data = self.request(
|
|
246
|
+
"POST",
|
|
247
|
+
auth_base.rstrip("/") + "/api/user/v3/tokens",
|
|
248
|
+
headers={
|
|
249
|
+
"user": user,
|
|
250
|
+
"accessKey": access_key,
|
|
251
|
+
"signature": signature,
|
|
252
|
+
"timestamp": timestamp,
|
|
253
|
+
},
|
|
254
|
+
)
|
|
255
|
+
if not isinstance(data, list):
|
|
256
|
+
raise BackendError("token endpoint returned an unexpected data shape")
|
|
257
|
+
self._regions_cache = [
|
|
258
|
+
item for item in data if isinstance(item, dict)
|
|
259
|
+
]
|
|
260
|
+
self._write_token_cache(fingerprint, self._regions_cache)
|
|
261
|
+
return self._regions_cache
|
|
262
|
+
finally:
|
|
263
|
+
self._release_token_cache_lock(lock)
|
|
172
264
|
|
|
173
265
|
def select_region(
|
|
174
266
|
self, requested: str | None
|