dsh-scnet 0.6.1 → 0.6.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +20 -3
- package/README_CN.md +17 -2
- package/docs/testing.md +3 -0
- package/docs/testing.zh-CN.md +16 -13
- package/index.mjs +119 -14
- package/package.json +1 -1
- package/skills/scnet-hpc/VERSION +1 -1
- package/skills/scnet-hpc/references/openapi.md +21 -0
- package/skills/scnet-hpc/scripts/scnet.py +80 -1
- package/skills/scnet-hpc/scripts/scnet_backends/openapi.py +114 -50
- package/skills/scnet-hpc/scripts/scnet_sdk/client.py +32 -18
package/README.md
CHANGED
|
@@ -4,11 +4,25 @@
|
|
|
4
4
|
|
|
5
5
|
DSH-SCNet (`dsh-scnet` on npm) is a community-maintained DSH bundle for operating Supercomputing Network (SCNet). It packages the source [`scnet-hpc`](https://github.com/lql341/scnet-hpc) Skill, profile-aware utilities, and bounded tools for SSH, Slurm, OpenAPI files, regional resources, and Notebook discovery.
|
|
6
6
|
|
|
7
|
-
Current release: **0.6.
|
|
7
|
+
Current release: **0.6.3**
|
|
8
8
|
|
|
9
9
|
> This is an independent community project. It is compatible with DeepSeek Harness but is not an official DeepSeek product and does not imply endorsement, partnership, or authorization by DeepSeek.
|
|
10
10
|
|
|
11
|
-
## What's new in 0.6.
|
|
11
|
+
## What's new in 0.6.3
|
|
12
|
+
|
|
13
|
+
- Added read-only account and resource summaries plus bounded `scnet_job_wait`.
|
|
14
|
+
- Submission queue preflight now fails closed and never silently selects a queue.
|
|
15
|
+
- Download paths can default to the remote filename; ordinary and chunked uploads return the same
|
|
16
|
+
directory/filename result shape.
|
|
17
|
+
|
|
18
|
+
## 0.6.2 highlights
|
|
19
|
+
|
|
20
|
+
- Completed jobs now resolve through the fast filtered history-list endpoint rather than timing
|
|
21
|
+
out on the empty history-detail response.
|
|
22
|
+
- Historical field aliases such as `workdir` are normalized for log-path discovery.
|
|
23
|
+
- OpenAPI operations continue without persistent caching when the cache directory is read-only.
|
|
24
|
+
|
|
25
|
+
## 0.6.1 highlights
|
|
12
26
|
|
|
13
27
|
- Job details now fall back to the history endpoint when realtime records expire, with complete
|
|
14
28
|
normalized terminal states.
|
|
@@ -60,11 +74,14 @@ See [Installation](./docs/installation.md) for npm, GitHub, and local-checkout w
|
|
|
60
74
|
| `scnet_run_compute_probe` | Submit a minimal compute-node capability probe |
|
|
61
75
|
| `scnet_status` | Read the saved backend, SSH profile, and OpenAPI configuration |
|
|
62
76
|
| `scnet_openapi_regions` | List authorized OpenAPI regions without exposing tokens |
|
|
77
|
+
| `scnet_account_summary` | Read account status and balance |
|
|
78
|
+
| `scnet_resource_summary` | Summarize regional queues and resource limits |
|
|
63
79
|
| `scnet_job_queues` | Query regional Slurm queues and live resource availability |
|
|
64
80
|
| `scnet_job_list` | List active or historical jobs with compact normalized records |
|
|
65
81
|
| `scnet_limits` | Read user and scheduler resource limits |
|
|
66
82
|
| `scnet_submit_job` | Submit a job through OpenAPI or SSH; `dry_run=true` only previews the request |
|
|
67
83
|
| `scnet_job_show` | Read one job's state, resources, and log paths |
|
|
84
|
+
| `scnet_job_wait` | Wait for a job to reach a terminal state with a bounded timeout |
|
|
68
85
|
| `scnet_job_logs` | Read job logs by explicit path, or by `job_id` + `work_dir` |
|
|
69
86
|
| `scnet_job_cancel` | Cancel a job; `dry_run=true` only previews the request |
|
|
70
87
|
| `scnet_file_list` | List shared-storage files through the common file API |
|
|
@@ -113,7 +130,7 @@ synchronizes the skill and version into `dsh-scnet` and `codex-scnet-hpc`. Publi
|
|
|
113
130
|
`dsh-scnet` npm package only after that downstream change is reviewed.
|
|
114
131
|
|
|
115
132
|
For a synchronized release, run the package validation on `main`, then trigger the repository's
|
|
116
|
-
`Publish DSH-SCNet` workflow with the exact value from `package.json` (currently `0.6.
|
|
133
|
+
`Publish DSH-SCNet` workflow with the exact value from `package.json` (currently `0.6.3`). The
|
|
117
134
|
workflow verifies the package/Skill version match, runs validation, creates the tarball, publishes
|
|
118
135
|
stable releases with the `latest` tag, and creates the matching Git tag. Do not bump only
|
|
119
136
|
`package.json`: `skills/scnet-hpc/VERSION`, the package version, and both README release markers
|
package/README_CN.md
CHANGED
|
@@ -4,10 +4,22 @@
|
|
|
4
4
|
|
|
5
5
|
<p align="center">在 DeepSeek Harness 里操作超算互联网(SCNet):配置 SSH、管理 Slurm 作业、查询 OpenAPI 区域和文件、检查 Notebook 资源与实例,并保留海光 DCU/DTK 诊断能力。</p>
|
|
6
6
|
|
|
7
|
-
当前版本:**0.6.
|
|
7
|
+
当前版本:**0.6.3**
|
|
8
8
|
|
|
9
9
|
> 本项目是独立维护的社区项目,与 DeepSeek Harness 兼容,但不是 DeepSeek 官方产品,也不表示 DeepSeek 对本项目提供背书、合作或授权。
|
|
10
10
|
|
|
11
|
+
## 0.6.3 更新
|
|
12
|
+
|
|
13
|
+
- 新增只读账户摘要、资源摘要和有界等待作业工具 `scnet_job_wait`。
|
|
14
|
+
- 提交前队列预检失败会停止提交,不再静默选择队列。
|
|
15
|
+
- 下载时可省略本地路径自动使用远端文件名;普通上传和分片上传返回统一结果结构。
|
|
16
|
+
|
|
17
|
+
## 0.6.2 更新
|
|
18
|
+
|
|
19
|
+
- 已结束作业改用快速的历史列表过滤查询,不再卡在返回空数据的历史详情接口。
|
|
20
|
+
- 兼容历史记录中的 `workdir` 等字段变体,可继续推导日志路径。
|
|
21
|
+
- 缓存目录只读时自动降级为无持久缓存,OpenAPI 主功能不再被缓存故障阻断。
|
|
22
|
+
|
|
11
23
|
## 0.6.1 更新
|
|
12
24
|
|
|
13
25
|
- 作业实时记录过期时自动回退历史接口,并补全终态映射。
|
|
@@ -65,11 +77,14 @@ git 源、本地目录安装以及验证步骤见[安装说明](./docs/installat
|
|
|
65
77
|
| scnet_run_compute_probe | 在计算节点运行最小能力探针 |
|
|
66
78
|
| scnet_status | 只读查看 backend、SSH profile 和 OpenAPI 配置 |
|
|
67
79
|
| scnet_openapi_regions | 列出授权 OpenAPI 区域,不暴露 token |
|
|
80
|
+
| scnet_account_summary | 查询账户状态和余额 |
|
|
81
|
+
| scnet_resource_summary | 汇总区域队列和资源限制 |
|
|
68
82
|
| scnet_job_queues | 查询区域 Slurm 队列和实时空闲资源 |
|
|
69
83
|
| scnet_job_list | 紧凑列出当前或历史作业 |
|
|
70
84
|
| scnet_limits | 查询用户和调度器资源限制 |
|
|
71
85
|
| scnet_submit_job | 通过 OpenAPI 或 SSH 提交作业;dry_run=true 只预览请求 |
|
|
72
86
|
| scnet_job_show | 查询单个作业的状态、资源与日志路径 |
|
|
87
|
+
| scnet_job_wait | 在限定时间内等待作业进入终态 |
|
|
73
88
|
| scnet_job_logs | 按显式 path,或按 job_id + work_dir 读取作业日志 |
|
|
74
89
|
| scnet_job_cancel | 取消作业;dry_run=true 只预览请求 |
|
|
75
90
|
| scnet_file_list | 通过通用文件 API 列出共享存储 |
|
|
@@ -126,7 +141,7 @@ git 源、本地目录安装以及验证步骤见[安装说明](./docs/installat
|
|
|
126
141
|
才能发布对应版本的 `dsh-scnet` npm 包。
|
|
127
142
|
|
|
128
143
|
下游同步合并到 `main` 后,先运行本地校验,再手动触发仓库的 `Publish DSH-SCNet` workflow,
|
|
129
|
-
输入 `package.json` 中的精确版本(当前为 `0.6.
|
|
144
|
+
输入 `package.json` 中的精确版本(当前为 `0.6.3`)。workflow 会校验 package 与 Skill
|
|
130
145
|
版本一致,执行验证、生成 tarball、以稳定版 `latest` 标签发布,并创建对应 Git tag。
|
|
131
146
|
不要只修改 `package.json`;`skills/scnet-hpc/VERSION`、package 版本和两份 README 的版本
|
|
132
147
|
标记必须保持一致。
|
package/docs/testing.md
CHANGED
|
@@ -39,11 +39,14 @@ Confirm that the composed configuration includes:
|
|
|
39
39
|
| `scnet_generate_job` | Writes a profile-aware Slurm script in the current directory |
|
|
40
40
|
| `scnet_status` | Returns redacted local SCNet configuration |
|
|
41
41
|
| `scnet_openapi_regions` | Returns authorized regions without tokens |
|
|
42
|
+
| `scnet_account_summary` | Returns account status and balance |
|
|
43
|
+
| `scnet_resource_summary` | Returns queues and resource limits for a region |
|
|
42
44
|
| `scnet_job_queues` | Returns live regional Slurm queues |
|
|
43
45
|
| `scnet_job_list` | Lists active or historical jobs |
|
|
44
46
|
| `scnet_limits` | Returns user and scheduler resource limits |
|
|
45
47
|
| `scnet_submit_job` | Submits a job through OpenAPI or SSH; `dry_run=true` returns the pending request only |
|
|
46
48
|
| `scnet_job_show` | Returns one job's state, resources, and log paths |
|
|
49
|
+
| `scnet_job_wait` | Waits for a job to reach a terminal state with a bounded timeout |
|
|
47
50
|
| `scnet_job_logs` | Reads job logs by explicit path, or by `job_id` + `work_dir` |
|
|
48
51
|
| `scnet_job_cancel` | Cancels one job; `dry_run=true` returns the pending request only |
|
|
49
52
|
| `scnet_file_list` | Lists shared-storage files without modification |
|
package/docs/testing.zh-CN.md
CHANGED
|
@@ -60,23 +60,26 @@ DEEPSEEK_API_KEY: sk-你的key
|
|
|
60
60
|
| 7 | 用 scnet_run_compute_probe 跑计算节点探针 | 返回 PROBE_ 结果 | 高(提交作业并等待) |
|
|
61
61
|
| 8 | 用 scnet_status 查看当前配置 | 返回脱敏配置 | 只读 |
|
|
62
62
|
| 9 | 用 scnet_openapi_regions 查看授权区域 | 返回区域且不含 token | 只读 |
|
|
63
|
+
| 9a | 用 scnet_account_summary 查看账户摘要 | 返回账户状态和余额 | 只读 |
|
|
64
|
+
| 9b | 用 scnet_resource_summary 查看资源摘要 | 返回队列和资源限制 | 只读 |
|
|
63
65
|
| 10 | 用 scnet_job_queues 查看昆山队列 | 返回实时队列和空闲资源 | 只读 |
|
|
64
|
-
|
|
|
65
|
-
|
|
|
66
|
-
|
|
|
67
|
-
|
|
|
68
|
-
|
|
|
69
|
-
|
|
|
70
|
-
|
|
|
71
|
-
|
|
|
72
|
-
|
|
|
73
|
-
|
|
|
74
|
-
|
|
|
75
|
-
|
|
|
66
|
+
| 11 | 用 scnet_job_list 列出当前或历史作业 | 返回紧凑作业记录 | 只读 |
|
|
67
|
+
| 12 | 用 scnet_limits 查询资源限制 | 返回用户和调度器限制 | 只读 |
|
|
68
|
+
| 13 | 用 scnet_submit_job(dry_run=true)预览一次提交 | 返回将要发送的请求,不真正提交 | 只读(dry_run) |
|
|
69
|
+
| 14 | 用 scnet_submit_job 提交一个 5 秒小作业 | 返回 job_id、作业目录与日志路径 | 高(会消耗配额) |
|
|
70
|
+
| 15 | 用 scnet_job_show 查该 job_id | 返回状态与 stdout/stderr 路径 | 只读 |
|
|
71
|
+
| 15a | 用 scnet_job_wait 等待作业终态 | 最多等待限定时间并返回状态 | 只读 |
|
|
72
|
+
| 16 | 用 scnet_job_logs 读该作业日志 | 返回日志内容(同时验证 path 与 job_id+work_dir 两种模式结果一致) | 只读 |
|
|
73
|
+
| 17 | 用 scnet_job_cancel 取消一个作业 | 建议先用 dry_run=true 预览 | 高(变更) |
|
|
74
|
+
| 18 | 用 scnet_file_list 查看默认目录 | 返回共享存储文件 | 只读 |
|
|
75
|
+
| 19 | 用 scnet_file_transfer 传输一个文件 | 默认不覆盖目标 | 有副作用 |
|
|
76
|
+
| 20 | 用 scnet_notebook_regions 查看区域 | 返回 Notebook 可用区域 | 只读 |
|
|
77
|
+
| 21 | 用 scnet_notebook_resources 查看昆山资源 | 返回 CPU/GPU/DCU 资源 | 只读 |
|
|
78
|
+
| 22 | 用 scnet_notebook_list 查看实例 | 返回脱敏实例列表 | 只读 |
|
|
76
79
|
|
|
77
80
|
第 4 至 7 步只有在你有私钥和目标集群时才测,测前确认副作用。
|
|
78
81
|
|
|
79
|
-
第
|
|
82
|
+
第 14、17 步是变更操作,会真实占用或释放调度器资源:先用 `dry_run=true` 预览,确认返回里的
|
|
80
83
|
目标区域/集群,并且**不要在提交超时后盲目重试**(平台没有幂等键),先查作业状态。
|
|
81
84
|
|
|
82
85
|
Notebook 创建、启动、停止和释放没有作为 DSH deterministic tool 暴露。此类操作通过
|
package/index.mjs
CHANGED
|
@@ -182,6 +182,16 @@ function errorTextFrom(res) {
|
|
|
182
182
|
return matched ? matched[1].trim() : raw
|
|
183
183
|
}
|
|
184
184
|
|
|
185
|
+
function errorCodeFrom(res) {
|
|
186
|
+
const raw = String(res.stdout || '').trim() || String(res.stderr || '').trim()
|
|
187
|
+
try {
|
|
188
|
+
const parsed = JSON.parse(raw)
|
|
189
|
+
return parsed && typeof parsed.error_code === 'string' ? parsed.error_code : ''
|
|
190
|
+
} catch {
|
|
191
|
+
return ''
|
|
192
|
+
}
|
|
193
|
+
}
|
|
194
|
+
|
|
185
195
|
// 一次只给一条可执行的提示(不展示整套参数清单)
|
|
186
196
|
function translateScnetError(text) {
|
|
187
197
|
const value = String(text || '')
|
|
@@ -197,17 +207,29 @@ function translateScnetError(text) {
|
|
|
197
207
|
'若只想看日志,请直接给日志文件的 path;若刚提交不久,可稍后重试。'
|
|
198
208
|
)
|
|
199
209
|
}
|
|
210
|
+
if (/realtime and history endpoints returned no job record/i.test(value)) {
|
|
211
|
+
return '平台的实时和最近 30 天历史记录中都没有找到这个作业,请核对 job_id 和区域。'
|
|
212
|
+
}
|
|
200
213
|
return value
|
|
201
214
|
}
|
|
202
215
|
|
|
216
|
+
function translateScnetFailure(res) {
|
|
217
|
+
const code = errorCodeFrom(res)
|
|
218
|
+
if (code === 'AUTHENTICATION_FAILED') return 'SCNet 认证失败:请检查 OpenAPI 配置或重新运行 setup。'
|
|
219
|
+
if (code === 'QUEUE_QUERY_FAILED') return '队列查询失败,已停止提交;请先检查区域、调度器和凭据。'
|
|
220
|
+
if (code === 'NO_QUEUE_AVAILABLE') return '当前资源条件下没有可用队列,请调整资源或选择其他区域。'
|
|
221
|
+
if (code === 'NETWORK_TIMEOUT') return 'SCNet 请求超时:请稍后重试;提交/取消类操作不要盲目重试。'
|
|
222
|
+
return translateScnetError(errorTextFrom(res))
|
|
223
|
+
}
|
|
224
|
+
|
|
203
225
|
function scnetJobResult(res, action) {
|
|
204
226
|
if (res.ok) return res.stdout.trim() || '{}'
|
|
205
|
-
return `${action}失败(exit ${res.code}):\n${
|
|
227
|
+
return `${action}失败(exit ${res.code}):\n${translateScnetFailure(res)}`
|
|
206
228
|
}
|
|
207
229
|
|
|
208
230
|
async function runScnetEnvelope(argv) {
|
|
209
231
|
const res = await runScnet(argv)
|
|
210
|
-
if (!res.ok) return { ok: false, error:
|
|
232
|
+
if (!res.ok) return { ok: false, error: translateScnetFailure(res) }
|
|
211
233
|
try {
|
|
212
234
|
const parsed = JSON.parse(res.stdout)
|
|
213
235
|
return { ok: true, envelope: parsed, data: parsed?.data }
|
|
@@ -233,22 +255,18 @@ function queueCandidates(queues, wantsAccelerator) {
|
|
|
233
255
|
.sort((a, b) => Number(b?.free_nodes ?? 0) - Number(a?.free_nodes ?? 0))
|
|
234
256
|
}
|
|
235
257
|
|
|
236
|
-
//
|
|
258
|
+
// 提交前队列预检:查询失败或空结果都必须阻止提交,不能把失败当成可提交。
|
|
237
259
|
async function precheckQueue(target, args) {
|
|
238
260
|
const check = await runScnetEnvelope([...target.argv, 'queues'])
|
|
239
|
-
if (!check.ok) return {
|
|
261
|
+
if (!check.ok) return { error: `队列查询失败,已停止提交:${check.error}` }
|
|
240
262
|
const queues = Array.isArray(check.data) ? check.data : []
|
|
241
|
-
if (queues.length === 0) return {
|
|
263
|
+
if (queues.length === 0) return { error: '队列查询成功但没有可用队列,已停止提交。' }
|
|
242
264
|
|
|
243
265
|
const queue = String(args.queue ?? '').trim()
|
|
244
266
|
const dcus = Number(String(args.dcus ?? '').trim() || 0)
|
|
245
267
|
|
|
246
268
|
if (!queue) {
|
|
247
269
|
const candidates = queueCandidates(queues, dcus > 0)
|
|
248
|
-
if (candidates.length === 1) {
|
|
249
|
-
const only = candidates[0]
|
|
250
|
-
return { queue: String(only.partition), note: `已自动选择队列 ${only.partition}` }
|
|
251
|
-
}
|
|
252
270
|
if (candidates.length > 1) {
|
|
253
271
|
const best = candidates[0]
|
|
254
272
|
return {
|
|
@@ -257,7 +275,10 @@ async function precheckQueue(target, args) {
|
|
|
257
275
|
`按你要的资源建议 ${best.partition}(空闲 ${best.free_nodes} 节点)。`,
|
|
258
276
|
}
|
|
259
277
|
}
|
|
260
|
-
|
|
278
|
+
if (candidates.length === 1) {
|
|
279
|
+
return { error: `请确认 queue:当前唯一可用队列是 ${candidates[0].partition}(不会自动选择计费队列)。` }
|
|
280
|
+
}
|
|
281
|
+
return { error: '没有找到匹配当前资源需求的可用队列。' }
|
|
261
282
|
}
|
|
262
283
|
|
|
263
284
|
const hit = queues.find((item) => String(item?.partition) === queue)
|
|
@@ -345,6 +366,51 @@ export function apply(ctx) {
|
|
|
345
366
|
}),
|
|
346
367
|
)
|
|
347
368
|
|
|
369
|
+
ctx.tools.register(
|
|
370
|
+
defineTool({
|
|
371
|
+
name: 'scnet_account_summary',
|
|
372
|
+
description:
|
|
373
|
+
'只读查询当前 SCNet 账户摘要,包括用户名、账户状态、余额、默认区域和区域信息;不需要额外参数。',
|
|
374
|
+
parameters: {},
|
|
375
|
+
output: {
|
|
376
|
+
schema: { type: 'string' },
|
|
377
|
+
render: (_args, value) => textBlock(value),
|
|
378
|
+
},
|
|
379
|
+
async execute() {
|
|
380
|
+
return scnetResult(
|
|
381
|
+
await runScnet(['--backend', 'openapi', 'account']),
|
|
382
|
+
'查询账户摘要',
|
|
383
|
+
)
|
|
384
|
+
},
|
|
385
|
+
}),
|
|
386
|
+
)
|
|
387
|
+
|
|
388
|
+
ctx.tools.register(
|
|
389
|
+
defineTool({
|
|
390
|
+
name: 'scnet_resource_summary',
|
|
391
|
+
description:
|
|
392
|
+
'只读汇总目标区域的队列、空闲资源和用户资源限制;region 可省略,后端会使用默认区域。',
|
|
393
|
+
parameters: {
|
|
394
|
+
region: { type: 'string', description: '区域名称或 ID;省略时使用默认区域' },
|
|
395
|
+
scheduler_id: { type: 'string', description: '调度器 ID;多调度器时填写' },
|
|
396
|
+
},
|
|
397
|
+
output: {
|
|
398
|
+
schema: { type: 'string' },
|
|
399
|
+
render: (_args, value) => textBlock(value),
|
|
400
|
+
},
|
|
401
|
+
async execute(args) {
|
|
402
|
+
const argv = [
|
|
403
|
+
'--backend',
|
|
404
|
+
'openapi',
|
|
405
|
+
...regionArgs(args.region),
|
|
406
|
+
...(args.scheduler_id ? ['--scheduler-id', String(args.scheduler_id)] : []),
|
|
407
|
+
'resource-summary',
|
|
408
|
+
]
|
|
409
|
+
return scnetResult(await runScnet(argv), '查询资源摘要')
|
|
410
|
+
},
|
|
411
|
+
}),
|
|
412
|
+
)
|
|
413
|
+
|
|
348
414
|
ctx.tools.register(
|
|
349
415
|
defineTool({
|
|
350
416
|
name: 'scnet_job_queues',
|
|
@@ -577,7 +643,7 @@ export function apply(ctx) {
|
|
|
577
643
|
defineTool({
|
|
578
644
|
name: 'scnet_job_show',
|
|
579
645
|
description:
|
|
580
|
-
'只读查询一个作业的状态与资源(state / exit_code / work_dir / stdout / stderr
|
|
646
|
+
'只读查询一个作业的状态与资源(state / exit_code / work_dir / stdout / stderr)。自动先查实时记录,再回退最近 30 天历史记录;用户只需提供 job_id。',
|
|
581
647
|
parameters: {
|
|
582
648
|
job_id: { type: 'string', required: true, description: '作业号' },
|
|
583
649
|
backend: { type: 'string', description: 'openapi(默认)或 ssh' },
|
|
@@ -599,6 +665,43 @@ export function apply(ctx) {
|
|
|
599
665
|
}),
|
|
600
666
|
)
|
|
601
667
|
|
|
668
|
+
ctx.tools.register(
|
|
669
|
+
defineTool({
|
|
670
|
+
name: 'scnet_job_wait',
|
|
671
|
+
description:
|
|
672
|
+
'只读等待一个作业进入终态。默认最多等待 300 秒、每 10 秒查询一次;不会提交、取消或自动重试作业。',
|
|
673
|
+
parameters: {
|
|
674
|
+
job_id: { type: 'string', required: true, description: '作业号' },
|
|
675
|
+
backend: { type: 'string', description: 'openapi(默认)或 ssh' },
|
|
676
|
+
region: { type: 'string', description: 'OpenAPI 区域;省略时使用默认区域' },
|
|
677
|
+
scheduler_id: { type: 'string', description: 'OpenAPI 调度器 ID;多调度器时填写' },
|
|
678
|
+
cluster: { type: 'string', description: 'SSH 集群短名;多个 profile 时必须指定' },
|
|
679
|
+
wait_timeout: { type: 'number', description: '最长等待秒数,默认 300,最大 600' },
|
|
680
|
+
interval: { type: 'number', description: '查询间隔秒数,默认 10,范围 1-60' },
|
|
681
|
+
},
|
|
682
|
+
timeoutMs: 660000,
|
|
683
|
+
output: {
|
|
684
|
+
schema: { type: 'string' },
|
|
685
|
+
render: (_args, value) => textBlock(value),
|
|
686
|
+
},
|
|
687
|
+
async execute(args) {
|
|
688
|
+
const jobId = String(args.job_id ?? '').trim()
|
|
689
|
+
if (!jobId) return MISSING_HINTS.job_id
|
|
690
|
+
const target = await resolveTarget(args)
|
|
691
|
+
if (target.error) return target.error
|
|
692
|
+
const argv = [...target.argv, 'wait', jobId]
|
|
693
|
+
for (const [flag, key] of [['--wait-timeout', 'wait_timeout'], ['--interval', 'interval']]) {
|
|
694
|
+
if (args[key] !== undefined) {
|
|
695
|
+
const checked = positiveInt(args[key], key)
|
|
696
|
+
if (checked?.error) return checked.error
|
|
697
|
+
argv.push(flag, checked.value)
|
|
698
|
+
}
|
|
699
|
+
}
|
|
700
|
+
return scnetJobResult(await runScnet(argv, 660000), '等待作业')
|
|
701
|
+
},
|
|
702
|
+
}),
|
|
703
|
+
)
|
|
704
|
+
|
|
602
705
|
ctx.tools.register(
|
|
603
706
|
defineTool({
|
|
604
707
|
name: 'scnet_job_logs',
|
|
@@ -703,7 +806,7 @@ export function apply(ctx) {
|
|
|
703
806
|
'在本地与 SCNet 区域共享存储之间传输一个文件。direction=upload 或 download;upload 的 remote_dir 是目录而不是文件名。默认不覆盖,执行前应确认区域和路径,dry_run=true 只预览。',
|
|
704
807
|
parameters: {
|
|
705
808
|
direction: { type: 'string', required: true, description: 'upload 或 download' },
|
|
706
|
-
local_path: { type: 'string',
|
|
809
|
+
local_path: { type: 'string', description: '本地文件路径;download 省略时使用远端文件名' },
|
|
707
810
|
remote_dir: { type: 'string', description: 'upload 时必填:区域共享存储中的绝对目录;文件名由 local_path 保留' },
|
|
708
811
|
remote_path: { type: 'string', description: 'download:区域共享存储中的绝对文件路径' },
|
|
709
812
|
backend: { type: 'string', description: 'openapi(默认)或 ssh' },
|
|
@@ -724,7 +827,7 @@ export function apply(ctx) {
|
|
|
724
827
|
const localPath = String(args.local_path ?? '').trim()
|
|
725
828
|
const remoteDir = String(args.remote_dir ?? '').trim()
|
|
726
829
|
const remotePath = String(args.remote_path ?? '').trim()
|
|
727
|
-
if (!localPath) return '缺少必填参数 local_path。'
|
|
830
|
+
if (direction === 'upload' && !localPath) return '缺少必填参数 local_path。'
|
|
728
831
|
if (direction === 'upload' && !remoteDir) return '缺少必填参数 remote_dir(目录,不是文件名)。'
|
|
729
832
|
if (direction === 'download' && !remotePath) return '缺少必填参数 remote_path。'
|
|
730
833
|
const target = await resolveTarget(args)
|
|
@@ -734,7 +837,8 @@ export function apply(ctx) {
|
|
|
734
837
|
if (direction === 'upload') {
|
|
735
838
|
argv.push('upload', localPath, remoteDir)
|
|
736
839
|
} else {
|
|
737
|
-
argv.push('download', remotePath
|
|
840
|
+
argv.push('download', remotePath)
|
|
841
|
+
if (localPath) argv.push(localPath)
|
|
738
842
|
}
|
|
739
843
|
if (args.overwrite === true) argv.push('--cover')
|
|
740
844
|
return scnetResult(
|
|
@@ -1064,6 +1168,7 @@ export const __testables = {
|
|
|
1064
1168
|
MISSING_HINTS,
|
|
1065
1169
|
SUBMIT_REQUIRED_ORDER,
|
|
1066
1170
|
errorTextFrom,
|
|
1171
|
+
errorCodeFrom,
|
|
1067
1172
|
firstMissingRequired,
|
|
1068
1173
|
parseWalltimeSeconds,
|
|
1069
1174
|
pickBackend,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-scnet",
|
|
3
|
-
"version": "0.6.
|
|
3
|
+
"version": "0.6.3",
|
|
4
4
|
"description": "Community DeepSeek Harness bundle for SCNet SSH and OpenAPI jobs, files, Notebook resources, Slurm diagnostics, and Hygon DCU/DTK guidance.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./index.mjs",
|
package/skills/scnet-hpc/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.6.
|
|
1
|
+
0.6.3
|
|
@@ -125,6 +125,27 @@ python3 scripts/scnet.py --backend openapi --region <id> \
|
|
|
125
125
|
jobs --scope history --days 30 --limit 20
|
|
126
126
|
```
|
|
127
127
|
|
|
128
|
+
`job <job-id>` checks the realtime endpoint first and automatically falls back to the filtered
|
|
129
|
+
history-list endpoint. The caller does not need to provide `acctTime`.
|
|
130
|
+
|
|
131
|
+
Account and resource summaries are read-only:
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
python3 scripts/scnet.py --backend openapi --region <id> account
|
|
135
|
+
python3 scripts/scnet.py --backend openapi --region <id> resource-summary
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
Wait for a job with a bounded timeout:
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
142
|
+
wait <job-id> --wait-timeout 300 --interval 10
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
Region tokens are cached under `${XDG_CACHE_HOME:-~/.cache}/scnet-hpc` when that location is
|
|
146
|
+
writable. Cache locking or writes are best-effort; an unavailable cache never blocks an OpenAPI
|
|
147
|
+
operation.
|
|
148
|
+
|
|
128
149
|
Uploads larger than 8 MiB automatically use the documented burst/merge flow. Override the
|
|
129
150
|
threshold and chunk size with `--chunk-size <bytes>`; use `--chunk-size 0` to force ordinary
|
|
130
151
|
multipart upload. The current implementation keeps one chunk in memory at a time.
|
|
@@ -15,6 +15,7 @@ import shlex
|
|
|
15
15
|
import shutil
|
|
16
16
|
import subprocess
|
|
17
17
|
import sys
|
|
18
|
+
import time
|
|
18
19
|
from pathlib import Path
|
|
19
20
|
from typing import Any, Mapping
|
|
20
21
|
|
|
@@ -120,9 +121,19 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
120
121
|
limits = subparsers.add_parser("limits", help="show scheduler or user resource limits")
|
|
121
122
|
limits.add_argument("--partition")
|
|
122
123
|
|
|
124
|
+
account = subparsers.add_parser("account", help="show account summary")
|
|
125
|
+
resource_summary = subparsers.add_parser(
|
|
126
|
+
"resource-summary", help="show region resources and limits"
|
|
127
|
+
)
|
|
128
|
+
|
|
123
129
|
job = subparsers.add_parser("job", help="show one job")
|
|
124
130
|
job.add_argument("job_id")
|
|
125
131
|
|
|
132
|
+
wait = subparsers.add_parser("wait", help="wait for one job to reach a terminal state")
|
|
133
|
+
wait.add_argument("job_id")
|
|
134
|
+
wait.add_argument("--wait-timeout", type=int, default=300)
|
|
135
|
+
wait.add_argument("--interval", type=int, default=10)
|
|
136
|
+
|
|
126
137
|
jobs = subparsers.add_parser("jobs", help="list active or historical jobs")
|
|
127
138
|
jobs.add_argument("--scope", choices=("active", "history"), default="active")
|
|
128
139
|
jobs.add_argument("--limit", type=int, default=20)
|
|
@@ -192,7 +203,7 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
192
203
|
|
|
193
204
|
download = subparsers.add_parser("download", help="download one file or folder")
|
|
194
205
|
download.add_argument("remote_path")
|
|
195
|
-
download.add_argument("local_path")
|
|
206
|
+
download.add_argument("local_path", nargs="?")
|
|
196
207
|
download.add_argument("--cover", action="store_true")
|
|
197
208
|
|
|
198
209
|
execute = subparsers.add_parser("exec", help="run an SSH command")
|
|
@@ -225,6 +236,68 @@ def compact_output(operation: str, data: Any) -> str:
|
|
|
225
236
|
return json.dumps(data, ensure_ascii=False, indent=2, sort_keys=True)
|
|
226
237
|
|
|
227
238
|
|
|
239
|
+
def _job_terminal(state: Any) -> bool:
|
|
240
|
+
value = str(state or "").upper()
|
|
241
|
+
return value in {
|
|
242
|
+
"COMPLETED",
|
|
243
|
+
"FAILED",
|
|
244
|
+
"CANCELLED",
|
|
245
|
+
"TIMEOUT",
|
|
246
|
+
"NODE_ERROR",
|
|
247
|
+
"ABNORMAL",
|
|
248
|
+
"EXITING",
|
|
249
|
+
"EXITED",
|
|
250
|
+
"DONE",
|
|
251
|
+
"OUT_OF_MEMORY",
|
|
252
|
+
"NODE_FAIL",
|
|
253
|
+
}
|
|
254
|
+
|
|
255
|
+
|
|
256
|
+
def wait_for_job(backend, options: Mapping[str, Any]) -> dict[str, Any]:
|
|
257
|
+
timeout = int(options.get("wait_timeout") or 300)
|
|
258
|
+
interval = int(options.get("interval") or 10)
|
|
259
|
+
if timeout < 1 or timeout > 600:
|
|
260
|
+
raise BackendError("wait_timeout must be between 1 and 600 seconds")
|
|
261
|
+
if interval < 1 or interval > 60:
|
|
262
|
+
raise BackendError("interval must be between 1 and 60 seconds")
|
|
263
|
+
started = time.monotonic()
|
|
264
|
+
while True:
|
|
265
|
+
latest = backend.execute("job", options)
|
|
266
|
+
if isinstance(latest, dict) and _job_terminal(latest.get("state")):
|
|
267
|
+
latest["waited_seconds"] = round(time.monotonic() - started, 1)
|
|
268
|
+
latest["timed_out"] = False
|
|
269
|
+
return latest
|
|
270
|
+
elapsed = time.monotonic() - started
|
|
271
|
+
if elapsed >= timeout:
|
|
272
|
+
return {
|
|
273
|
+
"job": latest,
|
|
274
|
+
"waited_seconds": round(elapsed, 1),
|
|
275
|
+
"timed_out": True,
|
|
276
|
+
}
|
|
277
|
+
time.sleep(min(interval, timeout - elapsed))
|
|
278
|
+
|
|
279
|
+
|
|
280
|
+
def error_code_for(text: str) -> str:
|
|
281
|
+
value = str(text).lower()
|
|
282
|
+
if "credentials are not configured" in value or "token" in value and "expired" in value:
|
|
283
|
+
return "AUTHENTICATION_FAILED"
|
|
284
|
+
if "region" in value and ("not available" in value or "missing" in value):
|
|
285
|
+
return "REGION_NOT_FOUND"
|
|
286
|
+
if "scheduler" in value and ("not available" in value or "multiple" in value):
|
|
287
|
+
return "SCHEDULER_NOT_FOUND"
|
|
288
|
+
if "queue" in value and ("not found" in value or "no available" in value):
|
|
289
|
+
return "QUEUE_QUERY_FAILED"
|
|
290
|
+
if "job" in value and ("not found" in value or "no job record" in value):
|
|
291
|
+
return "JOB_NOT_FOUND"
|
|
292
|
+
if "already exists" in value:
|
|
293
|
+
return "FILE_TARGET_EXISTS"
|
|
294
|
+
if "does not exist" in value or "not found" in value and "file" in value:
|
|
295
|
+
return "FILE_NOT_FOUND"
|
|
296
|
+
if "timed out" in value or "timeout" in value:
|
|
297
|
+
return "NETWORK_TIMEOUT"
|
|
298
|
+
return "BACKEND_ERROR"
|
|
299
|
+
|
|
300
|
+
|
|
228
301
|
def _ask(prompt: str, default: str = "") -> str:
|
|
229
302
|
suffix = f" [{default}]" if default else ""
|
|
230
303
|
answer = input(f"{prompt}{suffix}: ").strip()
|
|
@@ -1506,6 +1579,9 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
1506
1579
|
openapi_config = user_config.get("openapi", {})
|
|
1507
1580
|
if isinstance(openapi_config, dict):
|
|
1508
1581
|
apply_openapi_defaults(options, openapi_config)
|
|
1582
|
+
if args.operation == "download" and not options.get("local_path"):
|
|
1583
|
+
remote_path = str(options.get("remote_path") or "").rstrip("/")
|
|
1584
|
+
options["local_path"] = remote_path.rsplit("/", 1)[-1] or "download"
|
|
1509
1585
|
if args.operation == "notebook":
|
|
1510
1586
|
if backend_name != "openapi":
|
|
1511
1587
|
raise BackendError(
|
|
@@ -1532,6 +1608,8 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
1532
1608
|
"name": backend.name,
|
|
1533
1609
|
"capabilities": sorted(backend.capabilities),
|
|
1534
1610
|
}
|
|
1611
|
+
elif args.operation == "wait":
|
|
1612
|
+
data = wait_for_job(backend, options)
|
|
1535
1613
|
elif args.operation == "exec":
|
|
1536
1614
|
options["command"] = " ".join(args.command).strip()
|
|
1537
1615
|
data = backend.execute(args.operation, options)
|
|
@@ -1564,6 +1642,7 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
1564
1642
|
{
|
|
1565
1643
|
"ok": False,
|
|
1566
1644
|
"operation": args.operation,
|
|
1645
|
+
"error_code": error_code_for(str(exc)),
|
|
1567
1646
|
"error": str(exc),
|
|
1568
1647
|
},
|
|
1569
1648
|
ensure_ascii=False,
|
|
@@ -38,6 +38,7 @@ STATUS_MAP = {
|
|
|
38
38
|
"statRQ": "REQUEUED",
|
|
39
39
|
}
|
|
40
40
|
WALLTIME_RE = re.compile(r"^(?:[0-9]+-)?[0-9]{1,3}:[0-9]{2}:[0-9]{2}$")
|
|
41
|
+
SCNET_TIMEZONE = timezone(timedelta(hours=8))
|
|
41
42
|
|
|
42
43
|
|
|
43
44
|
def _bounded_int(options: Mapping[str, Any], name: str, minimum: int) -> int:
|
|
@@ -62,19 +63,23 @@ def normalize_job(item: Mapping[str, Any]) -> dict[str, Any]:
|
|
|
62
63
|
"name": item.get("jobName") or item.get("name"),
|
|
63
64
|
"state": STATUS_MAP.get(str(raw_state), raw_state),
|
|
64
65
|
"raw_state": raw_state,
|
|
65
|
-
"queue": item.get("queue"),
|
|
66
|
-
"user": item.get("user"),
|
|
67
|
-
"nodes": item.get("nodeNumReq") or item.get("nodeUsed"),
|
|
68
|
-
"cpus": item.get("procNumReq") or item.get("procNumUsed"),
|
|
66
|
+
"queue": item.get("queue") or item.get("queueName"),
|
|
67
|
+
"user": item.get("user") or item.get("userName"),
|
|
68
|
+
"nodes": item.get("nodeNumReq") or item.get("nodeUsed") or item.get("nodeNum"),
|
|
69
|
+
"cpus": item.get("procNumReq") or item.get("procNumUsed") or item.get("nproc"),
|
|
69
70
|
"gpus": item.get("gpuNumReq") or item.get("gpuNumUsed"),
|
|
70
71
|
"dcus": item.get("dcuNumReq") or item.get("dcuNumUsed"),
|
|
71
|
-
"elapsed": item.get("jobRunTime"),
|
|
72
|
-
"submitted_at":
|
|
72
|
+
"elapsed": item.get("jobRunTime") or item.get("jobWalltimeUsed"),
|
|
73
|
+
"submitted_at": (
|
|
74
|
+
item.get("jobSubmitTime")
|
|
75
|
+
or item.get("jobQueueTime")
|
|
76
|
+
or item.get("queueTime")
|
|
77
|
+
),
|
|
73
78
|
"started_at": item.get("jobStartTime"),
|
|
74
|
-
"ended_at": item.get("jobEndTime"),
|
|
79
|
+
"ended_at": item.get("jobEndTime") or item.get("acctTime"),
|
|
75
80
|
"exit_code": item.get("exitCode"),
|
|
76
81
|
"reason": item.get("reason"),
|
|
77
|
-
"work_dir": item.get("workDir"),
|
|
82
|
+
"work_dir": item.get("workDir") or item.get("workdir"),
|
|
78
83
|
"stdout": item.get("outputPath"),
|
|
79
84
|
"stderr": item.get("errorPath"),
|
|
80
85
|
}
|
|
@@ -87,6 +92,8 @@ class OpenAPIBackend(Backend):
|
|
|
87
92
|
"clusters",
|
|
88
93
|
"queues",
|
|
89
94
|
"limits",
|
|
95
|
+
"account",
|
|
96
|
+
"resource-summary",
|
|
90
97
|
"job",
|
|
91
98
|
"jobs",
|
|
92
99
|
"logs",
|
|
@@ -346,6 +353,37 @@ class OpenAPIBackend(Backend):
|
|
|
346
353
|
result = {target: data.get(source) for source, target in key_map.items()}
|
|
347
354
|
return {"data": result, "raw": data} if options.get("raw") else result
|
|
348
355
|
|
|
356
|
+
def op_account(self, options: Mapping[str, Any]) -> Any:
|
|
357
|
+
_, token, region = self._center(options)
|
|
358
|
+
url = self._env(
|
|
359
|
+
"SCNET_OPENAPI_USER_URL",
|
|
360
|
+
"https://www.scnet.cn/ac/openapi/v2/user",
|
|
361
|
+
)
|
|
362
|
+
data = self._json_request("GET", url, token=token)
|
|
363
|
+
if not isinstance(data, dict):
|
|
364
|
+
raise BackendError("user endpoint returned an unexpected data shape")
|
|
365
|
+
result = {
|
|
366
|
+
"user": data.get("userName"),
|
|
367
|
+
"account_name": data.get("accountName"),
|
|
368
|
+
"account_status": data.get("accountStatus"),
|
|
369
|
+
"balance": data.get("accountBalance"),
|
|
370
|
+
"computer_center": data.get("computerCenter"),
|
|
371
|
+
"region_id": region.get("clusterId"),
|
|
372
|
+
"region_name": region.get("clusterName"),
|
|
373
|
+
}
|
|
374
|
+
return {"data": result, "raw": data} if options.get("raw") else result
|
|
375
|
+
|
|
376
|
+
def op_resource_summary(self, options: Mapping[str, Any]) -> Any:
|
|
377
|
+
_, _, region = self._center(options)
|
|
378
|
+
queues = self.op_queues(options)
|
|
379
|
+
limits = self.op_limits(options)
|
|
380
|
+
return {
|
|
381
|
+
"region_id": region.get("clusterId"),
|
|
382
|
+
"region_name": region.get("clusterName"),
|
|
383
|
+
"queues": queues,
|
|
384
|
+
"limits": limits,
|
|
385
|
+
}
|
|
386
|
+
|
|
349
387
|
def op_job(self, options: Mapping[str, Any]) -> Any:
|
|
350
388
|
job_id = str(require_option(options, "job_id"))
|
|
351
389
|
hpc_url, token, scheduler_id, _, _ = self._hpc_context(options)
|
|
@@ -371,16 +409,23 @@ class OpenAPIBackend(Backend):
|
|
|
371
409
|
result["raw"] = candidate
|
|
372
410
|
return result
|
|
373
411
|
|
|
374
|
-
|
|
412
|
+
history_items, _, history_raw = self._history_jobs(
|
|
375
413
|
hpc_url,
|
|
376
414
|
token,
|
|
377
415
|
scheduler_id,
|
|
378
|
-
job_id,
|
|
379
|
-
|
|
416
|
+
job_id=job_id,
|
|
417
|
+
days=30,
|
|
418
|
+
limit=1,
|
|
380
419
|
)
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
420
|
+
history = next(
|
|
421
|
+
(
|
|
422
|
+
item
|
|
423
|
+
for item in history_items
|
|
424
|
+
if str(item.get("jobId") or item.get("job_id") or "") == job_id
|
|
425
|
+
),
|
|
426
|
+
None,
|
|
427
|
+
)
|
|
428
|
+
if history is None:
|
|
384
429
|
detail = "realtime and history endpoints returned no job record"
|
|
385
430
|
if realtime_error:
|
|
386
431
|
detail += f" ({realtime_error})"
|
|
@@ -388,7 +433,10 @@ class OpenAPIBackend(Backend):
|
|
|
388
433
|
result = normalize_job(history)
|
|
389
434
|
result["source"] = "history"
|
|
390
435
|
if options.get("raw"):
|
|
391
|
-
result["raw"] =
|
|
436
|
+
result["raw"] = {
|
|
437
|
+
"item": history,
|
|
438
|
+
"response": history_raw,
|
|
439
|
+
}
|
|
392
440
|
return result
|
|
393
441
|
|
|
394
442
|
@staticmethod
|
|
@@ -405,25 +453,53 @@ class OpenAPIBackend(Backend):
|
|
|
405
453
|
)
|
|
406
454
|
)
|
|
407
455
|
|
|
408
|
-
|
|
456
|
+
@staticmethod
|
|
457
|
+
def _job_items(data: Any) -> tuple[list[dict[str, Any]], int | None]:
|
|
458
|
+
if isinstance(data, dict):
|
|
459
|
+
items = data.get("items") or data.get("list") or data.get("rows") or []
|
|
460
|
+
total = data.get("total")
|
|
461
|
+
else:
|
|
462
|
+
items = data or []
|
|
463
|
+
total = None
|
|
464
|
+
if not isinstance(items, list):
|
|
465
|
+
raise BackendError("job list endpoint returned an unexpected data shape")
|
|
466
|
+
normalized_items = [item for item in items if isinstance(item, dict)]
|
|
467
|
+
return normalized_items, total
|
|
468
|
+
|
|
469
|
+
def _history_jobs(
|
|
409
470
|
self,
|
|
410
471
|
hpc_url: str,
|
|
411
472
|
token: str,
|
|
412
473
|
scheduler_id: str,
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
474
|
+
*,
|
|
475
|
+
job_id: str | None = None,
|
|
476
|
+
days: int = 30,
|
|
477
|
+
limit: int = 20,
|
|
478
|
+
) -> tuple[list[dict[str, Any]], int | None, Any]:
|
|
416
479
|
if not scheduler_id:
|
|
417
480
|
raise BackendError("history lookup requires a scheduler id")
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
481
|
+
end = datetime.now(SCNET_TIMEZONE)
|
|
482
|
+
start = end - timedelta(days=days)
|
|
483
|
+
params = {
|
|
484
|
+
"strClusterNameList": scheduler_id,
|
|
485
|
+
"timeType": "CUSTOM",
|
|
486
|
+
"isQueryByQueueTime": "false",
|
|
487
|
+
"start": 0,
|
|
488
|
+
"limit": limit,
|
|
489
|
+
"startTime": start.strftime("%Y-%m-%d %H:%M:%S"),
|
|
490
|
+
"endTime": end.strftime("%Y-%m-%d %H:%M:%S"),
|
|
491
|
+
}
|
|
492
|
+
if job_id:
|
|
493
|
+
params["jobId"] = job_id
|
|
494
|
+
data = self._json_request(
|
|
423
495
|
"GET",
|
|
424
|
-
service_endpoint(hpc_url, "hpc",
|
|
496
|
+
service_endpoint(hpc_url, "hpc", "/openapi/v2/historyjobs")
|
|
497
|
+
+ "?"
|
|
498
|
+
+ urlencode(params),
|
|
425
499
|
token=token,
|
|
426
500
|
)
|
|
501
|
+
items, total = self._job_items(data)
|
|
502
|
+
return items, total, data
|
|
427
503
|
|
|
428
504
|
def op_jobs(self, options: Mapping[str, Any]) -> Any:
|
|
429
505
|
hpc_url, token, scheduler_id, _, _ = self._hpc_context(options)
|
|
@@ -440,35 +516,23 @@ class OpenAPIBackend(Backend):
|
|
|
440
516
|
"start": 0,
|
|
441
517
|
"limit": limit,
|
|
442
518
|
}
|
|
519
|
+
data = self._json_request(
|
|
520
|
+
"GET",
|
|
521
|
+
service_endpoint(hpc_url, "hpc", path) + "?" + urlencode(params),
|
|
522
|
+
token=token,
|
|
523
|
+
)
|
|
524
|
+
items, total = self._job_items(data)
|
|
443
525
|
else:
|
|
444
526
|
days = _bounded_int(options, "days", 1)
|
|
445
527
|
if days > 90:
|
|
446
528
|
raise BackendError("days must be <= 90")
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
"start": 0,
|
|
455
|
-
"limit": limit,
|
|
456
|
-
"startTime": start.strftime("%Y-%m-%d %H:%M:%S"),
|
|
457
|
-
"endTime": end.strftime("%Y-%m-%d %H:%M:%S"),
|
|
458
|
-
}
|
|
459
|
-
data = self._json_request(
|
|
460
|
-
"GET",
|
|
461
|
-
service_endpoint(hpc_url, "hpc", path) + "?" + urlencode(params),
|
|
462
|
-
token=token,
|
|
463
|
-
)
|
|
464
|
-
if isinstance(data, dict):
|
|
465
|
-
items = data.get("items") or data.get("list") or data.get("rows") or []
|
|
466
|
-
total = data.get("total")
|
|
467
|
-
else:
|
|
468
|
-
items = data or []
|
|
469
|
-
total = None
|
|
470
|
-
if not isinstance(items, list):
|
|
471
|
-
raise BackendError("job list endpoint returned an unexpected data shape")
|
|
529
|
+
items, total, data = self._history_jobs(
|
|
530
|
+
hpc_url,
|
|
531
|
+
token,
|
|
532
|
+
scheduler_id,
|
|
533
|
+
days=days,
|
|
534
|
+
limit=limit,
|
|
535
|
+
)
|
|
472
536
|
result = [normalize_job(item) for item in items if isinstance(item, dict)]
|
|
473
537
|
payload = {
|
|
474
538
|
"scope": scope,
|
|
@@ -86,28 +86,39 @@ class SCNetClient:
|
|
|
86
86
|
@classmethod
|
|
87
87
|
def _write_token_cache(
|
|
88
88
|
cls, fingerprint: str, regions: list[dict[str, Any]]
|
|
89
|
-
) ->
|
|
89
|
+
) -> bool:
|
|
90
90
|
path, _ = cls._token_cache_paths()
|
|
91
|
-
path.parent.mkdir(parents=True, mode=0o700, exist_ok=True)
|
|
92
91
|
temporary = path.with_name(f".{path.name}.{os.getpid()}.tmp")
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
92
|
+
try:
|
|
93
|
+
path.parent.mkdir(parents=True, mode=0o700, exist_ok=True)
|
|
94
|
+
temporary.write_text(
|
|
95
|
+
json.dumps(
|
|
96
|
+
{
|
|
97
|
+
"created_at": time.time(),
|
|
98
|
+
"fingerprint": fingerprint,
|
|
99
|
+
"regions": regions,
|
|
100
|
+
},
|
|
101
|
+
ensure_ascii=False,
|
|
102
|
+
),
|
|
103
|
+
encoding="utf-8",
|
|
104
|
+
)
|
|
105
|
+
os.chmod(temporary, 0o600)
|
|
106
|
+
os.replace(temporary, path)
|
|
107
|
+
return True
|
|
108
|
+
except OSError:
|
|
109
|
+
try:
|
|
110
|
+
temporary.unlink()
|
|
111
|
+
except OSError:
|
|
112
|
+
pass
|
|
113
|
+
return False
|
|
106
114
|
|
|
107
115
|
@classmethod
|
|
108
116
|
def _acquire_token_cache_lock(cls) -> Path | None:
|
|
109
117
|
_, lock = cls._token_cache_paths()
|
|
110
|
-
|
|
118
|
+
try:
|
|
119
|
+
lock.parent.mkdir(parents=True, mode=0o700, exist_ok=True)
|
|
120
|
+
except OSError:
|
|
121
|
+
return None
|
|
111
122
|
for _ in range(50):
|
|
112
123
|
try:
|
|
113
124
|
fd = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
|
@@ -121,6 +132,8 @@ class SCNetClient:
|
|
|
121
132
|
except OSError:
|
|
122
133
|
pass
|
|
123
134
|
time.sleep(0.2)
|
|
135
|
+
except OSError:
|
|
136
|
+
return None
|
|
124
137
|
return None
|
|
125
138
|
|
|
126
139
|
@staticmethod
|
|
@@ -128,7 +141,7 @@ class SCNetClient:
|
|
|
128
141
|
if lock is not None:
|
|
129
142
|
try:
|
|
130
143
|
lock.unlink()
|
|
131
|
-
except
|
|
144
|
+
except OSError:
|
|
132
145
|
pass
|
|
133
146
|
|
|
134
147
|
def request(
|
|
@@ -257,7 +270,8 @@ class SCNetClient:
|
|
|
257
270
|
self._regions_cache = [
|
|
258
271
|
item for item in data if isinstance(item, dict)
|
|
259
272
|
]
|
|
260
|
-
|
|
273
|
+
if lock is not None:
|
|
274
|
+
self._write_token_cache(fingerprint, self._regions_cache)
|
|
261
275
|
return self._regions_cache
|
|
262
276
|
finally:
|
|
263
277
|
self._release_token_cache_lock(lock)
|