dsh-scnet 0.6.0 → 0.6.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -4,11 +4,26 @@
4
4
 
5
5
  DSH-SCNet (`dsh-scnet` on npm) is a community-maintained DSH bundle for operating Supercomputing Network (SCNet). It packages the source [`scnet-hpc`](https://github.com/lql341/scnet-hpc) Skill, profile-aware utilities, and bounded tools for SSH, Slurm, OpenAPI files, regional resources, and Notebook discovery.
6
6
 
7
- Current release: **0.6.0**
7
+ Current release: **0.6.2**
8
8
 
9
9
  > This is an independent community project. It is compatible with DeepSeek Harness but is not an official DeepSeek product and does not imply endorsement, partnership, or authorization by DeepSeek.
10
10
 
11
- ## What's new in 0.6.0
11
+ ## What's new in 0.6.2
12
+
13
+ - Completed jobs now resolve through the fast filtered history-list endpoint rather than timing
14
+ out on the empty history-detail response.
15
+ - Historical field aliases such as `workdir` are normalized for log-path discovery.
16
+ - OpenAPI operations continue without persistent caching when the cache directory is read-only.
17
+
18
+ ## 0.6.1 highlights
19
+
20
+ - Job details now fall back to the history endpoint when realtime records expire, with complete
21
+ normalized terminal states.
22
+ - Added compact `scnet_job_list`, `scnet_limits`, and `scnet_file_transfer` tools.
23
+ - File uploads now make the remote-directory contract explicit: the local filename is preserved
24
+ and must not be passed as the remote directory.
25
+
26
+ ## 0.6.0 highlights
12
27
 
13
28
  - Added first-class DSH tools for submitting jobs, inspecting job status, reading logs, and
14
29
  cancelling jobs through the existing SSH/OpenAPI backends.
@@ -53,11 +68,14 @@ See [Installation](./docs/installation.md) for npm, GitHub, and local-checkout w
53
68
  | `scnet_status` | Read the saved backend, SSH profile, and OpenAPI configuration |
54
69
  | `scnet_openapi_regions` | List authorized OpenAPI regions without exposing tokens |
55
70
  | `scnet_job_queues` | Query regional Slurm queues and live resource availability |
71
+ | `scnet_job_list` | List active or historical jobs with compact normalized records |
72
+ | `scnet_limits` | Read user and scheduler resource limits |
56
73
  | `scnet_submit_job` | Submit a job through OpenAPI or SSH; `dry_run=true` only previews the request |
57
74
  | `scnet_job_show` | Read one job's state, resources, and log paths |
58
75
  | `scnet_job_logs` | Read job logs by explicit path, or by `job_id` + `work_dir` |
59
76
  | `scnet_job_cancel` | Cancel a job; `dry_run=true` only previews the request |
60
77
  | `scnet_file_list` | List shared-storage files through the common file API |
78
+ | `scnet_file_transfer` | Upload or download one file without overwriting by default |
61
79
  | `scnet_notebook_regions` | List Notebook-capable regions |
62
80
  | `scnet_notebook_resources` | Query Notebook CPU/GPU/DCU resources |
63
81
  | `scnet_notebook_list` | List Notebook instances with sensitive fields redacted |
@@ -102,12 +120,18 @@ synchronizes the skill and version into `dsh-scnet` and `codex-scnet-hpc`. Publi
102
120
  `dsh-scnet` npm package only after that downstream change is reviewed.
103
121
 
104
122
  For a synchronized release, run the package validation on `main`, then trigger the repository's
105
- `Publish DSH-SCNet` workflow with the exact value from `package.json` (currently `0.6.0`). The
123
+ `Publish DSH-SCNet` workflow with the exact value from `package.json` (currently `0.6.2`). The
106
124
  workflow verifies the package/Skill version match, runs validation, creates the tarball, publishes
107
125
  stable releases with the `latest` tag, and creates the matching Git tag. Do not bump only
108
126
  `package.json`: `skills/scnet-hpc/VERSION`, the package version, and both README release markers
109
127
  must stay aligned.
110
128
 
129
+ The workflow uses npm Trusted Publishing through GitHub OIDC, so account 2FA/Passkey should remain
130
+ enabled and no long-lived `NPM_TOKEN` is required. Configure `lql341/dsh-scnet` and
131
+ `.github/workflows/publish.yml` as a trusted publisher in the npm package settings before the
132
+ first OIDC release. A granular read/write token with 2FA bypass is a fallback only when Trusted
133
+ Publishing cannot be enabled.
134
+
111
135
  ## Validation
112
136
 
113
137
  ```sh
package/README_CN.md CHANGED
@@ -4,10 +4,22 @@
4
4
 
5
5
  <p align="center">在 DeepSeek Harness 里操作超算互联网(SCNet):配置 SSH、管理 Slurm 作业、查询 OpenAPI 区域和文件、检查 Notebook 资源与实例,并保留海光 DCU/DTK 诊断能力。</p>
6
6
 
7
- 当前版本:**0.6.0**
7
+ 当前版本:**0.6.2**
8
8
 
9
9
  > 本项目是独立维护的社区项目,与 DeepSeek Harness 兼容,但不是 DeepSeek 官方产品,也不表示 DeepSeek 对本项目提供背书、合作或授权。
10
10
 
11
+ ## 0.6.2 更新
12
+
13
+ - 已结束作业改用快速的历史列表过滤查询,不再卡在返回空数据的历史详情接口。
14
+ - 兼容历史记录中的 `workdir` 等字段变体,可继续推导日志路径。
15
+ - 缓存目录只读时自动降级为无持久缓存,OpenAPI 主功能不再被缓存故障阻断。
16
+
17
+ ## 0.6.1 更新
18
+
19
+ - 作业实时记录过期时自动回退历史接口,并补全终态映射。
20
+ - 新增紧凑的 `scnet_job_list`、`scnet_limits`、`scnet_file_transfer` 工具。
21
+ - 明确 OpenAPI 上传参数是远端目录,文件名沿用本地文件名,不能把文件名当作远端目录传入。
22
+
11
23
  ## 0.6.0 更新
12
24
 
13
25
  - 新增作业提交、状态查询、日志读取和取消作业的 DSH 工具,复用现有 SSH/OpenAPI backend。
@@ -60,11 +72,14 @@ git 源、本地目录安装以及验证步骤见[安装说明](./docs/installat
60
72
  | scnet_status | 只读查看 backend、SSH profile 和 OpenAPI 配置 |
61
73
  | scnet_openapi_regions | 列出授权 OpenAPI 区域,不暴露 token |
62
74
  | scnet_job_queues | 查询区域 Slurm 队列和实时空闲资源 |
75
+ | scnet_job_list | 紧凑列出当前或历史作业 |
76
+ | scnet_limits | 查询用户和调度器资源限制 |
63
77
  | scnet_submit_job | 通过 OpenAPI 或 SSH 提交作业;dry_run=true 只预览请求 |
64
78
  | scnet_job_show | 查询单个作业的状态、资源与日志路径 |
65
79
  | scnet_job_logs | 按显式 path,或按 job_id + work_dir 读取作业日志 |
66
80
  | scnet_job_cancel | 取消作业;dry_run=true 只预览请求 |
67
81
  | scnet_file_list | 通过通用文件 API 列出共享存储 |
82
+ | scnet_file_transfer | 上传或下载单个文件,默认不覆盖 |
68
83
  | scnet_notebook_regions | 列出支持 Notebook 的区域 |
69
84
  | scnet_notebook_resources | 查询 Notebook CPU/GPU/DCU 资源 |
70
85
  | scnet_notebook_list | 列出 Notebook 实例并脱敏敏感字段 |
@@ -117,11 +132,16 @@ git 源、本地目录安装以及验证步骤见[安装说明](./docs/installat
117
132
  才能发布对应版本的 `dsh-scnet` npm 包。
118
133
 
119
134
  下游同步合并到 `main` 后,先运行本地校验,再手动触发仓库的 `Publish DSH-SCNet` workflow,
120
- 输入 `package.json` 中的精确版本(当前为 `0.6.0`)。workflow 会校验 package 与 Skill
135
+ 输入 `package.json` 中的精确版本(当前为 `0.6.2`)。workflow 会校验 package 与 Skill
121
136
  版本一致,执行验证、生成 tarball、以稳定版 `latest` 标签发布,并创建对应 Git tag。
122
137
  不要只修改 `package.json`;`skills/scnet-hpc/VERSION`、package 版本和两份 README 的版本
123
138
  标记必须保持一致。
124
139
 
140
+ 该 workflow 使用 GitHub OIDC 的 npm Trusted Publishing,因此应保留账号的 2FA/Passkey,
141
+ 不需要长期保存 `NPM_TOKEN`。首次使用 OIDC 发布前,请在 npm package 设置中把
142
+ `lql341/dsh-scnet` 与 `.github/workflows/publish.yml` 配置为 trusted publisher。只有无法
143
+ 启用 Trusted Publishing 时,才考虑使用具备 2FA bypass 的 granular read/write token。
144
+
125
145
  ## 隐私与安全
126
146
 
127
147
  - 仓库不含私钥、token、用户名、密钥指纹。
package/docs/testing.md CHANGED
@@ -40,11 +40,14 @@ Confirm that the composed configuration includes:
40
40
  | `scnet_status` | Returns redacted local SCNet configuration |
41
41
  | `scnet_openapi_regions` | Returns authorized regions without tokens |
42
42
  | `scnet_job_queues` | Returns live regional Slurm queues |
43
+ | `scnet_job_list` | Lists active or historical jobs |
44
+ | `scnet_limits` | Returns user and scheduler resource limits |
43
45
  | `scnet_submit_job` | Submits a job through OpenAPI or SSH; `dry_run=true` returns the pending request only |
44
46
  | `scnet_job_show` | Returns one job's state, resources, and log paths |
45
47
  | `scnet_job_logs` | Reads job logs by explicit path, or by `job_id` + `work_dir` |
46
48
  | `scnet_job_cancel` | Cancels one job; `dry_run=true` returns the pending request only |
47
49
  | `scnet_file_list` | Lists shared-storage files without modification |
50
+ | `scnet_file_transfer` | Uploads or downloads one file; defaults to no overwrite |
48
51
  | `scnet_notebook_regions` | Lists Notebook-capable regions |
49
52
  | `scnet_notebook_resources` | Returns Notebook resource groups |
50
53
  | `scnet_notebook_list` | Lists redacted Notebook instances |
@@ -61,19 +61,22 @@ DEEPSEEK_API_KEY: sk-你的key
61
61
  | 8 | 用 scnet_status 查看当前配置 | 返回脱敏配置 | 只读 |
62
62
  | 9 | 用 scnet_openapi_regions 查看授权区域 | 返回区域且不含 token | 只读 |
63
63
  | 10 | 用 scnet_job_queues 查看昆山队列 | 返回实时队列和空闲资源 | 只读 |
64
- | 10a | 用 scnet_submit_job(dry_run=true)预览一次提交 | 返回将要发送的请求,不真正提交 | 只读(dry_run) |
65
- | 10b | 用 scnet_submit_job 提交一个 5 秒小作业 | 返回 job_id、作业目录与日志路径 | 高(会消耗配额) |
66
- | 10c | 用 scnet_job_show 查该 job_id | 返回状态与 stdout/stderr 路径 | 只读 |
67
- | 10d | 用 scnet_job_logs 读该作业日志 | 返回日志内容(同时验证 path 与 job_id+work_dir 两种模式结果一致) | 只读 |
68
- | 10e | 用 scnet_job_cancel 取消一个作业 | 建议先用 dry_run=true 预览 | 高(变更) |
69
- | 11 | 用 scnet_file_list 查看默认目录 | 返回共享存储文件 | 只读 |
70
- | 12 | 用 scnet_notebook_regions 查看区域 | 返回 Notebook 可用区域 | 只读 |
71
- | 13 | 用 scnet_notebook_resources 查看昆山资源 | 返回 CPU/GPU/DCU 资源 | 只读 |
72
- | 14 | 用 scnet_notebook_list 查看实例 | 返回脱敏实例列表 | 只读 |
64
+ | 11 | 用 scnet_job_list 列出当前或历史作业 | 返回紧凑作业记录 | 只读 |
65
+ | 12 | 用 scnet_limits 查询资源限制 | 返回用户和调度器限制 | 只读 |
66
+ | 13 | 用 scnet_submit_job(dry_run=true)预览一次提交 | 返回将要发送的请求,不真正提交 | 只读(dry_run) |
67
+ | 14 | 用 scnet_submit_job 提交一个 5 秒小作业 | 返回 job_id、作业目录与日志路径 | 高(会消耗配额) |
68
+ | 15 | 用 scnet_job_show 查该 job_id | 返回状态与 stdout/stderr 路径 | 只读 |
69
+ | 16 | 用 scnet_job_logs 读该作业日志 | 返回日志内容(同时验证 path 与 job_id+work_dir 两种模式结果一致) | 只读 |
70
+ | 17 | 用 scnet_job_cancel 取消一个作业 | 建议先用 dry_run=true 预览 | 高(变更) |
71
+ | 18 | 用 scnet_file_list 查看默认目录 | 返回共享存储文件 | 只读 |
72
+ | 19 | 用 scnet_file_transfer 传输一个文件 | 默认不覆盖目标 | 有副作用 |
73
+ | 20 | 用 scnet_notebook_regions 查看区域 | 返回 Notebook 可用区域 | 只读 |
74
+ | 21 | 用 scnet_notebook_resources 查看昆山资源 | 返回 CPU/GPU/DCU 资源 | 只读 |
75
+ | 22 | 用 scnet_notebook_list 查看实例 | 返回脱敏实例列表 | 只读 |
73
76
 
74
77
  第 4 至 7 步只有在你有私钥和目标集群时才测,测前确认副作用。
75
78
 
76
- 第 10b、10e 步是变更操作,会真实占用或释放调度器资源:先用 `dry_run=true` 预览,确认返回里的
79
+ 第 14、17 步是变更操作,会真实占用或释放调度器资源:先用 `dry_run=true` 预览,确认返回里的
77
80
  目标区域/集群,并且**不要在提交超时后盲目重试**(平台没有幂等键),先查作业状态。
78
81
 
79
82
  Notebook 创建、启动、停止和释放没有作为 DSH deterministic tool 暴露。此类操作通过
package/index.mjs CHANGED
@@ -197,6 +197,9 @@ function translateScnetError(text) {
197
197
  '若只想看日志,请直接给日志文件的 path;若刚提交不久,可稍后重试。'
198
198
  )
199
199
  }
200
+ if (/realtime and history endpoints returned no job record/i.test(value)) {
201
+ return '平台的实时和最近 30 天历史记录中都没有找到这个作业,请核对 job_id 和区域。'
202
+ }
200
203
  return value
201
204
  }
202
205
 
@@ -365,6 +368,68 @@ export function apply(ctx) {
365
368
  }),
366
369
  )
367
370
 
371
+ ctx.tools.register(
372
+ defineTool({
373
+ name: 'scnet_job_list',
374
+ description:
375
+ '只读列出 SCNet 作业。默认列出当前区域正在排队或运行的作业;scope=history 时查询最近历史作业。区域、数量和历史天数都可省略,输出保持紧凑。',
376
+ parameters: {
377
+ scope: { type: 'string', description: 'active(默认)或 history' },
378
+ region: { type: 'string', description: '区域名称或 ID;省略时使用已保存的默认区域' },
379
+ limit: { type: 'number', description: '最多返回条数,默认 20,最大 100' },
380
+ days: { type: 'number', description: 'history 模式查询最近天数,默认 30,最大 90' },
381
+ },
382
+ output: {
383
+ schema: { type: 'string' },
384
+ render: (_args, value) => textBlock(value),
385
+ },
386
+ async execute(args) {
387
+ const scope = String(args.scope ?? 'active').trim().toLowerCase()
388
+ if (scope !== 'active' && scope !== 'history') {
389
+ return 'scope 只支持 active 或 history。'
390
+ }
391
+ const argv = ['--backend', 'openapi', ...regionArgs(args.region), 'jobs', '--scope', scope]
392
+ if (args.limit !== undefined) {
393
+ const checked = positiveInt(args.limit, 'limit')
394
+ if (checked?.error) return checked.error
395
+ argv.push('--limit', checked.value)
396
+ }
397
+ if (args.days !== undefined) {
398
+ const checked = positiveInt(args.days, 'days')
399
+ if (checked?.error) return checked.error
400
+ argv.push('--days', checked.value)
401
+ }
402
+ return scnetResult(await runScnet(argv), '查询作业列表')
403
+ },
404
+ }),
405
+ )
406
+
407
+ ctx.tools.register(
408
+ defineTool({
409
+ name: 'scnet_limits',
410
+ description:
411
+ '只读查询当前区域和调度器的用户资源限制。region、scheduler_id 都可省略,后端会自动解析唯一目标。',
412
+ parameters: {
413
+ region: { type: 'string', description: '区域名称或 ID;省略时使用默认区域' },
414
+ scheduler_id: { type: 'string', description: '调度器 ID;只有多个调度器且无法自动选择时填写' },
415
+ },
416
+ output: {
417
+ schema: { type: 'string' },
418
+ render: (_args, value) => textBlock(value),
419
+ },
420
+ async execute(args) {
421
+ const argv = [
422
+ '--backend',
423
+ 'openapi',
424
+ ...regionArgs(args.region),
425
+ ...(args.scheduler_id ? ['--scheduler-id', String(args.scheduler_id)] : []),
426
+ 'limits',
427
+ ]
428
+ return scnetResult(await runScnet(argv), '查询资源限制')
429
+ },
430
+ }),
431
+ )
432
+
368
433
  ctx.tools.register(
369
434
  defineTool({
370
435
  name: 'scnet_submit_job',
@@ -515,7 +580,7 @@ export function apply(ctx) {
515
580
  defineTool({
516
581
  name: 'scnet_job_show',
517
582
  description:
518
- '只读查询一个作业的状态与资源(state / exit_code / work_dir / stdout / stderr)。作业完成较久后平台可能不再返回记录,此时用 scnet_job_logs 的 path 模式读日志即可。',
583
+ '只读查询一个作业的状态与资源(state / exit_code / work_dir / stdout / stderr)。自动先查实时记录,再回退最近 30 天历史记录;用户只需提供 job_id。',
519
584
  parameters: {
520
585
  job_id: { type: 'string', required: true, description: '作业号' },
521
586
  backend: { type: 'string', description: 'openapi(默认)或 ssh' },
@@ -634,6 +699,55 @@ export function apply(ctx) {
634
699
  }),
635
700
  )
636
701
 
702
+ ctx.tools.register(
703
+ defineTool({
704
+ name: 'scnet_file_transfer',
705
+ description:
706
+ '在本地与 SCNet 区域共享存储之间传输一个文件。direction=upload 或 download;upload 的 remote_dir 是目录而不是文件名。默认不覆盖,执行前应确认区域和路径,dry_run=true 只预览。',
707
+ parameters: {
708
+ direction: { type: 'string', required: true, description: 'upload 或 download' },
709
+ local_path: { type: 'string', required: true, description: '本地文件路径' },
710
+ remote_dir: { type: 'string', description: 'upload 时必填:区域共享存储中的绝对目录;文件名由 local_path 保留' },
711
+ remote_path: { type: 'string', description: 'download:区域共享存储中的绝对文件路径' },
712
+ backend: { type: 'string', description: 'openapi(默认)或 ssh' },
713
+ region: { type: 'string', description: 'OpenAPI 区域;省略时使用默认区域' },
714
+ cluster: { type: 'string', description: 'SSH 集群短名;多个 profile 时必须指定' },
715
+ overwrite: { type: 'boolean', description: '是否覆盖已有目标,默认否' },
716
+ dry_run: { type: 'boolean', description: '只预览,不传输(默认否)' },
717
+ },
718
+ output: {
719
+ schema: { type: 'string' },
720
+ render: (_args, value) => textBlock(value),
721
+ },
722
+ async execute(args) {
723
+ const direction = String(args.direction ?? '').trim().toLowerCase()
724
+ if (direction !== 'upload' && direction !== 'download') {
725
+ return 'direction 只支持 upload 或 download。'
726
+ }
727
+ const localPath = String(args.local_path ?? '').trim()
728
+ const remoteDir = String(args.remote_dir ?? '').trim()
729
+ const remotePath = String(args.remote_path ?? '').trim()
730
+ if (!localPath) return '缺少必填参数 local_path。'
731
+ if (direction === 'upload' && !remoteDir) return '缺少必填参数 remote_dir(目录,不是文件名)。'
732
+ if (direction === 'download' && !remotePath) return '缺少必填参数 remote_path。'
733
+ const target = await resolveTarget(args)
734
+ if (target.error) return target.error
735
+ const argv = [...target.argv]
736
+ if (args.dry_run === true) argv.push('--dry-run')
737
+ if (direction === 'upload') {
738
+ argv.push('upload', localPath, remoteDir)
739
+ } else {
740
+ argv.push('download', remotePath, localPath)
741
+ }
742
+ if (args.overwrite === true) argv.push('--cover')
743
+ return scnetResult(
744
+ await runScnet(argv, 180000),
745
+ direction === 'upload' ? '上传文件' : '下载文件',
746
+ )
747
+ },
748
+ }),
749
+ )
750
+
637
751
  ctx.tools.register(
638
752
  defineTool({
639
753
  name: 'scnet_file_list',
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-scnet",
3
- "version": "0.6.0",
3
+ "version": "0.6.2",
4
4
  "description": "Community DeepSeek Harness bundle for SCNet SSH and OpenAPI jobs, files, Notebook resources, Slurm diagnostics, and Hygon DCU/DTK guidance.",
5
5
  "type": "module",
6
6
  "main": "./index.mjs",
@@ -1 +1 @@
1
- 0.6.0
1
+ 0.6.2
@@ -110,6 +110,28 @@ python3 scripts/scnet.py --backend openapi --region <id> \
110
110
  Downloads refuse to replace an existing local path unless `--cover` is supplied. Uploads send
111
111
  `uncover` by default and require `--cover` for replacement.
112
112
 
113
+ For uploads, the second argument is always a remote **directory**, not a complete filename.
114
+ The uploaded filename comes from the local file. For example, uploading `mesh.bin` to
115
+ `/public/home/alice/mesh` creates `/public/home/alice/mesh/mesh.bin`; passing
116
+ `/public/home/alice/mesh.bin` as the second argument asks the API to use `mesh.bin` as a
117
+ directory name.
118
+
119
+ List active or historical jobs without supplying a job ID:
120
+
121
+ ```bash
122
+ python3 scripts/scnet.py --backend openapi --region <id> \
123
+ jobs --scope active --limit 20
124
+ python3 scripts/scnet.py --backend openapi --region <id> \
125
+ jobs --scope history --days 30 --limit 20
126
+ ```
127
+
128
+ `job <job-id>` checks the realtime endpoint first and automatically falls back to the filtered
129
+ history-list endpoint. The caller does not need to provide `acctTime`.
130
+
131
+ Region tokens are cached under `${XDG_CACHE_HOME:-~/.cache}/scnet-hpc` when that location is
132
+ writable. Cache locking or writes are best-effort; an unavailable cache never blocks an OpenAPI
133
+ operation.
134
+
113
135
  Uploads larger than 8 MiB automatically use the documented burst/merge flow. Override the
114
136
  threshold and chunk size with `--chunk-size <bytes>`; use `--chunk-size 0` to force ordinary
115
137
  multipart upload. The current implementation keeps one chunk in memory at a time.
@@ -123,6 +123,11 @@ def build_parser() -> argparse.ArgumentParser:
123
123
  job = subparsers.add_parser("job", help="show one job")
124
124
  job.add_argument("job_id")
125
125
 
126
+ jobs = subparsers.add_parser("jobs", help="list active or historical jobs")
127
+ jobs.add_argument("--scope", choices=("active", "history"), default="active")
128
+ jobs.add_argument("--limit", type=int, default=20)
129
+ jobs.add_argument("--days", type=int, default=30)
130
+
126
131
  logs = subparsers.add_parser("logs", help="read a log file")
127
132
  logs.add_argument("--path", required=True)
128
133
  logs.add_argument("--lines", type=int, default=200)
@@ -166,9 +171,9 @@ def build_parser() -> argparse.ArgumentParser:
166
171
  default="name",
167
172
  )
168
173
 
169
- upload = subparsers.add_parser("upload", help="upload one file")
174
+ upload = subparsers.add_parser("upload", help="upload one file into a remote directory")
170
175
  upload.add_argument("local_path")
171
- upload.add_argument("remote_path")
176
+ upload.add_argument("remote_dir", help="remote destination directory, not a filename")
172
177
  upload.add_argument("--cover", action="store_true")
173
178
  upload.add_argument(
174
179
  "--chunk-size",
@@ -6,6 +6,7 @@ import json
6
6
  import mimetypes
7
7
  import re
8
8
  import uuid
9
+ from datetime import datetime, timedelta, timezone
9
10
  from pathlib import Path
10
11
  from typing import Any, Mapping
11
12
  from urllib.error import HTTPError, URLError
@@ -29,8 +30,15 @@ STATUS_MAP = {
29
30
  "statC": "COMPLETED",
30
31
  "statW": "WAITING",
31
32
  "statX": "OTHER",
33
+ "statDE": "CANCELLED",
34
+ "statD": "FAILED",
35
+ "statEX": "ABNORMAL",
36
+ "statT": "TIMEOUT",
37
+ "statN": "NODE_ERROR",
38
+ "statRQ": "REQUEUED",
32
39
  }
33
40
  WALLTIME_RE = re.compile(r"^(?:[0-9]+-)?[0-9]{1,3}:[0-9]{2}:[0-9]{2}$")
41
+ SCNET_TIMEZONE = timezone(timedelta(hours=8))
34
42
 
35
43
 
36
44
  def _bounded_int(options: Mapping[str, Any], name: str, minimum: int) -> int:
@@ -44,25 +52,34 @@ def _bounded_int(options: Mapping[str, Any], name: str, minimum: int) -> int:
44
52
 
45
53
 
46
54
  def normalize_job(item: Mapping[str, Any]) -> dict[str, Any]:
47
- raw_state = item.get("jobStatus") or item.get("state") or ""
55
+ raw_state = (
56
+ item.get("jobStatus")
57
+ or item.get("jobState")
58
+ or item.get("state")
59
+ or ""
60
+ )
48
61
  return {
49
62
  "job_id": str(item.get("jobId") or item.get("job_id") or ""),
50
63
  "name": item.get("jobName") or item.get("name"),
51
64
  "state": STATUS_MAP.get(str(raw_state), raw_state),
52
65
  "raw_state": raw_state,
53
- "queue": item.get("queue"),
54
- "user": item.get("user"),
55
- "nodes": item.get("nodeNumReq") or item.get("nodeUsed"),
56
- "cpus": item.get("procNumReq") or item.get("procNumUsed"),
66
+ "queue": item.get("queue") or item.get("queueName"),
67
+ "user": item.get("user") or item.get("userName"),
68
+ "nodes": item.get("nodeNumReq") or item.get("nodeUsed") or item.get("nodeNum"),
69
+ "cpus": item.get("procNumReq") or item.get("procNumUsed") or item.get("nproc"),
57
70
  "gpus": item.get("gpuNumReq") or item.get("gpuNumUsed"),
58
71
  "dcus": item.get("dcuNumReq") or item.get("dcuNumUsed"),
59
- "elapsed": item.get("jobRunTime"),
60
- "submitted_at": item.get("jobSubmitTime"),
72
+ "elapsed": item.get("jobRunTime") or item.get("jobWalltimeUsed"),
73
+ "submitted_at": (
74
+ item.get("jobSubmitTime")
75
+ or item.get("jobQueueTime")
76
+ or item.get("queueTime")
77
+ ),
61
78
  "started_at": item.get("jobStartTime"),
62
- "ended_at": item.get("jobEndTime"),
79
+ "ended_at": item.get("jobEndTime") or item.get("acctTime"),
63
80
  "exit_code": item.get("exitCode"),
64
81
  "reason": item.get("reason"),
65
- "work_dir": item.get("workDir"),
82
+ "work_dir": item.get("workDir") or item.get("workdir"),
66
83
  "stdout": item.get("outputPath"),
67
84
  "stderr": item.get("errorPath"),
68
85
  }
@@ -76,6 +93,7 @@ class OpenAPIBackend(Backend):
76
93
  "queues",
77
94
  "limits",
78
95
  "job",
96
+ "jobs",
79
97
  "logs",
80
98
  "submit",
81
99
  "cancel",
@@ -335,18 +353,160 @@ class OpenAPIBackend(Backend):
335
353
 
336
354
  def op_job(self, options: Mapping[str, Any]) -> Any:
337
355
  job_id = str(require_option(options, "job_id"))
338
- hpc_url, token, _, _, _ = self._hpc_context(options)
356
+ hpc_url, token, scheduler_id, _, _ = self._hpc_context(options)
357
+ realtime_error: str | None = None
358
+ data: Any = None
359
+ try:
360
+ data = self._json_request(
361
+ "GET",
362
+ service_endpoint(hpc_url, "hpc", f"/openapi/v2/jobs/{quote(job_id)}"),
363
+ token=token,
364
+ )
365
+ except BackendError as exc:
366
+ if not self._is_missing_job_error(str(exc)):
367
+ raise
368
+ realtime_error = str(exc)
369
+
370
+ if isinstance(data, dict):
371
+ candidate = data
372
+ if candidate.get("jobId") or candidate.get("job_id"):
373
+ result = normalize_job(candidate)
374
+ result["source"] = "realtime"
375
+ if options.get("raw"):
376
+ result["raw"] = candidate
377
+ return result
378
+
379
+ history_items, _, history_raw = self._history_jobs(
380
+ hpc_url,
381
+ token,
382
+ scheduler_id,
383
+ job_id=job_id,
384
+ days=30,
385
+ limit=1,
386
+ )
387
+ history = next(
388
+ (
389
+ item
390
+ for item in history_items
391
+ if str(item.get("jobId") or item.get("job_id") or "") == job_id
392
+ ),
393
+ None,
394
+ )
395
+ if history is None:
396
+ detail = "realtime and history endpoints returned no job record"
397
+ if realtime_error:
398
+ detail += f" ({realtime_error})"
399
+ raise BackendError(detail)
400
+ result = normalize_job(history)
401
+ result["source"] = "history"
402
+ if options.get("raw"):
403
+ result["raw"] = {
404
+ "item": history,
405
+ "response": history_raw,
406
+ }
407
+ return result
408
+
409
+ @staticmethod
410
+ def _is_missing_job_error(text: str) -> bool:
411
+ value = text.lower()
412
+ return any(
413
+ marker in value
414
+ for marker in (
415
+ "unexpected data shape",
416
+ "http 404",
417
+ "not found",
418
+ "does not exist",
419
+ "不存在",
420
+ )
421
+ )
422
+
423
+ @staticmethod
424
+ def _job_items(data: Any) -> tuple[list[dict[str, Any]], int | None]:
425
+ if isinstance(data, dict):
426
+ items = data.get("items") or data.get("list") or data.get("rows") or []
427
+ total = data.get("total")
428
+ else:
429
+ items = data or []
430
+ total = None
431
+ if not isinstance(items, list):
432
+ raise BackendError("job list endpoint returned an unexpected data shape")
433
+ normalized_items = [item for item in items if isinstance(item, dict)]
434
+ return normalized_items, total
435
+
436
+ def _history_jobs(
437
+ self,
438
+ hpc_url: str,
439
+ token: str,
440
+ scheduler_id: str,
441
+ *,
442
+ job_id: str | None = None,
443
+ days: int = 30,
444
+ limit: int = 20,
445
+ ) -> tuple[list[dict[str, Any]], int | None, Any]:
446
+ if not scheduler_id:
447
+ raise BackendError("history lookup requires a scheduler id")
448
+ end = datetime.now(SCNET_TIMEZONE)
449
+ start = end - timedelta(days=days)
450
+ params = {
451
+ "strClusterNameList": scheduler_id,
452
+ "timeType": "CUSTOM",
453
+ "isQueryByQueueTime": "false",
454
+ "start": 0,
455
+ "limit": limit,
456
+ "startTime": start.strftime("%Y-%m-%d %H:%M:%S"),
457
+ "endTime": end.strftime("%Y-%m-%d %H:%M:%S"),
458
+ }
459
+ if job_id:
460
+ params["jobId"] = job_id
339
461
  data = self._json_request(
340
462
  "GET",
341
- service_endpoint(hpc_url, "hpc", f"/openapi/v2/jobs/{quote(job_id)}"),
463
+ service_endpoint(hpc_url, "hpc", "/openapi/v2/historyjobs")
464
+ + "?"
465
+ + urlencode(params),
342
466
  token=token,
343
467
  )
344
- if not isinstance(data, dict):
345
- raise BackendError("job endpoint returned an unexpected data shape")
346
- result = normalize_job(data)
347
- if options.get("raw"):
348
- result["raw"] = data
349
- return result
468
+ items, total = self._job_items(data)
469
+ return items, total, data
470
+
471
+ def op_jobs(self, options: Mapping[str, Any]) -> Any:
472
+ hpc_url, token, scheduler_id, _, _ = self._hpc_context(options)
473
+ scope = str(options.get("scope") or "active").strip().lower()
474
+ if scope not in {"active", "history"}:
475
+ raise BackendError("scope must be active or history")
476
+ limit = _bounded_int(options, "limit", 1)
477
+ if limit > 100:
478
+ raise BackendError("limit must be <= 100")
479
+ if scope == "active":
480
+ path = "/openapi/v2/jobs"
481
+ params = {
482
+ "strClusterIDList": scheduler_id,
483
+ "start": 0,
484
+ "limit": limit,
485
+ }
486
+ data = self._json_request(
487
+ "GET",
488
+ service_endpoint(hpc_url, "hpc", path) + "?" + urlencode(params),
489
+ token=token,
490
+ )
491
+ items, total = self._job_items(data)
492
+ else:
493
+ days = _bounded_int(options, "days", 1)
494
+ if days > 90:
495
+ raise BackendError("days must be <= 90")
496
+ items, total, data = self._history_jobs(
497
+ hpc_url,
498
+ token,
499
+ scheduler_id,
500
+ days=days,
501
+ limit=limit,
502
+ )
503
+ result = [normalize_job(item) for item in items if isinstance(item, dict)]
504
+ payload = {
505
+ "scope": scope,
506
+ "items": result,
507
+ "total": total if total is not None else len(result),
508
+ }
509
+ return {"data": payload, "raw": data} if options.get("raw") else payload
350
510
 
351
511
  def op_logs(self, options: Mapping[str, Any]) -> Any:
352
512
  path = str(require_option(options, "path"))
@@ -580,14 +740,17 @@ class OpenAPIBackend(Backend):
580
740
 
581
741
  def op_upload(self, options: Mapping[str, Any]) -> Any:
582
742
  local_path = Path(str(require_option(options, "local_path"))).expanduser()
583
- remote_path = str(require_option(options, "remote_path"))
743
+ remote_dir = str(
744
+ options.get("remote_dir")
745
+ or require_option(options, "remote_path")
746
+ )
584
747
  if not local_path.is_file():
585
748
  raise BackendError(f"local file does not exist: {local_path}")
586
749
  if (
587
- not remote_path.startswith("/")
588
- or "\n" in remote_path
589
- or "\r" in remote_path
590
- or "\x00" in remote_path
750
+ not remote_dir.startswith("/")
751
+ or "\n" in remote_dir
752
+ or "\r" in remote_dir
753
+ or "\x00" in remote_dir
591
754
  ):
592
755
  raise BackendError("remote upload directory must be absolute")
593
756
  chunk_size = int(options.get("chunk_size") or 0)
@@ -596,7 +759,7 @@ class OpenAPIBackend(Backend):
596
759
  if chunk_size and chunk_size < 1024 * 1024:
597
760
  raise BackendError("chunk size must be at least 1 MiB or 0")
598
761
  if chunk_size and local_path.stat().st_size > chunk_size:
599
- return self._op_chunked_upload(local_path, remote_path, options, chunk_size)
762
+ return self._op_chunked_upload(local_path, remote_dir, options, chunk_size)
600
763
  efile_url, token, _ = self._efile_context(options)
601
764
  content_type = mimetypes.guess_type(local_path.name)[0] or "application/octet-stream"
602
765
  self._multipart_request(
@@ -604,13 +767,17 @@ class OpenAPIBackend(Backend):
604
767
  token,
605
768
  {
606
769
  "cover": "cover" if options.get("cover") else "uncover",
607
- "path": remote_path,
770
+ "path": remote_dir,
608
771
  },
609
772
  local_path.name,
610
773
  local_path.read_bytes(),
611
774
  content_type,
612
775
  )
613
- return {"local_path": str(local_path), "remote_path": remote_path}
776
+ return {
777
+ "local_path": str(local_path),
778
+ "remote_dir": remote_dir,
779
+ "remote_filename": local_path.name,
780
+ }
614
781
 
615
782
  def _multipart_request(
616
783
  self,
@@ -220,7 +220,10 @@ class SSHBackend(Backend):
220
220
 
221
221
  def op_upload(self, options: Mapping[str, Any]) -> Any:
222
222
  local_path = Path(str(require_option(options, "local_path"))).expanduser()
223
- remote_path = str(require_option(options, "remote_path"))
223
+ remote_path = str(
224
+ options.get("remote_dir")
225
+ or require_option(options, "remote_path")
226
+ )
224
227
  if not local_path.is_file():
225
228
  raise BackendError(f"local file does not exist: {local_path}")
226
229
  if not remote_path.startswith("/") or "\n" in remote_path:
@@ -7,6 +7,7 @@ import hmac
7
7
  import json
8
8
  import os
9
9
  import time
10
+ from pathlib import Path
10
11
  from typing import Any, Mapping
11
12
  from urllib.error import HTTPError, URLError
12
13
  from urllib.parse import urlencode, urlsplit, urlunsplit
@@ -45,6 +46,8 @@ def service_endpoint(base_url: str, service: str, suffix: str) -> str:
45
46
  class SCNetClient:
46
47
  """One authenticated OpenAPI client shared by service-specific domains."""
47
48
 
49
+ TOKEN_CACHE_TTL = 4 * 60 * 60
50
+
48
51
  def __init__(self, context: BackendContext):
49
52
  self.context = context
50
53
  self._regions_cache: list[dict[str, Any]] | None = None
@@ -55,6 +58,92 @@ class SCNetClient:
55
58
  value = os.environ.get(name)
56
59
  return value if value not in (None, "") else default
57
60
 
61
+ @classmethod
62
+ def _token_cache_paths(cls) -> tuple[Path, Path]:
63
+ root = Path(cls.env("XDG_CACHE_HOME", str(Path.home() / ".cache"))) / "scnet-hpc"
64
+ return root / "openapi-regions.json", root / "openapi-regions.lock"
65
+
66
+ @staticmethod
67
+ def _credential_fingerprint(user: str, access_key: str, secret_key: str) -> str:
68
+ value = "\0".join((user, access_key, secret_key)).encode("utf-8")
69
+ return hashlib.sha256(value).hexdigest()
70
+
71
+ @classmethod
72
+ def _read_token_cache(cls, fingerprint: str) -> list[dict[str, Any]] | None:
73
+ path, _ = cls._token_cache_paths()
74
+ try:
75
+ payload = json.loads(path.read_text(encoding="utf-8"))
76
+ if (
77
+ payload.get("fingerprint") != fingerprint
78
+ or time.time() - float(payload.get("created_at", 0)) > cls.TOKEN_CACHE_TTL
79
+ or not isinstance(payload.get("regions"), list)
80
+ ):
81
+ return None
82
+ return [item for item in payload["regions"] if isinstance(item, dict)]
83
+ except (OSError, TypeError, ValueError):
84
+ return None
85
+
86
+ @classmethod
87
+ def _write_token_cache(
88
+ cls, fingerprint: str, regions: list[dict[str, Any]]
89
+ ) -> bool:
90
+ path, _ = cls._token_cache_paths()
91
+ temporary = path.with_name(f".{path.name}.{os.getpid()}.tmp")
92
+ try:
93
+ path.parent.mkdir(parents=True, mode=0o700, exist_ok=True)
94
+ temporary.write_text(
95
+ json.dumps(
96
+ {
97
+ "created_at": time.time(),
98
+ "fingerprint": fingerprint,
99
+ "regions": regions,
100
+ },
101
+ ensure_ascii=False,
102
+ ),
103
+ encoding="utf-8",
104
+ )
105
+ os.chmod(temporary, 0o600)
106
+ os.replace(temporary, path)
107
+ return True
108
+ except OSError:
109
+ try:
110
+ temporary.unlink()
111
+ except OSError:
112
+ pass
113
+ return False
114
+
115
+ @classmethod
116
+ def _acquire_token_cache_lock(cls) -> Path | None:
117
+ _, lock = cls._token_cache_paths()
118
+ try:
119
+ lock.parent.mkdir(parents=True, mode=0o700, exist_ok=True)
120
+ except OSError:
121
+ return None
122
+ for _ in range(50):
123
+ try:
124
+ fd = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
125
+ os.close(fd)
126
+ return lock
127
+ except FileExistsError:
128
+ try:
129
+ if time.time() - lock.stat().st_mtime > 30:
130
+ lock.unlink()
131
+ continue
132
+ except OSError:
133
+ pass
134
+ time.sleep(0.2)
135
+ except OSError:
136
+ return None
137
+ return None
138
+
139
+ @staticmethod
140
+ def _release_token_cache_lock(lock: Path | None) -> None:
141
+ if lock is not None:
142
+ try:
143
+ lock.unlink()
144
+ except OSError:
145
+ pass
146
+
58
147
  def request(
59
148
  self,
60
149
  method: str,
@@ -150,25 +239,42 @@ class SCNetClient:
150
239
  "OpenAPI credentials are not configured; run setup or set "
151
240
  + ", ".join(missing)
152
241
  )
153
- timestamp = str(int(time.time()))
154
- signature = canonical_signature(access_key, timestamp, user, secret_key)
155
- auth_base = self.env("SCNET_OPENAPI_AUTH_BASE", "https://api.scnet.cn")
156
- data = self.request(
157
- "POST",
158
- auth_base.rstrip("/") + "/api/user/v3/tokens",
159
- headers={
160
- "user": user,
161
- "accessKey": access_key,
162
- "signature": signature,
163
- "timestamp": timestamp,
164
- },
165
- )
166
- if not isinstance(data, list):
167
- raise BackendError("token endpoint returned an unexpected data shape")
168
- self._regions_cache = [
169
- item for item in data if isinstance(item, dict)
170
- ]
171
- return self._regions_cache
242
+ fingerprint = self._credential_fingerprint(user, access_key, secret_key)
243
+ if not refresh:
244
+ cached = self._read_token_cache(fingerprint)
245
+ if cached:
246
+ self._regions_cache = cached
247
+ return cached
248
+ lock = self._acquire_token_cache_lock()
249
+ try:
250
+ if not refresh:
251
+ cached = self._read_token_cache(fingerprint)
252
+ if cached:
253
+ self._regions_cache = cached
254
+ return cached
255
+ timestamp = str(int(time.time()))
256
+ signature = canonical_signature(access_key, timestamp, user, secret_key)
257
+ auth_base = self.env("SCNET_OPENAPI_AUTH_BASE", "https://api.scnet.cn")
258
+ data = self.request(
259
+ "POST",
260
+ auth_base.rstrip("/") + "/api/user/v3/tokens",
261
+ headers={
262
+ "user": user,
263
+ "accessKey": access_key,
264
+ "signature": signature,
265
+ "timestamp": timestamp,
266
+ },
267
+ )
268
+ if not isinstance(data, list):
269
+ raise BackendError("token endpoint returned an unexpected data shape")
270
+ self._regions_cache = [
271
+ item for item in data if isinstance(item, dict)
272
+ ]
273
+ if lock is not None:
274
+ self._write_token_cache(fingerprint, self._regions_cache)
275
+ return self._regions_cache
276
+ finally:
277
+ self._release_token_cache_lock(lock)
172
278
 
173
279
  def select_region(
174
280
  self, requested: str | None