dsh-scnet 0.6.8 → 0.6.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +9 -7
- package/README_CN.md +6 -5
- package/package.json +1 -1
- package/skills/scnet-hpc/SKILL.md +5 -1
- package/skills/scnet-hpc/VERSION +1 -1
- package/skills/scnet-hpc/references/backends.md +22 -1
- package/skills/scnet-hpc/references/openapi.md +29 -4
- package/skills/scnet-hpc/references/quickstart-en.md +21 -5
- package/skills/scnet-hpc/references/setup.md +4 -2
- package/skills/scnet-hpc/scripts/scnet.py +116 -8
- package/skills/scnet-hpc/scripts/scnet_backends/openapi.py +39 -2
- package/skills/scnet-hpc/scripts/scnet_backends/ssh.py +9 -4
- package/skills/scnet-hpc/scripts/setup.sh +56 -11
package/README.md
CHANGED
|
@@ -4,15 +4,17 @@
|
|
|
4
4
|
|
|
5
5
|
DSH-SCNet (`dsh-scnet` on npm) is a community-maintained DSH bundle for operating Supercomputing Network (SCNet). It packages the source [`scnet-hpc`](https://github.com/lql341/scnet-hpc) Skill, profile-aware utilities, and bounded tools for SSH, Slurm, OpenAPI files, regional resources, and Notebook discovery.
|
|
6
6
|
|
|
7
|
-
Current release: **0.6.
|
|
7
|
+
Current release: **0.6.9**
|
|
8
8
|
|
|
9
9
|
<!-- scnet-release:start -->
|
|
10
|
-
## 0.6.
|
|
11
|
-
|
|
12
|
-
-
|
|
13
|
-
|
|
14
|
-
-
|
|
15
|
-
|
|
10
|
+
## 0.6.9 highlights
|
|
11
|
+
|
|
12
|
+
- Default CLI output is compact across OpenAPI and SSH, with `--details` and `--raw` for
|
|
13
|
+
progressively richer responses and bounded internal job waiting to reduce repeated output.
|
|
14
|
+
- OpenAPI completion is confirmed against accounting before it becomes terminal, and SSH job
|
|
15
|
+
lookup falls back to accounting after a job leaves the live queue.
|
|
16
|
+
- First-use setup defaults to OpenAPI, retains existing backend preferences, and preserves
|
|
17
|
+
unrelated configuration fields when settings are updated.
|
|
16
18
|
<!-- scnet-release:end -->
|
|
17
19
|
|
|
18
20
|
> This is an independent community project. It is compatible with DeepSeek Harness but is not an official DeepSeek product and does not imply endorsement, partnership, or authorization by DeepSeek.
|
package/README_CN.md
CHANGED
|
@@ -4,14 +4,15 @@
|
|
|
4
4
|
|
|
5
5
|
<p align="center">在 DeepSeek Harness 里操作超算互联网(SCNet):配置 SSH、管理 Slurm 作业、查询 OpenAPI 区域和文件、检查 Notebook 资源与实例,并保留海光 DCU/DTK 诊断能力。</p>
|
|
6
6
|
|
|
7
|
-
当前版本:**0.6.
|
|
7
|
+
当前版本:**0.6.9**
|
|
8
8
|
|
|
9
9
|
<!-- scnet-release:start -->
|
|
10
|
-
## 0.6.
|
|
10
|
+
## 0.6.9 更新
|
|
11
11
|
|
|
12
|
-
-
|
|
13
|
-
|
|
14
|
-
-
|
|
12
|
+
- OpenAPI 与 SSH 的 CLI 默认使用精简输出;需要更多信息时使用 `--details` 或 `--raw`,
|
|
13
|
+
作业等待改为程序内部有界轮询,减少重复输出。
|
|
14
|
+
- OpenAPI 在作业状态终结前通过账本确认,SSH 作业离开实时队列后自动回退到账本查询。
|
|
15
|
+
- 首次配置默认选择 OpenAPI,已有用户保留原 backend 偏好,更新设置时保留无关配置字段。
|
|
15
16
|
<!-- scnet-release:end -->
|
|
16
17
|
|
|
17
18
|
> 本项目是独立维护的社区项目,与 DeepSeek Harness 兼容,但不是 DeepSeek 官方产品,也不表示 DeepSeek 对本项目提供背书、合作或授权。
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-scnet",
|
|
3
|
-
"version": "0.6.
|
|
3
|
+
"version": "0.6.9",
|
|
4
4
|
"description": "Community DeepSeek Harness bundle for SCNet SSH and OpenAPI jobs, files, Notebook resources, Slurm diagnostics, and Hygon DCU/DTK guidance.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./index.mjs",
|
|
@@ -12,6 +12,8 @@ available through the API. Never silently switch backends.
|
|
|
12
12
|
## First use and selection
|
|
13
13
|
|
|
14
14
|
- Run `scripts/setup.sh new|modify|status|reset` for explicit configuration lifecycle.
|
|
15
|
+
OpenAPI setup requires Python 3. With no saved preference, default to OpenAPI; credentials
|
|
16
|
+
can come from environment variables or the supported OS credential store.
|
|
15
17
|
- Advanced OpenAPI discovery: `python3 scripts/scnet.py setup`.
|
|
16
18
|
- Read-only preflight: `python3 scripts/scnet.py doctor`.
|
|
17
19
|
- Backend precedence: `--backend`, `SCNET_HPC_BACKEND`, saved setup choice, profile
|
|
@@ -67,4 +69,6 @@ override examples.
|
|
|
67
69
|
- `scripts/probe-cluster.sh`: initial profile creation.
|
|
68
70
|
- `scripts/run-compute-probe.sh`: bounded accelerator probe.
|
|
69
71
|
|
|
70
|
-
Use compact normalized output by default.
|
|
72
|
+
Use compact normalized output by default. Use `--details` for full normalized fields and
|
|
73
|
+
`--raw` only for diagnosis. Prefer bounded `wait` over sending every polling response to the
|
|
74
|
+
model. `FINALIZING` means the OpenAPI accounting state is not yet confirmed.
|
package/skills/scnet-hpc/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.6.
|
|
1
|
+
0.6.9
|
|
@@ -45,7 +45,9 @@ backend.
|
|
|
45
45
|
|
|
46
46
|
## Common CLI
|
|
47
47
|
|
|
48
|
-
On a new computer, start with the
|
|
48
|
+
On a new computer, start with the Bash panel. OpenAPI setup requires Python 3; without it,
|
|
49
|
+
only first-time SSH setup is supported. Updating an existing main configuration requires Python
|
|
50
|
+
to preserve unrelated fields. New users default to OpenAPI; existing backend choices are retained:
|
|
49
51
|
|
|
50
52
|
```bash
|
|
51
53
|
./scripts/setup.sh
|
|
@@ -175,3 +177,22 @@ OPENAPI_USERNAME=""
|
|
|
175
177
|
```
|
|
176
178
|
|
|
177
179
|
Keep AK, SK, tokens, cookies, and passwords out of profiles.
|
|
180
|
+
|
|
181
|
+
## Compact output and waiting
|
|
182
|
+
|
|
183
|
+
Default job output includes ID, state, queue, and available exit code or failure reason.
|
|
184
|
+
Empty fields are omitted; zero and false are retained. Use `--details` for all normalized
|
|
185
|
+
fields and `--raw` for backend data. `--json` retains its envelope with compact formatting;
|
|
186
|
+
`--dry-run` always retains the complete preview. Consumers needing the previous complete
|
|
187
|
+
CLI payload must pass `--details`; SDK/backend return schemas are preserved.
|
|
188
|
+
|
|
189
|
+
```bash
|
|
190
|
+
python3 scripts/scnet.py --region <id> job <job-id>
|
|
191
|
+
python3 scripts/scnet.py --details --region <id> job <job-id>
|
|
192
|
+
python3 scripts/scnet.py --region <id> wait <job-id> --interval 3 --wait-timeout 180
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
`wait` polls internally and emits only the final result or timeout summary. OpenAPI live
|
|
196
|
+
`COMPLETED` can also describe a recently cancelled job: until accounting confirms the state,
|
|
197
|
+
output reports `FINALIZING`. Waiting may therefore include accounting update latency.
|
|
198
|
+
Transfers return the target path and byte count; logs return only the requested lines.
|
|
@@ -37,8 +37,9 @@ export SCNET_OPENAPI_ACCESS_KEY="<access-key>"
|
|
|
37
37
|
export SCNET_OPENAPI_SECRET_KEY="<secret-key>"
|
|
38
38
|
```
|
|
39
39
|
|
|
40
|
-
The client signs the canonical AK/timestamp/user JSON with HMAC-SHA256
|
|
41
|
-
|
|
40
|
+
The client signs the canonical AK/timestamp/user JSON with HMAC-SHA256 when requesting
|
|
41
|
+
region tokens. Tokens are reused from a permission-restricted disk cache for up to four hours
|
|
42
|
+
when available; an expired or unavailable cache triggers authentication. The setup panel can store the
|
|
42
43
|
platform credential set in macOS Keychain or Linux Secret Service; otherwise use environment
|
|
43
44
|
variables. Region IDs, region users, schedulers, and home paths are discovered and cached as
|
|
44
45
|
non-secret metadata.
|
|
@@ -78,14 +79,19 @@ python3 scripts/scnet.py --backend openapi --region <id> \
|
|
|
78
79
|
logs --path /absolute/path/std.out.123 --lines 200
|
|
79
80
|
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
80
81
|
files --path /absolute/path
|
|
81
|
-
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
82
|
-
mkdir /absolute/work/dir --parents
|
|
83
82
|
```
|
|
84
83
|
|
|
85
84
|
If a region exposes multiple schedulers, add `--scheduler-id`.
|
|
86
85
|
|
|
87
86
|
## Mutations
|
|
88
87
|
|
|
88
|
+
Create a remote directory:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
92
|
+
mkdir /absolute/work/dir --parents
|
|
93
|
+
```
|
|
94
|
+
|
|
89
95
|
OpenAPI submission uses the documented BASIC/BASE command mode. Extra scheduler directives can
|
|
90
96
|
be repeated:
|
|
91
97
|
|
|
@@ -174,3 +180,22 @@ the public documentation does not establish an idempotency key for submission.
|
|
|
174
180
|
interactive `srun`, performance profiling, or low-level scheduler diagnosis.
|
|
175
181
|
- API success is determined by both HTTP transport and the JSON `code`; HTTP 200 alone is not
|
|
176
182
|
sufficient.
|
|
183
|
+
|
|
184
|
+
## Compact output and waiting
|
|
185
|
+
|
|
186
|
+
Default job output includes ID, state, queue, and available exit code or failure reason.
|
|
187
|
+
Empty fields are omitted; zero and false are retained. Use `--details` for all normalized
|
|
188
|
+
fields and `--raw` for backend data. `--json` retains its envelope with compact formatting;
|
|
189
|
+
`--dry-run` always retains the complete preview. Consumers needing the previous complete
|
|
190
|
+
CLI payload must pass `--details`; SDK/backend return schemas are preserved.
|
|
191
|
+
|
|
192
|
+
```bash
|
|
193
|
+
python3 scripts/scnet.py --region <id> job <job-id>
|
|
194
|
+
python3 scripts/scnet.py --details --region <id> job <job-id>
|
|
195
|
+
python3 scripts/scnet.py --region <id> wait <job-id> --interval 3 --wait-timeout 180
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
`wait` polls internally and emits only the final result or timeout summary. OpenAPI live
|
|
199
|
+
`COMPLETED` can also describe a recently cancelled job: until accounting confirms the state,
|
|
200
|
+
output reports `FINALIZING`. Waiting may therefore include accounting update latency.
|
|
201
|
+
Transfers return the target path and byte count; logs return only the requested lines.
|
|
@@ -54,7 +54,23 @@ The repository is designed to be shared by Codex, Claude Code, and OpenCode. Ins
|
|
|
54
54
|
paths depend on the local agent layout; `install.sh --link` is useful when developing the
|
|
55
55
|
skill from a shared source directory.
|
|
56
56
|
|
|
57
|
-
## 3. Configure
|
|
57
|
+
## 3. Configure OpenAPI and use the default backend
|
|
58
|
+
|
|
59
|
+
Python 3 is required for OpenAPI setup. On first use the panel selects `new` and OpenAPI;
|
|
60
|
+
on later runs it selects `modify` and retains the saved backend choice. Supply platform AK/SK
|
|
61
|
+
through the panel, environment variables, or a supported OS credential store.
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
./scripts/setup.sh new --backend openapi
|
|
65
|
+
python3 scripts/scnet.py clusters
|
|
66
|
+
python3 scripts/scnet.py --region <region-id> queues
|
|
67
|
+
python3 scripts/scnet.py --region <region-id> jobs
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
For submission, cancellation, logs, and files, see `references/openapi.md`. These operations
|
|
71
|
+
use OpenAPI by default when no backend preference overrides it.
|
|
72
|
+
|
|
73
|
+
## 4. Configure SSH for shell workflows
|
|
58
74
|
|
|
59
75
|
Download the private key from the SCNet console, then run:
|
|
60
76
|
|
|
@@ -66,7 +82,7 @@ ssh <cluster>
|
|
|
66
82
|
The script installs the key under `~/.ssh`, adds an SSH config entry, and tests the
|
|
67
83
|
connection. Do not commit private keys, usernames, internal node names, or personal paths.
|
|
68
84
|
|
|
69
|
-
##
|
|
85
|
+
## 5. Generate a Slurm job for the SSH workflow
|
|
70
86
|
|
|
71
87
|
Accelerator job:
|
|
72
88
|
|
|
@@ -104,7 +120,7 @@ ssh <cluster> 'sbatch --test-only ~/scripts/myjob.slurm'
|
|
|
104
120
|
ssh <cluster> 'sbatch ~/scripts/myjob.slurm'
|
|
105
121
|
```
|
|
106
122
|
|
|
107
|
-
##
|
|
123
|
+
## 6. Monitor and debug over SSH
|
|
108
124
|
|
|
109
125
|
```bash
|
|
110
126
|
squeue -u "$USER"
|
|
@@ -119,7 +135,7 @@ ends with the real program exit code.
|
|
|
119
135
|
For short experiments, request an interactive allocation with `srun`. Use the partition,
|
|
120
136
|
GRES type, CPU count, and memory limit from the target profile.
|
|
121
137
|
|
|
122
|
-
##
|
|
138
|
+
## 7. SCNet environment rules
|
|
123
139
|
|
|
124
140
|
- Login nodes are for source management, dependency preparation, file operations, and
|
|
125
141
|
scheduler commands. Whether package indexes and model sites are reachable is cluster-specific.
|
|
@@ -130,7 +146,7 @@ GRES type, CPU count, and memory limit from the target profile.
|
|
|
130
146
|
- Loading a module on a login node does not prove that the runtime works on a compute node.
|
|
131
147
|
- Hygon DCU software capabilities depend on the exact DTK, driver, framework, and `gfx` target.
|
|
132
148
|
|
|
133
|
-
##
|
|
149
|
+
## 8. Refresh and probe a cluster
|
|
134
150
|
|
|
135
151
|
```bash
|
|
136
152
|
./scripts/refresh-cluster.sh --cluster <cluster> --dry-run
|
|
@@ -27,8 +27,9 @@ Host <集群> <主机名>
|
|
|
27
27
|
Port <端口>
|
|
28
28
|
IdentityFile ~/.ssh/id_rsa_<集群>
|
|
29
29
|
IdentitiesOnly yes
|
|
30
|
-
#
|
|
30
|
+
# UseKeychain 仅适用于 Apple OpenSSH;Linux 上省略
|
|
31
31
|
UseKeychain yes
|
|
32
|
+
# 标准 OpenSSH 选项,需要运行 ssh-agent
|
|
32
33
|
AddKeysToAgent yes
|
|
33
34
|
# 长连接
|
|
34
35
|
ControlMaster auto
|
|
@@ -66,7 +67,8 @@ client_global_hostkeys_prove_confirm: server gave bad signature for RSA key 0
|
|
|
66
67
|
|
|
67
68
|
| 项 | macOS | Linux |
|
|
68
69
|
|---|---|---|
|
|
69
|
-
| `UseKeychain`
|
|
70
|
+
| `UseKeychain` | Apple OpenSSH 支持 | 不支持,应省略 |
|
|
71
|
+
| `AddKeysToAgent` | 支持 | 支持,需要运行 ssh-agent |
|
|
70
72
|
| 下载文件隔离属性 | 需 `xattr -c` | 无此概念 |
|
|
71
73
|
| `sed -i` | 需 `sed -i ''`(BSD sed) | `sed -i`(GNU sed) |
|
|
72
74
|
|
|
@@ -79,6 +79,11 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
79
79
|
action="store_true",
|
|
80
80
|
help="request backend-specific raw fields when supported",
|
|
81
81
|
)
|
|
82
|
+
parser.add_argument(
|
|
83
|
+
"--details",
|
|
84
|
+
action="store_true",
|
|
85
|
+
help="include full normalized fields and formatted JSON; --raw includes backend data",
|
|
86
|
+
)
|
|
82
87
|
subparsers = parser.add_subparsers(dest="operation", required=True)
|
|
83
88
|
|
|
84
89
|
subparsers.add_parser("backends", help="list built-in and discovered backends")
|
|
@@ -224,20 +229,122 @@ def choose_backend(
|
|
|
224
229
|
)
|
|
225
230
|
|
|
226
231
|
|
|
227
|
-
def
|
|
232
|
+
def _without_empty(data: Any) -> Any:
|
|
233
|
+
if isinstance(data, dict):
|
|
234
|
+
return {
|
|
235
|
+
key: _without_empty(value) for key, value in data.items()
|
|
236
|
+
if value is not None and value != ""
|
|
237
|
+
}
|
|
238
|
+
if isinstance(data, list):
|
|
239
|
+
return [_without_empty(value) for value in data]
|
|
240
|
+
return data
|
|
241
|
+
|
|
242
|
+
|
|
243
|
+
def summarize_output(operation: str, data: Any) -> Any:
|
|
244
|
+
"""Project CLI output only; keep backend/SDK responses available to other callers."""
|
|
245
|
+
|
|
246
|
+
def pick(item, keys):
|
|
247
|
+
return _without_empty({key: item[key] for key in keys if key in item})
|
|
248
|
+
|
|
249
|
+
if operation in {"job", "wait"} and isinstance(data, dict):
|
|
250
|
+
if isinstance(data.get("job"), dict):
|
|
251
|
+
return {
|
|
252
|
+
"job": summarize_output("job", data["job"]),
|
|
253
|
+
**pick(data, ("waited_seconds", "timed_out")),
|
|
254
|
+
}
|
|
255
|
+
result = pick(
|
|
256
|
+
data,
|
|
257
|
+
(
|
|
258
|
+
"job_id",
|
|
259
|
+
"state",
|
|
260
|
+
"queue",
|
|
261
|
+
"exit_code",
|
|
262
|
+
"reason",
|
|
263
|
+
"state_confirmed",
|
|
264
|
+
"reported_state",
|
|
265
|
+
"waited_seconds",
|
|
266
|
+
"timed_out",
|
|
267
|
+
),
|
|
268
|
+
)
|
|
269
|
+
if not result.get("queue") and data.get("partition"):
|
|
270
|
+
result["queue"] = data["partition"]
|
|
271
|
+
if "state" in result:
|
|
272
|
+
result["state"] = (
|
|
273
|
+
str(result["state"])
|
|
274
|
+
.split(" BY ", 1)[0]
|
|
275
|
+
.split(" by ", 1)[0]
|
|
276
|
+
.rstrip("+")
|
|
277
|
+
)
|
|
278
|
+
return result
|
|
279
|
+
if operation == "jobs" and isinstance(data, dict):
|
|
280
|
+
return {
|
|
281
|
+
**pick(data, ("total", "scope")),
|
|
282
|
+
"items": [
|
|
283
|
+
summarize_output("job", item) for item in data.get("items", [])
|
|
284
|
+
],
|
|
285
|
+
}
|
|
286
|
+
if operation == "queues" and isinstance(data, list):
|
|
287
|
+
return [
|
|
288
|
+
pick(
|
|
289
|
+
item,
|
|
290
|
+
(
|
|
291
|
+
"partition",
|
|
292
|
+
"availability",
|
|
293
|
+
"nodes",
|
|
294
|
+
"free_nodes",
|
|
295
|
+
"free_cpus",
|
|
296
|
+
"max_walltime",
|
|
297
|
+
),
|
|
298
|
+
)
|
|
299
|
+
for item in data
|
|
300
|
+
]
|
|
301
|
+
if operation in {"upload", "download"} and isinstance(data, dict):
|
|
302
|
+
path = data.get("remote_path")
|
|
303
|
+
if not path and data.get("remote_dir") and data.get("remote_filename"):
|
|
304
|
+
path = str(data["remote_dir"]).rstrip("/") + "/" + str(data["remote_filename"])
|
|
305
|
+
result = {"path": path}
|
|
306
|
+
size = data.get("bytes")
|
|
307
|
+
if size is None and data.get("local_path"):
|
|
308
|
+
local = Path(str(data["local_path"]))
|
|
309
|
+
if local.is_file():
|
|
310
|
+
size = local.stat().st_size
|
|
311
|
+
if size is not None:
|
|
312
|
+
result["bytes"] = size
|
|
313
|
+
return _without_empty(result)
|
|
314
|
+
if operation == "submit" and isinstance(data, dict):
|
|
315
|
+
return pick(data, ("job_id",))
|
|
316
|
+
if operation == "cancel" and isinstance(data, dict):
|
|
317
|
+
return pick(data, ("job_id", "cancelled"))
|
|
318
|
+
if operation == "mkdir" and isinstance(data, dict):
|
|
319
|
+
return pick(data, ("path", "created"))
|
|
228
320
|
if operation == "logs" and isinstance(data, dict):
|
|
321
|
+
return pick(data, ("lines",))
|
|
322
|
+
# Diagnostic and configuration responses retain their schema.
|
|
323
|
+
return data
|
|
324
|
+
|
|
325
|
+
|
|
326
|
+
def compact_output(operation: str, data: Any, *, detailed: bool = False) -> str:
|
|
327
|
+
if operation == "logs" and isinstance(data, dict) and "lines" in data:
|
|
328
|
+
lines = "\n".join(data.get("lines") or [])
|
|
329
|
+
if not detailed:
|
|
330
|
+
return lines
|
|
229
331
|
header = (
|
|
230
332
|
f"# {data.get('path')} "
|
|
231
333
|
f"({data.get('direction')}, {len(data.get('lines') or [])} lines)"
|
|
232
334
|
)
|
|
233
|
-
return header + "\n" +
|
|
335
|
+
return header + "\n" + lines
|
|
234
336
|
if operation == "exec" and isinstance(data, dict):
|
|
235
337
|
return str(data.get("stdout") or "")
|
|
236
|
-
return json.dumps(
|
|
338
|
+
return json.dumps(
|
|
339
|
+
data,
|
|
340
|
+
ensure_ascii=False,
|
|
341
|
+
sort_keys=True,
|
|
342
|
+
**({"indent": 2} if detailed else {"separators": (",", ":")}),
|
|
343
|
+
)
|
|
237
344
|
|
|
238
345
|
|
|
239
346
|
def _job_terminal(state: Any) -> bool:
|
|
240
|
-
value = str(state or "").upper()
|
|
347
|
+
value = str(state or "").upper().split(" BY ", 1)[0].rstrip("+")
|
|
241
348
|
return value in {
|
|
242
349
|
"COMPLETED",
|
|
243
350
|
"FAILED",
|
|
@@ -245,7 +352,6 @@ def _job_terminal(state: Any) -> bool:
|
|
|
245
352
|
"TIMEOUT",
|
|
246
353
|
"NODE_ERROR",
|
|
247
354
|
"ABNORMAL",
|
|
248
|
-
"EXITING",
|
|
249
355
|
"EXITED",
|
|
250
356
|
"DONE",
|
|
251
357
|
"OUT_OF_MEMORY",
|
|
@@ -1615,6 +1721,8 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
1615
1721
|
data = backend.execute(args.operation, options)
|
|
1616
1722
|
else:
|
|
1617
1723
|
data = backend.execute(args.operation, options)
|
|
1724
|
+
detailed = args.details or args.raw or args.dry_run
|
|
1725
|
+
output_data = data if detailed else summarize_output(args.operation, data)
|
|
1618
1726
|
if args.json:
|
|
1619
1727
|
print(
|
|
1620
1728
|
json.dumps(
|
|
@@ -1623,15 +1731,15 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
1623
1731
|
"version": VERSION,
|
|
1624
1732
|
"backend": backend_name,
|
|
1625
1733
|
"operation": args.operation,
|
|
1626
|
-
"data":
|
|
1734
|
+
"data": output_data,
|
|
1627
1735
|
},
|
|
1628
1736
|
ensure_ascii=False,
|
|
1629
|
-
indent=2,
|
|
1630
1737
|
sort_keys=True,
|
|
1738
|
+
**({"indent": 2} if detailed else {"separators": (",", ":")}),
|
|
1631
1739
|
)
|
|
1632
1740
|
)
|
|
1633
1741
|
elif args.operation != "setup":
|
|
1634
|
-
print(compact_output(args.operation,
|
|
1742
|
+
print(compact_output(args.operation, output_data, detailed=detailed))
|
|
1635
1743
|
if args.operation == "doctor" and not data.get("ok", False):
|
|
1636
1744
|
return 1
|
|
1637
1745
|
return 0
|
|
@@ -65,8 +65,18 @@ def normalize_job(item: Mapping[str, Any]) -> dict[str, Any]:
|
|
|
65
65
|
"raw_state": raw_state,
|
|
66
66
|
"queue": item.get("queue") or item.get("queueName"),
|
|
67
67
|
"user": item.get("user") or item.get("userName"),
|
|
68
|
-
"nodes":
|
|
69
|
-
|
|
68
|
+
"nodes": (
|
|
69
|
+
item.get("nodeNumReq")
|
|
70
|
+
or item.get("nodeUsed")
|
|
71
|
+
or item.get("nodeNum")
|
|
72
|
+
or item.get("nodect")
|
|
73
|
+
),
|
|
74
|
+
"cpus": (
|
|
75
|
+
item.get("procNumReq")
|
|
76
|
+
or item.get("procNumUsed")
|
|
77
|
+
or item.get("nproc")
|
|
78
|
+
or item.get("jobProcNum")
|
|
79
|
+
),
|
|
70
80
|
"gpus": item.get("gpuNumReq") or item.get("gpuNumUsed"),
|
|
71
81
|
"dcus": item.get("dcuNumReq") or item.get("dcuNumUsed"),
|
|
72
82
|
"elapsed": item.get("jobRunTime") or item.get("jobWalltimeUsed"),
|
|
@@ -78,6 +88,7 @@ def normalize_job(item: Mapping[str, Any]) -> dict[str, Any]:
|
|
|
78
88
|
"started_at": item.get("jobStartTime"),
|
|
79
89
|
"ended_at": item.get("jobEndTime") or item.get("acctTime"),
|
|
80
90
|
"exit_code": item.get("exitCode"),
|
|
91
|
+
"platform_exit_status": item.get("jobExitStatus"),
|
|
81
92
|
"reason": item.get("reason"),
|
|
82
93
|
"work_dir": item.get("workDir") or item.get("workdir"),
|
|
83
94
|
"stdout": item.get("outputPath"),
|
|
@@ -405,6 +416,32 @@ class OpenAPIBackend(Backend):
|
|
|
405
416
|
if candidate.get("jobId") or candidate.get("job_id"):
|
|
406
417
|
result = normalize_job(candidate)
|
|
407
418
|
result["source"] = "realtime"
|
|
419
|
+
if result["state"] == "COMPLETED":
|
|
420
|
+
# The live service also reports statC immediately after cancellation.
|
|
421
|
+
# Prefer accounting evidence; until it exists, do not claim completion.
|
|
422
|
+
items, _, history_raw = self._history_jobs(
|
|
423
|
+
hpc_url, token, scheduler_id, job_id=job_id, days=30, limit=1
|
|
424
|
+
)
|
|
425
|
+
history = next(
|
|
426
|
+
(
|
|
427
|
+
item
|
|
428
|
+
for item in items
|
|
429
|
+
if str(item.get("jobId") or item.get("job_id"))
|
|
430
|
+
== job_id
|
|
431
|
+
),
|
|
432
|
+
None,
|
|
433
|
+
)
|
|
434
|
+
if history is not None:
|
|
435
|
+
result = normalize_job(history)
|
|
436
|
+
result["source"] = "history"
|
|
437
|
+
if options.get("raw"):
|
|
438
|
+
result["raw"] = {"item": history, "response": history_raw}
|
|
439
|
+
return result
|
|
440
|
+
result.update(
|
|
441
|
+
state="FINALIZING",
|
|
442
|
+
reported_state="COMPLETED",
|
|
443
|
+
state_confirmed=False,
|
|
444
|
+
)
|
|
408
445
|
if options.get("raw"):
|
|
409
446
|
result["raw"] = candidate
|
|
410
447
|
return result
|
|
@@ -146,10 +146,15 @@ class SSHBackend(Backend):
|
|
|
146
146
|
"cpus",
|
|
147
147
|
"reason_or_nodes",
|
|
148
148
|
]
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
149
|
+
try:
|
|
150
|
+
output = self._ssh(
|
|
151
|
+
"LC_ALL=C squeue -h "
|
|
152
|
+
f"-j {shlex.quote(job_id)} -o '%i|%T|%P|%M|%l|%D|%C|%R'"
|
|
153
|
+
).strip()
|
|
154
|
+
except BackendError as exc:
|
|
155
|
+
if "invalid job id specified" not in str(exc).lower():
|
|
156
|
+
raise
|
|
157
|
+
output = ""
|
|
153
158
|
if output:
|
|
154
159
|
return _split_pipe(output.splitlines()[0], fields)
|
|
155
160
|
output = self._ssh(
|
|
@@ -92,8 +92,11 @@ choose() {
|
|
|
92
92
|
shift
|
|
93
93
|
local choices=("$@")
|
|
94
94
|
local index choice
|
|
95
|
+
local default_index="${SCNET_SETUP_CHOICE_DEFAULT:-1}"
|
|
96
|
+
[[ "$default_index" =~ ^[0-9]+$ ]] || die "默认选项必须是数字"
|
|
97
|
+
[ "$default_index" -ge 1 ] && [ "$default_index" -le "${#choices[@]}" ] || die "默认选项超出范围"
|
|
95
98
|
if [ -t 0 ] && [ -t 2 ] && [ "${TERM:-dumb}" != "dumb" ]; then
|
|
96
|
-
local selected
|
|
99
|
+
local selected=$((default_index - 1))
|
|
97
100
|
local number=""
|
|
98
101
|
local key sequence value
|
|
99
102
|
local line_count=$(( ${#choices[@]} + 1 ))
|
|
@@ -165,11 +168,11 @@ choose() {
|
|
|
165
168
|
for index in "${!choices[@]}"; do
|
|
166
169
|
printf ' %d. %s\n' "$((index + 1))" "${choices[$index]}" >&2
|
|
167
170
|
done
|
|
168
|
-
printf '请输入 1-%d 的数字并按 Enter;直接按 Enter 使用默认项
|
|
169
|
-
"${#choices[@]}" >&2
|
|
171
|
+
printf '请输入 1-%d 的数字并按 Enter;直接按 Enter 使用默认项 %d。\n' \
|
|
172
|
+
"${#choices[@]}" "$default_index" >&2
|
|
170
173
|
printf '当前终端不支持交互导航,已使用数字输入模式。\n' >&2
|
|
171
174
|
local selected
|
|
172
|
-
selected=$(ask "选择" "
|
|
175
|
+
selected=$(ask "选择" "$default_index")
|
|
173
176
|
[[ "$selected" =~ ^[0-9]+$ ]] || die "选择必须是数字"
|
|
174
177
|
[ "$selected" -ge 1 ] && [ "$selected" -le "${#choices[@]}" ] \
|
|
175
178
|
|| die "选择超出范围"
|
|
@@ -195,6 +198,18 @@ json_string() {
|
|
|
195
198
|
write_config() {
|
|
196
199
|
local config_root="${XDG_CONFIG_HOME:-$HOME/.config}/scnet-hpc"
|
|
197
200
|
local config_path="$config_root/config.json"
|
|
201
|
+
if [ -n "${PYTHON_BIN:-}" ]; then
|
|
202
|
+
"$PYTHON_BIN" - "$SCRIPT_DIR" "$CLUSTER" "$DEFAULT_BACKEND" <<'PYCONFIG'
|
|
203
|
+
import sys
|
|
204
|
+
sys.path.insert(0, sys.argv[1])
|
|
205
|
+
from scnet_config import load_user_config, save_user_config
|
|
206
|
+
config = load_user_config()
|
|
207
|
+
config.update(version=1, cluster=sys.argv[2], default_backend=sys.argv[3])
|
|
208
|
+
print(save_user_config(config))
|
|
209
|
+
PYCONFIG
|
|
210
|
+
return
|
|
211
|
+
fi
|
|
212
|
+
[ ! -f "$config_path" ] || die "更新已有主配置需要 Python 3,以保留其他配置字段"
|
|
198
213
|
mkdir -p "$config_root"
|
|
199
214
|
chmod 700 "$config_root"
|
|
200
215
|
|
|
@@ -353,10 +368,25 @@ case "$(uname -s)" in
|
|
|
353
368
|
esac
|
|
354
369
|
printf '\n' >&2
|
|
355
370
|
|
|
356
|
-
|
|
357
|
-
|
|
371
|
+
config_root="${XDG_CONFIG_HOME:-$HOME/.config}/scnet-hpc"
|
|
372
|
+
PYTHON_BIN="$(resolve_python)" || PYTHON_BIN=""
|
|
373
|
+
saved_backend=$(json_string_value "$config_root/config.json" default_backend)
|
|
374
|
+
action_default=2
|
|
375
|
+
if [ -n "$PYTHON_BIN" ]; then
|
|
376
|
+
action_default=$("$PYTHON_BIN" - "$SCRIPT_DIR" <<'PYDEFAULT'
|
|
377
|
+
import sys
|
|
378
|
+
sys.path.insert(0, sys.argv[1])
|
|
379
|
+
from scnet_config import load_user_config
|
|
380
|
+
from scnet import _has_configuration
|
|
381
|
+
print(3 if _has_configuration(load_user_config()) else 2)
|
|
382
|
+
PYDEFAULT
|
|
383
|
+
)
|
|
384
|
+
elif [ -f "$config_root/config.json" ] || [ -f "$config_root/openapi.json" ] \
|
|
385
|
+
|| compgen -G "$config_root/ssh/*.json" >/dev/null; then
|
|
386
|
+
action_default=3
|
|
387
|
+
fi
|
|
358
388
|
if [ -z "$ACTION" ]; then
|
|
359
|
-
action_label=$(choose "选择配置操作:" \
|
|
389
|
+
action_label=$(SCNET_SETUP_CHOICE_DEFAULT="$action_default" choose "选择配置操作:" \
|
|
360
390
|
"查看当前状态(status)" \
|
|
361
391
|
"新增连接(new)" \
|
|
362
392
|
"修改现有配置(modify)" \
|
|
@@ -381,7 +411,9 @@ if [ "$ACTION" = status ] || [ "$ACTION" = reset ]; then
|
|
|
381
411
|
fi
|
|
382
412
|
|
|
383
413
|
if [ -z "$BACKEND" ]; then
|
|
384
|
-
|
|
414
|
+
backend_default=1
|
|
415
|
+
[ "$saved_backend" != ssh ] || backend_default=2
|
|
416
|
+
BACKEND=$(SCNET_SETUP_CHOICE_DEFAULT="$backend_default" choose "选择要管理的 backend:" \
|
|
385
417
|
"openapi(推荐:平台级 AK/SK 和多区域)" \
|
|
386
418
|
"ssh(区域级用户名和私钥)" \
|
|
387
419
|
"both(同时管理)")
|
|
@@ -396,6 +428,17 @@ case "$BACKEND" in
|
|
|
396
428
|
*) die "backend 必须是 ssh、openapi 或 both" ;;
|
|
397
429
|
esac
|
|
398
430
|
|
|
431
|
+
# Check requirements before installing keys or changing local configuration.
|
|
432
|
+
if [ "$BACKEND" = openapi ] || [ "$BACKEND" = both ]; then
|
|
433
|
+
[ -n "$PYTHON_BIN" ] || die "OpenAPI 配置需要 Python 3;请设置 DSH_SCNET_PYTHON 或 SCNET_PYTHON,或安装 python3"
|
|
434
|
+
fi
|
|
435
|
+
if [ "$BACKEND" = ssh ] || [ "$BACKEND" = both ]; then
|
|
436
|
+
[ -n "$PYTHON_BIN" ] || [ ! -f "$config_root/config.json" ] \
|
|
437
|
+
|| die "更新已有主配置需要 Python 3,以保留其他配置字段"
|
|
438
|
+
available=$(list_clusters)
|
|
439
|
+
[ -n "$available" ] || die "clusters/ 下没有 profile"
|
|
440
|
+
fi
|
|
441
|
+
|
|
399
442
|
if { [ "$BACKEND" = ssh ] || [ "$BACKEND" = both ]; } \
|
|
400
443
|
&& [ -z "$CLUSTER" ]; then
|
|
401
444
|
cluster_choices=()
|
|
@@ -429,7 +472,9 @@ fi
|
|
|
429
472
|
|
|
430
473
|
DEFAULT_BACKEND="$BACKEND"
|
|
431
474
|
if [ "$BACKEND" = both ]; then
|
|
432
|
-
|
|
475
|
+
backend_default=1
|
|
476
|
+
[ "$saved_backend" != ssh ] || backend_default=2
|
|
477
|
+
DEFAULT_BACKEND=$(SCNET_SETUP_CHOICE_DEFAULT="$backend_default" choose "选择默认 backend:" "openapi" "ssh")
|
|
433
478
|
fi
|
|
434
479
|
|
|
435
480
|
if [ "$BACKEND" = ssh ] || [ "$BACKEND" = both ]; then
|
|
@@ -458,8 +503,8 @@ if [ "$BACKEND" = openapi ] || [ "$BACKEND" = both ]; then
|
|
|
458
503
|
"$PYTHON_BIN" "$SCRIPT_DIR/scnet.py" "${openapi_args[@]}"
|
|
459
504
|
else
|
|
460
505
|
printf '%s\n' \
|
|
461
|
-
"OpenAPI backend 需要 Python 3
|
|
462
|
-
|
|
506
|
+
"OpenAPI backend 需要 Python 3。" >&2
|
|
507
|
+
exit 1
|
|
463
508
|
fi
|
|
464
509
|
fi
|
|
465
510
|
|