dsh-scnet 0.4.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +20 -113
- package/README_CN.md +20 -1
- package/docs/installation.md +10 -0
- package/docs/installation.zh-CN.md +13 -1
- package/docs/testing.md +11 -0
- package/docs/testing.zh-CN.md +10 -0
- package/index.mjs +184 -0
- package/package.json +4 -2
- package/skills/scnet-hpc/SKILL.md +63 -105
- package/skills/scnet-hpc/VERSION +1 -0
- package/skills/scnet-hpc/clusters/_template.conf +10 -4
- package/skills/scnet-hpc/references/adding-cluster.md +6 -1
- package/skills/scnet-hpc/references/backends.md +177 -0
- package/skills/scnet-hpc/references/notebook.md +60 -0
- package/skills/scnet-hpc/references/openapi.md +140 -0
- package/skills/scnet-hpc/references/quickstart-en.md +22 -0
- package/skills/scnet-hpc/references/setup.md +4 -0
- package/skills/scnet-hpc/scripts/__pycache__/compute-probe.cpython-312.pyc +0 -0
- package/skills/scnet-hpc/scripts/__pycache__/scnet.cpython-312.pyc +0 -0
- package/skills/scnet-hpc/scripts/__pycache__/scnet_config.cpython-312.pyc +0 -0
- package/skills/scnet-hpc/scripts/__pycache__/scnet_credentials.cpython-312.pyc +0 -0
- package/skills/scnet-hpc/scripts/__pycache__/scnet_version.cpython-312.pyc +0 -0
- package/skills/scnet-hpc/scripts/scnet.py +1573 -0
- package/skills/scnet-hpc/scripts/scnet_backends/__init__.py +53 -0
- package/skills/scnet-hpc/scripts/scnet_backends/base.py +64 -0
- package/skills/scnet-hpc/scripts/scnet_backends/external.py +92 -0
- package/skills/scnet-hpc/scripts/scnet_backends/openapi.py +771 -0
- package/skills/scnet-hpc/scripts/scnet_backends/profile.py +78 -0
- package/skills/scnet-hpc/scripts/scnet_backends/ssh.py +258 -0
- package/skills/scnet-hpc/scripts/scnet_config.py +175 -0
- package/skills/scnet-hpc/scripts/scnet_credentials.py +160 -0
- package/skills/scnet-hpc/scripts/scnet_sdk/__init__.py +11 -0
- package/skills/scnet-hpc/scripts/scnet_sdk/cli.py +394 -0
- package/skills/scnet-hpc/scripts/scnet_sdk/client.py +240 -0
- package/skills/scnet-hpc/scripts/scnet_sdk/notebook.py +379 -0
- package/skills/scnet-hpc/scripts/scnet_version.py +20 -0
- package/skills/scnet-hpc/scripts/setup-ssh.sh +33 -1
- package/skills/scnet-hpc/scripts/setup.sh +471 -0
|
@@ -1,111 +1,69 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: scnet-hpc
|
|
3
|
-
description: Configure
|
|
3
|
+
description: Configure or operate SCNet HPC through SSH, OpenAPI, and pluggable future backends.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# SCNet HPC
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
-
|
|
61
|
-
|
|
62
|
-
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
requested memory within `cpus-per-task × DEF_MEM_PER_CPU`. Use `scripts/new-job.sh` to
|
|
71
|
-
calculate a conservative value.
|
|
72
|
-
3. **Framework validation belongs on compute nodes.** A login node may lack accelerator
|
|
73
|
-
libraries even after modules are loaded. Use `srun`, `sbatch`, or the compute probe for
|
|
74
|
-
imports, kernels, collectives, and accelerator capabilities.
|
|
75
|
-
4. **Offline nodes require staged dependencies.** When `COMPUTE_NODE_OFFLINE=yes`, download or
|
|
76
|
-
build dependencies on a networked node and disable runtime downloads in the job.
|
|
77
|
-
5. **Test files use shared storage.** Create per-job temporary directories under
|
|
78
|
-
`$HOME/.scnet-hpc/tmp/$SLURM_JOB_ID`; do not depend on `/tmp` being shared across nodes.
|
|
79
|
-
6. **Job exit status must propagate.** Capture the workload return code and terminate the batch
|
|
80
|
-
script with `exit "$rc"`. Evaluate both accounting state and job logs.
|
|
81
|
-
7. **Compatibility claims require target-node evidence.** Package resolution or import success
|
|
82
|
-
alone is insufficient for native extensions and accelerator software. Validate the relevant
|
|
83
|
-
ABI, import, kernel, functional, and end-to-end layers.
|
|
84
|
-
|
|
85
|
-
## Repository tools
|
|
86
|
-
|
|
87
|
-
| Tool | Purpose |
|
|
88
|
-
|---|---|
|
|
89
|
-
| `scripts/setup-ssh.sh` | Install a supplied key and add an SSH host entry |
|
|
90
|
-
| `scripts/new-job.sh` | Generate a profile-aware Slurm script |
|
|
91
|
-
| `scripts/probe-cluster.sh` | Produce an initial profile from read-only login-node probes |
|
|
92
|
-
| `scripts/refresh-cluster.sh` | Refresh dynamic profile fields; compute probing is opt-in |
|
|
93
|
-
| `scripts/run-compute-probe.sh` | Submit a minimal accelerator capability probe |
|
|
94
|
-
|
|
95
|
-
Use `scripts/new-job.sh --cluster <name> ...` for routine job generation. Validate the generated
|
|
96
|
-
script with `sbatch --test-only` before submission when the target cluster supports it.
|
|
97
|
-
|
|
98
|
-
## Evidence and reporting
|
|
99
|
-
|
|
100
|
-
Distinguish:
|
|
101
|
-
|
|
102
|
-
- profile values;
|
|
103
|
-
- live scheduler observations;
|
|
104
|
-
- login-node observations;
|
|
105
|
-
- compute-node observations;
|
|
106
|
-
- conclusions inferred from those observations.
|
|
107
|
-
|
|
108
|
-
For failures, report the failing command, relevant log excerpt, scheduler state and exit code,
|
|
109
|
-
target profile, and the next bounded diagnostic action. For compatibility results, state the
|
|
110
|
-
software and toolchain versions, accelerator architecture, validation layer reached, and
|
|
111
|
-
untested boundaries.
|
|
8
|
+
Use the selected backend, cluster profile, and target-cluster evidence. SSH is the default;
|
|
9
|
+
OpenAPI is the structured control plane. Never silently switch backends.
|
|
10
|
+
|
|
11
|
+
## First use and selection
|
|
12
|
+
|
|
13
|
+
- Run `scripts/setup.sh new|modify|status|reset` for explicit configuration lifecycle.
|
|
14
|
+
- Advanced OpenAPI discovery: `python3 scripts/scnet.py setup`.
|
|
15
|
+
- Read-only preflight: `python3 scripts/scnet.py doctor`.
|
|
16
|
+
- Backend precedence: `--backend`, `SCNET_HPC_BACKEND`, saved setup choice, profile
|
|
17
|
+
`DEFAULT_BACKEND`, then `ssh`.
|
|
18
|
+
- Read `clusters/<cluster>.conf` before reporting static facts. Treat
|
|
19
|
+
`clusters/.cache/<cluster>.auto.conf` as optional, time-sensitive SSH probe data.
|
|
20
|
+
- With multiple profiles, require an explicit cluster before access changes, installation,
|
|
21
|
+
job generation, or submission. Query the selected backend for live queues, limits, and state.
|
|
22
|
+
- SSH setup is profile-scoped; record usernames and key expiry metadata per profile. OpenAPI
|
|
23
|
+
credentials are platform-scoped and discover all authorized regions/schedulers automatically.
|
|
24
|
+
- OpenAPI may enable multiple regions locally, but every mutating operation must resolve to one
|
|
25
|
+
explicit/default target region.
|
|
26
|
+
|
|
27
|
+
Read [references/backends.md](references/backends.md) for backend selection and extension.
|
|
28
|
+
Read [references/openapi.md](references/openapi.md) for OpenAPI credentials and limits.
|
|
29
|
+
|
|
30
|
+
## Route only as needed
|
|
31
|
+
|
|
32
|
+
- SSH access: `references/setup.md`
|
|
33
|
+
- Python/modules/offline dependencies: `references/environment.md`
|
|
34
|
+
- Scheduler/runtime/accelerator diagnosis: `references/troubleshooting.md`
|
|
35
|
+
- Add or refresh profiles: `references/adding-cluster.md`
|
|
36
|
+
- Hygon DCU/DTK/HIP/profiling: `references/hygon-dcu-development.md`
|
|
37
|
+
- Compatibility reports: `references/software-compatibility.md`
|
|
38
|
+
- English instructions: `references/quickstart-en.md`
|
|
39
|
+
|
|
40
|
+
Read only the selected profile and references needed for the current operation. Profile values
|
|
41
|
+
override examples.
|
|
42
|
+
|
|
43
|
+
## Boundaries and evidence
|
|
44
|
+
|
|
45
|
+
- Read-only inspection and local generation do not authorize remote mutation.
|
|
46
|
+
- Resolve backend, target, paths, command, and expected effect before submit, cancel, upload,
|
|
47
|
+
download, overwrite, deletion, or SSH configuration. Never silently retry submission.
|
|
48
|
+
- Compute probes consume scheduler resources; run them only when requested or required for
|
|
49
|
+
target-node evidence.
|
|
50
|
+
- Keep credentials, tokens, personal paths, internal hosts, nodes, job IDs, and private mirrors
|
|
51
|
+
out of profiles and public reports.
|
|
52
|
+
- Preserve existing SSH configuration; use `scripts/setup-ssh.sh` for bounded changes.
|
|
53
|
+
- Respect profile-specific partition, GRES, memory, walltime, offline-network, and shared-storage
|
|
54
|
+
constraints. Propagate workload exit status.
|
|
55
|
+
- Distinguish profile values, live scheduler state, login-node evidence, compute-node evidence,
|
|
56
|
+
and inference. Treat Hygon guidance as architecture/toolchain-specific.
|
|
57
|
+
|
|
58
|
+
## Stable entrypoints
|
|
59
|
+
|
|
60
|
+
- `scripts/setup.sh`: first-use configuration panel.
|
|
61
|
+
- `python3 scripts/scnet.py ...`: backend-neutral operations, dry-run, raw output, and doctor.
|
|
62
|
+
- `python3 scripts/scnet.py notebook ...`: Notebook discovery and confirmation-driven
|
|
63
|
+
lifecycle operations. Use `--dry-run` before create/start/stop/release.
|
|
64
|
+
- `scripts/new-job.sh`: profile-aware Slurm generation.
|
|
65
|
+
- `scripts/refresh-cluster.sh`: SSH-derived profile refresh.
|
|
66
|
+
- `scripts/probe-cluster.sh`: initial profile creation.
|
|
67
|
+
- `scripts/run-compute-probe.sh`: bounded accelerator probe.
|
|
68
|
+
|
|
69
|
+
Use compact normalized output by default. Request raw backend data only for diagnosis.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
0.5.0
|
|
@@ -5,13 +5,14 @@
|
|
|
5
5
|
# 2. 填下面的值(不确定的用 scripts/probe-cluster.sh 探测)
|
|
6
6
|
# 3. scripts/setup-ssh.sh --cluster <集群短名> <私钥文件> [用户名]
|
|
7
7
|
#
|
|
8
|
-
# CLUSTER_ID
|
|
9
|
-
#
|
|
10
|
-
# _common.sh
|
|
8
|
+
# CLUSTER_ID 是 profile 标识。SSH backend 还要求 SSH_HOST / SSH_PORT;OpenAPI backend
|
|
9
|
+
# 要求 OPENAPI_REGION_ID,并在多调度器区域填写 OPENAPI_SCHEDULER_ID。其余可留空。
|
|
10
|
+
# 旧 SSH profile 漏填 SSH_HOST 时,_common.sh 会从本机 ssh config 尝试补全。
|
|
11
11
|
|
|
12
|
-
# ----
|
|
12
|
+
# ---- 通用与 SSH 连接 ----
|
|
13
13
|
CLUSTER_ID="" # 短名,同时用作 ssh 别名,如 "zzeshell"
|
|
14
14
|
CLUSTER_DESC="" # 人类可读描述,如 "海光 DCU(郑州)"
|
|
15
|
+
DEFAULT_BACKEND="ssh" # ssh / openapi / 已注册的外部 backend
|
|
15
16
|
SSH_HOST="" # 如 "xxx.scnet.cn"
|
|
16
17
|
SSH_PORT="22"
|
|
17
18
|
HOME_BASE="/public/home" # 家目录父路径
|
|
@@ -19,6 +20,11 @@ HOME_BASE="/public/home" # 家目录父路径
|
|
|
19
20
|
KEY_NAME_MARKER="" # 私钥文件名里的主机标识,如 "_xxx.scnet.cn_"
|
|
20
21
|
NEEDS_UPDATE_HOSTKEYS_NO="no" # 平台主机密钥签名有问题时设 yes
|
|
21
22
|
|
|
23
|
+
# ---- OpenAPI(可选,不得写入凭据)----
|
|
24
|
+
OPENAPI_REGION_ID="" # SCNet 授权区域 ID
|
|
25
|
+
OPENAPI_SCHEDULER_ID="" # 区域存在多个调度器时填写
|
|
26
|
+
OPENAPI_USERNAME="" # 通常由区域信息自动返回
|
|
27
|
+
|
|
22
28
|
# ---- 调度器 ----
|
|
23
29
|
SCHEDULER="slurm"
|
|
24
30
|
PARTITION=""
|
|
@@ -134,12 +134,17 @@ Triton 是多种推理框架的运行前提;不可用时,对应的 Triton
|
|
|
134
134
|
|
|
135
135
|
## profile 字段参考
|
|
136
136
|
|
|
137
|
-
|
|
137
|
+
`CLUSTER_ID` 是 profile 标识。使用 SSH backend 时还需要 `SSH_HOST` 和 `SSH_PORT`;
|
|
138
|
+
使用 OpenAPI backend 时需要 `OPENAPI_REGION_ID`,区域存在多个调度器时再填写
|
|
139
|
+
`OPENAPI_SCHEDULER_ID`。不得把 AK、SK 或 token 写进 profile。
|
|
138
140
|
|
|
139
141
|
影响脚本行为的关键项:
|
|
140
142
|
|
|
141
143
|
| 字段 | 谁用 | 留空的后果 |
|
|
142
144
|
|---|---|---|
|
|
145
|
+
| `DEFAULT_BACKEND` | `scnet.py` | 默认使用 `ssh` |
|
|
146
|
+
| `OPENAPI_REGION_ID` | OpenAPI backend | 多区域账户需要每次显式传 `--region` |
|
|
147
|
+
| `OPENAPI_SCHEDULER_ID` | OpenAPI backend | 多调度器区域需要每次显式传 `--scheduler-id` |
|
|
143
148
|
| `MIN_GRES` | 文档提示 | 不提示 QOS 强制申请加速器 |
|
|
144
149
|
| `GRES_TYPE` | `new-job.sh` | 生成的脚本不含 `--gres` |
|
|
145
150
|
| `DEF_MEM_PER_CPU` | `new-job.sh` | 生成的脚本不含 `--mem`,需手填 |
|
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
# SCNet backend selection and extension
|
|
2
|
+
|
|
3
|
+
The skill separates cluster intent from transport. SSH and OpenAPI are built in; future MCP
|
|
4
|
+
servers, platform connectors, and site-specific gateways can be added without changing the
|
|
5
|
+
task-level commands.
|
|
6
|
+
|
|
7
|
+
## Selection
|
|
8
|
+
|
|
9
|
+
Backend precedence:
|
|
10
|
+
|
|
11
|
+
1. `--backend <name>`;
|
|
12
|
+
2. `SCNET_HPC_BACKEND`;
|
|
13
|
+
3. saved setup-panel `default_backend`;
|
|
14
|
+
4. selected profile `DEFAULT_BACKEND`;
|
|
15
|
+
5. `ssh`.
|
|
16
|
+
|
|
17
|
+
Do not silently fall back between backends. A failed OpenAPI request must not become an SSH
|
|
18
|
+
submission, and a failed SSH command must not be replayed through OpenAPI. Replaying mutations
|
|
19
|
+
can duplicate jobs or apply them to a different region.
|
|
20
|
+
|
|
21
|
+
List registered implementations and their declared capabilities:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
python3 scripts/scnet.py backends
|
|
25
|
+
python3 scripts/scnet.py --backend <name> capabilities
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Built-in capability summary:
|
|
29
|
+
|
|
30
|
+
| Domain / operation | SSH | OpenAPI |
|
|
31
|
+
|---|---:|---:|
|
|
32
|
+
| Local profiles / authorized regions | yes | yes |
|
|
33
|
+
| Queues and limits | yes | yes |
|
|
34
|
+
| Job status and logs | yes | yes |
|
|
35
|
+
| Submit and cancel | yes | yes |
|
|
36
|
+
| Upload and download | yes | yes |
|
|
37
|
+
| Create remote directory | no | yes |
|
|
38
|
+
| Arbitrary remote command | yes | no |
|
|
39
|
+
| Modules, builds, interactive diagnosis | yes | no |
|
|
40
|
+
| Notebook read-only discovery | no | yes |
|
|
41
|
+
|
|
42
|
+
Use OpenAPI for structured control-plane work. Use SSH for environment setup, compilation,
|
|
43
|
+
interactive inspection, full scheduler tooling, and operations not represented by a structured
|
|
44
|
+
backend.
|
|
45
|
+
|
|
46
|
+
## Common CLI
|
|
47
|
+
|
|
48
|
+
On a new computer, start with the dependency-light Bash panel:
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
./scripts/setup.sh
|
|
52
|
+
python3 scripts/scnet.py doctor
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Configuration lifecycle:
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
./scripts/setup.sh new
|
|
59
|
+
./scripts/setup.sh modify
|
|
60
|
+
./scripts/setup.sh status
|
|
61
|
+
./scripts/setup.sh reset
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
`new` adds a connection without replacing an existing profile/account. `modify` preserves
|
|
65
|
+
unrelated configuration. `status` is read-only. `reset` is scoped and does not remove real SSH
|
|
66
|
+
keys or `~/.ssh/config` unless that is performed separately and explicitly.
|
|
67
|
+
|
|
68
|
+
Use `./scripts/setup.sh --skip-connect` when only local selections should be saved. The Bash
|
|
69
|
+
panel can be rerun to update a profile's SSH user/key metadata. OpenAPI uses one platform
|
|
70
|
+
credential set and discovers regions, users, schedulers, and home paths automatically; no region
|
|
71
|
+
ID is required during setup.
|
|
72
|
+
Developers
|
|
73
|
+
can use `python3 scripts/scnet.py setup` for live OpenAPI region discovery. Both panels store no
|
|
74
|
+
secrets; provide OpenAPI credentials through a host credential manager or environment injection.
|
|
75
|
+
|
|
76
|
+
Region selection is multi-select for local enablement, followed by a single default region.
|
|
77
|
+
Mutating operations remain single-region by design.
|
|
78
|
+
|
|
79
|
+
Global options must appear before the operation:
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
python3 scripts/scnet.py --backend ssh --cluster <profile> queues
|
|
83
|
+
python3 scripts/scnet.py --backend openapi --region <region-id> limits
|
|
84
|
+
python3 scripts/scnet.py --backend openapi --region <region-id> job <job-id>
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Use `--dry-run` before a mutation to inspect the normalized action without contacting the
|
|
88
|
+
backend:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
python3 scripts/scnet.py --backend openapi --dry-run \
|
|
92
|
+
submit --name smoke --queue debug --work-dir /absolute/work/dir \
|
|
93
|
+
--command 'echo hello'
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Mutating examples:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
python3 scripts/scnet.py --backend ssh --cluster <profile> \
|
|
100
|
+
submit --remote-path /absolute/path/job.slurm
|
|
101
|
+
|
|
102
|
+
python3 scripts/scnet.py --backend openapi --region <region-id> submit \
|
|
103
|
+
--name probe --queue debug --work-dir /absolute/work/dir \
|
|
104
|
+
--command 'python3 probe.py' --dcus 1 --cpus 8 --walltime 00:10:00
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
The CLI does not interpret a command as authorization. Follow the skill's mutation boundaries
|
|
108
|
+
before invoking submit, cancel, upload, download, or `exec`.
|
|
109
|
+
|
|
110
|
+
## External backend protocol
|
|
111
|
+
|
|
112
|
+
An external backend is an executable named:
|
|
113
|
+
|
|
114
|
+
```text
|
|
115
|
+
scnet-hpc-backend-<name>
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Discover it either by:
|
|
119
|
+
|
|
120
|
+
- placing it in a directory listed by `SCNET_HPC_BACKEND_PATH`; or
|
|
121
|
+
- placing it on `PATH` and listing `<name>` in `SCNET_HPC_EXTERNAL_BACKENDS`.
|
|
122
|
+
|
|
123
|
+
The router writes one JSON request to standard input:
|
|
124
|
+
|
|
125
|
+
```json
|
|
126
|
+
{
|
|
127
|
+
"protocol": "scnet-hpc.backend",
|
|
128
|
+
"protocol_version": 1,
|
|
129
|
+
"operation": "queues",
|
|
130
|
+
"profile_name": "example",
|
|
131
|
+
"profile": {},
|
|
132
|
+
"options": {}
|
|
133
|
+
}
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
The executable returns one JSON object:
|
|
137
|
+
|
|
138
|
+
```json
|
|
139
|
+
{
|
|
140
|
+
"ok": true,
|
|
141
|
+
"protocol_version": 1,
|
|
142
|
+
"data": []
|
|
143
|
+
}
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
It must implement the `capabilities` operation and return:
|
|
147
|
+
|
|
148
|
+
```json
|
|
149
|
+
{"ok": true, "protocol_version": 1, "data": {"capabilities": ["clusters", "queues"]}}
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Return `{"ok": false, "error": "..."}` for a bounded failure. Never print credentials or tokens
|
|
153
|
+
to standard output or standard error.
|
|
154
|
+
|
|
155
|
+
## MCP and connector integration
|
|
156
|
+
|
|
157
|
+
If an official SCNet MCP server becomes available, prefer its native tools when the host exposes
|
|
158
|
+
them and the user has selected or authorized that connection. Keep the same normalized operation
|
|
159
|
+
semantics: clusters, queues, limits, job, logs, submit, cancel, files, upload, and download.
|
|
160
|
+
|
|
161
|
+
For hosts that cannot expose MCP tools directly, use a small
|
|
162
|
+
`scnet-hpc-backend-mcp` bridge implementing the protocol above. Platform connectors can use the
|
|
163
|
+
same adapter pattern. Do not hard-code speculative MCP tool names, schemas, or authentication
|
|
164
|
+
until the official interface is available and verified.
|
|
165
|
+
|
|
166
|
+
## Profiles
|
|
167
|
+
|
|
168
|
+
Profiles may contain non-secret backend hints:
|
|
169
|
+
|
|
170
|
+
```bash
|
|
171
|
+
DEFAULT_BACKEND="ssh"
|
|
172
|
+
OPENAPI_REGION_ID=""
|
|
173
|
+
OPENAPI_SCHEDULER_ID=""
|
|
174
|
+
OPENAPI_USERNAME=""
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Keep AK, SK, tokens, cookies, and passwords out of profiles.
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Notebook service
|
|
2
|
+
|
|
3
|
+
Notebook is a separate SCNet OpenAPI service domain. It reuses platform authentication and
|
|
4
|
+
region discovery, but it is not a Slurm job and does not use `hpcUrls`.
|
|
5
|
+
|
|
6
|
+
Read-only commands:
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
python3 scripts/scnet.py --backend openapi notebook regions
|
|
10
|
+
python3 scripts/scnet.py --backend openapi --region <id> notebook resources
|
|
11
|
+
python3 scripts/scnet.py --backend openapi --region <id> notebook images
|
|
12
|
+
python3 scripts/scnet.py --backend openapi --region <id> notebook list
|
|
13
|
+
python3 scripts/scnet.py --backend openapi --region <id> notebook show <notebook-id>
|
|
14
|
+
python3 scripts/scnet.py --backend openapi --region <id> notebook url <notebook-id>
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
The default output redacts SSH passwords and URL query credentials. Use `--reveal` only when
|
|
18
|
+
the user explicitly needs a URL to open, and do not place the resulting value in reports or
|
|
19
|
+
long-lived logs.
|
|
20
|
+
|
|
21
|
+
Lifecycle commands:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
python3 scripts/scnet.py --dry-run --backend openapi \
|
|
25
|
+
notebook create --region <id>
|
|
26
|
+
python3 scripts/scnet.py --backend openapi \
|
|
27
|
+
notebook create --region <id>
|
|
28
|
+
python3 scripts/scnet.py --backend openapi \
|
|
29
|
+
notebook start <id> --region <id>
|
|
30
|
+
python3 scripts/scnet.py --backend openapi \
|
|
31
|
+
notebook stop <id> --region <id>
|
|
32
|
+
python3 scripts/scnet.py --backend openapi \
|
|
33
|
+
notebook release <id> --region <id>
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
`create` automatically chooses the smallest currently available resource and the smallest
|
|
37
|
+
trusted preset Jupyter image. Advanced users may override the resource group or image ID.
|
|
38
|
+
|
|
39
|
+
`smoke` performs a bounded lifecycle:
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
python3 scripts/scnet.py --dry-run --backend openapi \
|
|
43
|
+
notebook smoke --region <id>
|
|
44
|
+
python3 scripts/scnet.py --backend openapi \
|
|
45
|
+
notebook smoke --region <id>
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
It creates one minimal Notebook, waits for `Running`, validates the redacted Jupyter URL, then
|
|
49
|
+
stops and releases the instance in a cleanup block.
|
|
50
|
+
|
|
51
|
+
Mutations require interactive confirmation. Non-interactive callers must pass `--yes`. Release
|
|
52
|
+
requires the Notebook ID in interactive mode because it is irreversible.
|
|
53
|
+
|
|
54
|
+
Official API sections:
|
|
55
|
+
|
|
56
|
+
- Notebook resource discovery;
|
|
57
|
+
- image discovery;
|
|
58
|
+
- instance list/detail;
|
|
59
|
+
- Jupyter URL lookup;
|
|
60
|
+
- instance lifecycle.
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# SCNet OpenAPI backend
|
|
2
|
+
|
|
3
|
+
This backend follows the official SCNet OpenAPI 2.0 documentation reviewed on
|
|
4
|
+
2026-09-26:
|
|
5
|
+
|
|
6
|
+
- Overview: <https://www.scnet.cn/ac/openapi/doc/2.0/index.html>
|
|
7
|
+
- Recommended AK/SK authentication:
|
|
8
|
+
<https://www.scnet.cn/ac/openapi/doc/2.0/api/safecertification/get-user-tokens-aksk.html>
|
|
9
|
+
- Job submission:
|
|
10
|
+
<https://www.scnet.cn/ac/openapi/doc/2.0/api/jobmanager/job.html>
|
|
11
|
+
- Creating a directory:
|
|
12
|
+
<https://www.scnet.cn/ac/openapi/doc/2.0/api/efile/folder-create.html>
|
|
13
|
+
- Chunked upload and merge:
|
|
14
|
+
<https://www.scnet.cn/ac/openapi/doc/2.0/api/efile/chunk-upload.html>
|
|
15
|
+
and <https://www.scnet.cn/ac/openapi/doc/2.0/api/efile/merge-file.html>
|
|
16
|
+
|
|
17
|
+
The local implementation intentionally normalizes large responses. Use `--raw` only when the
|
|
18
|
+
compact result is insufficient.
|
|
19
|
+
|
|
20
|
+
## Credentials
|
|
21
|
+
|
|
22
|
+
Obtain AK/SK from the SCNet website:
|
|
23
|
+
|
|
24
|
+
1. Sign in to SCNet.
|
|
25
|
+
2. Open **Personal Center / 个人中心**.
|
|
26
|
+
3. Open **Access Control / 访问控制**.
|
|
27
|
+
4. Generate and download the authorization file containing AccessKey and SecretKey.
|
|
28
|
+
|
|
29
|
+
Official instructions:
|
|
30
|
+
<https://www.scnet.cn/ac/openapi/doc/2.0/api/safecertification/get-user-tokens-aksk.html>
|
|
31
|
+
|
|
32
|
+
Preferred environment variables:
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
export SCNET_OPENAPI_USER="<platform-user>"
|
|
36
|
+
export SCNET_OPENAPI_ACCESS_KEY="<access-key>"
|
|
37
|
+
export SCNET_OPENAPI_SECRET_KEY="<secret-key>"
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The client signs the canonical AK/timestamp/user JSON with HMAC-SHA256 and obtains a fresh
|
|
41
|
+
per-region token for each invocation. It does not persist tokens. The setup panel can store the
|
|
42
|
+
platform credential set in macOS Keychain or Linux Secret Service; otherwise use environment
|
|
43
|
+
variables. Region IDs, region users, schedulers, and home paths are discovered and cached as
|
|
44
|
+
non-secret metadata.
|
|
45
|
+
|
|
46
|
+
Interactive terminals support ↑/↓ plus Enter and direct numeric selection. Press Enter without
|
|
47
|
+
typing a number to accept the current/default item. SecretKey input is intentionally invisible.
|
|
48
|
+
This behavior is the same in macOS Terminal/iTerm2 and Ubuntu/Debian terminals.
|
|
49
|
+
Ubuntu/Debian users can install `secret-tool` with `sudo apt install libsecret-tools`.
|
|
50
|
+
|
|
51
|
+
For an externally managed token:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
export SCNET_OPENAPI_TOKEN="<region-token>"
|
|
55
|
+
export SCNET_OPENAPI_REGION_ID="<region-id>"
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Optional overrides:
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
export SCNET_OPENAPI_AUTH_BASE="https://api.scnet.cn"
|
|
62
|
+
export SCNET_OPENAPI_CENTER_URL="https://www.scnet.cn/ac/openapi/v2/center"
|
|
63
|
+
export SCNET_OPENAPI_SCHEDULER_ID="<scheduler-id>"
|
|
64
|
+
export SCNET_OPENAPI_USERNAME="<region-user>"
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Never put these values in a prompt, profile, repository file, generated report, or shell trace.
|
|
68
|
+
Use the host's credential store or secret injection when available.
|
|
69
|
+
|
|
70
|
+
## Read-only operations
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
python3 scripts/scnet.py --backend openapi clusters
|
|
74
|
+
python3 scripts/scnet.py --backend openapi --region <id> queues
|
|
75
|
+
python3 scripts/scnet.py --backend openapi --region <id> limits
|
|
76
|
+
python3 scripts/scnet.py --backend openapi --region <id> job <job-id>
|
|
77
|
+
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
78
|
+
logs --path /absolute/path/std.out.123 --lines 200
|
|
79
|
+
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
80
|
+
files --path /absolute/path
|
|
81
|
+
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
82
|
+
mkdir /absolute/work/dir --parents
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
If a region exposes multiple schedulers, add `--scheduler-id`.
|
|
86
|
+
|
|
87
|
+
## Mutations
|
|
88
|
+
|
|
89
|
+
OpenAPI submission uses the documented BASIC/BASE command mode. Extra scheduler directives can
|
|
90
|
+
be repeated:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
python3 scripts/scnet.py --backend openapi --region <id> submit \
|
|
94
|
+
--name probe --queue debug --work-dir /absolute/work/dir \
|
|
95
|
+
--command $'module load ...\npython3 probe.py' \
|
|
96
|
+
--nodes 1 --cpus 8 --dcus 1 --walltime 00:10:00 \
|
|
97
|
+
--scheduler-option '#SBATCH --account=example'
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Other mutations:
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
python3 scripts/scnet.py --backend openapi --region <id> cancel <job-id>
|
|
104
|
+
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
105
|
+
upload local.file /absolute/remote/directory
|
|
106
|
+
python3 scripts/scnet.py --backend openapi --region <id> \
|
|
107
|
+
download /absolute/remote/file local.file
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Downloads refuse to replace an existing local path unless `--cover` is supplied. Uploads send
|
|
111
|
+
`uncover` by default and require `--cover` for replacement.
|
|
112
|
+
|
|
113
|
+
Uploads larger than 8 MiB automatically use the documented burst/merge flow. Override the
|
|
114
|
+
threshold and chunk size with `--chunk-size <bytes>`; use `--chunk-size 0` to force ordinary
|
|
115
|
+
multipart upload. The current implementation keeps one chunk in memory at a time.
|
|
116
|
+
|
|
117
|
+
Preview a mutation without contacting the backend:
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
python3 scripts/scnet.py --backend openapi --region <id> --dry-run \
|
|
121
|
+
submit --name probe --queue debug --work-dir /absolute/work/dir \
|
|
122
|
+
--command 'echo hello' --cpus 1 --dcus 1
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Do not automatically retry submission or cancellation after a timeout. Query job state first:
|
|
126
|
+
the public documentation does not establish an idempotency key for submission.
|
|
127
|
+
|
|
128
|
+
## Known validation boundaries
|
|
129
|
+
|
|
130
|
+
- The implementation is validated locally for parsing, signing, routing, normalization, and the
|
|
131
|
+
external-backend contract. Live OpenAPI operations still require testing against each
|
|
132
|
+
authorized SCNet region.
|
|
133
|
+
- The official cancellation page describes one method value in prose and uses `5` in all request
|
|
134
|
+
examples. The client defaults to `5`; override it with `SCNET_OPENAPI_CANCEL_METHOD` if a live
|
|
135
|
+
region requires a different documented value.
|
|
136
|
+
- Small ordinary uploads buffer one file in memory; large uploads use chunked transfer.
|
|
137
|
+
- OpenAPI does not replace SSH for module discovery, environment construction, compilation,
|
|
138
|
+
interactive `srun`, performance profiling, or low-level scheduler diagnosis.
|
|
139
|
+
- API success is determined by both HTTP transport and the JSON `code`; HTTP 200 alone is not
|
|
140
|
+
sufficient.
|
|
@@ -19,6 +19,28 @@ Check at least `PARTITION`, `PARTITION_CPU`, `GRES_TYPE`, `MIN_GRES`,
|
|
|
19
19
|
`DEF_MEM_PER_CPU`, `MODULE_LOADS`, `COMPUTE_NODE_OFFLINE`, and
|
|
20
20
|
`KNOWN_LIMITATIONS`.
|
|
21
21
|
|
|
22
|
+
On a new computer, run the Bash setup panel and then its read-only doctor:
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
./scripts/setup.sh
|
|
26
|
+
python3 scripts/scnet.py doctor
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
List available access backends:
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
python3 scripts/scnet.py backends
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
SSH is the default. Use OpenAPI for structured queue, limit, job, log, and file operations:
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
python3 scripts/scnet.py --backend openapi clusters
|
|
39
|
+
python3 scripts/scnet.py --backend openapi --region <region-id> queues
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
See `references/backends.md` and `references/openapi.md` before configuring another backend.
|
|
43
|
+
|
|
22
44
|
## 2. Install the skill
|
|
23
45
|
|
|
24
46
|
```bash
|
|
@@ -85,6 +85,10 @@ ssh -O exit <集群> # 手工断开(网络切换后卡住时用)
|
|
|
85
85
|
`<用户名>_<主机>_RsaKeyExpireTime_2026-10-29_12-51-40.txt` 表示 2026-10-29 过期。
|
|
86
86
|
到期前从控制台重新下载并重跑 setup 脚本。
|
|
87
87
|
|
|
88
|
+
可以重复运行 `scripts/setup.sh` 更新某个 profile 的用户名和私钥。面板会从密钥文件名
|
|
89
|
+
中的 `RsaKeyExpireTime_YYYY-MM-DD` 自动记录有效期,`python3 scripts/scnet.py doctor`
|
|
90
|
+
会报告剩余天数和已过期状态。密钥内容不会进入配置文件。
|
|
91
|
+
|
|
88
92
|
profile 里的 `KEY_NAME_MARKER` 就是用来从这种文件名里提取用户名的;公开
|
|
89
93
|
`scnet-hpc` profile 使用 `_<CLUSTER_HOST>_` 占位符,所以安装时通常需要显式传
|
|
90
94
|
用户名,或在本地私有 profile 中补齐真实标记。
|
|
Binary file
|