dsh-scnet 0.4.2 → 0.5.0

package/skills/scnet-hpc/references/backends.md

@@ -0,0 +1,177 @@
+# SCNet backend selection and extension
+
+The skill separates cluster intent from transport. SSH and OpenAPI are built in; future MCP
+servers, platform connectors, and site-specific gateways can be added without changing the
+task-level commands.
+
+## Selection
+
+Backend precedence:
+
+1. `--backend <name>`;
+2. `SCNET_HPC_BACKEND`;
+3. saved setup-panel `default_backend`;
+4. selected profile `DEFAULT_BACKEND`;
+5. `ssh`.
+
+Do not silently fall back between backends. A failed OpenAPI request must not become an SSH
+submission, and a failed SSH command must not be replayed through OpenAPI. Replaying mutations
+can duplicate jobs or apply them to a different region.
+
+List registered implementations and their declared capabilities:
+
+```bash
+python3 scripts/scnet.py backends
+python3 scripts/scnet.py --backend <name> capabilities
+```
+
+Built-in capability summary:
+
+| Domain / operation | SSH | OpenAPI |
+|---|---:|---:|
+| Local profiles / authorized regions | yes | yes |
+| Queues and limits | yes | yes |
+| Job status and logs | yes | yes |
+| Submit and cancel | yes | yes |
+| Upload and download | yes | yes |
+| Create remote directory | no | yes |
+| Arbitrary remote command | yes | no |
+| Modules, builds, interactive diagnosis | yes | no |
+| Notebook read-only discovery | no | yes |
+
+Use OpenAPI for structured control-plane work. Use SSH for environment setup, compilation,
+interactive inspection, full scheduler tooling, and operations not represented by a structured
+backend.
+
+## Common CLI
+
+On a new computer, start with the dependency-light Bash panel:
+
+```bash
+./scripts/setup.sh
+python3 scripts/scnet.py doctor
+```
+
+Configuration lifecycle:
+
+```bash
+./scripts/setup.sh new
+./scripts/setup.sh modify
+./scripts/setup.sh status
+./scripts/setup.sh reset
+```
+
+`new` adds a connection without replacing an existing profile/account. `modify` preserves
+unrelated configuration. `status` is read-only. `reset` is scoped and does not remove real SSH
+keys or `~/.ssh/config` unless that is performed separately and explicitly.
+
+Use `./scripts/setup.sh --skip-connect` when only local selections should be saved. The Bash
+panel can be rerun to update a profile's SSH user/key metadata. OpenAPI uses one platform
+credential set and discovers regions, users, schedulers, and home paths automatically; no region
+ID is required during setup.
+Developers
+can use `python3 scripts/scnet.py setup` for live OpenAPI region discovery. Both panels store no
+secrets; provide OpenAPI credentials through a host credential manager or environment injection.
+
+Region selection is multi-select for local enablement, followed by a single default region.
+Mutating operations remain single-region by design.
+
+Global options must appear before the operation:
+
+```bash
+python3 scripts/scnet.py --backend ssh --cluster <profile> queues
+python3 scripts/scnet.py --backend openapi --region <region-id> limits
+python3 scripts/scnet.py --backend openapi --region <region-id> job <job-id>
+```
+
+Use `--dry-run` before a mutation to inspect the normalized action without contacting the
+backend:
+
+```bash
+python3 scripts/scnet.py --backend openapi --dry-run \
+  submit --name smoke --queue debug --work-dir /absolute/work/dir \
+  --command 'echo hello'
+```
+
+Mutating examples:
+
+```bash
+python3 scripts/scnet.py --backend ssh --cluster <profile> \
+  submit --remote-path /absolute/path/job.slurm
+
+python3 scripts/scnet.py --backend openapi --region <region-id> submit \
+  --name probe --queue debug --work-dir /absolute/work/dir \
+  --command 'python3 probe.py' --dcus 1 --cpus 8 --walltime 00:10:00
+```
+
+The CLI does not interpret a command as authorization. Follow the skill's mutation boundaries
+before invoking submit, cancel, upload, download, or `exec`.
+
+## External backend protocol
+
+An external backend is an executable named:
+
+```text
+scnet-hpc-backend-<name>
+```
+
+Discover it either by:
+
+- placing it in a directory listed by `SCNET_HPC_BACKEND_PATH`; or
+- placing it on `PATH` and listing `<name>` in `SCNET_HPC_EXTERNAL_BACKENDS`.
+
+The router writes one JSON request to standard input:
+
+```json
+{
+  "protocol": "scnet-hpc.backend",
+  "protocol_version": 1,
+  "operation": "queues",
+  "profile_name": "example",
+  "profile": {},
+  "options": {}
+}
+```
+
+The executable returns one JSON object:
+
+```json
+{
+  "ok": true,
+  "protocol_version": 1,
+  "data": []
+}
+```
+
+It must implement the `capabilities` operation and return:
+
+```json
+{"ok": true, "protocol_version": 1, "data": {"capabilities": ["clusters", "queues"]}}
+```
+
+Return `{"ok": false, "error": "..."}` for a bounded failure. Never print credentials or tokens
+to standard output or standard error.
+
+## MCP and connector integration
+
+If an official SCNet MCP server becomes available, prefer its native tools when the host exposes
+them and the user has selected or authorized that connection. Keep the same normalized operation
+semantics: clusters, queues, limits, job, logs, submit, cancel, files, upload, and download.
+
+For hosts that cannot expose MCP tools directly, use a small
+`scnet-hpc-backend-mcp` bridge implementing the protocol above. Platform connectors can use the
+same adapter pattern. Do not hard-code speculative MCP tool names, schemas, or authentication
+until the official interface is available and verified.
+
+## Profiles
+
+Profiles may contain non-secret backend hints:
+
+```bash
+DEFAULT_BACKEND="ssh"
+OPENAPI_REGION_ID=""
+OPENAPI_SCHEDULER_ID=""
+OPENAPI_USERNAME=""
+```
+
+Keep AK, SK, tokens, cookies, and passwords out of profiles.

package/skills/scnet-hpc/references/notebook.md

@@ -0,0 +1,60 @@
+# Notebook service
+
+Notebook is a separate SCNet OpenAPI service domain. It reuses platform authentication and
+region discovery, but it is not a Slurm job and does not use `hpcUrls`.
+
+Read-only commands:
+
+```bash
+python3 scripts/scnet.py --backend openapi notebook regions
+python3 scripts/scnet.py --backend openapi --region <id> notebook resources
+python3 scripts/scnet.py --backend openapi --region <id> notebook images
+python3 scripts/scnet.py --backend openapi --region <id> notebook list
+python3 scripts/scnet.py --backend openapi --region <id> notebook show <notebook-id>
+python3 scripts/scnet.py --backend openapi --region <id> notebook url <notebook-id>
+```
+
+The default output redacts SSH passwords and URL query credentials. Use `--reveal` only when
+the user explicitly needs a URL to open, and do not place the resulting value in reports or
+long-lived logs.
+
+Lifecycle commands:
+
+```bash
+python3 scripts/scnet.py --dry-run --backend openapi \
+  notebook create --region <id>
+python3 scripts/scnet.py --backend openapi \
+  notebook create --region <id>
+python3 scripts/scnet.py --backend openapi \
+  notebook start <id> --region <id>
+python3 scripts/scnet.py --backend openapi \
+  notebook stop <id> --region <id>
+python3 scripts/scnet.py --backend openapi \
+  notebook release <id> --region <id>
+```
+
+`create` automatically chooses the smallest currently available resource and the smallest
+trusted preset Jupyter image. Advanced users may override the resource group or image ID.
+
+`smoke` performs a bounded lifecycle:
+
+```bash
+python3 scripts/scnet.py --dry-run --backend openapi \
+  notebook smoke --region <id>
+python3 scripts/scnet.py --backend openapi \
+  notebook smoke --region <id>
+```
+
+It creates one minimal Notebook, waits for `Running`, validates the redacted Jupyter URL, then
+stops and releases the instance in a cleanup block.
+
+Mutations require interactive confirmation. Non-interactive callers must pass `--yes`. Release
+requires the Notebook ID in interactive mode because it is irreversible.
+
+Official API sections:
+
+- Notebook resource discovery;
+- image discovery;
+- instance list/detail;
+- Jupyter URL lookup;
+- instance lifecycle.

package/skills/scnet-hpc/references/openapi.md

@@ -0,0 +1,140 @@
+# SCNet OpenAPI backend
+
+This backend follows the official SCNet OpenAPI 2.0 documentation reviewed on
+2026-09-26:
+
+- Overview: <https://www.scnet.cn/ac/openapi/doc/2.0/index.html>
+- Recommended AK/SK authentication:
+  <https://www.scnet.cn/ac/openapi/doc/2.0/api/safecertification/get-user-tokens-aksk.html>
+- Job submission:
+  <https://www.scnet.cn/ac/openapi/doc/2.0/api/jobmanager/job.html>
+- Creating a directory:
+  <https://www.scnet.cn/ac/openapi/doc/2.0/api/efile/folder-create.html>
+- Chunked upload and merge:
+  <https://www.scnet.cn/ac/openapi/doc/2.0/api/efile/chunk-upload.html>
+  and <https://www.scnet.cn/ac/openapi/doc/2.0/api/efile/merge-file.html>
+
+The local implementation intentionally normalizes large responses. Use `--raw` only when the
+compact result is insufficient.
+
+## Credentials
+
+Obtain AK/SK from the SCNet website:
+
+1. Sign in to SCNet.
+2. Open **Personal Center / 个人中心**.
+3. Open **Access Control / 访问控制**.
+4. Generate and download the authorization file containing AccessKey and SecretKey.
+
+Official instructions:
+<https://www.scnet.cn/ac/openapi/doc/2.0/api/safecertification/get-user-tokens-aksk.html>
+
+Preferred environment variables:
+
+```bash
+export SCNET_OPENAPI_USER="<platform-user>"
+export SCNET_OPENAPI_ACCESS_KEY="<access-key>"
+export SCNET_OPENAPI_SECRET_KEY="<secret-key>"
+```
+
+The client signs the canonical AK/timestamp/user JSON with HMAC-SHA256 and obtains a fresh
+per-region token for each invocation. It does not persist tokens. The setup panel can store the
+platform credential set in macOS Keychain or Linux Secret Service; otherwise use environment
+variables. Region IDs, region users, schedulers, and home paths are discovered and cached as
+non-secret metadata.
+
+Interactive terminals support ↑/↓ plus Enter and direct numeric selection. Press Enter without
+typing a number to accept the current/default item. SecretKey input is intentionally invisible.
+This behavior is the same in macOS Terminal/iTerm2 and Ubuntu/Debian terminals.
+Ubuntu/Debian users can install `secret-tool` with `sudo apt install libsecret-tools`.
+
+For an externally managed token:
+
+```bash
+export SCNET_OPENAPI_TOKEN="<region-token>"
+export SCNET_OPENAPI_REGION_ID="<region-id>"
+```
+
+Optional overrides:
+
+```bash
+export SCNET_OPENAPI_AUTH_BASE="https://api.scnet.cn"
+export SCNET_OPENAPI_CENTER_URL="https://www.scnet.cn/ac/openapi/v2/center"
+export SCNET_OPENAPI_SCHEDULER_ID="<scheduler-id>"
+export SCNET_OPENAPI_USERNAME="<region-user>"
+```
+
+Never put these values in a prompt, profile, repository file, generated report, or shell trace.
+Use the host's credential store or secret injection when available.
+
+## Read-only operations
+
+```bash
+python3 scripts/scnet.py --backend openapi clusters
+python3 scripts/scnet.py --backend openapi --region <id> queues
+python3 scripts/scnet.py --backend openapi --region <id> limits
+python3 scripts/scnet.py --backend openapi --region <id> job <job-id>
+python3 scripts/scnet.py --backend openapi --region <id> \
+  logs --path /absolute/path/std.out.123 --lines 200
+python3 scripts/scnet.py --backend openapi --region <id> \
+  files --path /absolute/path
+python3 scripts/scnet.py --backend openapi --region <id> \
+  mkdir /absolute/work/dir --parents
+```
+
+If a region exposes multiple schedulers, add `--scheduler-id`.
+
+## Mutations
+
+OpenAPI submission uses the documented BASIC/BASE command mode. Extra scheduler directives can
+be repeated:
+
+```bash
+python3 scripts/scnet.py --backend openapi --region <id> submit \
+  --name probe --queue debug --work-dir /absolute/work/dir \
+  --command $'module load ...\npython3 probe.py' \
+  --nodes 1 --cpus 8 --dcus 1 --walltime 00:10:00 \
+  --scheduler-option '#SBATCH --account=example'
+```
+
+Other mutations:
+
+```bash
+python3 scripts/scnet.py --backend openapi --region <id> cancel <job-id>
+python3 scripts/scnet.py --backend openapi --region <id> \
+  upload local.file /absolute/remote/directory
+python3 scripts/scnet.py --backend openapi --region <id> \
+  download /absolute/remote/file local.file
+```
+
+Downloads refuse to replace an existing local path unless `--cover` is supplied. Uploads send
+`uncover` by default and require `--cover` for replacement.
+
+Uploads larger than 8 MiB automatically use the documented burst/merge flow. Override the
+threshold and chunk size with `--chunk-size <bytes>`; use `--chunk-size 0` to force ordinary
+multipart upload. The current implementation keeps one chunk in memory at a time.
+
+Preview a mutation without contacting the backend:
+
+```bash
+python3 scripts/scnet.py --backend openapi --region <id> --dry-run \
+  submit --name probe --queue debug --work-dir /absolute/work/dir \
+  --command 'echo hello' --cpus 1 --dcus 1
+```
+
+Do not automatically retry submission or cancellation after a timeout. Query job state first:
+the public documentation does not establish an idempotency key for submission.
+
+## Known validation boundaries
+
+- The implementation is validated locally for parsing, signing, routing, normalization, and the
+  external-backend contract. Live OpenAPI operations still require testing against each
+  authorized SCNet region.
+- The official cancellation page describes one method value in prose and uses `5` in all request
+  examples. The client defaults to `5`; override it with `SCNET_OPENAPI_CANCEL_METHOD` if a live
+  region requires a different documented value.
+- Small ordinary uploads buffer one file in memory; large uploads use chunked transfer.
+- OpenAPI does not replace SSH for module discovery, environment construction, compilation,
+  interactive `srun`, performance profiling, or low-level scheduler diagnosis.
+- API success is determined by both HTTP transport and the JSON `code`; HTTP 200 alone is not
+  sufficient.

package/skills/scnet-hpc/references/quickstart-en.md

@@ -19,6 +19,28 @@ Check at least `PARTITION`, `PARTITION_CPU`, `GRES_TYPE`, `MIN_GRES`,
 `DEF_MEM_PER_CPU`, `MODULE_LOADS`, `COMPUTE_NODE_OFFLINE`, and
 `KNOWN_LIMITATIONS`.
 
+On a new computer, run the Bash setup panel and then its read-only doctor:
+
+```bash
+./scripts/setup.sh
+python3 scripts/scnet.py doctor
+```
+
+List available access backends:
+
+```bash
+python3 scripts/scnet.py backends
+```
+
+SSH is the default. Use OpenAPI for structured queue, limit, job, log, and file operations:
+
+```bash
+python3 scripts/scnet.py --backend openapi clusters
+python3 scripts/scnet.py --backend openapi --region <region-id> queues
+```
+
+See `references/backends.md` and `references/openapi.md` before configuring another backend.
+
 ## 2. Install the skill
 
 ```bash

package/skills/scnet-hpc/references/setup.md

@@ -85,6 +85,10 @@ ssh -O exit <集群>      # 手工断开（网络切换后卡住时用）
 `<用户名>_<主机>_RsaKeyExpireTime_2026-10-29_12-51-40.txt` 表示 2026-10-29 过期。
 到期前从控制台重新下载并重跑 setup 脚本。
 
+可以重复运行 `scripts/setup.sh` 更新某个 profile 的用户名和私钥。面板会从密钥文件名
+中的 `RsaKeyExpireTime_YYYY-MM-DD` 自动记录有效期，`python3 scripts/scnet.py doctor`
+会报告剩余天数和已过期状态。密钥内容不会进入配置文件。
+
 profile 里的 `KEY_NAME_MARKER` 就是用来从这种文件名里提取用户名的；公开
 `scnet-hpc` profile 使用 `_<CLUSTER_HOST>_` 占位符，所以安装时通常需要显式传
 用户名，或在本地私有 profile 中补齐真实标记。