@delta-infra/cli 1.1.21 → 1.1.25

package/README.md

@@ -121,12 +121,6 @@ DELTA_CLI_ARCHIVE=/path/to/delta-cli-linux-amd64.tar.gz npm install -g @delta-in
 | `DELTA_CLI_FATAL_ON_ERROR` | 安装失败时退出码非 0（默认警告，用于调试）。 |
 | `DELTA_CLI_DOWNLOAD_TIMEOUT` | 单个下载源最大等待时间（毫秒，默认 120000）。 |
 
-### 方式五：Go Install
-
-```bash
-go install github.com/delta-infra/delta-infra-cli/cmd/delta-cli@latest
-```
-
 ## 快速开始
 
 ### 1. 初始化配置
@@ -179,6 +173,14 @@ delta-cli auth status
 
 ```bash
 # 创建带 GPU 的 sandbox（--max-life 默认 30 分钟，长任务请调高）
+# ⚠️ 生命周期计时（详见 skills/delta-sandbox/references/lifecycle.md）：
+#   - 私有算力（OpenSandbox）：平台从 create 请求被接受时开始计时，镜像拉取/就绪
+#     时间也计入 max-life；到期由平台回收。
+#   - 第三方算力（AutoDL/RunPod）：从沙箱就绪（本地记录 started_at）开始计时，
+#     由服务端 cleanup 在 max-life + 最多 10 分钟宽限后回收。
+#   - 自动回收（含平台提前回收）不会做 OSS 写回；需要保留产物请提前 pull 或
+#     显式 `sandbox finish`。`sandbox status` 会返回 expires_at / reclaim_source /
+#     termination_reason，沙箱结束后仍可查询。
 # ⚠️ 资源语义：`--memory` 是容器 CPU 内存【总量】（每沙箱一份）；`--gpu-mem` 是【每张卡】的显存（MiB）——
 # `--gpu 2 --gpu-mem 8000` = 每卡 8000MiB、总显存 16000MiB（内部 OpenSandbox/HAMi 后端直接写入 HAMi 每卡注解；
 # 注意云端 runpod 后端将 --gpu-mem 视为总显存÷卡数）。单卡显存上限可查 `sandbox resources`（memory_mib）。
@@ -188,19 +190,20 @@ delta-cli auth status
 # `--devices` 申请宿主机设备直通（逗号分隔，如 kvm → 容器内 /dev/kvm，用于嵌套虚拟化）：属权限提升，
 # 服务端白名单未开启时创建会被明确拒绝（不会静默忽略）。两者仅私有算力生效，第三方算力忽略。
 delta-cli sandbox create \
-  --image image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
+  --image-name image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
   --cpu 4 --memory 16Gi --gpu 1 \
   --max-life 120
 
 # DataLoader 多进程/需要嵌套虚拟化时：调大 /dev/shm 并申请 /dev/kvm（shm 不得超过 memory）
 delta-cli sandbox create \
-  --image image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
+  --image-name image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
   --cpu 4 --memory 8Gi --shm-size 4Gi --devices kvm \
   --max-life 120
 
-# 需要长期存活、不被自动清理的 sandbox：加 --no-auto-cleanup（仅显式 kill/finish 可销毁）
+# 需要长期存活、不被自动清理的 sandbox：加 --no-auto-cleanup
+# （同时关闭平台 TTL 与服务端 cleanup，仅显式 kill/finish 可销毁）
 delta-cli sandbox create \
-  --image image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
+  --image-name image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
   --cpu 4 --memory 16Gi --gpu 1 \
   --no-auto-cleanup
 
@@ -209,15 +212,29 @@ delta-cli sandbox run <sandbox_id> \
   --command "<命令>" \
   --timeout 3600
 
-# v1.0.85+ sandbox run 走 SSE 实时流：stdout 为原始 data: 帧（无末尾信封），exit_code/log_file 在 complete 帧中
-# 需要 stdout 末尾 JSON 摘要时：读取 log_file 解析（sandbox read <id> --path <log_file>）
+# v1.0.85+ sandbox run 默认走 SSE 实时流：stdout 为原始 data: 帧（无末尾信封），
+# exit_code / log_file / stdout_tail / duration_seconds 在 complete 帧中
 delta-cli sandbox run <sandbox_id> --command "echo '{\"status\":\"ok\",\"result\":\"done\"}'" --timeout 60
 
+# 自动化/编排推荐：--format json（等价 --json-result）只输出一个 JSON 信封，
+# data 含 execution_id / finished / exit_code / stdout_tail / stderr_tail /
+# stdout_size / stderr_size / log_file / duration_seconds / result_summary。
+# 字段名与 complete 帧一致；失败时仍是单个对象：
+# {"ok":false,"error":{...},"data":{...}}，进程退出码非零。
+delta-cli sandbox run <sandbox_id> --command "python run.py" --timeout 600 --format json
+
+# 后台任务 + 等待 + 统一结果
+delta-cli sandbox run-bg <sandbox_id> --command "python train.py" --timeout 3600 --wait --format json
+
 # 读取结果文件
 delta-cli sandbox read <sandbox_id> --path /workspace/result.json
 
-# 销毁 sandbox
+# 销毁 sandbox（销毁在服务端继续执行，响应带 request_id）
 delta-cli sandbox kill <sandbox_id>
+
+# 创建重试幂等 + 事后对账
+delta-cli sandbox create --image-name <镜像名> --idempotency-key retry-001
+delta-cli sandbox operation <request_id>
 ```
 
 ## 命令列表
@@ -244,6 +261,8 @@ delta-cli sandbox kill <sandbox_id>
 ### Sandbox 命令
 
 > **提示**：`sandbox list`（列沙箱实例）与 `sandbox ls <id>`（列目录）不同，注意区分。
+>
+> **沙箱内路径**：指向沙箱内部的路径（`--path`、`write-multiple --entry` 的远端部分、`upload --target`、`pull --source`、`run`/`run-bg --log-file`）若为相对路径，会自动拼到沙箱工作目录（`sandbox working-directory <id>` 可查），绝对路径原样使用；指向本地机器的路径（`write --source`、`upload --source`、`pull --target`、`read --output`）不做任何改写。
 
 | 命令 | 说明 |
 |------|------|
@@ -252,14 +271,15 @@ delta-cli sandbox kill <sandbox_id>
 | `delta-cli sandbox resources [--provider opensandbox\|autodl]` | 查看各后端当前剩余可申请资源（GPU/显存/核心） |
 | `delta-cli sandbox images` | 查看可用镜像列表（每条含 `image_name`/`tags`/`description`/`supports_shared_workspace`；`supports_shared_workspace=true` 表示该算力可开共享工作区（opensandbox/autodl），可与 `--shared-workspace` 搭配；实际播种/写回由平台创建时探测镜像内 rclone 决定） |
 | `delta-cli sandbox recommend --cpu N --memory XGi` | 获取资源配置推荐 |
-| `delta-cli sandbox create --image <img> [--shm-size XGi] [--devices kvm] [--no-auto-cleanup] [--shared-workspace=true\|false]` | 创建 sandbox 容器（`--no-auto-cleanup` 不被自动清理；`--shared-workspace` 默认 true：启用共享工作区（播种/写回由平台创建时探测镜像内 rclone 决定，镜像无 rclone 时跳过、数据不跨沙箱持久）；false 跳过播种/写回，适合快速任务；`--shm-size` 调大 /dev/shm（默认 64Mi，**必须带 Gi/Mi 单位**且 ≤ `--memory`，PyTorch DataLoader/multiprocessing 场景常用）；`--devices` 逗号分隔的宿主机设备令牌（`kvm` → /dev/kvm，嵌套虚拟化），**属权限提升**，服务端白名单未开启时**明确拒绝**；`--shm-size`/`--devices` 仅私有算力生效、第三方算力忽略） |
-| `delta-cli sandbox extend <id> --max-life N` | **动态延长运行中** sandbox 的 max-life（可增可减，N 需大于已存活分钟且 ≤10080；提前回收用 `kill`/`finish`）；不带 `--max-life` 时查询当前生命周期（已存活/剩余/到期时间）；`status` 输出含 `data.lifecycle` 生命周期摘要（已存活/剩余/到期；生命周期接口不可用时省略） |
+| `delta-cli sandbox create --image-name <镜像名> [--idempotency-key <key>] [--shm-size XGi] [--devices kvm] [--no-auto-cleanup] [--shared-workspace=true\|false]` | 创建 sandbox 容器（`--image` 为旧别名；**`--idempotency-key`**：网络重试时用同一个键会回放原 sandbox 而不会重复创建，同键换参数会被拒绝；响应含 `request_id`/`operation_status`/`replayed`；`--no-auto-cleanup` 同时关闭平台 TTL 与服务端 cleanup，仅显式 kill/finish 可销毁；`--shared-workspace` 默认 true：启用共享工作区（播种/写回由平台创建时探测镜像内 rclone 决定，镜像无 rclone 时跳过、数据不跨沙箱持久）；false 跳过播种/写回，适合快速任务；`--shm-size` 调大 /dev/shm（默认 64Mi，**必须带 Gi/Mi 单位**且 ≤ `--memory`，PyTorch DataLoader/multiprocessing 场景常用）；`--devices` 逗号分隔的宿主机设备令牌（`kvm` → /dev/kvm，嵌套虚拟化），**属权限提升**，服务端白名单未开启时**明确拒绝**；`--shm-size`/`--devices` 仅私有算力生效、第三方算力忽略） |
+| `delta-cli sandbox extend <id> --max-life N` | **动态延长运行中** sandbox 的 max-life（N 必须大于当前到期时间且 ≤10080；不接受缩短，提前回收用 `kill`/`finish`）；不带 `--max-life` 时查询当前生命周期（已存活/剩余/到期时间）；`status` 输出含 `data.lifecycle` 生命周期摘要（已存活/剩余/到期；生命周期接口不可用时省略） |
 | `delta-cli sandbox connect <id>` | 连接 sandbox |
-| `delta-cli sandbox status <id>` | 查看 sandbox 状态 |
-| `delta-cli sandbox finish <id>` | 保存结果并销毁 |
-| `delta-cli sandbox kill <id>` | 销毁 sandbox |
-| `delta-cli sandbox run <id> --command "..." [--timeout <秒>] [--summary/--no-summary] [--artifacts]` | 同步运行命令（SSE 原始 `data:` 帧透传到 stdout，无末尾信封，`exit_code`/`log_file` 在 `complete` 帧）；`--summary`/`--artifacts` 仅旧服务器回退路径生效 |
-| `delta-cli sandbox run-bg <id> --command "..." [--timeout <秒>] [--wait] [--summary/--no-summary] [--artifacts]` | 后台运行命令；不加 `--wait` 立即返回 `{execution_id, sandbox_id}`；加 `--wait` 通过 SSE 实时流跟随（原始 `data:` 帧透传到 stdout，无末尾信封） |
+| `delta-cli sandbox status <id>` | 查看 sandbox 状态；返回 `expires_at` / `reclaim_source`，已结束的沙箱仍可查询并带 `termination_reason` / `termination_remark` / `terminated_at`（如 `max_life_exceeded` / `platform_gone` / `user_killed` / `user_finished`） |
+| `delta-cli sandbox finish <id> [--no-wait]` | 保存结果并销毁；销毁是**服务端任务**，响应含 `request_id`/`operation_status`（`--no-wait` 立即返回 `pending`，断线后任务继续，可用 `sandbox operation <request_id>` 对账） |
+| `delta-cli sandbox kill <id> [--no-wait]` | 销毁 sandbox（不保存结果）；响应含 `request_id`/`operation_status`，中断或网络报错后可用 `sandbox operation` 查询真实终态 |
+| `delta-cli sandbox operation [request_id] [--sandbox-id <id>] [--operation create\|destroy] [--limit N]` | 查询服务端操作账本：带 `request_id` 查单条，否则按条件列出最近的创建/销毁记录（终态、时间、原因），用于计费与资源对账 |
+| `delta-cli sandbox run <id> --command "..." [--timeout <秒>] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | 同步运行命令。默认（`--format sse`）把 SSE 原始 `data:` 帧透传到 stdout，无末尾信封，`exit_code`/`log_file`/`stdout_tail`/`duration_seconds` 在 `complete` 帧；`--format json`（或 `--json-result`）只输出**一个** JSON 信封，`data` 含 `execution_id`/`finished`/`exit_code`/`stdout_tail`/`stderr_tail`/`stdout_size`/`stderr_size`/`log_file`/`duration_seconds`/`result_summary`，失败时为 `{ok:false,error,data}` 且进程非零退出；`--summary`/`--artifacts` 仅旧服务器回退路径生效 |
+| `delta-cli sandbox run-bg <id> --command "..." [--timeout <秒>] [--wait] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | 后台运行命令；不加 `--wait` 立即返回 `{execution_id, sandbox_id}`（`--format json` 时为统一结果对象，`finished=false`）；加 `--wait` 通过 SSE 实时流跟随（默认透传原始 `data:` 帧）；`--format json` 时只输出一个与 `sandbox run --format json` 相同结构的信封 |
 | `delta-cli sandbox logs <id> --execution-id <eid> [--tail N --grep <pattern> --context N --max-bytes N]` | 查看后台日志；默认返回 `stderr_size + stderr_tail`（避免上下文爆炸）；可用 `--tail/--grep` 过滤 |
 | `delta-cli sandbox cancel <id> --execution-id <eid>` | 中断后台命令 |
 | `delta-cli sandbox read <id> --path <path> [--output <本地路径>] [--tail N] [--grep <pattern>] [--offset N] [--limit N] [--context N] [--max-bytes N] [--parse-json]` | 读取容器内文件；`--output <path>` 保存到本地；`--tail/--grep` 过滤；非 UTF-8 文件 CLI 自动走 base64 fallback |
@@ -318,9 +338,11 @@ Skill 文档 `skills/delta-science/references/qe.md` 与
 {"ok":true,"data":{"id":"sb-xxx","status":"running"}}
 
 // 错误
-{"ok":false,"error":{"type":"auth","message":"not authenticated","hint":"run 'delta-cli auth login'"}}
+{"ok":false,"error":{"type":"auth","message":"not authenticated","hint":"run 'delta-cli auth login'","request_id":"create_a1b2c3"}}
 ```
 
+`error.hint` 给出可执行的下一步（创建失败时会说明具体原因，例如「镜像未在目标节点缓存，重试或换一个已缓存该镜像的 GPU 型号」），`error.request_id` 用于在服务端日志中定位同一次失败。
+
 错误类型及退出码：
 
 | Type | Exit Code | 场景 |
@@ -388,8 +410,8 @@ make install
 make PREFIX=$HOME/.local install
 
 # Windows 源码安装（二选一）
-# 方式 A：直接用 go install 安装到 GOPATH/bin
-go install github.com/delta-infra/delta-infra-cli/cmd/delta-cli@latest
+# 方式 A：在仓库根目录用 go install 安装到 GOPATH/bin
+go install ./cmd/delta-cli
 
 # 方式 B：在仓库根目录运行 PowerShell 脚本，构建并添加到用户 PATH
 # .\install-from-source.ps1

package/checksums.txt

@@ -1,5 +1,5 @@
-146e95ba3dca9f41c4e7751244178ce461c2aa428a6057212d7e7065be17f5ff  delta-cli-darwin-amd64.tar.gz
-6d339b78e509d215a0e25042d42d806ae7f9a22cb54fbb66ebd26bbf374ced76  delta-cli-darwin-arm64.tar.gz
-d65dacbc85b09a775403407d0779f0c040f5f17b84eb0a6e19f3c5985b7dc952  delta-cli-linux-amd64.tar.gz
-68f6aa27e13b3cd5168fe48f1ab810ad0d203a70553d881f21d584593a7114fc  delta-cli-linux-arm64.tar.gz
-9fe0d9ad21ba47792caee7ebf60a423db1fbb24f142f6cd9430c125a0381eb74  delta-cli-windows-amd64.zip
+c5f6872c10ed7c875b9a9bdad77ba52a1a25b0c5696311e7eac4b02ec1f473a9  delta-cli-darwin-amd64.tar.gz
+9553a81ec84f8a48e6f0963b294bc6da7630a20f9da01acb89eb151f8009003d  delta-cli-darwin-arm64.tar.gz
+37193622609e8175418565168f22014d876d91fe26d20e1cfa2f9c012ef6595d  delta-cli-linux-amd64.tar.gz
+0fe68346288688c10a7be32e84243bb18fccfcd5766f359388590373a016664f  delta-cli-linux-arm64.tar.gz
+f059b6c9dd9c58afc38c3c52421d9ce039e786d4be26a6fc605dade7787ff0bc  delta-cli-windows-amd64.zip

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@delta-infra/cli",
-  "version": "1.1.21",
+  "version": "1.1.25",
   "description": "Delta Agent Cloud CLI for sandbox compute and science tools.",
   "bin": {
     "delta-cli": "scripts/run.js"

package/skills/delta-protein/AGENTS.md

@@ -54,11 +54,11 @@ delta-protein/
   `## Output Template` sections in every reference from
   `scripts/output_metadata.json` (single source of truth for field projection /
   presentation form / downstream reuse / provenance marker). Stdlib only, idempotent.
-- Workflow: update `output_metadata.json` → run
+- Workflow: update `scripts/output_metadata.json` → run
   `python scripts/gen_references.py` → run `python scripts/gen_references.py --check`.
   `--check` fails loudly on missing sections, drifting titles, broken SKILL links or
   references not covered by metadata.
-- `output_metadata.json` `provenance` is `tested` (verified by a real `delta-cli`
+- `scripts/output_metadata.json` `provenance` is `tested` (verified by a real `delta-cli`
   invocation) or `derived` (from ToolSpec descriptions / agent templates, marked
   "pending live calibration"). Newly verified endpoints should be flipped to `tested`.
 - After changing references/SKILL, resync the opencode install dir:

package/skills/delta-sandbox/AGENTS.md

@@ -0,0 +1,37 @@
+# skills/delta-sandbox — Sandbox AI Agent Skill
+
+**Generated:** 2026-10-21
+**Parent:** `../AGENTS.md`
+
+## OVERVIEW
+
+The primary sandbox skill. It teaches an AI agent (Planner → executor) how to drive `delta-cli sandbox` end to end: create a sandbox, run commands in it, read and write files, tunnel ports, and collect artifacts. Ships as one skill directory in the npm tarball (`package.json` `files`) and is installed to every supported agent platform by `scripts/install-wizard.js`.
+
+## WHERE TO LOOK
+
+| Path | Content |
+|------|---------|
+| `SKILL.md` | The whole contract (401 lines): when to use, scope, sandbox environment, mandatory rules, request wording, required_outputs, quick routing, full lifecycle, tunnelling, structured-output convention, reading output, Planner guidance, recovery |
+| `references/commands.md` | Command cheat sheet (72 lines): discovery, lifecycle, command execution, file operations, tunnelling |
+| `references/lifecycle.md` | Full create-to-kill workflow (111 lines) incl. background tasks and polling patterns |
+| `references/recipes.md` | Standard task request patterns (139 lines): required_outputs templates, script output template, GPU/CUDA check, run-a-script, install-and-run, train-and-save-metrics, general data processing |
+| `agents/openai.yaml` | OpenAI-format agent descriptor |
+
+## CONVENTIONS
+
+- YAML frontmatter carries `name`, `description`, `metadata.requires.bins: ["delta-cli"]` and `metadata.cliHelp: "delta-cli sandbox --help"`
+- The `description` is what the Planner matches against — it encodes the output convention and the `required_outputs` requirement
+- `SKILL.md` opens with a **CRITICAL** instruction to read `../delta-shared/SKILL.md` first (auth, config, error handling)
+- "Mandatory rules" is a failure section: breaking any of them makes the task a failure
+- Request wording: describe commands with neutral verbs (run/execute); use create/write/save only when a file is actually written
+- When the Planner calls this skill it must declare `[{kind: 'file', extensions: ['.json']}]` in `required_outputs` for the plan step
+- Sandbox scripts/commands print structured JSON at the end of stdout; on the SSE path the CLI detects the trailing JSON and exposes it as `result_summary` in the complete frame
+- Cross-link to sibling skills and references with relative paths only
+
+## ANTI-PATTERNS
+
+- Do not embed realistic credentials, API keys, or tokens in the examples
+- Do not break the relative links between `SKILL.md`, `references/` and `../delta-shared/SKILL.md`
+- Do not reference internal Go package paths — use CLI command names
+- Do not document a command that `delta-cli sandbox` does not expose
+- Do not state that a sandbox has no TTL or auto-reclaim: the platform TTL is the primary expiry path, and the server also reclaims stale sandboxes — keep the wording consistent with `references/lifecycle.md` and the CLI help

package/skills/delta-sandbox/SKILL.md

@@ -35,7 +35,11 @@ metadata:
 1. **You must use delta-cli**: perform all sandbox operations with `delta-cli sandbox <subcommand>`. **Do not** call the HTTP API directly with Python (`urllib`/`httpx`/`requests`) or any SDK. delta-cli provides typed error output (`error.type`), the only reliable way for an AI agent to interpret errors.
 2. **You must destroy the sandbox**: after each successful `sandbox create`, you **must** run `sandbox kill <id>` or `sandbox finish <id>` to destroy it before the current task's final `step_finish`. Failing to destroy it counts as task failure.
 3. **Destroy it even after an error**: if a command inside the sandbox fails, times out or reports an error, you **must still** run `sandbox kill <id>`. An error is no reason to skip cleanup.
-4. **Do not rely on external cleanup**: the server has **no** automatic sandbox expiry or TTL mechanism. Without `kill`, the sandbox will occupy GPU resources indefinitely.
+4. **Do not rely on external cleanup**: sandboxes **do** have automatic reclamation, but it does not replace destroying them yourself.
+   - The default `--max-life` is 30 minutes. On private compute (OpenSandbox) the timer starts when the create request is accepted and the platform reclaims the sandbox; on third-party compute (AutoDL/RunPod) it starts when the sandbox is ready, and the server-side cleanup reclaims it at max-life plus up to 10 minutes of grace.
+   - The platform may still reclaim a sandbox early (cluster eviction, node failure, etc.; `termination_reason=platform_gone`).
+   - **Automatic reclamation does not write back to OSS**, so artifacts that were not pulled are lost.
+   - Without `kill`, the sandbox keeps occupying GPU resources until it expires, and the reclamation time cannot be used to preserve results. When the task ends you must still run `kill` / `finish` explicitly (only `finish` writes back to OSS).
 5. **Complete synchronous tasks in one command; background tasks may span turns**:
    - Synchronous mode (`sandbox run`): complete the full lifecycle from `create` to `kill` in the same execution. Do not retain a sandbox across tasks.
    - Background mode (`sandbox run-bg`): you may report `sandbox_id` and `execution_id` to the user during the current execution, then retrieve the result in a later turn with `sandbox status` and `sandbox logs`. You must still run `sandbox kill` when the task is complete.
@@ -49,6 +53,15 @@ metadata:
    ```
     The CLI follows output in real time over SSE (`/logs/stream`) and passes each raw SSE frame (`data: {...}`) through to stdout, as with `curl -N`. The JSON `type` field identifies each frame: `init` (with `execution_id`), `stdout`, `stderr`, `error` or `complete`. The command ends when `complete` arrives. **The CLI fills in fields in the complete frame**: the server supplies `exit_code`/`execution_id`/`error`; the CLI adds the accumulated streamed `stdout`/`stderr`, a recovered `log_file` as a fallback, and a trailing-JSON `result_summary` when it detects a valid JSON object in the last 50 lines of stdout. Use `exit_code` in the `complete` frame to determine success. **If the command exits non-zero or reports an error, the CLI appends an `error.type: command_failed` envelope and exits with a non-zero code (process exit code 8)**; the envelope's `message` includes the end of stderr, which makes it easier to use with `&&`, branch on `error.type` and scan the failure reason. The CLI also best-effort appends the stderr tail to the `message` of a `type:"error"` frame; it does not append an envelope on success. If no `complete` frame arrives within `--timeout` (300s by default, serving as both the server command timeout and the client wait deadline), the CLI appends a snapshot envelope with `finished=false` after the frames already passed through, indicating that the command is still running. After `finished`, full stdout is written to `log_file` inside the sandbox (default: `/workspace/{user_id}/sandbox_logs/delta-result-{sandbox_id}-{execution_id}.json`; use the server-returned `log_file` field as authoritative). To read the full output, use `sandbox read <id> --path <log_file>` (the trailing JSON summary is already in the complete frame, so this is usually unnecessary). **`--summary`/`--artifacts` enrichment, including hints, is available only on the older-server fallback path** (when the server lacks the `/logs/stream` endpoint, the CLI automatically falls back to polling every 5s and returns a `CommandResult` envelope). Use `execution_id` for later `sandbox logs` or `cancel` calls.
 
+    **For automation: `--format json` (same as `--json-result`)** — emits no `data:` frames, just **one** JSON envelope, so you can parse `data` directly instead of stitching stdout frames together:
+
+   ```bash
+   delta-cli sandbox run <id> --command "<command>" --timeout <seconds> --format json
+   delta-cli sandbox run-bg <id> --command "<command>" --timeout <seconds> --wait --format json
+   ```
+
+   The `data` fields have the same names and meaning as in the `complete` frame: `execution_id`, `finished`, `exit_code`, `stdout_tail`, `stderr_tail`, `stdout_size`, `stderr_size`, `log_file`, `duration_seconds` and `result_summary`. `stdout_tail`/`stderr_tail` are the last 2000 characters of the accumulated streamed output (newlines preserved), so there is no need to aggregate the `stdout` frames. When the command exits non-zero the CLI emits `{"ok":false,"error":{...},"data":{...}}`, where `data` is the same object (including `exit_code`), and the process exit code is 8. This mode shares field semantics with SSE mode, so you can switch between them at any time.
+
     **Live following / manual polling (when you need intermediate progress)**:
     `sandbox logs <id> --execution-id <exec_id> --stream` follows the task in real time (SSE), passing each raw `data:` frame through to stdout until the `complete` event, with no trailing envelope (`--stream` requires `--execution-id`). Without `--stream`, `sandbox logs <id> --execution-id <exec_id>` returns a snapshot envelope. You may omit `--execution-id`; if omitted, the server queries only by sandbox and automatically uses ES (Elasticsearch) because `sandbox_id` and `execution_id` were not both supplied (it returns an **array** aggregated by execution ID rather than a single snapshot object):
     - `finished=true` and `exit_code=0` → complete
@@ -152,8 +165,8 @@ The reason is that, after running a command, this skill reads `log_file` from th
 | Finish a sandbox | `sandbox finish <id> [--results '{...}']` |
 | Destroy a sandbox | `sandbox kill <id>` |
 | **Command execution** | |
-| Run a command synchronously (≤60s) | `sandbox run <id> --command "..." [--timeout <seconds>] [--summary/--no-summary] [--artifacts]` |
-| Run a command in the background (>60s) | `sandbox run-bg <id> --command "..." [--timeout <seconds>] [--wait] [--summary/--no-summary] [--artifacts]` |
+| Run a command synchronously (≤60s) | `sandbox run <id> --command "..." [--timeout <seconds>] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` (`--format json` = a single unified result envelope, recommended for automation) |
+| Run a command in the background (>60s) | `sandbox run-bg <id> --command "..." [--timeout <seconds>] [--wait] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` (`--wait --format json` = a single unified result envelope) |
 | View background logs | `sandbox logs <id> [--execution-id <exec_id>] [--tail N --grep <pattern> --context N] [--stream]` (`--execution-id` is optional; if omitted, query ES by sandbox only; only `--stream` requires it) |
 | Interrupt a background command | `sandbox cancel <id> --execution-id <exec_id>` |
 | **Tunnelling (port exposure)** | |
@@ -164,7 +177,7 @@ The reason is that, after running a command, this skill reads `log_file` from th
 | Read a file | `sandbox read <id> --path <path> [--output <local-path> --tail N --grep <pattern> --offset N --limit N --context N --max-bytes N --parse-json]` |
 | Pull a file/directory | `sandbox pull <id> --source <sandbox-path> --target <local-path> [--recursive] [--pattern <glob>]` — pull a single file or recursively pull a directory, with sha1 verification (the reverse of upload: source=remote, target=local) |
 | View the working directory | `sandbox working-directory <id>` (returns `data.path`, such as `/workspace/{user_id}/{sandbox_id}`) |
-| Write a file | `sandbox write <id> --source <filename>` (without `--path`, writes to the working directory by default); a relative `--path <path>` is prefixed with the working directory, while an absolute path is used as given |
+| Write a file | `sandbox write <id> --source <filename>` (without `--path`, writes to the working directory by default); **every sandbox-side path** — `--path`, the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, `run`/`run-bg --log-file` — is prefixed with the working directory when relative, while an absolute path is used as given |
 | Write multiple files | `sandbox write-multiple <id> --entry <remote-path>=<local-path> [--entry ...]` |
 | Upload a directory | `sandbox upload <id> --source <local-directory> --target <sandbox-path>` |
 | List a directory | `sandbox ls <id> --path <path>` (defaults to `.`) |
@@ -176,7 +189,6 @@ The reason is that, after running a command, this skill reads `log_file` from th
 | Delete directories | `sandbox rmdir <id> --path <path> [--path ...]` |
 | Create directories | `sandbox mkdir <id> --path <path> [--path ...]` |
 | Search files | `sandbox search <id> --path <root-directory> --pattern <glob>` |
-| Upload a directory | `sandbox upload <id> --source <local-directory> --target <sandbox-path>` |
 
 ## Full lifecycle
 
@@ -192,17 +204,17 @@ The reason is that, after running a command, this skill reads `log_file` from th
     - **`--gpu-type <model>` (optional) selects a GPU model**: for example, `"RTX 4090"` or `H100`. This schedules the task on that GPU model. Use `sandbox resources` to see available models (`gpu_types[].gpu_name`). Specify a model only when the user explicitly requests one; otherwise, the system schedules automatically.
     - **`--shm-size <size>` (optional) increases the container's `/dev/shm`**: the default is only **64Mi**. Workloads such as PyTorch `DataLoader(num_workers>0)`, `torch.multiprocessing`/shared memory and Chromium-based engines (Playwright/Selenium) may fail with `No space left on device` / `Bus error` when shared memory is insufficient; increase it as needed (for example, `--shm-size 4Gi`). Hard constraints: **include Gi/Mi units** (`4Gi`/`512Mi`; the CLI normalises `4G`/`512M` to `4Gi`/`512Mi`, while the server rejects a bare number such as `4` with a clear error because its meaning differs from the server's); **do not exceed `--memory`** (`/dev/shm` is a memory filesystem counted against the container's memory cgroup; filling it can cause an OOMKill). Since `--memory` defaults to only `1Gi`, increasing shm usually means increasing `--memory` too (for example, `--memory 8Gi --shm-size 4Gi`). This parameter is applied only on private compute (third-party compute ignores it); however, unit and limit checks happen before creation regardless of backend, so invalid values are rejected even when using third-party compute. The create response echoes the effective `resource.shmSize` for verification.
     - **`--devices <token>` (optional) requests host-device passthrough**: a comma-separated token list (currently `kvm` is supported, which adds `/dev/kvm` inside the container), used for **nested virtualisation** workloads such as QEMU/KVM, Android emulators and Kata inside the sandbox. **This is a privilege escalation, not an ordinary resource request**: device passthrough exposes host devices to the container, so the server uses an explicit allowlist. If operations staff have not enabled it, creation is **explicitly rejected (not silently ignored)**. Do not retry or change the syntax after this error; use an approach that does not require the device. This parameter is applied only on private compute (third-party compute ignores it), but the allowlist check happens before creation regardless of backend.
-    - `--no-auto-cleanup`: when this flag is set, the sandbox **will not be cleaned up automatically** (both server-side timeout reclamation and the local periodic `cleanup_stale` process are skipped). It can be destroyed only by an explicit `sandbox kill`/`finish`. Use this only when the task genuinely needs to survive past periodic cleanup; you must destroy it explicitly when the task is done to avoid leaking resources.
+    - `--no-auto-cleanup`: when this flag is set, the sandbox **will not be cleaned up automatically** (OpenSandbox gets `timeout=None`, so platform TTL and the server's periodic cleanup are both skipped). It can be destroyed only by an explicit `sandbox kill`/`finish`, and `status` reports `expires_at`/`remaining` as null. Use this only when the task genuinely needs to survive past periodic cleanup; you must destroy it explicitly when the task is done to avoid leaking resources.
     - **Do not repeatedly poll with `sandbox status` after a successful create**. The sandbox is ready as soon as `sandbox create` returns; use its returned `data.sandbox_id` directly for `write`/`run`. Extra polling adds tool calls without providing any benefit.
 3. **Write code/data**:
-   - **Single file**: `delta-cli sandbox write <id> --source <filename>` — omit `--path` to write to the sandbox's working directory by default (`/workspace/{user_id}/{sandbox_id}/`; query it with `delta-cli sandbox working-directory <id>`). Pass only a local filename to `--source` (for example, `--source train.py`). **Do not** use shell-variable paths such as `"$WORKSPACE_ROOT/train.py"` or `$(pwd)/train.py`. With `--source`, the CLI reads the local file itself without shell expansion, making this the safest option. If you need a specific path, a relative `--path <relative-path>` is automatically prefixed with the working directory; absolute paths are used as given (**avoid** paths such as `/workspace/<filename>` outside `{user_id}`, as they do not persist after `kill`). The returned `size` field is the actual number of bytes on disk (verified with stat), so you can check that the write is complete. For small configuration files (<20 lines), you may use `--data "..."`, but you must also pass `--path`. File paths and extensions depend on the runtime in the image.
+   - **Single file**: `delta-cli sandbox write <id> --source <filename>` — omit `--path` to write to the sandbox's working directory by default (`/workspace/{user_id}/{sandbox_id}/`; query it with `delta-cli sandbox working-directory <id>`). Pass only a local filename to `--source` (for example, `--source train.py`). **Do not** use shell-variable paths such as `"$WORKSPACE_ROOT/train.py"` or `$(pwd)/train.py`. With `--source`, the CLI reads the local file itself without shell expansion, making this the safest option. If you need a specific path, a relative `--path <relative-path>` is automatically prefixed with the working directory; absolute paths are used as given (the same rule applies to every sandbox-side path: the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, and `run`/`run-bg --log-file`) (**avoid** paths such as `/workspace/<filename>` outside `{user_id}`, as they do not persist after `kill`). The returned `size` field is the actual number of bytes on disk (verified with stat), so you can check that the write is complete. For small configuration files (<20 lines), you may use `--data "..."`, but you must also pass `--path`. File paths and extensions depend on the runtime in the image.
    - **Write multiple files**: `sandbox write-multiple <id> --entry <remote-path>=<local-path> [--entry ...]` (remote path is on the left of `=`, local path on the right; bulk writes with `--data` are not supported)
    - **Upload a directory**: `sandbox upload <id> --source <local-directory> --target <sandbox-path>` — the CLI packages the local directory as tar.gz and uploads it using multipart/form-data; the server extracts it into the target directory. The response lists each file's path, size and mode. After upload, the CLI checks the local and remote file lists for integrity and warns about mismatched sizes, extra files and similar issues.
      - **Note**: `--source` is a **local directory path**; `--target` is the **target directory inside the sandbox**. The target directory is created if it does not exist.
      - Nested directories are supported, and empty directories are created too.
      - **Optional post-write check**: use `sandbox stat <id> --path <path>` to confirm that the file exists and its size is as expected. **Do not** use `sandbox ls` plus `sandbox read` to read the just-written file back to the host and compare it character by character; if there is no issue, no read-back is needed.
 4. **Run the command**:
-    - **Short task (expected to take ≤ 60 seconds)**: run synchronously with `delta-cli sandbox run <id> --command "<command>" --timeout <seconds>`. The command's stdout/stderr is passed to stdout over SSE as **raw `data:` frames, one at a time** (with no trailing envelope). Read `exit_code` / `log_file` / `execution_id` from the `complete` frame. After completion, the full `stdout` is written to the run-envelope log file (the `log_file` field); use `sandbox read` to read it in full or extract trailing JSON. **Do not** then call `sandbox logs`. You can customise the run-envelope log path with `--log-file <path>`; the default is `/workspace/{user_id}/sandbox_logs/delta-result-{sandbox_id}-{execution_id}.json` (use the server-returned `log_file` field as authoritative). A script written with `sandbox write` is in the working directory; run it using its full path, `<working-directory>/<filename>` (first query `sandbox working-directory <id>`). Build the command for the runtime in the image. Common examples (where `<wd>` is the working directory):
+    - **Short task (expected to take ≤ 60 seconds)**: run synchronously with `delta-cli sandbox run <id> --command "<command>" --timeout <seconds>`. The command's stdout/stderr is passed to stdout over SSE as **raw `data:` frames, one at a time** (with no trailing envelope). Read `exit_code` / `log_file` / `execution_id` / `stdout_tail` / `stderr_tail` / `duration_seconds` from the `complete` frame. For automation, add `--format json` (or `--json-result`) so the CLI emits a single unified result envelope instead of making you stitch frames together yourself. After completion, the full `stdout` is written to the run-envelope log file (the `log_file` field); use `sandbox read` to read it in full or extract trailing JSON. **Do not** then call `sandbox logs`. You can customise the run-envelope log path with `--log-file <path>`; the default is `/workspace/{user_id}/sandbox_logs/delta-result-{sandbox_id}-{execution_id}.json` (use the server-returned `log_file` field as authoritative). A script written with `sandbox write` is in the working directory; run it using its full path, `<working-directory>/<filename>` (first query `sandbox working-directory <id>`). Build the command for the runtime in the image. Common examples (where `<wd>` is the working directory):
      - Python: `python <wd>/train.py`
      - Node.js: `node <wd>/app.js`
      - Go: `go run <wd>/main.go`
@@ -383,6 +395,7 @@ This skill is designed to make the conclusion self-contained: when execution end
 | `error.type: validation` | Check parameter formats and required flags |
 | `error.type: auth / permission` | Follow the permission-denied guidance in delta-shared |
 | `error.type: api` (5xx) | Server error; check the API service status |
+| `sandbox create` failed | Read `error.hint` — it states the cause and the next step (for example, the image was not cached on the target node, so the initial pull exceeded the readiness timeout: retry once, or pick a GPU type whose node already has the image). A failed create leaves no sandbox behind, so retrying is safe. `error.request_id` identifies the failure in the server log. |
 | Command execution failed (`exit_code ≠ 0`) | Check stdout/stderr returned by `run` |
 | Command timed out | Increase `--timeout` or use `run-bg` |
 | `error.type: internal` | Client-side internal error; check the delta-cli version or contact the maintainer |

package/skills/delta-sandbox/references/commands.md

@@ -1,5 +1,7 @@
 > **Prerequisite:** Read [`../../delta-shared/SKILL.md`](../../delta-shared/SKILL.md) first for authentication, configuration and general error handling.
 
+> **Paths inside the sandbox:** every path that points *inside* the sandbox — `--path`, the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, and `run`/`run-bg --log-file` — is resolved against the sandbox working directory when it is relative; an absolute path is used as-is. Flags that point at the *local* machine (`write --source`, `upload --source`, `pull --target`, `read --output`) are local and are never rewritten.
+
 # Command reference
 
 ## Discovery
@@ -15,18 +17,20 @@
 
 | Command | Description |
 |------|------|
-| `sandbox create --image-name <image-name> [--cpu N --memory XGi --gpu N --gpu-mem N --gpu-type <model> --shm-size XGi --devices kvm --max-life M --no-auto-cleanup]` | Create a sandbox container (`--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; the underlying image identifier is hidden from users. `--gpu-type` selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish); `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, `1G`/`512M`; lowercase is also accepted), which the CLI converts to the `Gi`/`Mi` units accepted by the server; `--shm-size` increases the container's `/dev/shm` (the **default is only 64Mi**; it must be increased if PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error` because shared memory is insufficient; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number); it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only `1Gi`, so increasing shm usually requires increasing memory too); validation is backend-independent and rejects invalid values before creation, but this setting is applied only on private compute (third-party compute ignores it); `--devices` requests host-device passthrough using comma-separated tokens (for example, `kvm` makes `/dev/kvm` available inside the container for nested virtualisation; this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** rather than silently ignored; this setting is applied only on private compute); **resource limits per sandbox**: `--cpu` ≤ 512, `--memory` ≤ 1024Gi, `--gpu` ≤ 64, `--gpu-mem` ≤ 1024Gi, and `--shm-size` ≤ `--memory`. Exceeding a limit returns a **validation error promptly at request time** (it does not hang); normal tasks are well below these limits. The response echoes the requested image name/resource (request values fill fields not returned by the server; server values take precedence); compute backend is an internal detail and is selected automatically, so `--provider` is no longer supported. |
+| `sandbox create --image-name <image-name> [--idempotency-key <key>] [--cpu N --memory XGi --gpu N --gpu-mem N --gpu-type <model> --shm-size XGi --devices kvm --max-life M --no-auto-cleanup]` | Create a sandbox container (`--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; the underlying image identifier is hidden from users. **`--idempotency-key`**: a retry with the same key replays the original sandbox instead of creating a duplicate, and the same key with different parameters is rejected; the response includes `request_id`/`operation_status`/`replayed`. `--gpu-type` selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish); `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, `1G`/`512M`; lowercase is also accepted), which the CLI converts to the `Gi`/`Mi` units accepted by the server; `--shm-size` increases the container's `/dev/shm` (the **default is only 64Mi**; it must be increased if PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error` because shared memory is insufficient; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number); it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only `1Gi`, so increasing shm usually requires increasing memory too); validation is backend-independent and rejects invalid values before creation, but this setting is applied only on private compute (third-party compute ignores it); `--devices` requests host-device passthrough using comma-separated tokens (for example, `kvm` makes `/dev/kvm` available inside the container for nested virtualisation; this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** rather than silently ignored; this setting is applied only on private compute); **resource limits per sandbox**: `--cpu` ≤ 512, `--memory` ≤ 1024Gi, `--gpu` ≤ 64, `--gpu-mem` ≤ 1024Gi, and `--shm-size` ≤ `--memory`. Exceeding a limit returns a **validation error promptly at request time** (it does not hang); normal tasks are well below these limits. The response echoes the requested image name/resource (request values fill fields not returned by the server; server values take precedence); compute backend is an internal detail and is selected automatically, so `--provider` is no longer supported. If creation fails, `error.hint` states the cause and the next step (for example, the image was not cached on the target node: retry, or pick a GPU type whose node already has the image), and `error.request_id` identifies the failure in the server log. |
 | `sandbox connect <id>` | Connect to an existing sandbox. |
 | `sandbox status <id>` | Check sandbox status (running / done). |
-| `sandbox finish <id> [--results '{...}']` | Save results and destroy the sandbox automatically. |
-| `sandbox kill <id>` | Destroy the sandbox immediately without saving results. |
+| `sandbox finish <id> [--results '{...}'] [--no-wait]` | Save results and destroy the sandbox automatically. Destruction is a server-side operation: the response includes `request_id`/`operation_status`; `--no-wait` returns immediately with `pending`. |
+| `sandbox kill <id> [--no-wait]` | Destroy the sandbox immediately without saving results; the response includes `request_id`/`operation_status`. |
+| `sandbox operation [request_id] [--sandbox-id <id>] [--operation create\|destroy] [--limit N]` | Query the server-side operation ledger (terminal state, time and reason). Use it to confirm the real outcome of a create or destroy after a client disconnect, timeout or network error. |
+| `sandbox extend <id> [--max-life N]` | Query or adjust the max lifetime of a running sandbox (N in minutes; omit `--max-life` to query the current lifecycle). `status` also reports the lifecycle summary under `data.lifecycle`. |
 
 ## Command execution
 
 | Command | Description |
 |------|------|
-| `sandbox run <id> --command "..." [--timeout N] [--summary/--no-summary] [--artifacts]` | Run a command synchronously (live SSE stream; raw `data:` frames are passed through to stdout one by one; **there is no trailing envelope on success**; the CLI fills in `exit_code`/`execution_id`/`log_file`/`stdout`/`stderr`/`result_summary` in the `complete` frame; **if the command exits non-zero or errors, an `error.type: command_failed` envelope is appended and the process exits non-zero**; the envelope's `message` includes the stderr tail, as does the message in a `type:"error"` frame on a best-effort basis). `--summary`/`--artifacts` take effect only on the legacy-server fallback path (when there is no SSE endpoint and a `CommandResult` envelope is returned). |
-| `sandbox run-bg <id> --command "..." [--timeout N] [--wait] [--summary/--no-summary] [--artifacts]` | Run a command in the background. Without `--wait`, return `{execution_id, sandbox_id}` immediately. With `--wait`, follow the live SSE stream until `complete` (raw `data:` frames are passed through to stdout; the CLI fills in `log_file`/`stdout`/`stderr`/`result_summary` in the `complete` frame; **on command failure, a `command_failed` envelope is appended and the process exits non-zero**; the envelope's `message` includes the stderr tail; on timeout, a `finished=false` snapshot envelope is appended). `--summary`/`--artifacts` take effect only on the legacy-server fallback path. |
+| `sandbox run <id> --command "..." [--timeout N] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | Run a command synchronously (live SSE stream; raw `data:` frames are passed through to stdout one by one; **there is no trailing envelope on success**; the CLI fills in `exit_code`/`execution_id`/`log_file`/`stdout`/`stderr`/`stdout_tail`/`stderr_tail`/`stdout_size`/`stderr_size`/`duration_seconds`/`result_summary` in the `complete` frame; **if the command exits non-zero or errors, an `error.type: command_failed` envelope is appended and the process exits non-zero**; the envelope's `message` includes the stderr tail, as does the message in a `type:"error"` frame on a best-effort basis). `--format json` (same as `--json-result`) emits no frames, only one envelope whose `data` is the unified result object above; on failure it is `{ok:false,error,data}`. `--summary`/`--artifacts` take effect only on the legacy-server fallback path (when there is no SSE endpoint and a `CommandResult` envelope is returned). |
+| `sandbox run-bg <id> --command "..." [--timeout N] [--wait] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | Run a command in the background. Without `--wait`, return `{execution_id, sandbox_id}` immediately (with `--format json`, the unified result object with `finished=false`). With `--wait`, follow the live SSE stream until `complete` (raw `data:` frames are passed through to stdout; the CLI fills in `log_file`/`stdout`/`stderr`/`result_summary` in the `complete` frame; **on command failure, a `command_failed` envelope is appended and the process exits non-zero**; the envelope's `message` includes the stderr tail; on timeout, a `finished=false` snapshot envelope is appended). With `--format json`, emit only one envelope with the same structure as `sandbox run --format json`. `--summary`/`--artifacts` take effect only on the legacy-server fallback path. |
 | `sandbox logs <id> --execution-id <eid> [--tail N --grep <pattern> --context N] [--stream]` | Retrieve background-command logs. **By default, returns `stdout_tail` (last 800 bytes) + `stderr_tail` (last 200 bytes) + `stderr_size` + `cursor`**, about 1 KB in total, to keep context small. `--stream` follows the live `/logs/stream` SSE endpoint, passing raw `data:` frames through to stdout until `complete` (the CLI fills in `stdout`/`stderr`/`log_file`/`result_summary` in the complete frame; there is no trailing envelope). `--tail N` applies to both stdout and stderr (last N lines from each); `--grep <pattern>` filters both stdout and stderr (regular expression); `--context N` includes N lines before and after each grep match (for both stdout and stderr). The `cursor` field has no `omitempty` and is always present, even when 0. Snapshots include `last_updated` (filled by the CLI on the SSE path; the server snapshot path is awaiting a server-side update). **If `--execution-id` is omitted, query by sandbox** (the server uses ES); the response is an **array** grouped by execution_id (each item contains `execution_id`/`stdout`/`stderr`/`exit_code`/`command`/`finished`/`error`/`log_file`), and the CLI applies the same tail/grep trimming to each item. |
 | `sandbox cancel <id> --execution-id <eid>` | Interrupt a background command that is running. |
 
@@ -38,17 +42,27 @@
 | `sandbox working-directory <id>` | Get the sandbox's current working directory (absolute path `/workspace/{user_id}/{sandbox_id}`). |
 | `sandbox write <id> [--path <path>] --source <filename>` | Write a file (recommended). If `--path` is omitted, write to the working directory (`<working-directory>/<filename>`); a relative `--path` is also resolved under the working directory. An absolute `--path` is used as-is (but the `/workspace/<filename>` root path is outside the OSS sync scope and is not recommended). |
 | `sandbox write <id> --path <absolute-path> --data "..."` | Write a small amount of inline content (`--data` has no source filename, so `--path` must be specified explicitly). |
-| `sandbox write-multiple <id> --entry <src=path> [--entry ...]` | Write multiple files in one operation (a relative target `path` is also resolved under the working directory). |
+| `sandbox write-multiple <id> --entry <remote-path>=<local-path> [--entry ...]` | Write multiple files in one operation (a relative target `path` is also resolved under the working directory). |
 | `sandbox pull <id> --source <sandbox-path> --target <local-path> [--recursive] [--pattern <glob>]` | Pull files/directories to the local machine (the reverse of `upload`; flag directions are reversed: `--source`=remote sandbox path, `--target`=local path). Single files are detected automatically; directories are traversed recursively by default; SHA-1 integrity is checked by both the CLI and server. |
+| `sandbox upload <id> --source <local-directory> --target <sandbox-path>` | Upload a local directory (packaged as tar.gz, extracted server-side) and verify the resulting file list against the local manifest. `--source` is local, `--target` is inside the sandbox. |
 | `sandbox ls <id> --path <path>` | List directory contents (defaults to `.`). |
 | `sandbox stat <id> --path <path>` | Get file metadata (size / mode / owner / group). |
 | `sandbox mv <id> --entry <source=dest> [--entry ...]` | Move or rename files. |
 | `sandbox replace <id> --path <path> --old <text> --new <text> [--regex] [--ignore-case]` | Replace file contents. |
 | `sandbox chmod <id> --path <path> --mode <octal>` | Change file permissions. |
 | `sandbox rm <id> --path <path> [--path ...]` | Delete files. |
+| `sandbox rmdir <id> --path <path> [--path ...]` | Delete directories recursively (cannot be undone). |
 | `sandbox mkdir <id> --path <path> [--path ...]` | Create directories (`mkdir -p`). |
 | `sandbox search <id> --path <root-directory> --pattern <glob>` | Search for files. |
 
+## Tunnelling (port exposure)
+
+| Command | Description |
+|------|------|
+| `sandbox expose <id> [--port N]` | Expose a service port inside the sandbox to the public internet and return the `public_url` (the platform default inner port is 8700). |
+| `sandbox exposes <id>` | List the public tunnels currently active for this sandbox. |
+| `sandbox dispose <id> --port N` | Recycle one tunnel (stop frpc and release the public port); killing the sandbox also reclaims its tunnels. |
+
 ## Notes
 
 - `list` ≠ `ls`: `sandbox list` lists sandbox instances; `sandbox ls <id>` lists directories inside a sandbox.

package/skills/delta-sandbox/references/lifecycle.md

@@ -6,15 +6,21 @@
 
 1. **Check available images** `delta-cli sandbox images` — list the images supported by the server. The image determines which language runtimes are available in the container (Python / Node.js / Go / Java / Rust, etc.); choose one that fits the user's technology stack and GPU/CPU requirements.
 2. **List existing sandboxes (optional)** `delta-cli sandbox list` — check the active sandboxes already created by the current user to avoid creating duplicates.
-3. **Create** `delta-cli sandbox create --image-name <image-name> --cpu 4 --memory 16Gi --gpu 1 --gpu-mem 8000 --max-life 120 [--gpu-type <model>] [--shm-size XGi] [--devices kvm]` (the sandbox is ready to use immediately; no connection step is needed. `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, 1G/512M; lowercase is also accepted), which the CLI converts to Gi/Mi. `--max-life` sets the sandbox's maximum lifetime in minutes; the default is 30. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish. `--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; compute backend selection is automatic and does not need to be specified. `--gpu-type` (optional) selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--shm-size` (optional) increases the container's `/dev/shm` (the **default is only 64Mi**)—it must be increased when PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error`; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number), and it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only 1Gi, so increasing shm usually requires increasing memory too, for example, `--memory 8Gi --shm-size 4Gi`). Only private compute (`private`) applies this setting (third-party compute ignores it), but unit/limit validation is backend-independent and rejects invalid values before creation. `--devices` (optional) requests host-device passthrough using comma-separated tokens (for example, `--devices kvm` makes `/dev/kvm` available inside the container for nested virtualisation); this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** (not silently ignored; validation is backend-independent). If this error occurs, do not retry; use an approach that does not depend on that device. Only private compute applies this setting. **Resource limits per sandbox**: cpu ≤ 512, memory ≤ 1024Gi, gpu ≤ 64, gpuMem ≤ 1024Gi, shmSize ≤ memory. Exceeding a limit returns a validation error promptly at request time rather than hanging.)
-4. **Write code/data** `delta-cli sandbox write <id> --source <filename>` — write a local file directly to the sandbox. If `--path` is omitted, the file is written to that sandbox's working directory by default (`/workspace/{user_id}/{sandbox_id}/<filename>`, which is within the OSS sync scope, so the file persists after kill). Before writing, use `delta-cli sandbox working-directory <id>` to confirm the path. **You must** use a relative source path (for example, `--source infer.py`); do not use `"$WORKSPACE_ROOT/<filename>"` or `$(pwd)/...`. A relative `--path` is also resolved under the working directory; an absolute `--path` is used as-is (`/workspace/<filename>` is outside the OSS sync scope and is not recommended). You can also use `--data "..."` to write a small amount of inline content (requires an explicit `--path`), and `--mode 755` to set file permissions. The returned `size` field is the actual on-disk byte count verified with stat, so it can confirm that the write completed. Use `write-multiple` to write several files at once. File paths and extensions depend on the runtimes in the image.
+3. **Create** `delta-cli sandbox create --image-name <image-name> --cpu 4 --memory 16Gi --gpu 1 --gpu-mem 8000 --max-life 120 [--gpu-type <model>] [--shm-size XGi] [--devices kvm] [--idempotency-key <key>]` (the sandbox is ready to use immediately; no connection step is needed. `--idempotency-key` (optional) makes network retries safe: the same key replays the original sandbox instead of creating a duplicate, and the response includes `request_id`/`operation_status`/`replayed`; `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, 1G/512M; lowercase is also accepted), which the CLI converts to Gi/Mi. `--max-life` sets the sandbox's maximum lifetime in minutes; the default is 30. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish. `--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; compute backend selection is automatic and does not need to be specified. `--gpu-type` (optional) selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--shm-size` (optional) increases the container's `/dev/shm` (the **default is only 64Mi**)—it must be increased when PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error`; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number), and it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only 1Gi, so increasing shm usually requires increasing memory too, for example, `--memory 8Gi --shm-size 4Gi`). Only private compute (`private`) applies this setting (third-party compute ignores it), but unit/limit validation is backend-independent and rejects invalid values before creation. `--devices` (optional) requests host-device passthrough using comma-separated tokens (for example, `--devices kvm` makes `/dev/kvm` available inside the container for nested virtualisation); this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** (not silently ignored; validation is backend-independent). If this error occurs, do not retry; use an approach that does not depend on that device. Only private compute applies this setting. **Resource limits per sandbox**: cpu ≤ 512, memory ≤ 1024Gi, gpu ≤ 64, gpuMem ≤ 1024Gi, shmSize ≤ memory. Exceeding a limit returns a validation error promptly at request time rather than hanging.)
+   - **If `create` fails**: the error envelope's `hint` states the cause and the next step. A common one is "the image may not be cached on the target node" — the initial pull exceeded the readiness timeout, so retry once (the pull continues in the background and later creates succeed) or pick a GPU type whose node already has the image. A failed create leaves no sandbox behind, so retrying is safe; `error.request_id` identifies the failure in the server log.
+4. **Write code/data** `delta-cli sandbox write <id> --source <filename>` — write a local file directly to the sandbox. If `--path` is omitted, the file is written to that sandbox's working directory by default (`/workspace/{user_id}/{sandbox_id}/<filename>`, which is within the OSS sync scope, so the file persists after kill). Before writing, use `delta-cli sandbox working-directory <id>` to confirm the path. **You must** use a relative source path (for example, `--source infer.py`); do not use `"$WORKSPACE_ROOT/<filename>"` or `$(pwd)/...`. A relative `--path` is also resolved under the working directory; an absolute `--path` is used as-is (the same rule applies to every sandbox-side path: the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, and `run`/`run-bg --log-file`) (`/workspace/<filename>` is outside the OSS sync scope and is not recommended). You can also use `--data "..."` to write a small amount of inline content (requires an explicit `--path`), and `--mode 755` to set file permissions. The returned `size` field is the actual on-disk byte count verified with stat, so it can confirm that the write completed. Use `write-multiple` to write several files at once. File paths and extensions depend on the runtimes in the image.
 5. **Run a command** — first use `delta-cli sandbox working-directory <id>` to get `<wd>`, then construct a command for a runtime in the image. Common examples: `python <wd>/train.py` (Python), `node <wd>/app.js` (Node.js), `go run <wd>/main.go` (Go), `bash <wd>/run.sh` (Shell).
    - **Short tasks ≤60s**: `delta-cli sandbox run <id> --command "<command>" --timeout <seconds>` runs synchronously; raw SSE `data:` frames are passed through to stdout (with no trailing envelope), `exit_code`/`log_file` are read from the `complete` frame, and full stdout is written to the log_file.
    - **Long tasks >60s**: run `delta-cli sandbox run-bg <id> --command "<command>" --timeout <seconds> --wait` in the background (recommended), or use `run-bg` + `logs` for manual polling (`run-bg --wait` includes `--summary` by default; without `--wait`, it returns `{execution_id, sandbox_id}` immediately).
 6. **Read the results** `delta-cli sandbox read <id> --path <wd>/result.json` (`<wd>` is the working directory; use `delta-cli sandbox working-directory <id>` to find it) — returns `content` (contents), `size` (disk bytes, from stat) and `content_length` (character count). Reading a non-existent file returns an error rather than empty content.
-7. **Destroy the sandbox** `delta-cli sandbox kill <id>` (if you need to save results, use finish; finish destroys the sandbox automatically).
+7. **Destroy the sandbox** `delta-cli sandbox kill <id>` (if you need to save results, use finish; finish destroys the sandbox automatically). Both are **server-side operations**: the response includes `request_id`/`operation_status`, they continue after the client disconnects, and `delta-cli sandbox operation <request_id>` reconciles the real outcome.
 
-> **Note:** The server has no automatic sandbox expiry or TTL mechanism. If you do not kill a sandbox, it will occupy GPU resources indefinitely. Always destroy it when the task is complete.
+> **Note:** sandboxes **do** get reclaimed automatically, but when and from which starting point depends on the backend:
+> - Private compute (OpenSandbox): `--max-life` (default 30 minutes) starts when the create request is accepted and the platform reclaims the sandbox at expiry; image pull / readiness time counts towards it too.
+> - Third-party compute (AutoDL/RunPod): the clock starts when the sandbox is ready, and the server-side cleanup reclaims it at max-life plus up to 10 minutes of grace.
+> - The platform may still reclaim a sandbox early (cluster eviction, node failure; `termination_reason=platform_gone`), and automatic reclamation **does not** write back to OSS, so artifacts that were not pulled are lost.
+> - Therefore you must still run `kill`/`finish` explicitly when the task ends; to preserve artifacts use `finish` or `pull` beforehand. `sandbox status` returns `expires_at` / `reclaim_source` / `termination_reason`, and remains queryable after the sandbox has ended.
+> - `--no-auto-cleanup` disables both the platform TTL and the server-side cleanup, so `expires_at`/`remaining` are null.
 
 ## Background tasks
 
@@ -29,7 +35,7 @@ The CLI follows the live SSE stream (`/logs/stream`) and passes raw SSE frames (
 
 > On the SSE path, `--wait` puts `result_summary` directly in the **complete frame** (if trailing JSON is detected), so there is no need to scan backwards with `sandbox read <id> --path <log_file>`. Read the log_file manually only if no valid JSON is found at the end. `--summary`/`--artifacts` enrichment (including hints) still takes effect only on the legacy-server fallback path.
 
-> **Output contract (SSE path):** On success, `run-bg --wait` writes only raw SSE frames to stdout, with **no trailing envelope**. On command failure, it appends an `{"ok":false,"error":{"type":"command_failed",...}}` envelope after the complete frame. If the command has not finished within `--timeout`, it appends a `finished=false` `CommandLogsResult` snapshot envelope. For an older server (without the `/logs/stream` endpoint), the CLI falls back to polling every 5s and outputs a `CommandResult` envelope (the `finished` key is always present; when `finished=true`, it includes `result_summary`/`hints` enrichment).
+> **Output contract (SSE path):** On success, `run-bg --wait` writes only raw SSE frames to stdout, with **no trailing envelope**. On command failure, it appends an `{"ok":false,"error":{"type":"command_failed",...}}` envelope after the complete frame. If the command has not finished within `--timeout`, it appends a `finished=false` `CommandLogsResult` snapshot envelope. For an older server (without the `/logs/stream` endpoint), the CLI falls back to polling every 5s and outputs a `CommandResult` envelope (the `finished` key is always present; when `finished=true`, it includes `result_summary`/`hints` enrichment). **For automation, add `--format json` (same as `--json-result`)**: it emits no frames, only one envelope whose `data` contains `execution_id`/`finished`/`exit_code`/`stdout_tail`/`stderr_tail`/`stdout_size`/`stderr_size`/`log_file`/`duration_seconds`/`result_summary`; on failure it is `{ok:false,error,data}` and the process exits non-zero. When those fields are missing from the `complete` frame, the CLI fills them from the accumulated stream, so the field semantics match `--format json` exactly.
 
 ### Method 2: poll manually (when you need to see live output)
 
@@ -49,8 +55,13 @@ delta-cli sandbox logs <id> --execution-id <exec_id>
 #   the response is then an **array** grouped by execution_id (each item contains execution_id/stdout/stderr/exit_code/command/finished/error/log_file),
 #   rather than a single snapshot object; the CLI also applies 800/200-byte tail trimming and --tail/--grep/--context filters to each item
 
-# 3. Destroy after completion
+# 3. Destroy after completion (returns request_id / operation_status; destruction continues server-side)
 delta-cli sandbox kill <id>
+
+# 4. Reconcile: after a client disconnect, a create error or a network error on kill, use the operation ledger to confirm the real outcome
+delta-cli sandbox operation <request_id>
+# or list the create/destroy records for a sandbox
+delta-cli sandbox operation --sandbox-id <id>
 ```
 
 ### How to interpret polling results

package/skills/delta-sandbox/references/recipes.md

@@ -129,6 +129,7 @@ Notes:
 ## Local `result.json` content contract
 
 - **Preferred SSE path (recommended, one step):** The **complete frame for `sandbox run` / `run-bg --wait` includes `result_summary` directly** (the CLI injects detected trailing JSON from stdout), so the skill can extract it from the complete frame without reading and parsing the log.
+- **Preferred path for automation:** add `--format json` (same as `--json-result`) and the CLI emits a single envelope; read `data.exit_code` / `data.result_summary` / `data.stdout_tail` directly, without parsing multiple frames or reassembling stdout (newlines are preserved). On failure `data` is still present (`{ok:false,error,data}`).
 - **SSE fallback path:** If the complete frame lacks `result_summary` (there is no valid JSON at the end of stdout), `delta-cli sandbox read <id> --path <log_file>` reads the run-envelope log file, then scans backwards through stdout to extract trailing JSON. The `log_file` path comes from the SSE `complete` frame (the CLI fills it in if needed) or the `sandbox logs` snapshot (after `finished=true`).
 - **Legacy-server fallback path:** When `run` / `run-bg --wait` falls back to polling, it returns a `CommandResult` envelope whose `data.result_summary` field can be reused directly.
 - The local `result.json` is a **concise summary** for host deliverable validation, not a complete log archive.

package/skills/delta-shared/SKILL.md

@@ -65,19 +65,20 @@ This returns the current user's information and remaining credits (through the g
 - **Do not install the CLI proactively**: if the user says it is not installed, direct them to the README or `install.sh`; do not choose an installation method yourself.
 - **Confirm intent before write or delete operations**: tell the user what commands such as `sandbox kill`, `sandbox finish` and `sandbox write` will affect.
 - `~/.delta-infra/config.json` must have `0600` permissions; do not loosen them manually.
-- **The sandbox must be destroyed**: after calling `sandbox create`, call `sandbox kill` before the current task ends. The server does not clean up automatically; failure to destroy a sandbox leaves GPU resources occupied indefinitely. Do not skip this, even if an error occurs.
+- **The sandbox must be destroyed explicitly**: after calling `sandbox create`, call `sandbox kill` before the current task ends (use `finish` when results must be preserved). Sandboxes do get reclaimed automatically (default `--max-life` is 30 minutes; OpenSandbox starts the clock when create is accepted and the platform reclaims, while AutoDL/RunPod start at readiness and the server reclaims at max-life plus up to 10 minutes of grace), but the platform may reclaim early and automatic reclamation does not write back to OSS — relying on it loses artifacts. Do not skip destruction, even if an error occurs.
 
 ## JSON output conventions
 
 ```json
 {"ok":true,"data":{...}}
-{"ok":false,"error":{"type":"auth","message":"...","hint":"..."}}
+{"ok":false,"error":{"type":"auth","message":"...","hint":"...","request_id":"..."}}
 ```
 
 AI agent parsing rules:
 - Always check `ok`; do not rely on the exit code alone.
 - `error.type` is the stable field for branching on: `auth`, `permission`, `not_found`, `validation`, `network`, `api`, `command_failed`, `internal`.
-- `error.hint` may be shown to the user.
+- `error.hint` may be shown to the user. **Act on it rather than retrying blindly**: for a failed `sandbox create` the hint names the concrete cause and the next step (for example, the image was not cached on the target node — retry, or pick a GPU type whose node already has the image).
+- `error.request_id` identifies the same failure in the server log; quote it when reporting the failure.
 
 ## Exit code conventions
 
@@ -110,4 +111,5 @@ delta-cli upgrade
 | `error.type: network` | Check `base_url` and network connectivity |
 | `error.type: validation` | Check required flags and argument formats |
 | `error.type: command_failed` | Command failed inside the sandbox: the envelope's `message` already includes the stderr tail to help locate the issue quickly; the full `stderr` / `result_summary` are in the `complete` frame. Retry or correct the command |
+| `sandbox create` failed (`error.type: api` with a specific `hint`) | Follow the hint — it names the cause and the next step. "the image may not be cached on the target node" → retry once (the pull continues in the background, so later creates succeed) or pick a GPU type whose node already has the image. "check the target node's free disk space" → a retry will not help until an operator frees the node. A failed create leaves no sandbox behind, so retrying is safe. Quote `error.request_id` when reporting it. |
 | Configuration file parsing failed | Check that `~/.delta-infra/config.json` contains valid JSON |