@delta-infra/cli 1.1.23 → 1.1.26

package/README.md

@@ -121,12 +121,6 @@ DELTA_CLI_ARCHIVE=/path/to/delta-cli-linux-amd64.tar.gz npm install -g @delta-in
 | `DELTA_CLI_FATAL_ON_ERROR` | 安装失败时退出码非 0（默认警告，用于调试）。 |
 | `DELTA_CLI_DOWNLOAD_TIMEOUT` | 单个下载源最大等待时间（毫秒，默认 120000）。 |
 
-### 方式五：Go Install
-
-```bash
-go install github.com/delta-infra/delta-infra-cli/cmd/delta-cli@latest
-```
-
 ## 快速开始
 
 ### 1. 初始化配置
@@ -179,6 +173,14 @@ delta-cli auth status
 
 ```bash
 # 创建带 GPU 的 sandbox（--max-life 默认 30 分钟，长任务请调高）
+# ⚠️ 生命周期计时（详见 skills/delta-sandbox/references/lifecycle.md）：
+#   - 私有算力（OpenSandbox）：平台从 create 请求被接受时开始计时，镜像拉取/就绪
+#     时间也计入 max-life；到期由平台回收。
+#   - 第三方算力（AutoDL/RunPod）：从沙箱就绪（本地记录 started_at）开始计时，
+#     由服务端 cleanup 在 max-life + 最多 10 分钟宽限后回收。
+#   - 自动回收（含平台提前回收）不会做 OSS 写回；需要保留产物请提前 pull 或
+#     显式 `sandbox finish`。`sandbox status` 会返回 expires_at / reclaim_source /
+#     termination_reason，沙箱结束后仍可查询。
 # ⚠️ 资源语义：`--memory` 是容器 CPU 内存【总量】（每沙箱一份）；`--gpu-mem` 是【每张卡】的显存（MiB）——
 # `--gpu 2 --gpu-mem 8000` = 每卡 8000MiB、总显存 16000MiB（内部 OpenSandbox/HAMi 后端直接写入 HAMi 每卡注解；
 # 注意云端 runpod 后端将 --gpu-mem 视为总显存÷卡数）。单卡显存上限可查 `sandbox resources`（memory_mib）。
@@ -188,19 +190,20 @@ delta-cli auth status
 # `--devices` 申请宿主机设备直通（逗号分隔，如 kvm → 容器内 /dev/kvm，用于嵌套虚拟化）：属权限提升，
 # 服务端白名单未开启时创建会被明确拒绝（不会静默忽略）。两者仅私有算力生效，第三方算力忽略。
 delta-cli sandbox create \
-  --image image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
+  --image-name image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
   --cpu 4 --memory 16Gi --gpu 1 \
   --max-life 120
 
 # DataLoader 多进程/需要嵌套虚拟化时：调大 /dev/shm 并申请 /dev/kvm（shm 不得超过 memory）
 delta-cli sandbox create \
-  --image image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
+  --image-name image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
   --cpu 4 --memory 8Gi --shm-size 4Gi --devices kvm \
   --max-life 120
 
-# 需要长期存活、不被自动清理的 sandbox：加 --no-auto-cleanup（仅显式 kill/finish 可销毁）
+# 需要长期存活、不被自动清理的 sandbox：加 --no-auto-cleanup
+# （同时关闭平台 TTL 与服务端 cleanup，仅显式 kill/finish 可销毁）
 delta-cli sandbox create \
-  --image image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
+  --image-name image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
   --cpu 4 --memory 16Gi --gpu 1 \
   --no-auto-cleanup
 
@@ -209,15 +212,29 @@ delta-cli sandbox run <sandbox_id> \
   --command "<命令>" \
   --timeout 3600
 
-# v1.0.85+ sandbox run 走 SSE 实时流：stdout 为原始 data: 帧（无末尾信封），exit_code/log_file 在 complete 帧中
-# 需要 stdout 末尾 JSON 摘要时：读取 log_file 解析（sandbox read <id> --path <log_file>）
+# v1.0.85+ sandbox run 默认走 SSE 实时流：stdout 为原始 data: 帧（无末尾信封），
+# exit_code / log_file / stdout_tail / duration_seconds 在 complete 帧中
 delta-cli sandbox run <sandbox_id> --command "echo '{\"status\":\"ok\",\"result\":\"done\"}'" --timeout 60
 
+# 自动化/编排推荐：--format json（等价 --json-result）只输出一个 JSON 信封，
+# data 含 execution_id / finished / exit_code / stdout_tail / stderr_tail /
+# stdout_size / stderr_size / log_file / duration_seconds / result_summary。
+# 字段名与 complete 帧一致；失败时仍是单个对象：
+# {"ok":false,"error":{...},"data":{...}}，进程退出码非零。
+delta-cli sandbox run <sandbox_id> --command "python run.py" --timeout 600 --format json
+
+# 后台任务 + 等待 + 统一结果
+delta-cli sandbox run-bg <sandbox_id> --command "python train.py" --timeout 3600 --wait --format json
+
 # 读取结果文件
 delta-cli sandbox read <sandbox_id> --path /workspace/result.json
 
-# 销毁 sandbox
+# 销毁 sandbox（销毁在服务端继续执行，响应带 request_id）
 delta-cli sandbox kill <sandbox_id>
+
+# 创建重试幂等 + 事后对账
+delta-cli sandbox create --image-name <镜像名> --idempotency-key retry-001
+delta-cli sandbox operation <request_id>
 ```
 
 ## 命令列表
@@ -254,19 +271,20 @@ delta-cli sandbox kill <sandbox_id>
 | `delta-cli sandbox resources [--provider opensandbox\|autodl]` | 查看各后端当前剩余可申请资源（GPU/显存/核心） |
 | `delta-cli sandbox images` | 查看可用镜像列表（每条含 `image_name`/`tags`/`description`/`supports_shared_workspace`；`supports_shared_workspace=true` 表示该算力可开共享工作区（opensandbox/autodl），可与 `--shared-workspace` 搭配；实际播种/写回由平台创建时探测镜像内 rclone 决定） |
 | `delta-cli sandbox recommend --cpu N --memory XGi` | 获取资源配置推荐 |
-| `delta-cli sandbox create --image <img> [--shm-size XGi] [--devices kvm] [--no-auto-cleanup] [--shared-workspace=true\|false]` | 创建 sandbox 容器（`--no-auto-cleanup` 不被自动清理；`--shared-workspace` 默认 true：启用共享工作区（播种/写回由平台创建时探测镜像内 rclone 决定，镜像无 rclone 时跳过、数据不跨沙箱持久）；false 跳过播种/写回，适合快速任务；`--shm-size` 调大 /dev/shm（默认 64Mi，**必须带 Gi/Mi 单位**且 ≤ `--memory`，PyTorch DataLoader/multiprocessing 场景常用）；`--devices` 逗号分隔的宿主机设备令牌（`kvm` → /dev/kvm，嵌套虚拟化），**属权限提升**，服务端白名单未开启时**明确拒绝**；`--shm-size`/`--devices` 仅私有算力生效、第三方算力忽略） |
-| `delta-cli sandbox extend <id> --max-life N` | **动态延长运行中** sandbox 的 max-life（可增可减，N 需大于已存活分钟且 ≤10080；提前回收用 `kill`/`finish`）；不带 `--max-life` 时查询当前生命周期（已存活/剩余/到期时间）；`status` 输出含 `data.lifecycle` 生命周期摘要（已存活/剩余/到期；生命周期接口不可用时省略） |
+| `delta-cli sandbox create --image-name <镜像名> [--idempotency-key <key>] [--shm-size XGi] [--devices kvm] [--no-auto-cleanup] [--shared-workspace=true\|false]` | 创建 sandbox 容器（`--image` 为旧别名；**`--idempotency-key`**：网络重试时用同一个键会回放原 sandbox 而不会重复创建，同键换参数会被拒绝；响应含 `request_id`/`operation_status`/`replayed`；`--no-auto-cleanup` 同时关闭平台 TTL 与服务端 cleanup，仅显式 kill/finish 可销毁；`--shared-workspace` 默认 true：启用共享工作区（播种/写回由平台创建时探测镜像内 rclone 决定，镜像无 rclone 时跳过、数据不跨沙箱持久）；false 跳过播种/写回，适合快速任务；`--shm-size` 调大 /dev/shm（默认 64Mi，**必须带 Gi/Mi 单位**且 ≤ `--memory`，PyTorch DataLoader/multiprocessing 场景常用）；`--devices` 逗号分隔的宿主机设备令牌（`kvm` → /dev/kvm，嵌套虚拟化），**属权限提升**，服务端白名单未开启时**明确拒绝**；`--shm-size`/`--devices` 仅私有算力生效、第三方算力忽略） |
+| `delta-cli sandbox extend <id> --max-life N` | **动态延长运行中** sandbox 的 max-life（N 必须大于当前到期时间且 ≤10080；不接受缩短，提前回收用 `kill`/`finish`）；不带 `--max-life` 时查询当前生命周期（已存活/剩余/到期时间）；`status` 输出含 `data.lifecycle` 生命周期摘要（已存活/剩余/到期；生命周期接口不可用时省略） |
 | `delta-cli sandbox connect <id>` | 连接 sandbox |
-| `delta-cli sandbox status <id>` | 查看 sandbox 状态 |
-| `delta-cli sandbox finish <id>` | 保存结果并销毁 |
-| `delta-cli sandbox kill <id>` | 销毁 sandbox |
-| `delta-cli sandbox run <id> --command "..." [--timeout <秒>] [--summary/--no-summary] [--artifacts]` | 同步运行命令（SSE 原始 `data:` 帧透传到 stdout，无末尾信封，`exit_code`/`log_file` 在 `complete` 帧）；`--summary`/`--artifacts` 仅旧服务器回退路径生效 |
-| `delta-cli sandbox run-bg <id> --command "..." [--timeout <秒>] [--wait] [--summary/--no-summary] [--artifacts]` | 后台运行命令；不加 `--wait` 立即返回 `{execution_id, sandbox_id}`；加 `--wait` 通过 SSE 实时流跟随（原始 `data:` 帧透传到 stdout，无末尾信封） |
+| `delta-cli sandbox status <id>` | 查看 sandbox 状态；返回 `expires_at` / `reclaim_source`，已结束的沙箱仍可查询并带 `termination_reason` / `termination_remark` / `terminated_at`（如 `max_life_exceeded` / `platform_gone` / `user_killed` / `user_finished`） |
+| `delta-cli sandbox finish <id> [--no-wait]` | 保存结果并销毁；销毁是**服务端任务**，响应含 `request_id`/`operation_status`（`--no-wait` 立即返回 `pending`，断线后任务继续，可用 `sandbox operation <request_id>` 对账） |
+| `delta-cli sandbox kill <id> [--no-wait]` | 销毁 sandbox（不保存结果）；响应含 `request_id`/`operation_status`，中断或网络报错后可用 `sandbox operation` 查询真实终态 |
+| `delta-cli sandbox operation [request_id] [--sandbox-id <id>] [--operation create\|destroy] [--limit N]` | 查询服务端操作账本：带 `request_id` 查单条，否则按条件列出最近的创建/销毁记录（终态、时间、原因），用于计费与资源对账 |
+| `delta-cli sandbox run <id> --command "..." [--timeout <秒>] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | 同步运行命令。默认（`--format sse`）把 SSE 原始 `data:` 帧透传到 stdout，无末尾信封，`exit_code`/`log_file`/`stdout_tail`/`duration_seconds` 在 `complete` 帧；`--format json`（或 `--json-result`）只输出**一个** JSON 信封，`data` 含 `execution_id`/`finished`/`exit_code`/`stdout_tail`/`stderr_tail`/`stdout_size`/`stderr_size`/`log_file`/`duration_seconds`/`result_summary`，失败时为 `{ok:false,error,data}` 且进程非零退出；`--summary`/`--artifacts` 仅旧服务器回退路径生效。**退出码语义**：命令非零退出 → `error.type=command_failed`，进程退出码 **8**；命令已结束但**退出状态不可知**（流在 `complete` 帧前中断、或只有 error 帧）→ `error.type=unknown_execution_state`，进程退出码 1，**绝不会合成成功** |
+| `delta-cli sandbox run-bg <id> --command "..." [--timeout <秒>] [--wait] [--cancel-on-timeout] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | 后台运行命令；不加 `--wait` 立即返回 `{execution_id, sandbox_id}`（`--format json` 时为统一结果对象，`finished=false`）；加 `--wait` 通过 SSE 实时流跟随（默认透传原始 `data:` 帧）；`--format json` 时只输出一个与 `sandbox run --format json` 相同结构的信封。**等待超时** → `error.type=timeout`、进程**非零退出**，`error.details` 保留 `sandbox_id`/`execution_id`/`remote_state`（`not_cancelled`/`cancelled`/`cancel_failed`）以便事后查询或取消；`remote_state=not_cancelled` 表示**只是本地停止等待，远端命令可能仍在运行**。默认不取消远端命令，加 `--cancel-on-timeout` 才会下发取消（取消失败时报 `cancel_failed`） |
 | `delta-cli sandbox logs <id> --execution-id <eid> [--tail N --grep <pattern> --context N --max-bytes N]` | 查看后台日志；默认返回 `stderr_size + stderr_tail`（避免上下文爆炸）；可用 `--tail/--grep` 过滤 |
 | `delta-cli sandbox cancel <id> --execution-id <eid>` | 中断后台命令 |
-| `delta-cli sandbox read <id> --path <path> [--output <本地路径>] [--tail N] [--grep <pattern>] [--offset N] [--limit N] [--context N] [--max-bytes N] [--parse-json]` | 读取容器内文件；`--output <path>` 保存到本地；`--tail/--grep` 过滤；非 UTF-8 文件 CLI 自动走 base64 fallback |
-| `delta-cli sandbox pull <id> --source <沙箱路径> --target <本地路径> [--recursive] [--pattern <glob>]` | 从沙箱拉取文件/目录到本地（mirror of `upload`，flag 方向相反：source=远程，target=本地）；单文件或递归目录；CLI 端 + 服务端双向 sha1 完整性校验 |
-| `delta-cli sandbox write <id> --path <path> --source <本地路径>` | 写入文件 |
+| `delta-cli sandbox read <id> --path <path> [--output <本地路径>] [--tail N] [--grep <pattern>] [--offset N] [--limit N] [--context N] [--max-bytes N] [--parse-json]` | 读取容器内文件；`--output <path>` 保存到本地；`--tail/--grep` 过滤。**整份拷贝（不带 `--tail/--grep/--offset/--limit/--max-bytes`）走分块下载通道，字节精确**（二进制安全，响应含 `byte_exact:true`/`integrity:"sha256"`/`digest_verified`）；带上述过滤参数时走文本通道（服务端按 UTF-8 解码，非法字节会被替换为 U+FFFD），此时响应含 `byte_exact:false`/`integrity:"text_decoded"` 且 stderr 给出警告——**二进制文件请用 `sandbox pull` 拷贝** |
+| `delta-cli sandbox pull <id> --source <沙箱路径> --target <本地路径> [--pattern <glob>] [--concurrency N] [--no-resume] [--no-verify]` | 从沙箱拉取文件/目录到本地（mirror of `upload`，flag 方向相反：source=远程，target=本地）。**目录始终递归**（`--recursive` 已废弃，无需再加）；走**分块下载**通道：二进制安全、端到端 **sha256** 校验（响应含 `digest_verified`）、**可断点续传**（`.delta.part`）、多文件/多块**并行**传输（`--concurrency`，默认 4），因此不受单条命令 300s 超时限制。服务端公开的上限随每次响应返回（`limits`）：块大小 4 MiB（最大 16 MiB）、单文件最大 100 GiB、每块超时 120s、整文件摘要超时 600s。`--no-resume` 忽略残留的 `.delta.part`，`--no-verify` 跳过沙箱侧 sha256 比对 |
+| `delta-cli sandbox write <id> --path <path> --source <本地路径>` | 写入文件。**二进制安全**：内容按 base64 传输，CLI 同时声明 `size`/`sha256`，服务端在落盘前核对，不匹配直接拒绝；响应含 `server_sha256`/`sha256`/`size`/`size_verified`。超过 32 MiB 的 `--source` 走流式上传通道，不整份读入内存 |
 | `delta-cli sandbox write-multiple <id> --entry <远程路径>=<本地路径>` | 批量写入 |
 | `delta-cli sandbox ls <id> --path <path>` | 列出目录 |
 | `delta-cli sandbox stat <id> --path <path>` | 文件元数据 |
@@ -392,8 +410,8 @@ make install
 make PREFIX=$HOME/.local install
 
 # Windows 源码安装（二选一）
-# 方式 A：直接用 go install 安装到 GOPATH/bin
-go install github.com/delta-infra/delta-infra-cli/cmd/delta-cli@latest
+# 方式 A：在仓库根目录用 go install 安装到 GOPATH/bin
+go install ./cmd/delta-cli
 
 # 方式 B：在仓库根目录运行 PowerShell 脚本，构建并添加到用户 PATH
 # .\install-from-source.ps1

package/checksums.txt

@@ -1,5 +1,5 @@
-9eb9fe9e0d02bf4e67695e306a3108c0b1c7cdccca74c91ec3a0e31ab1445e85  delta-cli-darwin-amd64.tar.gz
-0bad652d1b34bda5023cc33dbb45c69728652eda998563eae51f43c1b9e69a9c  delta-cli-darwin-arm64.tar.gz
-a209b2e939ca8a1774a1c1a6ccf382484faa07bdc7a69b4f7ee25574a182022a  delta-cli-linux-amd64.tar.gz
-9cbcf8c8ef861fbcb36e63d3d7ede2087eb16cf774e9a4ee9e17c2b8094d19e0  delta-cli-linux-arm64.tar.gz
-da62e711c7e1f93912982a380d96b32584df2ade08eeb9a4a99872bb70304238  delta-cli-windows-amd64.zip
+443bab8338178a51c774f2df119b6cd5caa1abf26fa9a12cae35e4052aa71c07  delta-cli-darwin-amd64.tar.gz
+04375b629ec29f16e66d64618c333326c92a8df2514a5ee7f0b59d50c08956e7  delta-cli-darwin-arm64.tar.gz
+5735ceeb9e2d7daad0032d2337847a88c52cc181bb813dfd2e130776cf9a1e0e  delta-cli-linux-amd64.tar.gz
+217f08d2fd641ed653adcb908ec62b681274887dbcfc78d5362f6118c669b01b  delta-cli-linux-arm64.tar.gz
+42e17577a3d9cde1b7595010725ffc5326f10aa1b2dd928744813b645a4fe4e5  delta-cli-windows-amd64.zip

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@delta-infra/cli",
-  "version": "1.1.23",
+  "version": "1.1.26",
   "description": "Delta Agent Cloud CLI for sandbox compute and science tools.",
   "bin": {
     "delta-cli": "scripts/run.js"

package/skills/delta-protein/AGENTS.md

@@ -54,11 +54,11 @@ delta-protein/
   `## Output Template` sections in every reference from
   `scripts/output_metadata.json` (single source of truth for field projection /
   presentation form / downstream reuse / provenance marker). Stdlib only, idempotent.
-- Workflow: update `output_metadata.json` → run
+- Workflow: update `scripts/output_metadata.json` → run
   `python scripts/gen_references.py` → run `python scripts/gen_references.py --check`.
   `--check` fails loudly on missing sections, drifting titles, broken SKILL links or
   references not covered by metadata.
-- `output_metadata.json` `provenance` is `tested` (verified by a real `delta-cli`
+- `scripts/output_metadata.json` `provenance` is `tested` (verified by a real `delta-cli`
   invocation) or `derived` (from ToolSpec descriptions / agent templates, marked
   "pending live calibration"). Newly verified endpoints should be flipped to `tested`.
 - After changing references/SKILL, resync the opencode install dir:

package/skills/delta-sandbox/AGENTS.md

@@ -0,0 +1,37 @@
+# skills/delta-sandbox — Sandbox AI Agent Skill
+
+**Generated:** 2026-10-21
+**Parent:** `../AGENTS.md`
+
+## OVERVIEW
+
+The primary sandbox skill. It teaches an AI agent (Planner → executor) how to drive `delta-cli sandbox` end to end: create a sandbox, run commands in it, read and write files, tunnel ports, and collect artifacts. Ships as one skill directory in the npm tarball (`package.json` `files`) and is installed to every supported agent platform by `scripts/install-wizard.js`.
+
+## WHERE TO LOOK
+
+| Path | Content |
+|------|---------|
+| `SKILL.md` | The whole contract (401 lines): when to use, scope, sandbox environment, mandatory rules, request wording, required_outputs, quick routing, full lifecycle, tunnelling, structured-output convention, reading output, Planner guidance, recovery |
+| `references/commands.md` | Command cheat sheet (72 lines): discovery, lifecycle, command execution, file operations, tunnelling |
+| `references/lifecycle.md` | Full create-to-kill workflow (111 lines) incl. background tasks and polling patterns |
+| `references/recipes.md` | Standard task request patterns (139 lines): required_outputs templates, script output template, GPU/CUDA check, run-a-script, install-and-run, train-and-save-metrics, general data processing |
+| `agents/openai.yaml` | OpenAI-format agent descriptor |
+
+## CONVENTIONS
+
+- YAML frontmatter carries `name`, `description`, `metadata.requires.bins: ["delta-cli"]` and `metadata.cliHelp: "delta-cli sandbox --help"`
+- The `description` is what the Planner matches against — it encodes the output convention and the `required_outputs` requirement
+- `SKILL.md` opens with a **CRITICAL** instruction to read `../delta-shared/SKILL.md` first (auth, config, error handling)
+- "Mandatory rules" is a failure section: breaking any of them makes the task a failure
+- Request wording: describe commands with neutral verbs (run/execute); use create/write/save only when a file is actually written
+- When the Planner calls this skill it must declare `[{kind: 'file', extensions: ['.json']}]` in `required_outputs` for the plan step
+- Sandbox scripts/commands print structured JSON at the end of stdout; on the SSE path the CLI detects the trailing JSON and exposes it as `result_summary` in the complete frame
+- Cross-link to sibling skills and references with relative paths only
+
+## ANTI-PATTERNS
+
+- Do not embed realistic credentials, API keys, or tokens in the examples
+- Do not break the relative links between `SKILL.md`, `references/` and `../delta-shared/SKILL.md`
+- Do not reference internal Go package paths — use CLI command names
+- Do not document a command that `delta-cli sandbox` does not expose
+- Do not state that a sandbox has no TTL or auto-reclaim: the platform TTL is the primary expiry path, and the server also reclaims stale sandboxes — keep the wording consistent with `references/lifecycle.md` and the CLI help

package/skills/delta-sandbox/SKILL.md

@@ -35,7 +35,11 @@ metadata:
 1. **You must use delta-cli**: perform all sandbox operations with `delta-cli sandbox <subcommand>`. **Do not** call the HTTP API directly with Python (`urllib`/`httpx`/`requests`) or any SDK. delta-cli provides typed error output (`error.type`), the only reliable way for an AI agent to interpret errors.
 2. **You must destroy the sandbox**: after each successful `sandbox create`, you **must** run `sandbox kill <id>` or `sandbox finish <id>` to destroy it before the current task's final `step_finish`. Failing to destroy it counts as task failure.
 3. **Destroy it even after an error**: if a command inside the sandbox fails, times out or reports an error, you **must still** run `sandbox kill <id>`. An error is no reason to skip cleanup.
-4. **Do not rely on external cleanup**: the server has **no** automatic sandbox expiry or TTL mechanism. Without `kill`, the sandbox will occupy GPU resources indefinitely.
+4. **Do not rely on external cleanup**: sandboxes **do** have automatic reclamation, but it does not replace destroying them yourself.
+   - The default `--max-life` is 30 minutes. On private compute (OpenSandbox) the timer starts when the create request is accepted and the platform reclaims the sandbox; on third-party compute (AutoDL/RunPod) it starts when the sandbox is ready, and the server-side cleanup reclaims it at max-life plus up to 10 minutes of grace.
+   - The platform may still reclaim a sandbox early (cluster eviction, node failure, etc.; `termination_reason=platform_gone`).
+   - **Automatic reclamation does not write back to OSS**, so artifacts that were not pulled are lost.
+   - Without `kill`, the sandbox keeps occupying GPU resources until it expires, and the reclamation time cannot be used to preserve results. When the task ends you must still run `kill` / `finish` explicitly (only `finish` writes back to OSS).
 5. **Complete synchronous tasks in one command; background tasks may span turns**:
    - Synchronous mode (`sandbox run`): complete the full lifecycle from `create` to `kill` in the same execution. Do not retain a sandbox across tasks.
    - Background mode (`sandbox run-bg`): you may report `sandbox_id` and `execution_id` to the user during the current execution, then retrieve the result in a later turn with `sandbox status` and `sandbox logs`. You must still run `sandbox kill` when the task is complete.
@@ -49,6 +53,15 @@ metadata:
    ```
     The CLI follows output in real time over SSE (`/logs/stream`) and passes each raw SSE frame (`data: {...}`) through to stdout, as with `curl -N`. The JSON `type` field identifies each frame: `init` (with `execution_id`), `stdout`, `stderr`, `error` or `complete`. The command ends when `complete` arrives. **The CLI fills in fields in the complete frame**: the server supplies `exit_code`/`execution_id`/`error`; the CLI adds the accumulated streamed `stdout`/`stderr`, a recovered `log_file` as a fallback, and a trailing-JSON `result_summary` when it detects a valid JSON object in the last 50 lines of stdout. Use `exit_code` in the `complete` frame to determine success. **If the command exits non-zero or reports an error, the CLI appends an `error.type: command_failed` envelope and exits with a non-zero code (process exit code 8)**; the envelope's `message` includes the end of stderr, which makes it easier to use with `&&`, branch on `error.type` and scan the failure reason. The CLI also best-effort appends the stderr tail to the `message` of a `type:"error"` frame; it does not append an envelope on success. If no `complete` frame arrives within `--timeout` (300s by default, serving as both the server command timeout and the client wait deadline), the CLI appends a snapshot envelope with `finished=false` after the frames already passed through, indicating that the command is still running. After `finished`, full stdout is written to `log_file` inside the sandbox (default: `/workspace/{user_id}/sandbox_logs/delta-result-{sandbox_id}-{execution_id}.json`; use the server-returned `log_file` field as authoritative). To read the full output, use `sandbox read <id> --path <log_file>` (the trailing JSON summary is already in the complete frame, so this is usually unnecessary). **`--summary`/`--artifacts` enrichment, including hints, is available only on the older-server fallback path** (when the server lacks the `/logs/stream` endpoint, the CLI automatically falls back to polling every 5s and returns a `CommandResult` envelope). Use `execution_id` for later `sandbox logs` or `cancel` calls.
 
+    **For automation: `--format json` (same as `--json-result`)** — emits no `data:` frames, just **one** JSON envelope, so you can parse `data` directly instead of stitching stdout frames together:
+
+   ```bash
+   delta-cli sandbox run <id> --command "<command>" --timeout <seconds> --format json
+   delta-cli sandbox run-bg <id> --command "<command>" --timeout <seconds> --wait --format json
+   ```
+
+   The `data` fields have the same names and meaning as in the `complete` frame: `execution_id`, `finished`, `exit_code`, `stdout_tail`, `stderr_tail`, `stdout_size`, `stderr_size`, `log_file`, `duration_seconds` and `result_summary`. `stdout_tail`/`stderr_tail` are the last 2000 characters of the accumulated streamed output (newlines preserved), so there is no need to aggregate the `stdout` frames. When the command exits non-zero the CLI emits `{"ok":false,"error":{...},"data":{...}}`, where `data` is the same object (including `exit_code`), and the process exit code is 8. This mode shares field semantics with SSE mode, so you can switch between them at any time.
+
     **Live following / manual polling (when you need intermediate progress)**:
     `sandbox logs <id> --execution-id <exec_id> --stream` follows the task in real time (SSE), passing each raw `data:` frame through to stdout until the `complete` event, with no trailing envelope (`--stream` requires `--execution-id`). Without `--stream`, `sandbox logs <id> --execution-id <exec_id>` returns a snapshot envelope. You may omit `--execution-id`; if omitted, the server queries only by sandbox and automatically uses ES (Elasticsearch) because `sandbox_id` and `execution_id` were not both supplied (it returns an **array** aggregated by execution ID rather than a single snapshot object):
     - `finished=true` and `exit_code=0` → complete
@@ -88,14 +101,14 @@ metadata:
 
 13. **`result.json` is required before completion**
     - Before calling `step_finish`, confirm that:
-      1. `delta-cli sandbox run` / `run-bg --wait` has returned with `finished=true`;
+      1. `delta-cli sandbox run` / `run-bg --wait` has returned with `finished=true` **and `exit_code` = 0**. `finished=true` alone is not a success signal — a command that fails still finishes. A non-zero exit is reported as `error.type: command_failed` (process exit code 8), an **unknown** exit status as `error.type: unknown_execution_state` (exit code 1), and a wait timeout as `error.type: timeout` (non-zero exit, with `sandbox_id`/`execution_id` in `error.details`); in all three cases the task has **not** succeeded — report the failure instead of claiming a result;
       2. (By default) the `result_summary` in the SSE `complete` frame from `sandbox run` / `run-bg --wait` has been written to `result.json`; if it is missing (fallback), read `log_file` with `sandbox read` and parse it with Python.
       3. `sandbox kill <id>` has been run to destroy the sandbox;
       4. `final_response` consists of the single line `RESULT: <result_summary>`.
 
 14. **Avoid redundant checks and validation**
     - If `skill_request` or the user's request already specifies the image and resources, **do not** call `sandbox images` or `sandbox recommend` before `sandbox create`.
-    - After writing a file to the sandbox, it is enough to use `sandbox stat` (optionally) to confirm the size. **Do not** use `sandbox ls` plus `sandbox read` to read the file back to the host and verify its contents; that is an unnecessary round trip.
+    - After writing a file to the sandbox, it is enough to use `sandbox stat` (optionally) to confirm the size. **Do not** use `sandbox ls` plus `sandbox read` to read the file back to the host and verify its contents; that is an unnecessary round trip. (The `sandbox write` response already carries `server_sha256` and `size_verified`, so a local digest comparison is enough if the bytes matter.)
     - Do not repeatedly poll with `sandbox status` after a successful `sandbox create`; the sandbox is ready as soon as create returns.
 
 ## Request wording conventions
@@ -152,8 +165,8 @@ The reason is that, after running a command, this skill reads `log_file` from th
 | Finish a sandbox | `sandbox finish <id> [--results '{...}']` |
 | Destroy a sandbox | `sandbox kill <id>` |
 | **Command execution** | |
-| Run a command synchronously (≤60s) | `sandbox run <id> --command "..." [--timeout <seconds>] [--summary/--no-summary] [--artifacts]` |
-| Run a command in the background (>60s) | `sandbox run-bg <id> --command "..." [--timeout <seconds>] [--wait] [--summary/--no-summary] [--artifacts]` |
+| Run a command synchronously (≤60s) | `sandbox run <id> --command "..." [--timeout <seconds>] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` (`--format json` = a single unified result envelope, recommended for automation). **A non-zero exit is reported as `error.type: command_failed` with process exit code 8; an unknown exit status is reported as `error.type: unknown_execution_state` with exit code 1 — the CLI never reports an unknown outcome as success** |
+| Run a command in the background (>60s) | `sandbox run-bg <id> --command "..." [--timeout <seconds>] [--wait] [--cancel-on-timeout] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` (`--wait --format json` = a single unified result envelope). **If the wait times out the CLI exits non-zero with `error.type: timeout` and keeps `sandbox_id`/`execution_id`/`remote_state` in `error.details`; `remote_state: not_cancelled` means only the local wait stopped and the remote command may still be running — add `--cancel-on-timeout` to cancel it** |
 | View background logs | `sandbox logs <id> [--execution-id <exec_id>] [--tail N --grep <pattern> --context N] [--stream]` (`--execution-id` is optional; if omitted, query ES by sandbox only; only `--stream` requires it) |
 | Interrupt a background command | `sandbox cancel <id> --execution-id <exec_id>` |
 | **Tunnelling (port exposure)** | |
@@ -161,10 +174,10 @@ The reason is that, after running a command, this skill reads `log_file` from th
 | List active tunnels | `sandbox exposes <id>` — list the public tunnels currently active for this sandbox |
 | Reclaim a tunnel | `sandbox dispose <id> --port N` — explicitly reclaim a single tunnel (stop frpc and release the public port); killing the sandbox also reclaims all its tunnels |
 | **File operations** | |
-| Read a file | `sandbox read <id> --path <path> [--output <local-path> --tail N --grep <pattern> --offset N --limit N --context N --max-bytes N --parse-json]` |
-| Pull a file/directory | `sandbox pull <id> --source <sandbox-path> --target <local-path> [--recursive] [--pattern <glob>]` — pull a single file or recursively pull a directory, with sha1 verification (the reverse of upload: source=remote, target=local) |
+| Read a file | `sandbox read <id> --path <path> [--output <local-path> --tail N --grep <pattern> --offset N --limit N --context N --max-bytes N --parse-json]` — **a full copy is byte-exact** (chunked download, sha256-verified); with `--tail`/`--grep`/`--offset`/`--limit` it uses the text channel and the envelope reports `byte_exact:false` — use `sandbox pull` for binary files |
+| Pull a file/directory | `sandbox pull <id> --source <sandbox-path> --target <local-path> [--pattern <glob>] [--concurrency N] [--no-resume] [--no-verify]` — pull a single file or **recursively** pull a directory (directories are always recursive), over the **chunked** path: binary-safe, sha256-verified, resumable and parallel (the reverse of upload: source=remote, target=local) |
 | View the working directory | `sandbox working-directory <id>` (returns `data.path`, such as `/workspace/{user_id}/{sandbox_id}`) |
-| Write a file | `sandbox write <id> --source <filename>` (without `--path`, writes to the working directory by default); **every sandbox-side path** — `--path`, the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, `run`/`run-bg --log-file` — is prefixed with the working directory when relative, while an absolute path is used as given |
+| Write a file | `sandbox write <id> --source <filename>` (without `--path`, writes to the working directory by default); **every sandbox-side path** — `--path`, the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, `run`/`run-bg --log-file` — is prefixed with the working directory when relative, while an absolute path is used as given. The content is **binary-safe** (base64) and the server verifies the declared `size`/`sha256` before writing, so the response's `server_sha256`/`size_verified` can be compared with the local file |
 | Write multiple files | `sandbox write-multiple <id> --entry <remote-path>=<local-path> [--entry ...]` |
 | Upload a directory | `sandbox upload <id> --source <local-directory> --target <sandbox-path>` |
 | List a directory | `sandbox ls <id> --path <path>` (defaults to `.`) |
@@ -191,7 +204,7 @@ The reason is that, after running a command, this skill reads `log_file` from th
     - **`--gpu-type <model>` (optional) selects a GPU model**: for example, `"RTX 4090"` or `H100`. This schedules the task on that GPU model. Use `sandbox resources` to see available models (`gpu_types[].gpu_name`). Specify a model only when the user explicitly requests one; otherwise, the system schedules automatically.
     - **`--shm-size <size>` (optional) increases the container's `/dev/shm`**: the default is only **64Mi**. Workloads such as PyTorch `DataLoader(num_workers>0)`, `torch.multiprocessing`/shared memory and Chromium-based engines (Playwright/Selenium) may fail with `No space left on device` / `Bus error` when shared memory is insufficient; increase it as needed (for example, `--shm-size 4Gi`). Hard constraints: **include Gi/Mi units** (`4Gi`/`512Mi`; the CLI normalises `4G`/`512M` to `4Gi`/`512Mi`, while the server rejects a bare number such as `4` with a clear error because its meaning differs from the server's); **do not exceed `--memory`** (`/dev/shm` is a memory filesystem counted against the container's memory cgroup; filling it can cause an OOMKill). Since `--memory` defaults to only `1Gi`, increasing shm usually means increasing `--memory` too (for example, `--memory 8Gi --shm-size 4Gi`). This parameter is applied only on private compute (third-party compute ignores it); however, unit and limit checks happen before creation regardless of backend, so invalid values are rejected even when using third-party compute. The create response echoes the effective `resource.shmSize` for verification.
     - **`--devices <token>` (optional) requests host-device passthrough**: a comma-separated token list (currently `kvm` is supported, which adds `/dev/kvm` inside the container), used for **nested virtualisation** workloads such as QEMU/KVM, Android emulators and Kata inside the sandbox. **This is a privilege escalation, not an ordinary resource request**: device passthrough exposes host devices to the container, so the server uses an explicit allowlist. If operations staff have not enabled it, creation is **explicitly rejected (not silently ignored)**. Do not retry or change the syntax after this error; use an approach that does not require the device. This parameter is applied only on private compute (third-party compute ignores it), but the allowlist check happens before creation regardless of backend.
-    - `--no-auto-cleanup`: when this flag is set, the sandbox **will not be cleaned up automatically** (both server-side timeout reclamation and the local periodic `cleanup_stale` process are skipped). It can be destroyed only by an explicit `sandbox kill`/`finish`. Use this only when the task genuinely needs to survive past periodic cleanup; you must destroy it explicitly when the task is done to avoid leaking resources.
+    - `--no-auto-cleanup`: when this flag is set, the sandbox **will not be cleaned up automatically** (OpenSandbox gets `timeout=None`, so platform TTL and the server's periodic cleanup are both skipped). It can be destroyed only by an explicit `sandbox kill`/`finish`, and `status` reports `expires_at`/`remaining` as null. Use this only when the task genuinely needs to survive past periodic cleanup; you must destroy it explicitly when the task is done to avoid leaking resources.
     - **Do not repeatedly poll with `sandbox status` after a successful create**. The sandbox is ready as soon as `sandbox create` returns; use its returned `data.sandbox_id` directly for `write`/`run`. Extra polling adds tool calls without providing any benefit.
 3. **Write code/data**:
    - **Single file**: `delta-cli sandbox write <id> --source <filename>` — omit `--path` to write to the sandbox's working directory by default (`/workspace/{user_id}/{sandbox_id}/`; query it with `delta-cli sandbox working-directory <id>`). Pass only a local filename to `--source` (for example, `--source train.py`). **Do not** use shell-variable paths such as `"$WORKSPACE_ROOT/train.py"` or `$(pwd)/train.py`. With `--source`, the CLI reads the local file itself without shell expansion, making this the safest option. If you need a specific path, a relative `--path <relative-path>` is automatically prefixed with the working directory; absolute paths are used as given (the same rule applies to every sandbox-side path: the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, and `run`/`run-bg --log-file`) (**avoid** paths such as `/workspace/<filename>` outside `{user_id}`, as they do not persist after `kill`). The returned `size` field is the actual number of bytes on disk (verified with stat), so you can check that the write is complete. For small configuration files (<20 lines), you may use `--data "..."`, but you must also pass `--path`. File paths and extensions depend on the runtime in the image.
@@ -201,7 +214,7 @@ The reason is that, after running a command, this skill reads `log_file` from th
      - Nested directories are supported, and empty directories are created too.
      - **Optional post-write check**: use `sandbox stat <id> --path <path>` to confirm that the file exists and its size is as expected. **Do not** use `sandbox ls` plus `sandbox read` to read the just-written file back to the host and compare it character by character; if there is no issue, no read-back is needed.
 4. **Run the command**:
-    - **Short task (expected to take ≤ 60 seconds)**: run synchronously with `delta-cli sandbox run <id> --command "<command>" --timeout <seconds>`. The command's stdout/stderr is passed to stdout over SSE as **raw `data:` frames, one at a time** (with no trailing envelope). Read `exit_code` / `log_file` / `execution_id` from the `complete` frame. After completion, the full `stdout` is written to the run-envelope log file (the `log_file` field); use `sandbox read` to read it in full or extract trailing JSON. **Do not** then call `sandbox logs`. You can customise the run-envelope log path with `--log-file <path>`; the default is `/workspace/{user_id}/sandbox_logs/delta-result-{sandbox_id}-{execution_id}.json` (use the server-returned `log_file` field as authoritative). A script written with `sandbox write` is in the working directory; run it using its full path, `<working-directory>/<filename>` (first query `sandbox working-directory <id>`). Build the command for the runtime in the image. Common examples (where `<wd>` is the working directory):
+    - **Short task (expected to take ≤ 60 seconds)**: run synchronously with `delta-cli sandbox run <id> --command "<command>" --timeout <seconds>`. The command's stdout/stderr is passed to stdout over SSE as **raw `data:` frames, one at a time** (with no trailing envelope). Read `exit_code` / `log_file` / `execution_id` / `stdout_tail` / `stderr_tail` / `duration_seconds` from the `complete` frame. For automation, add `--format json` (or `--json-result`) so the CLI emits a single unified result envelope instead of making you stitch frames together yourself. After completion, the full `stdout` is written to the run-envelope log file (the `log_file` field); use `sandbox read` to read it in full or extract trailing JSON. **Do not** then call `sandbox logs`. You can customise the run-envelope log path with `--log-file <path>`; the default is `/workspace/{user_id}/sandbox_logs/delta-result-{sandbox_id}-{execution_id}.json` (use the server-returned `log_file` field as authoritative). A script written with `sandbox write` is in the working directory; run it using its full path, `<working-directory>/<filename>` (first query `sandbox working-directory <id>`). Build the command for the runtime in the image. Common examples (where `<wd>` is the working directory):
      - Python: `python <wd>/train.py`
      - Node.js: `node <wd>/app.js`
      - Go: `go run <wd>/main.go`

package/skills/delta-sandbox/references/commands.md

@@ -17,19 +17,20 @@
 
 | Command | Description |
 |------|------|
-| `sandbox create --image-name <image-name> [--cpu N --memory XGi --gpu N --gpu-mem N --gpu-type <model> --shm-size XGi --devices kvm --max-life M --no-auto-cleanup]` | Create a sandbox container (`--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; the underlying image identifier is hidden from users. `--gpu-type` selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish); `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, `1G`/`512M`; lowercase is also accepted), which the CLI converts to the `Gi`/`Mi` units accepted by the server; `--shm-size` increases the container's `/dev/shm` (the **default is only 64Mi**; it must be increased if PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error` because shared memory is insufficient; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number); it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only `1Gi`, so increasing shm usually requires increasing memory too); validation is backend-independent and rejects invalid values before creation, but this setting is applied only on private compute (third-party compute ignores it); `--devices` requests host-device passthrough using comma-separated tokens (for example, `kvm` makes `/dev/kvm` available inside the container for nested virtualisation; this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** rather than silently ignored; this setting is applied only on private compute); **resource limits per sandbox**: `--cpu` ≤ 512, `--memory` ≤ 1024Gi, `--gpu` ≤ 64, `--gpu-mem` ≤ 1024Gi, and `--shm-size` ≤ `--memory`. Exceeding a limit returns a **validation error promptly at request time** (it does not hang); normal tasks are well below these limits. The response echoes the requested image name/resource (request values fill fields not returned by the server; server values take precedence); compute backend is an internal detail and is selected automatically, so `--provider` is no longer supported. If creation fails, `error.hint` states the cause and the next step (for example, the image was not cached on the target node: retry, or pick a GPU type whose node already has the image), and `error.request_id` identifies the failure in the server log. |
+| `sandbox create --image-name <image-name> [--idempotency-key <key>] [--cpu N --memory XGi --gpu N --gpu-mem N --gpu-type <model> --shm-size XGi --devices kvm --max-life M --no-auto-cleanup]` | Create a sandbox container (`--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; the underlying image identifier is hidden from users. **`--idempotency-key`**: a retry with the same key replays the original sandbox instead of creating a duplicate, and the same key with different parameters is rejected; the response includes `request_id`/`operation_status`/`replayed`. `--gpu-type` selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish); `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, `1G`/`512M`; lowercase is also accepted), which the CLI converts to the `Gi`/`Mi` units accepted by the server; `--shm-size` increases the container's `/dev/shm` (the **default is only 64Mi**; it must be increased if PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error` because shared memory is insufficient; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number); it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only `1Gi`, so increasing shm usually requires increasing memory too); validation is backend-independent and rejects invalid values before creation, but this setting is applied only on private compute (third-party compute ignores it); `--devices` requests host-device passthrough using comma-separated tokens (for example, `kvm` makes `/dev/kvm` available inside the container for nested virtualisation; this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** rather than silently ignored; this setting is applied only on private compute); **resource limits per sandbox**: `--cpu` ≤ 512, `--memory` ≤ 1024Gi, `--gpu` ≤ 64, `--gpu-mem` ≤ 1024Gi, and `--shm-size` ≤ `--memory`. Exceeding a limit returns a **validation error promptly at request time** (it does not hang); normal tasks are well below these limits. The response echoes the requested image name/resource (request values fill fields not returned by the server; server values take precedence); compute backend is an internal detail and is selected automatically, so `--provider` is no longer supported. If creation fails, `error.hint` states the cause and the next step (for example, the image was not cached on the target node: retry, or pick a GPU type whose node already has the image), and `error.request_id` identifies the failure in the server log. |
 | `sandbox connect <id>` | Connect to an existing sandbox. |
 | `sandbox status <id>` | Check sandbox status (running / done). |
-| `sandbox finish <id> [--results '{...}']` | Save results and destroy the sandbox automatically. |
-| `sandbox kill <id>` | Destroy the sandbox immediately without saving results. |
+| `sandbox finish <id> [--results '{...}'] [--no-wait]` | Save results and destroy the sandbox automatically. Destruction is a server-side operation: the response includes `request_id`/`operation_status`; `--no-wait` returns immediately with `pending`. |
+| `sandbox kill <id> [--no-wait]` | Destroy the sandbox immediately without saving results; the response includes `request_id`/`operation_status`. |
+| `sandbox operation [request_id] [--sandbox-id <id>] [--operation create\|destroy] [--limit N]` | Query the server-side operation ledger (terminal state, time and reason). Use it to confirm the real outcome of a create or destroy after a client disconnect, timeout or network error. |
 | `sandbox extend <id> [--max-life N]` | Query or adjust the max lifetime of a running sandbox (N in minutes; omit `--max-life` to query the current lifecycle). `status` also reports the lifecycle summary under `data.lifecycle`. |
 
 ## Command execution
 
 | Command | Description |
 |------|------|
-| `sandbox run <id> --command "..." [--timeout N] [--summary/--no-summary] [--artifacts]` | Run a command synchronously (live SSE stream; raw `data:` frames are passed through to stdout one by one; **there is no trailing envelope on success**; the CLI fills in `exit_code`/`execution_id`/`log_file`/`stdout`/`stderr`/`result_summary` in the `complete` frame; **if the command exits non-zero or errors, an `error.type: command_failed` envelope is appended and the process exits non-zero**; the envelope's `message` includes the stderr tail, as does the message in a `type:"error"` frame on a best-effort basis). `--summary`/`--artifacts` take effect only on the legacy-server fallback path (when there is no SSE endpoint and a `CommandResult` envelope is returned). |
-| `sandbox run-bg <id> --command "..." [--timeout N] [--wait] [--summary/--no-summary] [--artifacts]` | Run a command in the background. Without `--wait`, return `{execution_id, sandbox_id}` immediately. With `--wait`, follow the live SSE stream until `complete` (raw `data:` frames are passed through to stdout; the CLI fills in `log_file`/`stdout`/`stderr`/`result_summary` in the `complete` frame; **on command failure, a `command_failed` envelope is appended and the process exits non-zero**; the envelope's `message` includes the stderr tail; on timeout, a `finished=false` snapshot envelope is appended). `--summary`/`--artifacts` take effect only on the legacy-server fallback path. |
+| `sandbox run <id> --command "..." [--timeout N] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | Run a command synchronously (live SSE stream; raw `data:` frames are passed through to stdout one by one; **there is no trailing envelope on success**; the CLI fills in `exit_code`/`execution_id`/`log_file`/`stdout`/`stderr`/`stdout_tail`/`stderr_tail`/`stdout_size`/`stderr_size`/`duration_seconds`/`result_summary` in the `complete` frame; **if the command exits non-zero or errors, an `error.type: command_failed` envelope is appended and the process exits non-zero (exit code 8)**; if the command finished but its **exit status is unknown** (the stream ended before the `complete` frame, or only an error frame arrived), the envelope is `error.type: unknown_execution_state` with exit code 1 — the CLI **never synthesizes a success** in that case; the envelope's `message` includes the stderr tail, as does the message in a `type:"error"` frame on a best-effort basis). `--format json` (same as `--json-result`) emits no frames, only one envelope whose `data` is the unified result object above; on failure it is `{ok:false,error,data}`. `--summary`/`--artifacts` take effect only on the legacy-server fallback path (when there is no SSE endpoint and a `CommandResult` envelope is returned). |
+| `sandbox run-bg <id> --command "..." [--timeout N] [--wait] [--cancel-on-timeout] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | Run a command in the background. Without `--wait`, return `{execution_id, sandbox_id}` immediately (with `--format json`, the unified result object with `finished=false`). With `--wait`, follow the live SSE stream until `complete` (raw `data:` frames are passed through to stdout; the CLI fills in `log_file`/`stdout`/`stderr`/`result_summary` in the `complete` frame; **on command failure, a `command_failed` envelope is appended and the process exits non-zero (exit code 8)**; the envelope's `message` includes the stderr tail). **When the wait times out, the CLI exits non-zero with `error.type: timeout`** and `error.details` carrying `sandbox_id`, `execution_id` and `remote_state` (`not_cancelled` / `cancelled` / `cancel_failed`) so the command can be queried or cancelled afterwards; `remote_state: not_cancelled` means only the local wait stopped and **the remote command may still be running**. The remote command is left running by default — pass `--cancel-on-timeout` to cancel it (a failed cancel is reported as `cancel_failed`). With `--format json`, emit only one envelope with the same structure as `sandbox run --format json`. `--summary`/`--artifacts` take effect only on the legacy-server fallback path. |
 | `sandbox logs <id> --execution-id <eid> [--tail N --grep <pattern> --context N] [--stream]` | Retrieve background-command logs. **By default, returns `stdout_tail` (last 800 bytes) + `stderr_tail` (last 200 bytes) + `stderr_size` + `cursor`**, about 1 KB in total, to keep context small. `--stream` follows the live `/logs/stream` SSE endpoint, passing raw `data:` frames through to stdout until `complete` (the CLI fills in `stdout`/`stderr`/`log_file`/`result_summary` in the complete frame; there is no trailing envelope). `--tail N` applies to both stdout and stderr (last N lines from each); `--grep <pattern>` filters both stdout and stderr (regular expression); `--context N` includes N lines before and after each grep match (for both stdout and stderr). The `cursor` field has no `omitempty` and is always present, even when 0. Snapshots include `last_updated` (filled by the CLI on the SSE path; the server snapshot path is awaiting a server-side update). **If `--execution-id` is omitted, query by sandbox** (the server uses ES); the response is an **array** grouped by execution_id (each item contains `execution_id`/`stdout`/`stderr`/`exit_code`/`command`/`finished`/`error`/`log_file`), and the CLI applies the same tail/grep trimming to each item. |
 | `sandbox cancel <id> --execution-id <eid>` | Interrupt a background command that is running. |
 
@@ -37,12 +38,12 @@
 
 | Command | Description |
 |------|------|
-| `sandbox read <id> --path <path> [--output <local-path>] [--tail N] [--grep <pattern>] [--offset N] [--limit N] [--context N] [--max-bytes N] [--parse-json]` | Read a file; returns `content` + `size` + `content_length`. `--output <path>` saves it locally; `--tail`/`--grep` filter the output. For non-UTF-8 files, the CLI automatically falls back to base64. |
+| `sandbox read <id> --path <path> [--output <local-path>] [--tail N] [--grep <pattern>] [--offset N] [--limit N] [--context N] [--max-bytes N] [--parse-json]` | Read a file; returns `content` + `size` + `content_length`. `--output <path>` saves it locally; `--tail`/`--grep` filter the output. **A full copy (no `--tail`/`--grep`/`--offset`/`--limit`/`--max-bytes`) is byte-exact**: it goes through the chunked download path (binary-safe, sha256-verified — the envelope reports `byte_exact:true`, `integrity:"sha256"`, `digest_verified`). With any range filter it falls back to the text channel, where the server decodes as UTF-8 and replaces invalid bytes with U+FFFD; the envelope then reports `byte_exact:false` / `integrity:"text_decoded"` and a warning is printed to stderr. **Use `sandbox pull` to copy a binary file.** |
 | `sandbox working-directory <id>` | Get the sandbox's current working directory (absolute path `/workspace/{user_id}/{sandbox_id}`). |
-| `sandbox write <id> [--path <path>] --source <filename>` | Write a file (recommended). If `--path` is omitted, write to the working directory (`<working-directory>/<filename>`); a relative `--path` is also resolved under the working directory. An absolute `--path` is used as-is (but the `/workspace/<filename>` root path is outside the OSS sync scope and is not recommended). |
+| `sandbox write <id> [--path <path>] --source <filename>` | Write a file (recommended). If `--path` is omitted, write to the working directory (`<working-directory>/<filename>`); a relative `--path` is also resolved under the working directory. An absolute `--path` is used as-is (but the `/workspace/<filename>` root path is outside the OSS sync scope and is not recommended). **Binary-safe**: the content travels as base64 and the CLI declares its `size`/`sha256`, which the server verifies before writing (a mismatch is rejected). The response carries `server_sha256`/`sha256`/`size`/`size_verified`. |
 | `sandbox write <id> --path <absolute-path> --data "..."` | Write a small amount of inline content (`--data` has no source filename, so `--path` must be specified explicitly). |
 | `sandbox write-multiple <id> --entry <remote-path>=<local-path> [--entry ...]` | Write multiple files in one operation (a relative target `path` is also resolved under the working directory). |
-| `sandbox pull <id> --source <sandbox-path> --target <local-path> [--recursive] [--pattern <glob>]` | Pull files/directories to the local machine (the reverse of `upload`; flag directions are reversed: `--source`=remote sandbox path, `--target`=local path). Single files are detected automatically; directories are traversed recursively by default; SHA-1 integrity is checked by both the CLI and server. |
+| `sandbox pull <id> --source <sandbox-path> --target <local-path> [--pattern <glob>] [--concurrency N] [--no-resume] [--no-verify]` | Pull files/directories to the local machine (the reverse of `upload`; flag directions are reversed: `--source`=remote sandbox path, `--target`=local path). Single files are detected automatically; **directories are always traversed recursively** (`--recursive` is deprecated and no longer needed). The transfer uses the **chunked download** path: binary-safe, end-to-end **sha256** verified (`digest_verified`), **resumable** (a partial `<target>.delta.part` is reused), and **parallel** across files and chunks (`--concurrency`, default 4) — so a large file is not bound by a single command timeout. The server publishes its limits with every response (`limits`): 4 MiB chunks (16 MiB max), 100 GiB max per file, 120 s per chunk, 600 s for the whole-file digest. `--no-resume` ignores a leftover `.delta.part`; `--no-verify` skips the sandbox-side sha256 comparison. |
 | `sandbox upload <id> --source <local-directory> --target <sandbox-path>` | Upload a local directory (packaged as tar.gz, extracted server-side) and verify the resulting file list against the local manifest. `--source` is local, `--target` is inside the sandbox. |
 | `sandbox ls <id> --path <path>` | List directory contents (defaults to `.`). |
 | `sandbox stat <id> --path <path>` | Get file metadata (size / mode / owner / group). |

package/skills/delta-sandbox/references/lifecycle.md

@@ -6,16 +6,21 @@
 
 1. **Check available images** `delta-cli sandbox images` — list the images supported by the server. The image determines which language runtimes are available in the container (Python / Node.js / Go / Java / Rust, etc.); choose one that fits the user's technology stack and GPU/CPU requirements.
 2. **List existing sandboxes (optional)** `delta-cli sandbox list` — check the active sandboxes already created by the current user to avoid creating duplicates.
-3. **Create** `delta-cli sandbox create --image-name <image-name> --cpu 4 --memory 16Gi --gpu 1 --gpu-mem 8000 --max-life 120 [--gpu-type <model>] [--shm-size XGi] [--devices kvm]` (the sandbox is ready to use immediately; no connection step is needed. `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, 1G/512M; lowercase is also accepted), which the CLI converts to Gi/Mi. `--max-life` sets the sandbox's maximum lifetime in minutes; the default is 30. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish. `--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; compute backend selection is automatic and does not need to be specified. `--gpu-type` (optional) selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--shm-size` (optional) increases the container's `/dev/shm` (the **default is only 64Mi**)—it must be increased when PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error`; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number), and it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only 1Gi, so increasing shm usually requires increasing memory too, for example, `--memory 8Gi --shm-size 4Gi`). Only private compute (`private`) applies this setting (third-party compute ignores it), but unit/limit validation is backend-independent and rejects invalid values before creation. `--devices` (optional) requests host-device passthrough using comma-separated tokens (for example, `--devices kvm` makes `/dev/kvm` available inside the container for nested virtualisation); this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** (not silently ignored; validation is backend-independent). If this error occurs, do not retry; use an approach that does not depend on that device. Only private compute applies this setting. **Resource limits per sandbox**: cpu ≤ 512, memory ≤ 1024Gi, gpu ≤ 64, gpuMem ≤ 1024Gi, shmSize ≤ memory. Exceeding a limit returns a validation error promptly at request time rather than hanging.)
+3. **Create** `delta-cli sandbox create --image-name <image-name> --cpu 4 --memory 16Gi --gpu 1 --gpu-mem 8000 --max-life 120 [--gpu-type <model>] [--shm-size XGi] [--devices kvm] [--idempotency-key <key>]` (the sandbox is ready to use immediately; no connection step is needed. `--idempotency-key` (optional) makes network retries safe: the same key replays the original sandbox instead of creating a duplicate, and the response includes `request_id`/`operation_status`/`replayed`; `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, 1G/512M; lowercase is also accepted), which the CLI converts to Gi/Mi. `--max-life` sets the sandbox's maximum lifetime in minutes; the default is 30. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish. `--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; compute backend selection is automatic and does not need to be specified. `--gpu-type` (optional) selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--shm-size` (optional) increases the container's `/dev/shm` (the **default is only 64Mi**)—it must be increased when PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error`; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number), and it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only 1Gi, so increasing shm usually requires increasing memory too, for example, `--memory 8Gi --shm-size 4Gi`). Only private compute (`private`) applies this setting (third-party compute ignores it), but unit/limit validation is backend-independent and rejects invalid values before creation. `--devices` (optional) requests host-device passthrough using comma-separated tokens (for example, `--devices kvm` makes `/dev/kvm` available inside the container for nested virtualisation); this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** (not silently ignored; validation is backend-independent). If this error occurs, do not retry; use an approach that does not depend on that device. Only private compute applies this setting. **Resource limits per sandbox**: cpu ≤ 512, memory ≤ 1024Gi, gpu ≤ 64, gpuMem ≤ 1024Gi, shmSize ≤ memory. Exceeding a limit returns a validation error promptly at request time rather than hanging.)
    - **If `create` fails**: the error envelope's `hint` states the cause and the next step. A common one is "the image may not be cached on the target node" — the initial pull exceeded the readiness timeout, so retry once (the pull continues in the background and later creates succeed) or pick a GPU type whose node already has the image. A failed create leaves no sandbox behind, so retrying is safe; `error.request_id` identifies the failure in the server log.
 4. **Write code/data** `delta-cli sandbox write <id> --source <filename>` — write a local file directly to the sandbox. If `--path` is omitted, the file is written to that sandbox's working directory by default (`/workspace/{user_id}/{sandbox_id}/<filename>`, which is within the OSS sync scope, so the file persists after kill). Before writing, use `delta-cli sandbox working-directory <id>` to confirm the path. **You must** use a relative source path (for example, `--source infer.py`); do not use `"$WORKSPACE_ROOT/<filename>"` or `$(pwd)/...`. A relative `--path` is also resolved under the working directory; an absolute `--path` is used as-is (the same rule applies to every sandbox-side path: the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, and `run`/`run-bg --log-file`) (`/workspace/<filename>` is outside the OSS sync scope and is not recommended). You can also use `--data "..."` to write a small amount of inline content (requires an explicit `--path`), and `--mode 755` to set file permissions. The returned `size` field is the actual on-disk byte count verified with stat, so it can confirm that the write completed. Use `write-multiple` to write several files at once. File paths and extensions depend on the runtimes in the image.
 5. **Run a command** — first use `delta-cli sandbox working-directory <id>` to get `<wd>`, then construct a command for a runtime in the image. Common examples: `python <wd>/train.py` (Python), `node <wd>/app.js` (Node.js), `go run <wd>/main.go` (Go), `bash <wd>/run.sh` (Shell).
    - **Short tasks ≤60s**: `delta-cli sandbox run <id> --command "<command>" --timeout <seconds>` runs synchronously; raw SSE `data:` frames are passed through to stdout (with no trailing envelope), `exit_code`/`log_file` are read from the `complete` frame, and full stdout is written to the log_file.
    - **Long tasks >60s**: run `delta-cli sandbox run-bg <id> --command "<command>" --timeout <seconds> --wait` in the background (recommended), or use `run-bg` + `logs` for manual polling (`run-bg --wait` includes `--summary` by default; without `--wait`, it returns `{execution_id, sandbox_id}` immediately).
 6. **Read the results** `delta-cli sandbox read <id> --path <wd>/result.json` (`<wd>` is the working directory; use `delta-cli sandbox working-directory <id>` to find it) — returns `content` (contents), `size` (disk bytes, from stat) and `content_length` (character count). Reading a non-existent file returns an error rather than empty content.
-7. **Destroy the sandbox** `delta-cli sandbox kill <id>` (if you need to save results, use finish; finish destroys the sandbox automatically).
+7. **Destroy the sandbox** `delta-cli sandbox kill <id>` (if you need to save results, use finish; finish destroys the sandbox automatically). Both are **server-side operations**: the response includes `request_id`/`operation_status`, they continue after the client disconnects, and `delta-cli sandbox operation <request_id>` reconciles the real outcome.
 
-> **Note:** The server has no automatic sandbox expiry or TTL mechanism. If you do not kill a sandbox, it will occupy GPU resources indefinitely. Always destroy it when the task is complete.
+> **Note:** sandboxes **do** get reclaimed automatically, but when and from which starting point depends on the backend:
+> - Private compute (OpenSandbox): `--max-life` (default 30 minutes) starts when the create request is accepted and the platform reclaims the sandbox at expiry; image pull / readiness time counts towards it too.
+> - Third-party compute (AutoDL/RunPod): the clock starts when the sandbox is ready, and the server-side cleanup reclaims it at max-life plus up to 10 minutes of grace.
+> - The platform may still reclaim a sandbox early (cluster eviction, node failure; `termination_reason=platform_gone`), and automatic reclamation **does not** write back to OSS, so artifacts that were not pulled are lost.
+> - Therefore you must still run `kill`/`finish` explicitly when the task ends; to preserve artifacts use `finish` or `pull` beforehand. `sandbox status` returns `expires_at` / `reclaim_source` / `termination_reason`, and remains queryable after the sandbox has ended.
+> - `--no-auto-cleanup` disables both the platform TTL and the server-side cleanup, so `expires_at`/`remaining` are null.
 
 ## Background tasks
 
@@ -30,7 +35,7 @@ The CLI follows the live SSE stream (`/logs/stream`) and passes raw SSE frames (
 
 > On the SSE path, `--wait` puts `result_summary` directly in the **complete frame** (if trailing JSON is detected), so there is no need to scan backwards with `sandbox read <id> --path <log_file>`. Read the log_file manually only if no valid JSON is found at the end. `--summary`/`--artifacts` enrichment (including hints) still takes effect only on the legacy-server fallback path.
 
-> **Output contract (SSE path):** On success, `run-bg --wait` writes only raw SSE frames to stdout, with **no trailing envelope**. On command failure, it appends an `{"ok":false,"error":{"type":"command_failed",...}}` envelope after the complete frame. If the command has not finished within `--timeout`, it appends a `finished=false` `CommandLogsResult` snapshot envelope. For an older server (without the `/logs/stream` endpoint), the CLI falls back to polling every 5s and outputs a `CommandResult` envelope (the `finished` key is always present; when `finished=true`, it includes `result_summary`/`hints` enrichment).
+> **Output contract (SSE path):** On success, `run-bg --wait` writes only raw SSE frames to stdout, with **no trailing envelope**. On command failure, it appends an `{"ok":false,"error":{"type":"command_failed",...}}` envelope after the complete frame and exits non-zero (process exit code 8). If the command has not finished within `--timeout`, the CLI exits **non-zero** with `{"ok":false,"error":{"type":"timeout",...}}`; `error.details` carries `sandbox_id`, `execution_id` and `remote_state` so the command can be queried or cancelled afterwards (`remote_state: not_cancelled` means only the local wait stopped and the remote command may still be running — pass `--cancel-on-timeout` to cancel it). For an older server (without the `/logs/stream` endpoint), the CLI falls back to polling every 5s and outputs a `CommandResult` envelope (the `finished` key is always present; when `finished=true`, it includes `result_summary`/`hints` enrichment). **For automation, add `--format json` (same as `--json-result`)**: it emits no frames, only one envelope whose `data` contains `execution_id`/`finished`/`exit_code`/`stdout_tail`/`stderr_tail`/`stdout_size`/`stderr_size`/`log_file`/`duration_seconds`/`result_summary`; on failure it is `{ok:false,error,data}` and the process exits non-zero. When those fields are missing from the `complete` frame, the CLI fills them from the accumulated stream, so the field semantics match `--format json` exactly.
 
 ### Method 2: poll manually (when you need to see live output)
 
@@ -50,8 +55,13 @@ delta-cli sandbox logs <id> --execution-id <exec_id>
 #   the response is then an **array** grouped by execution_id (each item contains execution_id/stdout/stderr/exit_code/command/finished/error/log_file),
 #   rather than a single snapshot object; the CLI also applies 800/200-byte tail trimming and --tail/--grep/--context filters to each item
 
-# 3. Destroy after completion
+# 3. Destroy after completion (returns request_id / operation_status; destruction continues server-side)
 delta-cli sandbox kill <id>
+
+# 4. Reconcile: after a client disconnect, a create error or a network error on kill, use the operation ledger to confirm the real outcome
+delta-cli sandbox operation <request_id>
+# or list the create/destroy records for a sandbox
+delta-cli sandbox operation --sandbox-id <id>
 ```
 
 ### How to interpret polling results
@@ -97,5 +107,5 @@ Interpret task status using these rules, in order:
 **Notes:**
 - Use `sandbox logs` only with `sandbox run-bg`; synchronous `sandbox run` returns its result directly, so `logs` cannot and should not be called for it.
 - Without `--wait`, the `sandbox run-bg` HTTP request itself returns quickly with `{execution_id, sandbox_id}`. If the command times out, it exits with a non-zero code; retrieve `exit_code` later with `logs`.
-- In `sandbox run-bg --wait` mode, `--timeout` serves **both** as the server-side command timeout and the client-side waiting deadline (default 300s). The CLI follows the live SSE stream (`/logs/stream`), passes raw `data:` frames through to stdout and ends when `complete` arrives (with no trailing envelope); on timeout, it appends a `finished=false` snapshot envelope. The initial `/run-background` POST has its own 60s HTTP timeout.
+- In `sandbox run-bg --wait` mode, `--timeout` serves **both** as the server-side command timeout and the client-side waiting deadline (default 300s). The CLI follows the live SSE stream (`/logs/stream`), passes raw `data:` frames through to stdout and ends when `complete` arrives (with no trailing envelope); on timeout it exits **non-zero** with `error.type: timeout` and keeps `sandbox_id`/`execution_id`/`remote_state` in `error.details` (add `--cancel-on-timeout` to also cancel the remote command). The initial `/run-background` POST has its own 60s HTTP timeout.
 - Background commands can be queried across tool-call turns: save `sandbox_id` and `execution_id`, then retrieve results in later turns with `sandbox status` / `sandbox logs`.

package/skills/delta-sandbox/references/recipes.md

@@ -129,6 +129,7 @@ Notes:
 ## Local `result.json` content contract
 
 - **Preferred SSE path (recommended, one step):** The **complete frame for `sandbox run` / `run-bg --wait` includes `result_summary` directly** (the CLI injects detected trailing JSON from stdout), so the skill can extract it from the complete frame without reading and parsing the log.
+- **Preferred path for automation:** add `--format json` (same as `--json-result`) and the CLI emits a single envelope; read `data.exit_code` / `data.result_summary` / `data.stdout_tail` directly, without parsing multiple frames or reassembling stdout (newlines are preserved). On failure `data` is still present (`{ok:false,error,data}`).
 - **SSE fallback path:** If the complete frame lacks `result_summary` (there is no valid JSON at the end of stdout), `delta-cli sandbox read <id> --path <log_file>` reads the run-envelope log file, then scans backwards through stdout to extract trailing JSON. The `log_file` path comes from the SSE `complete` frame (the CLI fills it in if needed) or the `sandbox logs` snapshot (after `finished=true`).
 - **Legacy-server fallback path:** When `run` / `run-bg --wait` falls back to polling, it returns a `CommandResult` envelope whose `data.result_summary` field can be reused directly.
 - The local `result.json` is a **concise summary** for host deliverable validation, not a complete log archive.

package/skills/delta-shared/SKILL.md

@@ -65,7 +65,7 @@ This returns the current user's information and remaining credits (through the g
 - **Do not install the CLI proactively**: if the user says it is not installed, direct them to the README or `install.sh`; do not choose an installation method yourself.
 - **Confirm intent before write or delete operations**: tell the user what commands such as `sandbox kill`, `sandbox finish` and `sandbox write` will affect.
 - `~/.delta-infra/config.json` must have `0600` permissions; do not loosen them manually.
-- **The sandbox must be destroyed**: after calling `sandbox create`, call `sandbox kill` before the current task ends. The server does not clean up automatically; failure to destroy a sandbox leaves GPU resources occupied indefinitely. Do not skip this, even if an error occurs.
+- **The sandbox must be destroyed explicitly**: after calling `sandbox create`, call `sandbox kill` before the current task ends (use `finish` when results must be preserved). Sandboxes do get reclaimed automatically (default `--max-life` is 30 minutes; OpenSandbox starts the clock when create is accepted and the platform reclaims, while AutoDL/RunPod start at readiness and the server reclaims at max-life plus up to 10 minutes of grace), but the platform may reclaim early and automatic reclamation does not write back to OSS — relying on it loses artifacts. Do not skip destruction, even if an error occurs.
 
 ## JSON output conventions
 
@@ -76,7 +76,7 @@ This returns the current user's information and remaining credits (through the g
 
 AI agent parsing rules:
 - Always check `ok`; do not rely on the exit code alone.
-- `error.type` is the stable field for branching on: `auth`, `permission`, `not_found`, `validation`, `network`, `api`, `command_failed`, `internal`.
+- `error.type` is the stable field for branching on: `auth`, `permission`, `not_found`, `validation`, `network`, `api`, `command_failed`, `timeout`, `unknown_execution_state`, `internal`.
 - `error.hint` may be shown to the user. **Act on it rather than retrying blindly**: for a failed `sandbox create` the hint names the concrete cause and the next step (for example, the image was not cached on the target node — retry, or pick a GPU type whose node already has the image).
 - `error.request_id` identifies the same failure in the server log; quote it when reporting the failure.
 
@@ -91,6 +91,8 @@ AI agent parsing rules:
 | network | 6 | DNS, timeout or connection failure |
 | api | 7 | Server returned a non-2xx response |
 | command_failed | 8 | Command failed inside the sandbox (non-zero `exit_code` or an error); on the SSE path, an error envelope is appended after the `complete` frame, with the stderr tail in `message` for quick reference |
+| timeout | 1 | A local wait deadline elapsed (`sandbox run-bg --wait`): the remote command may still be running — `error.details` keeps `sandbox_id`, `execution_id` and `remote_state` (`not_cancelled` / `cancelled` / `cancel_failed`) so it can be queried or cancelled |
+| unknown_execution_state | 1 | The command's outcome is **unknown** (the stream ended before the `complete` frame, or only an error frame arrived). Never treat this as success and never guess the exit code |
 | internal | 10 | Internal client error |
 
 ## Upgrade check
@@ -111,5 +113,7 @@ delta-cli upgrade
 | `error.type: network` | Check `base_url` and network connectivity |
 | `error.type: validation` | Check required flags and argument formats |
 | `error.type: command_failed` | Command failed inside the sandbox: the envelope's `message` already includes the stderr tail to help locate the issue quickly; the full `stderr` / `result_summary` are in the `complete` frame. Retry or correct the command |
+| `error.type: timeout` | A local wait deadline elapsed, **not** a command failure. Read `error.details.execution_id` and query it with `sandbox logs <id> --execution-id <exec_id>`; cancel the still-running command with `sandbox cancel <id> --execution-id <exec_id>` (or re-run with `--cancel-on-timeout` so the CLI cancels it for you) |
+| `error.type: unknown_execution_state` | The remote outcome is unknown — the CLI will not guess. Re-check the command with `sandbox logs <id> --execution-id <exec_id>` (the execution id is in the error details) before deciding whether it succeeded |
 | `sandbox create` failed (`error.type: api` with a specific `hint`) | Follow the hint — it names the cause and the next step. "the image may not be cached on the target node" → retry once (the pull continues in the background, so later creates succeed) or pick a GPU type whose node already has the image. "check the target node's free disk space" → a retry will not help until an operator frees the node. A failed create leaves no sandbox behind, so retrying is safe. Quote `error.request_id` when reporting it. |
 | Configuration file parsing failed | Check that `~/.delta-infra/config.json` contains valid JSON |