@delta-agentcloud/cli 1.1.24 → 1.1.28

package/README.md

@@ -120,12 +120,6 @@ Common environment variables:
 | `DELTA_CLI_FATAL_ON_ERROR` | Exit non-zero on install failure (default: warn only; useful for debugging). |
 | `DELTA_CLI_DOWNLOAD_TIMEOUT` | Max wait time per download source (ms, default 120000). |
 
-### Option 5: Go Install
-
-```bash
-go install github.com/delta-infra/delta-infra-cli/cmd/delta-cli@latest
-```
-
 ## Quick Start
 
 ### 1. Initialize configuration
@@ -178,6 +172,14 @@ You can also provide credentials via environment variables: `DELTA_INFRA_API_KEY
 
 ```bash
 # Create a sandbox with a GPU (--max-life defaults to 30 minutes; raise it for long tasks)
+# ⚠️ Lifecycle timing (see skills/delta-sandbox/references/lifecycle.md):
+#   - Private compute (OpenSandbox): the clock starts when the create request is accepted, and the
+#     image pull / readiness time counts towards max-life; the platform reclaims the sandbox at expiry.
+#   - Third-party compute (AutoDL/RunPod): the clock starts when the sandbox is ready (started_at),
+#     and the server-side cleanup reclaims it at max-life plus up to 10 minutes of grace.
+#   - Auto-reclaim (including an early platform reclaim) does NOT write back to OSS; pull the
+#     artifacts or call `sandbox finish` explicitly if you need to keep them. `sandbox status`
+#     returns expires_at / reclaim_source / termination_reason, and a finished sandbox stays queryable.
 # ⚠️ Resource semantics: `--memory` is the container CPU memory TOTAL (one per sandbox); `--gpu-mem` is the memory PER CARD (MiB) —
 # `--gpu 2 --gpu-mem 8000` = 8000MiB per card, 16000MiB total (the private OpenSandbox/HAMi backend writes it straight to the HAMi per-card annotation;
 # the cloud RunPod backend treats --gpu-mem as total memory ÷ card count). Per-card memory limit is visible via `sandbox resources` (memory_mib).
@@ -187,19 +189,20 @@ You can also provide credentials via environment variables: `DELTA_INFRA_API_KEY
 # `--devices` requests host devices (comma-separated, e.g. kvm → /dev/kvm inside the container, for nested virtualization): a privilege escalation,
 # rejected by the server when the allowlist is disabled (never silently ignored). Both apply to private compute only; third-party compute ignores them.
 delta-cli sandbox create \
-  --image image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
+  --image-name image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
   --cpu 4 --memory 16Gi --gpu 1 \
   --max-life 120
 
 # For DataLoader multiprocessing / nested virtualization: raise /dev/shm and request /dev/kvm (shm must not exceed memory)
 delta-cli sandbox create \
-  --image image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
+  --image-name image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
   --cpu 4 --memory 8Gi --shm-size 4Gi --devices kvm \
   --max-life 120
 
-# Sandbox that must live long and not be auto-cleaned: add --no-auto-cleanup (only explicit kill/finish destroys it)
+# Sandbox that must live long and not be auto-cleaned: add --no-auto-cleanup
+# (it disables both the platform TTL and the server-side cleanup; only explicit kill/finish destroys it)
 delta-cli sandbox create \
-  --image image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
+  --image-name image.yangtzeailab.com/opensandbox/pytorch-cuda13:latest \
   --cpu 4 --memory 16Gi --gpu 1 \
   --no-auto-cleanup
 
@@ -208,15 +211,29 @@ delta-cli sandbox run <sandbox_id> \
   --command "<command>" \
   --timeout 3600
 
-# v1.0.85+ sandbox run uses an SSE real-time stream: stdout carries raw data: frames (no trailing envelope); exit_code/log_file are in the complete frame
-# To get the trailing JSON summary in stdout: read the log_file and parse it (sandbox read <id> --path <log_file>)
+# v1.0.85+ sandbox run uses an SSE real-time stream: stdout carries raw data: frames (no trailing envelope);
+# exit_code / log_file / stdout_tail / duration_seconds are in the complete frame
 delta-cli sandbox run <sandbox_id> --command "echo '{\"status\":\"ok\",\"result\":\"done\"}'" --timeout 60
 
+# Automation/orchestration: --format json (same as --json-result) prints exactly ONE JSON envelope,
+# whose data holds execution_id / finished / exit_code / stdout_tail / stderr_tail /
+# stdout_size / stderr_size / log_file / duration_seconds / result_summary.
+# The field names match the complete frame; on failure it is still a single object:
+# {"ok":false,"error":{...},"data":{...}} and the process exits non-zero.
+delta-cli sandbox run <sandbox_id> --command "python run.py" --timeout 600 --format json
+
+# Background task + wait + unified result
+delta-cli sandbox run-bg <sandbox_id> --command "python train.py" --timeout 3600 --wait --format json
+
 # Read a result file
 delta-cli sandbox read <sandbox_id> --path /workspace/result.json
 
-# Destroy the sandbox
+# Destroy the sandbox (the destroy keeps running server-side; the response carries request_id)
 delta-cli sandbox kill <sandbox_id>
+
+# Idempotent create retries + reconciliation afterwards
+delta-cli sandbox create --image-name <image name> --idempotency-key retry-001
+delta-cli sandbox operation <request_id>
 ```
 
 ## Command list
@@ -253,19 +270,20 @@ delta-cli sandbox kill <sandbox_id>
 | `delta-cli sandbox resources [--provider opensandbox\|autodl]` | Show remaining requestable resources (GPU/memory/cores) per backend |
 | `delta-cli sandbox images` | Show the available image list (each entry has `image_name`/`tags`/`description`/`supports_shared_workspace`; `supports_shared_workspace=true` means the compute supports a shared workspace (opensandbox/autodl) and can pair with `--shared-workspace`; the actual seed/write-back is decided at create time by probing for rclone inside the image) |
 | `delta-cli sandbox recommend --cpu N --memory XGi` | Get a resource configuration recommendation |
-| `delta-cli sandbox create --image <img> [--shm-size XGi] [--devices kvm] [--no-auto-cleanup] [--shared-workspace=true\|false]` | Create a sandbox container (`--no-auto-cleanup` disables auto-cleanup; `--shared-workspace` defaults to true: enables the shared workspace (seed/write-back determined at create time by probing for rclone in the image; skipped when absent, data not persistent across sandboxes); false skips seed/write-back, suited to fast tasks; `--shm-size` raises /dev/shm (default 64Mi, **must carry a Gi/Mi unit** and be ≤ `--memory`, common for PyTorch DataLoader/multiprocessing); `--devices` is a comma-separated list of host device tokens (`kvm` → /dev/kvm, nested virtualization), **a privilege escalation** the server **explicitly rejects** when the allowlist is off; `--shm-size`/`--devices` apply to private compute only, third-party compute ignores them) |
-| `delta-cli sandbox extend <id> --max-life N` | **Dynamically extend** a running sandbox's max-life (increase or decrease; N must exceed elapsed minutes and be ≤10080; use `kill`/`finish` for early reclaim); without `--max-life` it queries the current lifecycle (elapsed/remaining/expiry); `status` output includes a `data.lifecycle` summary (elapsed/remaining/expiry; omitted when the lifecycle API is unavailable) |
+| `delta-cli sandbox create --image-name <image name> [--idempotency-key <key>] [--shm-size XGi] [--devices kvm] [--no-auto-cleanup] [--shared-workspace=true\|false]` | Create a sandbox container (`--image` is the legacy alias; **`--idempotency-key`**: retrying with the same key replays the original sandbox instead of creating a second one, and a different request under the same key is rejected; the response carries `request_id`/`operation_status`/`replayed`; `--no-auto-cleanup` disables both the platform TTL and the server-side cleanup, so only an explicit kill/finish destroys it; `--shared-workspace` defaults to true: enables the shared workspace (seed/write-back determined at create time by probing for rclone in the image; skipped when absent, data not persistent across sandboxes); false skips seed/write-back, suited to fast tasks; `--shm-size` raises /dev/shm (default 64Mi, **must carry a Gi/Mi unit** and be ≤ `--memory`, common for PyTorch DataLoader/multiprocessing); `--devices` is a comma-separated list of host device tokens (`kvm` → /dev/kvm, nested virtualization), **a privilege escalation** the server **explicitly rejects** when the allowlist is off; `--shm-size`/`--devices` apply to private compute only, third-party compute ignores them) |
+| `delta-cli sandbox extend <id> --max-life N` | **Dynamically extend** a running sandbox's max-life (N must be greater than the current expiry and ≤10080; shortening is not accepted — use `kill`/`finish` for an early reclaim); without `--max-life` it queries the current lifecycle (elapsed/remaining/expiry); `status` output includes a `data.lifecycle` summary (elapsed/remaining/expiry; omitted when the lifecycle API is unavailable) |
 | `delta-cli sandbox connect <id>` | Connect to a sandbox |
-| `delta-cli sandbox status <id>` | Show sandbox status |
-| `delta-cli sandbox finish <id>` | Save results and destroy |
-| `delta-cli sandbox kill <id>` | Destroy a sandbox |
-| `delta-cli sandbox run <id> --command "..." [--timeout <sec>] [--summary/--no-summary] [--artifacts]` | Run a command synchronously (raw SSE `data:` frames to stdout, no trailing envelope; `exit_code`/`log_file` in the `complete` frame); `--summary`/`--artifacts` only apply on the legacy-server fallback path |
-| `delta-cli sandbox run-bg <id> --command "..." [--timeout <sec>] [--wait] [--summary/--no-summary] [--artifacts]` | Run a command in the background; without `--wait` it returns `{execution_id, sandbox_id}` immediately; with `--wait` it follows via SSE (raw `data:` frames to stdout, no trailing envelope) |
+| `delta-cli sandbox status <id>` | Show sandbox status; returns `expires_at` / `reclaim_source`, and a finished sandbox stays queryable with `termination_reason` / `termination_remark` / `terminated_at` (e.g. `max_life_exceeded` / `platform_gone` / `user_killed` / `user_finished`) |
+| `delta-cli sandbox finish <id> [--no-wait]` | Save results and destroy; the destroy is a **server-side task** and the response carries `request_id`/`operation_status` (`--no-wait` returns `pending` immediately and the task continues after a disconnect — reconcile it with `sandbox operation <request_id>`) |
+| `delta-cli sandbox kill <id> [--no-wait]` | Destroy a sandbox (without saving results); the response carries `request_id`/`operation_status`, so after an interrupted call or a network error the real terminal state can be queried with `sandbox operation` |
+| `delta-cli sandbox operation [request_id] [--sandbox-id <id>] [--operation create\|destroy] [--limit N]` | Query the server-side operation ledger: with a `request_id` it fetches that single operation, otherwise it lists the recent create/destroy records (terminal state, time, reason) for billing and resource reconciliation |
+| `delta-cli sandbox run <id> --command "..." [--timeout <sec>] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | Run a command synchronously. By default (`--format sse`) the raw SSE `data:` frames are passed through to stdout with no trailing envelope, and `exit_code`/`log_file`/`stdout_tail`/`duration_seconds` are in the `complete` frame; `--format json` (or `--json-result`) prints exactly **one** JSON envelope whose `data` holds `execution_id`/`finished`/`exit_code`/`stdout_tail`/`stderr_tail`/`stdout_size`/`stderr_size`/`log_file`/`duration_seconds`/`result_summary`, and on failure it is `{ok:false,error,data}` with a non-zero process exit code; `--summary`/`--artifacts` only apply on the legacy-server fallback path. **Exit-code semantics**: a non-zero command exit → `error.type=command_failed`, process exit code **8**; the command finished but its **exit status is unknown** (the stream ended before the `complete` frame, or only an error frame arrived) → `error.type=unknown_execution_state`, process exit code 1 — the CLI **never synthesizes a success** |
+| `delta-cli sandbox run-bg <id> --command "..." [--timeout <sec>] [--wait] [--cancel-on-timeout] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | Run a command in the background; without `--wait` it returns `{execution_id, sandbox_id}` immediately (with `--format json`, the unified result object with `finished=false`); with `--wait` it follows via SSE (raw `data:` frames to stdout, no trailing envelope); `--format json` prints exactly one envelope with the same structure as `sandbox run --format json`. **On a wait timeout** → `error.type=timeout` and a **non-zero** process exit code, with `error.details` keeping `sandbox_id`/`execution_id`/`remote_state` (`not_cancelled`/`cancelled`/`cancel_failed`) so the command can be queried or cancelled afterwards; `remote_state=not_cancelled` means only the local wait stopped and **the remote command may still be running**. The remote command is left running by default — add `--cancel-on-timeout` to cancel it (a failed cancel is reported as `cancel_failed`) |
 | `delta-cli sandbox logs <id> --execution-id <eid> [--tail N --grep <pattern> --context N --max-bytes N]` | Show background logs; returns `stderr_size + stderr_tail` by default (avoids context blow-up); filter with `--tail/--grep` |
 | `delta-cli sandbox cancel <id> --execution-id <eid>` | Interrupt a background command |
-| `delta-cli sandbox read <id> --path <path> [--output <local path>] [--tail N] [--grep <pattern>] [--offset N] [--limit N] [--context N] [--max-bytes N] [--parse-json]` | Read a file in the container; `--output <path>` saves locally; filter with `--tail/--grep`; non-UTF-8 files fall back to base64 automatically |
-| `delta-cli sandbox pull <id> --source <sandbox path> --target <local path> [--recursive] [--pattern <glob>]` | Pull files/dirs from the sandbox to local (mirror of `upload`; flag direction reversed: source=remote, target=local); single file or recursive dir; bidirectional sha1 integrity check on both CLI and server |
-| `delta-cli sandbox write <id> --path <path> --source <local path>` | Write a file |
+| `delta-cli sandbox read <id> --path <path> [--output <local path>] [--tail N] [--grep <pattern>] [--offset N] [--limit N] [--context N] [--max-bytes N] [--parse-json]` | Read a file in the container; `--output <path>` saves locally; filter with `--tail/--grep`. **A full copy (without `--tail/--grep/--offset/--limit/--max-bytes`) is byte-exact**: it goes through the chunked download path (binary-safe, sha256-verified — the envelope reports `byte_exact:true`, `integrity:"sha256"`, `digest_verified`); with any range filter it falls back to the text channel, where the server decodes as UTF-8 and replaces invalid bytes with U+FFFD, so the envelope then reports `byte_exact:false` / `integrity:"text_decoded"` and a warning is printed to stderr — **use `sandbox pull` to copy a binary file**. When the result is printed inline and the file is not valid UTF-8, `content` is the lossy text and the **exact original bytes** are in `content_base64` (base64-decode it); the envelope is marked `byte_exact:true`. Text files carry no `content_base64` |
+| `delta-cli sandbox pull <id> --source <sandbox path> --target <local path> [--pattern <glob>] [--concurrency N] [--no-resume] [--no-verify]` | Pull files/dirs from the sandbox to local (mirror of `upload`; flag direction reversed: source=remote, target=local). **Directories are always pulled recursively** (`--recursive` is deprecated and no longer needed); the transfer uses the **chunked download** path: binary-safe, end-to-end **sha256** verified (`digest_verified`), **resumable** (a partial `<target>.delta.part` is reused), and **parallel** across files and chunks (`--concurrency`, default 4) — so a large file is not bound by a single command timeout. The server publishes its limits with every response (`limits`): 4 MiB chunks (16 MiB max), 100 GiB max per file, 120 s per chunk, 600 s for the whole-file digest. `--no-resume` ignores a leftover `.delta.part`; `--no-verify` skips the sandbox-side sha256 comparison |
+| `delta-cli sandbox write <id> --path <path> --source <local path>` | Write a file. **Binary-safe**: the content travels as base64 and the CLI declares its `size`/`sha256`, which the server verifies before writing (a mismatch is rejected); the response carries `server_sha256`/`sha256`/`size`/`size_verified`. A `--source` above 32 MiB goes through the streaming upload path instead of being read into memory |
 | `delta-cli sandbox write-multiple <id> --entry <remote path>=<local path>` | Write multiple files |
 | `delta-cli sandbox ls <id> --path <path>` | List a directory |
 | `delta-cli sandbox stat <id> --path <path>` | File metadata |
@@ -392,8 +410,8 @@ make install
 make PREFIX=$HOME/.local install
 
 # Windows source install (pick one)
-# A: install to GOPATH/bin with go install
-go install github.com/delta-infra/delta-infra-cli/cmd/delta-cli@latest
+# A: install to GOPATH/bin with go install, from the repository root
+go install ./cmd/delta-cli
 
 # B: run the PowerShell script from the repo root to build and add to the user PATH
 # .\install-from-source.ps1

package/checksums.txt

@@ -1,5 +1,5 @@
-ae8095f3a2c9dc94ee4167bae557a86f1907093eeba3d84cef61ab4f85a17e4f  delta-cli-darwin-amd64.tar.gz
-371e22d3b7b72586dd3d9d2886ad1c9ab73e1e620a8f62247b8d6a7f3257321f  delta-cli-darwin-arm64.tar.gz
-add5f9953d771fc13164aa4bee8dffc103f281213f704b4b5ef08118a80f3637  delta-cli-linux-amd64.tar.gz
-4f57edbbde69750ee8052a14fb56c09cc3bc5d24f02e5aca4ae08460c1ba69a0  delta-cli-linux-arm64.tar.gz
-fd6261b30d542ccf6bfa13722060a502b97697407efea22b7755632d998fe573  delta-cli-windows-amd64.zip
+0cec396238f1fd86d2fe90ff1415f06aa36b77c81ed4fa4aeea16d8b8c78b480  delta-cli-darwin-amd64.tar.gz
+7f0d9f2261fe8686aead83eeebf24f544b2b71526bed0edff5ddcae0b6f9ce09  delta-cli-darwin-arm64.tar.gz
+a60ee4ef16553e6f8427c25c91c1b4a7154bcfb646a59f88fce66c755fcc7fbf  delta-cli-linux-amd64.tar.gz
+46d709d1b91f56c4930c4601b3099a80834a622dbcedc42f2b1d3e4fddb555db  delta-cli-linux-arm64.tar.gz
+d6f5ec032f743ffa7c59bdf555e669259cbcdf64f3484b9182d854d0480909f6  delta-cli-windows-amd64.zip

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@delta-agentcloud/cli",
-  "version": "1.1.24",
+  "version": "1.1.28",
   "description": "Delta Agent Cloud CLI for sandbox compute and science tools.",
   "bin": {
     "delta-cli": "scripts/run.js"

package/skills/delta-protein/AGENTS.md

@@ -54,11 +54,11 @@ delta-protein/
   `## Output Template` sections in every reference from
   `scripts/output_metadata.json` (single source of truth for field projection /
   presentation form / downstream reuse / provenance marker). Stdlib only, idempotent.
-- Workflow: update `output_metadata.json` → run
+- Workflow: update `scripts/output_metadata.json` → run
   `python scripts/gen_references.py` → run `python scripts/gen_references.py --check`.
   `--check` fails loudly on missing sections, drifting titles, broken SKILL links or
   references not covered by metadata.
-- `output_metadata.json` `provenance` is `tested` (verified by a real `delta-cli`
+- `scripts/output_metadata.json` `provenance` is `tested` (verified by a real `delta-cli`
   invocation) or `derived` (from ToolSpec descriptions / agent templates, marked
   "pending live calibration"). Newly verified endpoints should be flipped to `tested`.
 - After changing references/SKILL, resync the opencode install dir:

package/skills/delta-sandbox/AGENTS.md

@@ -0,0 +1,37 @@
+# skills/delta-sandbox — Sandbox AI Agent Skill
+
+**Generated:** 2026-10-21
+**Parent:** `../AGENTS.md`
+
+## OVERVIEW
+
+The primary sandbox skill. It teaches an AI agent (Planner → executor) how to drive `delta-cli sandbox` end to end: create a sandbox, run commands in it, read and write files, tunnel ports, and collect artifacts. Ships as one skill directory in the npm tarball (`package.json` `files`) and is installed to every supported agent platform by `scripts/install-wizard.js`.
+
+## WHERE TO LOOK
+
+| Path | Content |
+|------|---------|
+| `SKILL.md` | The whole contract (401 lines): when to use, scope, sandbox environment, mandatory rules, request wording, required_outputs, quick routing, full lifecycle, tunnelling, structured-output convention, reading output, Planner guidance, recovery |
+| `references/commands.md` | Command cheat sheet (72 lines): discovery, lifecycle, command execution, file operations, tunnelling |
+| `references/lifecycle.md` | Full create-to-kill workflow (111 lines) incl. background tasks and polling patterns |
+| `references/recipes.md` | Standard task request patterns (139 lines): required_outputs templates, script output template, GPU/CUDA check, run-a-script, install-and-run, train-and-save-metrics, general data processing |
+| `agents/openai.yaml` | OpenAI-format agent descriptor |
+
+## CONVENTIONS
+
+- YAML frontmatter carries `name`, `description`, `metadata.requires.bins: ["delta-cli"]` and `metadata.cliHelp: "delta-cli sandbox --help"`
+- The `description` is what the Planner matches against — it encodes the output convention and the `required_outputs` requirement
+- `SKILL.md` opens with a **CRITICAL** instruction to read `../delta-shared/SKILL.md` first (auth, config, error handling)
+- "Mandatory rules" is a failure section: breaking any of them makes the task a failure
+- Request wording: describe commands with neutral verbs (run/execute); use create/write/save only when a file is actually written
+- When the Planner calls this skill it must declare `[{kind: 'file', extensions: ['.json']}]` in `required_outputs` for the plan step
+- Sandbox scripts/commands print structured JSON at the end of stdout; on the SSE path the CLI detects the trailing JSON and exposes it as `result_summary` in the complete frame
+- Cross-link to sibling skills and references with relative paths only
+
+## ANTI-PATTERNS
+
+- Do not embed realistic credentials, API keys, or tokens in the examples
+- Do not break the relative links between `SKILL.md`, `references/` and `../delta-shared/SKILL.md`
+- Do not reference internal Go package paths — use CLI command names
+- Do not document a command that `delta-cli sandbox` does not expose
+- Do not state that a sandbox has no TTL or auto-reclaim: the platform TTL is the primary expiry path, and the server also reclaims stale sandboxes — keep the wording consistent with `references/lifecycle.md` and the CLI help

package/skills/delta-sandbox/SKILL.md

@@ -35,7 +35,11 @@ metadata:
 1. **You must use delta-cli**: perform all sandbox operations with `delta-cli sandbox <subcommand>`. **Do not** call the HTTP API directly with Python (`urllib`/`httpx`/`requests`) or any SDK. delta-cli provides typed error output (`error.type`), the only reliable way for an AI agent to interpret errors.
 2. **You must destroy the sandbox**: after each successful `sandbox create`, you **must** run `sandbox kill <id>` or `sandbox finish <id>` to destroy it before the current task's final `step_finish`. Failing to destroy it counts as task failure.
 3. **Destroy it even after an error**: if a command inside the sandbox fails, times out or reports an error, you **must still** run `sandbox kill <id>`. An error is no reason to skip cleanup.
-4. **Do not rely on external cleanup**: the server has **no** automatic sandbox expiry or TTL mechanism. Without `kill`, the sandbox will occupy GPU resources indefinitely.
+4. **Do not rely on external cleanup**: sandboxes **do** have automatic reclamation, but it does not replace destroying them yourself.
+   - The default `--max-life` is 30 minutes. On private compute (OpenSandbox) the timer starts when the create request is accepted and the platform reclaims the sandbox; on third-party compute (AutoDL/RunPod) it starts when the sandbox is ready, and the server-side cleanup reclaims it at max-life plus up to 10 minutes of grace.
+   - The platform may still reclaim a sandbox early (cluster eviction, node failure, etc.; `termination_reason=platform_gone`).
+   - **Automatic reclamation does not write back to OSS**, so artifacts that were not pulled are lost.
+   - Without `kill`, the sandbox keeps occupying GPU resources until it expires, and the reclamation time cannot be used to preserve results. When the task ends you must still run `kill` / `finish` explicitly (only `finish` writes back to OSS).
 5. **Complete synchronous tasks in one command; background tasks may span turns**:
    - Synchronous mode (`sandbox run`): complete the full lifecycle from `create` to `kill` in the same execution. Do not retain a sandbox across tasks.
    - Background mode (`sandbox run-bg`): you may report `sandbox_id` and `execution_id` to the user during the current execution, then retrieve the result in a later turn with `sandbox status` and `sandbox logs`. You must still run `sandbox kill` when the task is complete.
@@ -49,6 +53,15 @@ metadata:
    ```
     The CLI follows output in real time over SSE (`/logs/stream`) and passes each raw SSE frame (`data: {...}`) through to stdout, as with `curl -N`. The JSON `type` field identifies each frame: `init` (with `execution_id`), `stdout`, `stderr`, `error` or `complete`. The command ends when `complete` arrives. **The CLI fills in fields in the complete frame**: the server supplies `exit_code`/`execution_id`/`error`; the CLI adds the accumulated streamed `stdout`/`stderr`, a recovered `log_file` as a fallback, and a trailing-JSON `result_summary` when it detects a valid JSON object in the last 50 lines of stdout. Use `exit_code` in the `complete` frame to determine success. **If the command exits non-zero or reports an error, the CLI appends an `error.type: command_failed` envelope and exits with a non-zero code (process exit code 8)**; the envelope's `message` includes the end of stderr, which makes it easier to use with `&&`, branch on `error.type` and scan the failure reason. The CLI also best-effort appends the stderr tail to the `message` of a `type:"error"` frame; it does not append an envelope on success. If no `complete` frame arrives within `--timeout` (300s by default, serving as both the server command timeout and the client wait deadline), the CLI appends a snapshot envelope with `finished=false` after the frames already passed through, indicating that the command is still running. After `finished`, full stdout is written to `log_file` inside the sandbox (default: `/workspace/{user_id}/sandbox_logs/delta-result-{sandbox_id}-{execution_id}.json`; use the server-returned `log_file` field as authoritative). To read the full output, use `sandbox read <id> --path <log_file>` (the trailing JSON summary is already in the complete frame, so this is usually unnecessary). **`--summary`/`--artifacts` enrichment, including hints, is available only on the older-server fallback path** (when the server lacks the `/logs/stream` endpoint, the CLI automatically falls back to polling every 5s and returns a `CommandResult` envelope). Use `execution_id` for later `sandbox logs` or `cancel` calls.
 
+    **For automation: `--format json` (same as `--json-result`)** — emits no `data:` frames, just **one** JSON envelope, so you can parse `data` directly instead of stitching stdout frames together:
+
+   ```bash
+   delta-cli sandbox run <id> --command "<command>" --timeout <seconds> --format json
+   delta-cli sandbox run-bg <id> --command "<command>" --timeout <seconds> --wait --format json
+   ```
+
+   The `data` fields have the same names and meaning as in the `complete` frame: `execution_id`, `finished`, `exit_code`, `stdout_tail`, `stderr_tail`, `stdout_size`, `stderr_size`, `log_file`, `duration_seconds` and `result_summary`. `stdout_tail`/`stderr_tail` are the last 2000 characters of the accumulated streamed output (newlines preserved), so there is no need to aggregate the `stdout` frames. When the command exits non-zero the CLI emits `{"ok":false,"error":{...},"data":{...}}`, where `data` is the same object (including `exit_code`), and the process exit code is 8. This mode shares field semantics with SSE mode, so you can switch between them at any time.
+
     **Live following / manual polling (when you need intermediate progress)**:
     `sandbox logs <id> --execution-id <exec_id> --stream` follows the task in real time (SSE), passing each raw `data:` frame through to stdout until the `complete` event, with no trailing envelope (`--stream` requires `--execution-id`). Without `--stream`, `sandbox logs <id> --execution-id <exec_id>` returns a snapshot envelope. You may omit `--execution-id`; if omitted, the server queries only by sandbox and automatically uses ES (Elasticsearch) because `sandbox_id` and `execution_id` were not both supplied (it returns an **array** aggregated by execution ID rather than a single snapshot object):
     - `finished=true` and `exit_code=0` → complete
@@ -88,14 +101,14 @@ metadata:
 
 13. **`result.json` is required before completion**
     - Before calling `step_finish`, confirm that:
-      1. `delta-cli sandbox run` / `run-bg --wait` has returned with `finished=true`;
+      1. `delta-cli sandbox run` / `run-bg --wait` has returned with `finished=true` **and `exit_code` = 0**. `finished=true` alone is not a success signal — a command that fails still finishes. A non-zero exit is reported as `error.type: command_failed` (process exit code 8), an **unknown** exit status as `error.type: unknown_execution_state` (exit code 1), and a wait timeout as `error.type: timeout` (non-zero exit, with `sandbox_id`/`execution_id` in `error.details`); in all three cases the task has **not** succeeded — report the failure instead of claiming a result;
       2. (By default) the `result_summary` in the SSE `complete` frame from `sandbox run` / `run-bg --wait` has been written to `result.json`; if it is missing (fallback), read `log_file` with `sandbox read` and parse it with Python.
       3. `sandbox kill <id>` has been run to destroy the sandbox;
       4. `final_response` consists of the single line `RESULT: <result_summary>`.
 
 14. **Avoid redundant checks and validation**
     - If `skill_request` or the user's request already specifies the image and resources, **do not** call `sandbox images` or `sandbox recommend` before `sandbox create`.
-    - After writing a file to the sandbox, it is enough to use `sandbox stat` (optionally) to confirm the size. **Do not** use `sandbox ls` plus `sandbox read` to read the file back to the host and verify its contents; that is an unnecessary round trip.
+    - After writing a file to the sandbox, it is enough to use `sandbox stat` (optionally) to confirm the size. **Do not** use `sandbox ls` plus `sandbox read` to read the file back to the host and verify its contents; that is an unnecessary round trip. (The `sandbox write` response already carries `server_sha256` and `size_verified`, so a local digest comparison is enough if the bytes matter.)
     - Do not repeatedly poll with `sandbox status` after a successful `sandbox create`; the sandbox is ready as soon as create returns.
 
 ## Request wording conventions
@@ -152,8 +165,8 @@ The reason is that, after running a command, this skill reads `log_file` from th
 | Finish a sandbox | `sandbox finish <id> [--results '{...}']` |
 | Destroy a sandbox | `sandbox kill <id>` |
 | **Command execution** | |
-| Run a command synchronously (≤60s) | `sandbox run <id> --command "..." [--timeout <seconds>] [--summary/--no-summary] [--artifacts]` |
-| Run a command in the background (>60s) | `sandbox run-bg <id> --command "..." [--timeout <seconds>] [--wait] [--summary/--no-summary] [--artifacts]` |
+| Run a command synchronously (≤60s) | `sandbox run <id> --command "..." [--timeout <seconds>] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` (`--format json` = a single unified result envelope, recommended for automation). **A non-zero exit is reported as `error.type: command_failed` with process exit code 8; an unknown exit status is reported as `error.type: unknown_execution_state` with exit code 1 — the CLI never reports an unknown outcome as success** |
+| Run a command in the background (>60s) | `sandbox run-bg <id> --command "..." [--timeout <seconds>] [--wait] [--cancel-on-timeout] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` (`--wait --format json` = a single unified result envelope). **If the wait times out the CLI exits non-zero with `error.type: timeout` and keeps `sandbox_id`/`execution_id`/`remote_state` in `error.details`; `remote_state: not_cancelled` means only the local wait stopped and the remote command may still be running — add `--cancel-on-timeout` to cancel it** |
 | View background logs | `sandbox logs <id> [--execution-id <exec_id>] [--tail N --grep <pattern> --context N] [--stream]` (`--execution-id` is optional; if omitted, query ES by sandbox only; only `--stream` requires it) |
 | Interrupt a background command | `sandbox cancel <id> --execution-id <exec_id>` |
 | **Tunnelling (port exposure)** | |
@@ -161,10 +174,10 @@ The reason is that, after running a command, this skill reads `log_file` from th
 | List active tunnels | `sandbox exposes <id>` — list the public tunnels currently active for this sandbox |
 | Reclaim a tunnel | `sandbox dispose <id> --port N` — explicitly reclaim a single tunnel (stop frpc and release the public port); killing the sandbox also reclaims all its tunnels |
 | **File operations** | |
-| Read a file | `sandbox read <id> --path <path> [--output <local-path> --tail N --grep <pattern> --offset N --limit N --context N --max-bytes N --parse-json]` |
-| Pull a file/directory | `sandbox pull <id> --source <sandbox-path> --target <local-path> [--recursive] [--pattern <glob>]` — pull a single file or recursively pull a directory, with sha1 verification (the reverse of upload: source=remote, target=local) |
+| Read a file | `sandbox read <id> --path <path> [--output <local-path> --tail N --grep <pattern> --offset N --limit N --context N --max-bytes N --parse-json]` — **a full copy is byte-exact** (chunked download, sha256-verified); with `--tail`/`--grep`/`--offset`/`--limit` it uses the text channel and the envelope reports `byte_exact:false` — use `sandbox pull` for binary files. Printed inline, a non-UTF-8 file puts the **exact bytes** in `content_base64` (`content` is the lossy text) |
+| Pull a file/directory | `sandbox pull <id> --source <sandbox-path> --target <local-path> [--pattern <glob>] [--concurrency N] [--no-resume] [--no-verify]` — pull a single file or **recursively** pull a directory (directories are always recursive), over the **chunked** path: binary-safe, sha256-verified, resumable and parallel (the reverse of upload: source=remote, target=local) |
 | View the working directory | `sandbox working-directory <id>` (returns `data.path`, such as `/workspace/{user_id}/{sandbox_id}`) |
-| Write a file | `sandbox write <id> --source <filename>` (without `--path`, writes to the working directory by default); **every sandbox-side path** — `--path`, the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, `run`/`run-bg --log-file` — is prefixed with the working directory when relative, while an absolute path is used as given |
+| Write a file | `sandbox write <id> --source <filename>` (without `--path`, writes to the working directory by default); **every sandbox-side path** — `--path`, the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, `run`/`run-bg --log-file` — is prefixed with the working directory when relative, while an absolute path is used as given. The content is **binary-safe** (base64) and the server verifies the declared `size`/`sha256` before writing, so the response's `server_sha256`/`size_verified` can be compared with the local file |
 | Write multiple files | `sandbox write-multiple <id> --entry <remote-path>=<local-path> [--entry ...]` |
 | Upload a directory | `sandbox upload <id> --source <local-directory> --target <sandbox-path>` |
 | List a directory | `sandbox ls <id> --path <path>` (defaults to `.`) |
@@ -191,7 +204,7 @@ The reason is that, after running a command, this skill reads `log_file` from th
     - **`--gpu-type <model>` (optional) selects a GPU model**: for example, `"RTX 4090"` or `H100`. This schedules the task on that GPU model. Use `sandbox resources` to see available models (`gpu_types[].gpu_name`). Specify a model only when the user explicitly requests one; otherwise, the system schedules automatically.
     - **`--shm-size <size>` (optional) increases the container's `/dev/shm`**: the default is only **64Mi**. Workloads such as PyTorch `DataLoader(num_workers>0)`, `torch.multiprocessing`/shared memory and Chromium-based engines (Playwright/Selenium) may fail with `No space left on device` / `Bus error` when shared memory is insufficient; increase it as needed (for example, `--shm-size 4Gi`). Hard constraints: **include Gi/Mi units** (`4Gi`/`512Mi`; the CLI normalises `4G`/`512M` to `4Gi`/`512Mi`, while the server rejects a bare number such as `4` with a clear error because its meaning differs from the server's); **do not exceed `--memory`** (`/dev/shm` is a memory filesystem counted against the container's memory cgroup; filling it can cause an OOMKill). Since `--memory` defaults to only `1Gi`, increasing shm usually means increasing `--memory` too (for example, `--memory 8Gi --shm-size 4Gi`). This parameter is applied only on private compute (third-party compute ignores it); however, unit and limit checks happen before creation regardless of backend, so invalid values are rejected even when using third-party compute. The create response echoes the effective `resource.shmSize` for verification.
     - **`--devices <token>` (optional) requests host-device passthrough**: a comma-separated token list (currently `kvm` is supported, which adds `/dev/kvm` inside the container), used for **nested virtualisation** workloads such as QEMU/KVM, Android emulators and Kata inside the sandbox. **This is a privilege escalation, not an ordinary resource request**: device passthrough exposes host devices to the container, so the server uses an explicit allowlist. If operations staff have not enabled it, creation is **explicitly rejected (not silently ignored)**. Do not retry or change the syntax after this error; use an approach that does not require the device. This parameter is applied only on private compute (third-party compute ignores it), but the allowlist check happens before creation regardless of backend.
-    - `--no-auto-cleanup`: when this flag is set, the sandbox **will not be cleaned up automatically** (both server-side timeout reclamation and the local periodic `cleanup_stale` process are skipped). It can be destroyed only by an explicit `sandbox kill`/`finish`. Use this only when the task genuinely needs to survive past periodic cleanup; you must destroy it explicitly when the task is done to avoid leaking resources.
+    - `--no-auto-cleanup`: when this flag is set, the sandbox **will not be cleaned up automatically** (OpenSandbox gets `timeout=None`, so platform TTL and the server's periodic cleanup are both skipped). It can be destroyed only by an explicit `sandbox kill`/`finish`, and `status` reports `expires_at`/`remaining` as null. Use this only when the task genuinely needs to survive past periodic cleanup; you must destroy it explicitly when the task is done to avoid leaking resources.
     - **Do not repeatedly poll with `sandbox status` after a successful create**. The sandbox is ready as soon as `sandbox create` returns; use its returned `data.sandbox_id` directly for `write`/`run`. Extra polling adds tool calls without providing any benefit.
 3. **Write code/data**:
    - **Single file**: `delta-cli sandbox write <id> --source <filename>` — omit `--path` to write to the sandbox's working directory by default (`/workspace/{user_id}/{sandbox_id}/`; query it with `delta-cli sandbox working-directory <id>`). Pass only a local filename to `--source` (for example, `--source train.py`). **Do not** use shell-variable paths such as `"$WORKSPACE_ROOT/train.py"` or `$(pwd)/train.py`. With `--source`, the CLI reads the local file itself without shell expansion, making this the safest option. If you need a specific path, a relative `--path <relative-path>` is automatically prefixed with the working directory; absolute paths are used as given (the same rule applies to every sandbox-side path: the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, and `run`/`run-bg --log-file`) (**avoid** paths such as `/workspace/<filename>` outside `{user_id}`, as they do not persist after `kill`). The returned `size` field is the actual number of bytes on disk (verified with stat), so you can check that the write is complete. For small configuration files (<20 lines), you may use `--data "..."`, but you must also pass `--path`. File paths and extensions depend on the runtime in the image.
@@ -201,7 +214,7 @@ The reason is that, after running a command, this skill reads `log_file` from th
      - Nested directories are supported, and empty directories are created too.
      - **Optional post-write check**: use `sandbox stat <id> --path <path>` to confirm that the file exists and its size is as expected. **Do not** use `sandbox ls` plus `sandbox read` to read the just-written file back to the host and compare it character by character; if there is no issue, no read-back is needed.
 4. **Run the command**:
-    - **Short task (expected to take ≤ 60 seconds)**: run synchronously with `delta-cli sandbox run <id> --command "<command>" --timeout <seconds>`. The command's stdout/stderr is passed to stdout over SSE as **raw `data:` frames, one at a time** (with no trailing envelope). Read `exit_code` / `log_file` / `execution_id` from the `complete` frame. After completion, the full `stdout` is written to the run-envelope log file (the `log_file` field); use `sandbox read` to read it in full or extract trailing JSON. **Do not** then call `sandbox logs`. You can customise the run-envelope log path with `--log-file <path>`; the default is `/workspace/{user_id}/sandbox_logs/delta-result-{sandbox_id}-{execution_id}.json` (use the server-returned `log_file` field as authoritative). A script written with `sandbox write` is in the working directory; run it using its full path, `<working-directory>/<filename>` (first query `sandbox working-directory <id>`). Build the command for the runtime in the image. Common examples (where `<wd>` is the working directory):
+    - **Short task (expected to take ≤ 60 seconds)**: run synchronously with `delta-cli sandbox run <id> --command "<command>" --timeout <seconds>`. The command's stdout/stderr is passed to stdout over SSE as **raw `data:` frames, one at a time** (with no trailing envelope). Read `exit_code` / `log_file` / `execution_id` / `stdout_tail` / `stderr_tail` / `duration_seconds` from the `complete` frame. For automation, add `--format json` (or `--json-result`) so the CLI emits a single unified result envelope instead of making you stitch frames together yourself. After completion, the full `stdout` is written to the run-envelope log file (the `log_file` field); use `sandbox read` to read it in full or extract trailing JSON. **Do not** then call `sandbox logs`. You can customise the run-envelope log path with `--log-file <path>`; the default is `/workspace/{user_id}/sandbox_logs/delta-result-{sandbox_id}-{execution_id}.json` (use the server-returned `log_file` field as authoritative). A script written with `sandbox write` is in the working directory; run it using its full path, `<working-directory>/<filename>` (first query `sandbox working-directory <id>`). Build the command for the runtime in the image. Common examples (where `<wd>` is the working directory):
      - Python: `python <wd>/train.py`
      - Node.js: `node <wd>/app.js`
      - Go: `go run <wd>/main.go`

package/skills/delta-sandbox/references/commands.md

@@ -17,19 +17,20 @@
 
 | Command | Description |
 |------|------|
-| `sandbox create --image-name <image-name> [--cpu N --memory XGi --gpu N --gpu-mem N --gpu-type <model> --shm-size XGi --devices kvm --max-life M --no-auto-cleanup]` | Create a sandbox container (`--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; the underlying image identifier is hidden from users. `--gpu-type` selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish); `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, `1G`/`512M`; lowercase is also accepted), which the CLI converts to the `Gi`/`Mi` units accepted by the server; `--shm-size` increases the container's `/dev/shm` (the **default is only 64Mi**; it must be increased if PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error` because shared memory is insufficient; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number); it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only `1Gi`, so increasing shm usually requires increasing memory too); validation is backend-independent and rejects invalid values before creation, but this setting is applied only on private compute (third-party compute ignores it); `--devices` requests host-device passthrough using comma-separated tokens (for example, `kvm` makes `/dev/kvm` available inside the container for nested virtualisation; this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** rather than silently ignored; this setting is applied only on private compute); **resource limits per sandbox**: `--cpu` ≤ 512, `--memory` ≤ 1024Gi, `--gpu` ≤ 64, `--gpu-mem` ≤ 1024Gi, and `--shm-size` ≤ `--memory`. Exceeding a limit returns a **validation error promptly at request time** (it does not hang); normal tasks are well below these limits. The response echoes the requested image name/resource (request values fill fields not returned by the server; server values take precedence); compute backend is an internal detail and is selected automatically, so `--provider` is no longer supported. If creation fails, `error.hint` states the cause and the next step (for example, the image was not cached on the target node: retry, or pick a GPU type whose node already has the image), and `error.request_id` identifies the failure in the server log. |
+| `sandbox create --image-name <image-name> [--idempotency-key <key>] [--cpu N --memory XGi --gpu N --gpu-mem N --gpu-type <model> --shm-size XGi --devices kvm --max-life M --no-auto-cleanup]` | Create a sandbox container (`--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; the underlying image identifier is hidden from users. **`--idempotency-key`**: a retry with the same key replays the original sandbox instead of creating a duplicate, and the same key with different parameters is rejected; the response includes `request_id`/`operation_status`/`replayed`. `--gpu-type` selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish); `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, `1G`/`512M`; lowercase is also accepted), which the CLI converts to the `Gi`/`Mi` units accepted by the server; `--shm-size` increases the container's `/dev/shm` (the **default is only 64Mi**; it must be increased if PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error` because shared memory is insufficient; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number); it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only `1Gi`, so increasing shm usually requires increasing memory too); validation is backend-independent and rejects invalid values before creation, but this setting is applied only on private compute (third-party compute ignores it); `--devices` requests host-device passthrough using comma-separated tokens (for example, `kvm` makes `/dev/kvm` available inside the container for nested virtualisation; this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** rather than silently ignored; this setting is applied only on private compute); **resource limits per sandbox**: `--cpu` ≤ 512, `--memory` ≤ 1024Gi, `--gpu` ≤ 64, `--gpu-mem` ≤ 1024Gi, and `--shm-size` ≤ `--memory`. Exceeding a limit returns a **validation error promptly at request time** (it does not hang); normal tasks are well below these limits. The response echoes the requested image name/resource (request values fill fields not returned by the server; server values take precedence); compute backend is an internal detail and is selected automatically, so `--provider` is no longer supported. If creation fails, `error.hint` states the cause and the next step (for example, the image was not cached on the target node: retry, or pick a GPU type whose node already has the image), and `error.request_id` identifies the failure in the server log. |
 | `sandbox connect <id>` | Connect to an existing sandbox. |
 | `sandbox status <id>` | Check sandbox status (running / done). |
-| `sandbox finish <id> [--results '{...}']` | Save results and destroy the sandbox automatically. |
-| `sandbox kill <id>` | Destroy the sandbox immediately without saving results. |
+| `sandbox finish <id> [--results '{...}'] [--no-wait]` | Save results and destroy the sandbox automatically. Destruction is a server-side operation: the response includes `request_id`/`operation_status`; `--no-wait` returns immediately with `pending`. |
+| `sandbox kill <id> [--no-wait]` | Destroy the sandbox immediately without saving results; the response includes `request_id`/`operation_status`. |
+| `sandbox operation [request_id] [--sandbox-id <id>] [--operation create\|destroy] [--limit N]` | Query the server-side operation ledger (terminal state, time and reason). Use it to confirm the real outcome of a create or destroy after a client disconnect, timeout or network error. |
 | `sandbox extend <id> [--max-life N]` | Query or adjust the max lifetime of a running sandbox (N in minutes; omit `--max-life` to query the current lifecycle). `status` also reports the lifecycle summary under `data.lifecycle`. |
 
 ## Command execution
 
 | Command | Description |
 |------|------|
-| `sandbox run <id> --command "..." [--timeout N] [--summary/--no-summary] [--artifacts]` | Run a command synchronously (live SSE stream; raw `data:` frames are passed through to stdout one by one; **there is no trailing envelope on success**; the CLI fills in `exit_code`/`execution_id`/`log_file`/`stdout`/`stderr`/`result_summary` in the `complete` frame; **if the command exits non-zero or errors, an `error.type: command_failed` envelope is appended and the process exits non-zero**; the envelope's `message` includes the stderr tail, as does the message in a `type:"error"` frame on a best-effort basis). `--summary`/`--artifacts` take effect only on the legacy-server fallback path (when there is no SSE endpoint and a `CommandResult` envelope is returned). |
-| `sandbox run-bg <id> --command "..." [--timeout N] [--wait] [--summary/--no-summary] [--artifacts]` | Run a command in the background. Without `--wait`, return `{execution_id, sandbox_id}` immediately. With `--wait`, follow the live SSE stream until `complete` (raw `data:` frames are passed through to stdout; the CLI fills in `log_file`/`stdout`/`stderr`/`result_summary` in the `complete` frame; **on command failure, a `command_failed` envelope is appended and the process exits non-zero**; the envelope's `message` includes the stderr tail; on timeout, a `finished=false` snapshot envelope is appended). `--summary`/`--artifacts` take effect only on the legacy-server fallback path. |
+| `sandbox run <id> --command "..." [--timeout N] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | Run a command synchronously (live SSE stream; raw `data:` frames are passed through to stdout one by one; **there is no trailing envelope on success**; the CLI fills in `exit_code`/`execution_id`/`log_file`/`stdout`/`stderr`/`stdout_tail`/`stderr_tail`/`stdout_size`/`stderr_size`/`duration_seconds`/`result_summary` in the `complete` frame; **if the command exits non-zero or errors, an `error.type: command_failed` envelope is appended and the process exits non-zero (exit code 8)**; if the command finished but its **exit status is unknown** (the stream ended before the `complete` frame, or only an error frame arrived), the envelope is `error.type: unknown_execution_state` with exit code 1 — the CLI **never synthesizes a success** in that case; the envelope's `message` includes the stderr tail, as does the message in a `type:"error"` frame on a best-effort basis). `--format json` (same as `--json-result`) emits no frames, only one envelope whose `data` is the unified result object above; on failure it is `{ok:false,error,data}`. `--summary`/`--artifacts` take effect only on the legacy-server fallback path (when there is no SSE endpoint and a `CommandResult` envelope is returned). |
+| `sandbox run-bg <id> --command "..." [--timeout N] [--wait] [--cancel-on-timeout] [--format sse\|json] [--json-result] [--summary/--no-summary] [--artifacts]` | Run a command in the background. Without `--wait`, return `{execution_id, sandbox_id}` immediately (with `--format json`, the unified result object with `finished=false`). With `--wait`, follow the live SSE stream until `complete` (raw `data:` frames are passed through to stdout; the CLI fills in `log_file`/`stdout`/`stderr`/`result_summary` in the `complete` frame; **on command failure, a `command_failed` envelope is appended and the process exits non-zero (exit code 8)**; the envelope's `message` includes the stderr tail). **When the wait times out, the CLI exits non-zero with `error.type: timeout`** and `error.details` carrying `sandbox_id`, `execution_id` and `remote_state` (`not_cancelled` / `cancelled` / `cancel_failed`) so the command can be queried or cancelled afterwards; `remote_state: not_cancelled` means only the local wait stopped and **the remote command may still be running**. The remote command is left running by default — pass `--cancel-on-timeout` to cancel it (a failed cancel is reported as `cancel_failed`). With `--format json`, emit only one envelope with the same structure as `sandbox run --format json`. `--summary`/`--artifacts` take effect only on the legacy-server fallback path. |
 | `sandbox logs <id> --execution-id <eid> [--tail N --grep <pattern> --context N] [--stream]` | Retrieve background-command logs. **By default, returns `stdout_tail` (last 800 bytes) + `stderr_tail` (last 200 bytes) + `stderr_size` + `cursor`**, about 1 KB in total, to keep context small. `--stream` follows the live `/logs/stream` SSE endpoint, passing raw `data:` frames through to stdout until `complete` (the CLI fills in `stdout`/`stderr`/`log_file`/`result_summary` in the complete frame; there is no trailing envelope). `--tail N` applies to both stdout and stderr (last N lines from each); `--grep <pattern>` filters both stdout and stderr (regular expression); `--context N` includes N lines before and after each grep match (for both stdout and stderr). The `cursor` field has no `omitempty` and is always present, even when 0. Snapshots include `last_updated` (filled by the CLI on the SSE path; the server snapshot path is awaiting a server-side update). **If `--execution-id` is omitted, query by sandbox** (the server uses ES); the response is an **array** grouped by execution_id (each item contains `execution_id`/`stdout`/`stderr`/`exit_code`/`command`/`finished`/`error`/`log_file`), and the CLI applies the same tail/grep trimming to each item. |
 | `sandbox cancel <id> --execution-id <eid>` | Interrupt a background command that is running. |
 
@@ -37,12 +38,12 @@
 
 | Command | Description |
 |------|------|
-| `sandbox read <id> --path <path> [--output <local-path>] [--tail N] [--grep <pattern>] [--offset N] [--limit N] [--context N] [--max-bytes N] [--parse-json]` | Read a file; returns `content` + `size` + `content_length`. `--output <path>` saves it locally; `--tail`/`--grep` filter the output. For non-UTF-8 files, the CLI automatically falls back to base64. |
+| `sandbox read <id> --path <path> [--output <local-path>] [--tail N] [--grep <pattern>] [--offset N] [--limit N] [--context N] [--max-bytes N] [--parse-json]` | Read a file; returns `content` + `size` + `content_length`. `--output <path>` saves it locally; `--tail`/`--grep` filter the output. **A full copy (no `--tail`/`--grep`/`--offset`/`--limit`/`--max-bytes`) is byte-exact**: it goes through the chunked download path (binary-safe, sha256-verified — the envelope reports `byte_exact:true`, `integrity:"sha256"`, `digest_verified`). With any range filter it falls back to the text channel, where the server decodes as UTF-8 and replaces invalid bytes with U+FFFD; the envelope then reports `byte_exact:false` / `integrity:"text_decoded"` and a warning is printed to stderr. **Use `sandbox pull` to copy a binary file.** When the result is printed inline and the file is not valid UTF-8, `content` is the lossy text and the **exact original bytes** are in `content_base64` (base64-decode it); the envelope is marked `byte_exact:true`. Text files carry no `content_base64`. |
 | `sandbox working-directory <id>` | Get the sandbox's current working directory (absolute path `/workspace/{user_id}/{sandbox_id}`). |
-| `sandbox write <id> [--path <path>] --source <filename>` | Write a file (recommended). If `--path` is omitted, write to the working directory (`<working-directory>/<filename>`); a relative `--path` is also resolved under the working directory. An absolute `--path` is used as-is (but the `/workspace/<filename>` root path is outside the OSS sync scope and is not recommended). |
+| `sandbox write <id> [--path <path>] --source <filename>` | Write a file (recommended). If `--path` is omitted, write to the working directory (`<working-directory>/<filename>`); a relative `--path` is also resolved under the working directory. An absolute `--path` is used as-is (but the `/workspace/<filename>` root path is outside the OSS sync scope and is not recommended). **Binary-safe**: the content travels as base64 and the CLI declares its `size`/`sha256`, which the server verifies before writing (a mismatch is rejected). The response carries `server_sha256`/`sha256`/`size`/`size_verified`. |
 | `sandbox write <id> --path <absolute-path> --data "..."` | Write a small amount of inline content (`--data` has no source filename, so `--path` must be specified explicitly). |
 | `sandbox write-multiple <id> --entry <remote-path>=<local-path> [--entry ...]` | Write multiple files in one operation (a relative target `path` is also resolved under the working directory). |
-| `sandbox pull <id> --source <sandbox-path> --target <local-path> [--recursive] [--pattern <glob>]` | Pull files/directories to the local machine (the reverse of `upload`; flag directions are reversed: `--source`=remote sandbox path, `--target`=local path). Single files are detected automatically; directories are traversed recursively by default; SHA-1 integrity is checked by both the CLI and server. |
+| `sandbox pull <id> --source <sandbox-path> --target <local-path> [--pattern <glob>] [--concurrency N] [--no-resume] [--no-verify]` | Pull files/directories to the local machine (the reverse of `upload`; flag directions are reversed: `--source`=remote sandbox path, `--target`=local path). Single files are detected automatically; **directories are always traversed recursively** (`--recursive` is deprecated and no longer needed). The transfer uses the **chunked download** path: binary-safe, end-to-end **sha256** verified (`digest_verified`), **resumable** (a partial `<target>.delta.part` is reused), and **parallel** across files and chunks (`--concurrency`, default 4) — so a large file is not bound by a single command timeout. The server publishes its limits with every response (`limits`): 4 MiB chunks (16 MiB max), 100 GiB max per file, 120 s per chunk, 600 s for the whole-file digest. `--no-resume` ignores a leftover `.delta.part`; `--no-verify` skips the sandbox-side sha256 comparison. |
 | `sandbox upload <id> --source <local-directory> --target <sandbox-path>` | Upload a local directory (packaged as tar.gz, extracted server-side) and verify the resulting file list against the local manifest. `--source` is local, `--target` is inside the sandbox. |
 | `sandbox ls <id> --path <path>` | List directory contents (defaults to `.`). |
 | `sandbox stat <id> --path <path>` | Get file metadata (size / mode / owner / group). |

package/skills/delta-sandbox/references/lifecycle.md

@@ -6,16 +6,21 @@
 
 1. **Check available images** `delta-cli sandbox images` — list the images supported by the server. The image determines which language runtimes are available in the container (Python / Node.js / Go / Java / Rust, etc.); choose one that fits the user's technology stack and GPU/CPU requirements.
 2. **List existing sandboxes (optional)** `delta-cli sandbox list` — check the active sandboxes already created by the current user to avoid creating duplicates.
-3. **Create** `delta-cli sandbox create --image-name <image-name> --cpu 4 --memory 16Gi --gpu 1 --gpu-mem 8000 --max-life 120 [--gpu-type <model>] [--shm-size XGi] [--devices kvm]` (the sandbox is ready to use immediately; no connection step is needed. `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, 1G/512M; lowercase is also accepted), which the CLI converts to Gi/Mi. `--max-life` sets the sandbox's maximum lifetime in minutes; the default is 30. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish. `--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; compute backend selection is automatic and does not need to be specified. `--gpu-type` (optional) selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--shm-size` (optional) increases the container's `/dev/shm` (the **default is only 64Mi**)—it must be increased when PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error`; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number), and it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only 1Gi, so increasing shm usually requires increasing memory too, for example, `--memory 8Gi --shm-size 4Gi`). Only private compute (`private`) applies this setting (third-party compute ignores it), but unit/limit validation is backend-independent and rejects invalid values before creation. `--devices` (optional) requests host-device passthrough using comma-separated tokens (for example, `--devices kvm` makes `/dev/kvm` available inside the container for nested virtualisation); this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** (not silently ignored; validation is backend-independent). If this error occurs, do not retry; use an approach that does not depend on that device. Only private compute applies this setting. **Resource limits per sandbox**: cpu ≤ 512, memory ≤ 1024Gi, gpu ≤ 64, gpuMem ≤ 1024Gi, shmSize ≤ memory. Exceeding a limit returns a validation error promptly at request time rather than hanging.)
+3. **Create** `delta-cli sandbox create --image-name <image-name> --cpu 4 --memory 16Gi --gpu 1 --gpu-mem 8000 --max-life 120 [--gpu-type <model>] [--shm-size XGi] [--devices kvm] [--idempotency-key <key>]` (the sandbox is ready to use immediately; no connection step is needed. `--idempotency-key` (optional) makes network retries safe: the same key replays the original sandbox instead of creating a duplicate, and the response includes `request_id`/`operation_status`/`replayed`; `--memory`/`--gpu-mem` accept g/m/G/M formats (for example, 1G/512M; lowercase is also accepted), which the CLI converts to Gi/Mi. `--max-life` sets the sandbox's maximum lifetime in minutes; the default is 30. `--no-auto-cleanup` prevents automatic cleanup; the sandbox can then be destroyed only with an explicit kill/finish. `--image-name` takes an image name, such as `"PyTorch CUDA13 (GPU)"`; compute backend selection is automatic and does not need to be specified. `--gpu-type` (optional) selects a GPU model, such as `"RTX 4090"` or `H100`; the server maps this to the HAMi use-gputype annotation for model-based scheduling. To see available models, check `sandbox resources` (the `third_party` group's `gpu_types[].gpu_name`); if omitted, scheduling is automatic. `--shm-size` (optional) increases the container's `/dev/shm` (the **default is only 64Mi**)—it must be increased when PyTorch `DataLoader(num_workers>0)`/`multiprocessing`, Chromium-based engines or similar report `No space left on device`/`Bus error`; **a Gi/Mi unit is required** (`4G`/`512M` are normalised by the CLI; the server rejects a bare number), and it **must not exceed `--memory`** (it is a memory filesystem and filling it causes OOMKill; `--memory` defaults to only 1Gi, so increasing shm usually requires increasing memory too, for example, `--memory 8Gi --shm-size 4Gi`). Only private compute (`private`) applies this setting (third-party compute ignores it), but unit/limit validation is backend-independent and rejects invalid values before creation. `--devices` (optional) requests host-device passthrough using comma-separated tokens (for example, `--devices kvm` makes `/dev/kvm` available inside the container for nested virtualisation); this raises privileges, and if the server allowlist is not enabled, creation is **explicitly rejected** (not silently ignored; validation is backend-independent). If this error occurs, do not retry; use an approach that does not depend on that device. Only private compute applies this setting. **Resource limits per sandbox**: cpu ≤ 512, memory ≤ 1024Gi, gpu ≤ 64, gpuMem ≤ 1024Gi, shmSize ≤ memory. Exceeding a limit returns a validation error promptly at request time rather than hanging.)
    - **If `create` fails**: the error envelope's `hint` states the cause and the next step. A common one is "the image may not be cached on the target node" — the initial pull exceeded the readiness timeout, so retry once (the pull continues in the background and later creates succeed) or pick a GPU type whose node already has the image. A failed create leaves no sandbox behind, so retrying is safe; `error.request_id` identifies the failure in the server log.
 4. **Write code/data** `delta-cli sandbox write <id> --source <filename>` — write a local file directly to the sandbox. If `--path` is omitted, the file is written to that sandbox's working directory by default (`/workspace/{user_id}/{sandbox_id}/<filename>`, which is within the OSS sync scope, so the file persists after kill). Before writing, use `delta-cli sandbox working-directory <id>` to confirm the path. **You must** use a relative source path (for example, `--source infer.py`); do not use `"$WORKSPACE_ROOT/<filename>"` or `$(pwd)/...`. A relative `--path` is also resolved under the working directory; an absolute `--path` is used as-is (the same rule applies to every sandbox-side path: the remote part of `write-multiple --entry`, `upload --target`, `pull --source`, and `run`/`run-bg --log-file`) (`/workspace/<filename>` is outside the OSS sync scope and is not recommended). You can also use `--data "..."` to write a small amount of inline content (requires an explicit `--path`), and `--mode 755` to set file permissions. The returned `size` field is the actual on-disk byte count verified with stat, so it can confirm that the write completed. Use `write-multiple` to write several files at once. File paths and extensions depend on the runtimes in the image.
 5. **Run a command** — first use `delta-cli sandbox working-directory <id>` to get `<wd>`, then construct a command for a runtime in the image. Common examples: `python <wd>/train.py` (Python), `node <wd>/app.js` (Node.js), `go run <wd>/main.go` (Go), `bash <wd>/run.sh` (Shell).
    - **Short tasks ≤60s**: `delta-cli sandbox run <id> --command "<command>" --timeout <seconds>` runs synchronously; raw SSE `data:` frames are passed through to stdout (with no trailing envelope), `exit_code`/`log_file` are read from the `complete` frame, and full stdout is written to the log_file.
    - **Long tasks >60s**: run `delta-cli sandbox run-bg <id> --command "<command>" --timeout <seconds> --wait` in the background (recommended), or use `run-bg` + `logs` for manual polling (`run-bg --wait` includes `--summary` by default; without `--wait`, it returns `{execution_id, sandbox_id}` immediately).
 6. **Read the results** `delta-cli sandbox read <id> --path <wd>/result.json` (`<wd>` is the working directory; use `delta-cli sandbox working-directory <id>` to find it) — returns `content` (contents), `size` (disk bytes, from stat) and `content_length` (character count). Reading a non-existent file returns an error rather than empty content.
-7. **Destroy the sandbox** `delta-cli sandbox kill <id>` (if you need to save results, use finish; finish destroys the sandbox automatically).
+7. **Destroy the sandbox** `delta-cli sandbox kill <id>` (if you need to save results, use finish; finish destroys the sandbox automatically). Both are **server-side operations**: the response includes `request_id`/`operation_status`, they continue after the client disconnects, and `delta-cli sandbox operation <request_id>` reconciles the real outcome.
 
-> **Note:** The server has no automatic sandbox expiry or TTL mechanism. If you do not kill a sandbox, it will occupy GPU resources indefinitely. Always destroy it when the task is complete.
+> **Note:** sandboxes **do** get reclaimed automatically, but when and from which starting point depends on the backend:
+> - Private compute (OpenSandbox): `--max-life` (default 30 minutes) starts when the create request is accepted and the platform reclaims the sandbox at expiry; image pull / readiness time counts towards it too.
+> - Third-party compute (AutoDL/RunPod): the clock starts when the sandbox is ready, and the server-side cleanup reclaims it at max-life plus up to 10 minutes of grace.
+> - The platform may still reclaim a sandbox early (cluster eviction, node failure; `termination_reason=platform_gone`), and automatic reclamation **does not** write back to OSS, so artifacts that were not pulled are lost.
+> - Therefore you must still run `kill`/`finish` explicitly when the task ends; to preserve artifacts use `finish` or `pull` beforehand. `sandbox status` returns `expires_at` / `reclaim_source` / `termination_reason`, and remains queryable after the sandbox has ended.
+> - `--no-auto-cleanup` disables both the platform TTL and the server-side cleanup, so `expires_at`/`remaining` are null.
 
 ## Background tasks
 
@@ -30,7 +35,7 @@ The CLI follows the live SSE stream (`/logs/stream`) and passes raw SSE frames (
 
 > On the SSE path, `--wait` puts `result_summary` directly in the **complete frame** (if trailing JSON is detected), so there is no need to scan backwards with `sandbox read <id> --path <log_file>`. Read the log_file manually only if no valid JSON is found at the end. `--summary`/`--artifacts` enrichment (including hints) still takes effect only on the legacy-server fallback path.
 
-> **Output contract (SSE path):** On success, `run-bg --wait` writes only raw SSE frames to stdout, with **no trailing envelope**. On command failure, it appends an `{"ok":false,"error":{"type":"command_failed",...}}` envelope after the complete frame. If the command has not finished within `--timeout`, it appends a `finished=false` `CommandLogsResult` snapshot envelope. For an older server (without the `/logs/stream` endpoint), the CLI falls back to polling every 5s and outputs a `CommandResult` envelope (the `finished` key is always present; when `finished=true`, it includes `result_summary`/`hints` enrichment).
+> **Output contract (SSE path):** On success, `run-bg --wait` writes only raw SSE frames to stdout, with **no trailing envelope**. On command failure, it appends an `{"ok":false,"error":{"type":"command_failed",...}}` envelope after the complete frame and exits non-zero (process exit code 8). If the command has not finished within `--timeout`, the CLI exits **non-zero** with `{"ok":false,"error":{"type":"timeout",...}}`; `error.details` carries `sandbox_id`, `execution_id` and `remote_state` so the command can be queried or cancelled afterwards (`remote_state: not_cancelled` means only the local wait stopped and the remote command may still be running — pass `--cancel-on-timeout` to cancel it). For an older server (without the `/logs/stream` endpoint), the CLI falls back to polling every 5s and outputs a `CommandResult` envelope (the `finished` key is always present; when `finished=true`, it includes `result_summary`/`hints` enrichment). **For automation, add `--format json` (same as `--json-result`)**: it emits no frames, only one envelope whose `data` contains `execution_id`/`finished`/`exit_code`/`stdout_tail`/`stderr_tail`/`stdout_size`/`stderr_size`/`log_file`/`duration_seconds`/`result_summary`; on failure it is `{ok:false,error,data}` and the process exits non-zero. When those fields are missing from the `complete` frame, the CLI fills them from the accumulated stream, so the field semantics match `--format json` exactly.
 
 ### Method 2: poll manually (when you need to see live output)
 
@@ -50,8 +55,13 @@ delta-cli sandbox logs <id> --execution-id <exec_id>
 #   the response is then an **array** grouped by execution_id (each item contains execution_id/stdout/stderr/exit_code/command/finished/error/log_file),
 #   rather than a single snapshot object; the CLI also applies 800/200-byte tail trimming and --tail/--grep/--context filters to each item
 
-# 3. Destroy after completion
+# 3. Destroy after completion (returns request_id / operation_status; destruction continues server-side)
 delta-cli sandbox kill <id>
+
+# 4. Reconcile: after a client disconnect, a create error or a network error on kill, use the operation ledger to confirm the real outcome
+delta-cli sandbox operation <request_id>
+# or list the create/destroy records for a sandbox
+delta-cli sandbox operation --sandbox-id <id>
 ```
 
 ### How to interpret polling results
@@ -97,5 +107,5 @@ Interpret task status using these rules, in order:
 **Notes:**
 - Use `sandbox logs` only with `sandbox run-bg`; synchronous `sandbox run` returns its result directly, so `logs` cannot and should not be called for it.
 - Without `--wait`, the `sandbox run-bg` HTTP request itself returns quickly with `{execution_id, sandbox_id}`. If the command times out, it exits with a non-zero code; retrieve `exit_code` later with `logs`.
-- In `sandbox run-bg --wait` mode, `--timeout` serves **both** as the server-side command timeout and the client-side waiting deadline (default 300s). The CLI follows the live SSE stream (`/logs/stream`), passes raw `data:` frames through to stdout and ends when `complete` arrives (with no trailing envelope); on timeout, it appends a `finished=false` snapshot envelope. The initial `/run-background` POST has its own 60s HTTP timeout.
+- In `sandbox run-bg --wait` mode, `--timeout` serves **both** as the server-side command timeout and the client-side waiting deadline (default 300s). The CLI follows the live SSE stream (`/logs/stream`), passes raw `data:` frames through to stdout and ends when `complete` arrives (with no trailing envelope); on timeout it exits **non-zero** with `error.type: timeout` and keeps `sandbox_id`/`execution_id`/`remote_state` in `error.details` (add `--cancel-on-timeout` to also cancel the remote command). The initial `/run-background` POST has its own 60s HTTP timeout.
 - Background commands can be queried across tool-call turns: save `sandbox_id` and `execution_id`, then retrieve results in later turns with `sandbox status` / `sandbox logs`.

package/skills/delta-sandbox/references/recipes.md

@@ -129,6 +129,7 @@ Notes:
 ## Local `result.json` content contract
 
 - **Preferred SSE path (recommended, one step):** The **complete frame for `sandbox run` / `run-bg --wait` includes `result_summary` directly** (the CLI injects detected trailing JSON from stdout), so the skill can extract it from the complete frame without reading and parsing the log.
+- **Preferred path for automation:** add `--format json` (same as `--json-result`) and the CLI emits a single envelope; read `data.exit_code` / `data.result_summary` / `data.stdout_tail` directly, without parsing multiple frames or reassembling stdout (newlines are preserved). On failure `data` is still present (`{ok:false,error,data}`).
 - **SSE fallback path:** If the complete frame lacks `result_summary` (there is no valid JSON at the end of stdout), `delta-cli sandbox read <id> --path <log_file>` reads the run-envelope log file, then scans backwards through stdout to extract trailing JSON. The `log_file` path comes from the SSE `complete` frame (the CLI fills it in if needed) or the `sandbox logs` snapshot (after `finished=true`).
 - **Legacy-server fallback path:** When `run` / `run-bg --wait` falls back to polling, it returns a `CommandResult` envelope whose `data.result_summary` field can be reused directly.
 - The local `result.json` is a **concise summary** for host deliverable validation, not a complete log archive.

package/skills/delta-shared/SKILL.md

@@ -65,7 +65,7 @@ This returns the current user's information and remaining credits (through the g
 - **Do not install the CLI proactively**: if the user says it is not installed, direct them to the README or `install.sh`; do not choose an installation method yourself.
 - **Confirm intent before write or delete operations**: tell the user what commands such as `sandbox kill`, `sandbox finish` and `sandbox write` will affect.
 - `~/.delta-infra/config.json` must have `0600` permissions; do not loosen them manually.
-- **The sandbox must be destroyed**: after calling `sandbox create`, call `sandbox kill` before the current task ends. The server does not clean up automatically; failure to destroy a sandbox leaves GPU resources occupied indefinitely. Do not skip this, even if an error occurs.
+- **The sandbox must be destroyed explicitly**: after calling `sandbox create`, call `sandbox kill` before the current task ends (use `finish` when results must be preserved). Sandboxes do get reclaimed automatically (default `--max-life` is 30 minutes; OpenSandbox starts the clock when create is accepted and the platform reclaims, while AutoDL/RunPod start at readiness and the server reclaims at max-life plus up to 10 minutes of grace), but the platform may reclaim early and automatic reclamation does not write back to OSS — relying on it loses artifacts. Do not skip destruction, even if an error occurs.
 
 ## JSON output conventions
 
@@ -76,7 +76,7 @@ This returns the current user's information and remaining credits (through the g
 
 AI agent parsing rules:
 - Always check `ok`; do not rely on the exit code alone.
-- `error.type` is the stable field for branching on: `auth`, `permission`, `not_found`, `validation`, `network`, `api`, `command_failed`, `internal`.
+- `error.type` is the stable field for branching on: `auth`, `permission`, `not_found`, `validation`, `network`, `api`, `command_failed`, `timeout`, `unknown_execution_state`, `internal`.
 - `error.hint` may be shown to the user. **Act on it rather than retrying blindly**: for a failed `sandbox create` the hint names the concrete cause and the next step (for example, the image was not cached on the target node — retry, or pick a GPU type whose node already has the image).
 - `error.request_id` identifies the same failure in the server log; quote it when reporting the failure.
 
@@ -91,6 +91,8 @@ AI agent parsing rules:
 | network | 6 | DNS, timeout or connection failure |
 | api | 7 | Server returned a non-2xx response |
 | command_failed | 8 | Command failed inside the sandbox (non-zero `exit_code` or an error); on the SSE path, an error envelope is appended after the `complete` frame, with the stderr tail in `message` for quick reference |
+| timeout | 1 | A local wait deadline elapsed (`sandbox run-bg --wait`): the remote command may still be running — `error.details` keeps `sandbox_id`, `execution_id` and `remote_state` (`not_cancelled` / `cancelled` / `cancel_failed`) so it can be queried or cancelled |
+| unknown_execution_state | 1 | The command's outcome is **unknown** (the stream ended before the `complete` frame, or only an error frame arrived). Never treat this as success and never guess the exit code |
 | internal | 10 | Internal client error |
 
 ## Upgrade check
@@ -111,5 +113,7 @@ delta-cli upgrade
 | `error.type: network` | Check `base_url` and network connectivity |
 | `error.type: validation` | Check required flags and argument formats |
 | `error.type: command_failed` | Command failed inside the sandbox: the envelope's `message` already includes the stderr tail to help locate the issue quickly; the full `stderr` / `result_summary` are in the `complete` frame. Retry or correct the command |
+| `error.type: timeout` | A local wait deadline elapsed, **not** a command failure. Read `error.details.execution_id` and query it with `sandbox logs <id> --execution-id <exec_id>`; cancel the still-running command with `sandbox cancel <id> --execution-id <exec_id>` (or re-run with `--cancel-on-timeout` so the CLI cancels it for you) |
+| `error.type: unknown_execution_state` | The remote outcome is unknown — the CLI will not guess. Re-check the command with `sandbox logs <id> --execution-id <exec_id>` (the execution id is in the error details) before deciding whether it succeeded |
 | `sandbox create` failed (`error.type: api` with a specific `hint`) | Follow the hint — it names the cause and the next step. "the image may not be cached on the target node" → retry once (the pull continues in the background, so later creates succeed) or pick a GPU type whose node already has the image. "check the target node's free disk space" → a retry will not help until an operator frees the node. A failed create leaves no sandbox behind, so retrying is safe. Quote `error.request_id` when reporting it. |
 | Configuration file parsing failed | Check that `~/.delta-infra/config.json` contains valid JSON |