@shardflux/sdk 0.13.0 → 0.13.1

package/CHANGELOG.md

@@ -3,7 +3,86 @@
 Every API the README shows is available from the version named here. Breaking changes ship in minor releases and are
 marked **Breaking**.
 
-## 0.13.0 (not yet published)
+## 0.14.0 (not yet published)
+
+### Workspaces working at once
+
+Additive. The plan's workspace number caps workspaces working at the same moment; stored workspaces are bounded by
+Retained state.
+
+- A tool call that finds every workspace the plan allows to work at once busy is refused with 429 `quota_exceeded`
+  (retryable, `Retry-After`; `details.limit` `concurrent_workspaces`, `limit_value`, `current`,
+  `retry_after_seconds`). The call did not run, so the SDK sends the same request again after `Retry-After` (at most
+  5 s) within `maxRetries`, for every call: exec starts, stdin, PTY create, keepalive and process signals included.
+  A refusal that outlasts the retries is a `ShardfluxApiError` with `retryAfterSeconds`. `executions.run()` keeps
+  these retries and does not add its own on top; the wake hint is not retried (it never waits).
+- `pty.read()` rejects with the refusal (`ShardfluxApiError`, e.g. 429 `quota_exceeded`, retryable) when an accepted
+  attach is closed with an error body and a 4xxx close code, instead of returning empty output.
+- The API's 403 `quota_exceeded` is not retried, as before. New `details.limit`: `retained_state` (`limit_value` and
+  `current` in GiB): opening a new key and forking are refused while the plan's Retained state is used up; existing
+  workspaces keep running, waking, suspending and resuming. Usage summaries report that allowance with `enforcement`
+  `storage_block` and `cap_state` `storage_blocked`.
+- Regenerated contract types: `ErrorCode` includes `quota_exceeded` for the workspace gateway; allowance
+  `enforcement` adds `storage_block`, `cap_state` adds `storage_blocked`.
+
+## 0.13.1 (not yet published)
+
+Patch, additive: the transport, a documented value, a fix for tool calls across a move and self-healing workspaces.
+`@shardflux/cli` 0.8.1, `@shardflux/mcp` 0.7.1 and the `shardflux` bundle 0.10.1 follow (`^0.13.1`).
+
+### Workspaces recover by themselves
+
+Additive. When the machine a workspace runs on fails, the workspace is suspended, and its next use (a tool call's wake,
+`resume()`, `open()`) restores it: from its own disk, files kept, or from its newest checkpoint. Nothing changes for
+code that already handles a cold resume.
+
+- `cold_boot_reason` / `ServerTiming.coldBootReason` can be `host_lost`: the resume booted the workspace's disk
+  (`resumePath` `cold_boot`, `memoryRestored` false, processes restarted). New type `ColdBootReason`
+  (`'runtime_changed' | 'host_lost'`, open to other strings; the field still accepts any string).
+- `hostLostOf(op)` and `ServerTiming.hostLost`: `result.host_lost` camelCased (`HostLost`: `detectedAt`,
+  `restoredFrom` `'disk' | 'checkpoint'`, `restoredCheckpointId`, `stateAsOf`). A resume from the checkpoint has its
+  state as of `stateAsOf`; a suspend that found the machine failed succeeds with `durable: true` and only
+  `detectedAt` (`restoredFrom` null).
+- `formatTiming()` prints `processes restarted (host_lost)` for the disk, `restored checkpoint <id> (host_lost; state as
+  of <time>)` for the checkpoint, and `host_lost (detected <time>)` for the suspend.
+- Documented operation error codes: `resume_required` (a fork or snapshot of such a workspace before its resume; resume
+  it first; not retryable) and `workspace_storage_unavailable` with `details.reason` `host_lost` (not retryable).
+  `KnownErrorReason` gains `host_lost`.
+
+### Connection reuse
+
+Additive. The call after an agent's pause between tool calls reuses its connection instead of opening a new one.
+
+- On Node 26 the pooled transport keeps an idle connection reusable for 5 minutes (was 4 s), so a request seconds or
+  minutes after the last one skips the TCP and TLS handshake. Pooled sockets send TCP keepalive probes after 60 s
+  idle, so NATs and firewalls along the way keep the connection open. Idle connections never keep a process alive. A
+  supplied `fetch`, other runtimes and `SHARDFLUX_HTTP_KEEPALIVE` work as before.
+- A request the SDK retries (GET, a request with an `Idempotency-Key`, a read-only POST such as files search) whose
+  connection closes before the response arrives is sent again at once on a new connection. That one resend has no
+  backoff and does not count against `maxRetries`; timing records and progress `retry` events list it with
+  `delayMs: 0`. A further failure follows `maxRetries` and backoff as before, and a POST without an
+  `Idempotency-Key` is never sent twice.
+
+### Faster suspend and resume
+
+Additive. `resumePath` / `resume_path` can be `thaw`: a resume that arrives while its suspend is still being
+written continues the same VM in place. Nothing else changes for your code.
+
+### Tool calls across a move
+
+Fix. A tool call made while its workspace finishes a resume or a move now runs once the workspace runs, instead of
+failing with `409 conflict` (`workspace_not_running`).
+
+- When the call's tool token was refused as not running and the wake then found the workspace already running (the
+  resume or move committed in between; `wake` resolves `false`), the call is retried with a current token: the held
+  resume's, or a new one. A refused call never ran, so the retry cannot run it twice. Before, the refusal surfaced.
+- A refusal that comes back while the API keeps reporting the workspace running is retried after 500 ms, within the
+  same 3 wakes and `transitionTimeoutMs`, then surfaces as before.
+- `onProgress` sees the retry as a `tool` event of type `retry`, cause `conflict workspace_not_running (the workspace
+  runs; new token)`.
+- The CLI, the MCP server and the `shardflux` bundle get it through their `^0.13.0` dependency.
+
+## 0.13.0
 
 ### Elastic memory
 
@@ -446,7 +525,6 @@ behavior is the automatic version check (below), which makes one background requ
 - New exports: `FeedbackCategory`, `FeedbackContext`, `FeedbackReceipt`, `SendFeedbackParams`, `AccountFeedbackParams`,
   `FEEDBACK_CATEGORIES`, `FEEDBACK_MESSAGE_MAX_LENGTH`.
 
-
 ## 0.8.0 (2026-09-28)
 
 Types only; nothing changes at run time and the API is unchanged.

package/README.md

@@ -296,10 +296,10 @@ await cell.exec.run(['make', 'test']);                         // resumes the wo
 ```
 
 **Detecting a cold resume (0.11.0+).** A resume restores memory and running processes from the checkpoint. After a
-platform runtime update, a resume can boot from the saved disk instead of restoring memory (a cold boot), also when a
-tool call wakes the workspace; check `memoryRestored`. The workspace runs with its files as of the suspend, and its
-processes start fresh, as after a reboot: start dev servers, databases and background jobs again. The resume's timing
-says which it was:
+platform runtime update, or after the machine the workspace ran on failed (0.13.1+), a resume can boot from the saved
+disk instead of restoring memory (a cold boot), also when a tool call wakes the workspace; check `memoryRestored`. The
+workspace runs with its files kept, and its processes start fresh, as after a reboot: start dev servers, databases and
+background jobs again. The resume's timing says which it was:
 
 ```ts
 const t = workspace.lastTiming; // after resume({ wait: true }), wake(), or a tool call that woke the workspace
@@ -310,11 +310,20 @@ if (t?.server?.memoryRestored === false) {
 
 - `ServerTiming.memoryRestored`: `true` when memory and processes came back, `false` when the workspace booted
   (`resumePath` `cold_boot`, or `reset_blank_layer` after a reset), `null` when the API did not say (older APIs).
-- `ServerTiming.coldBootReason`: why it booted (`runtime_changed`), else null. `formatTiming()` prints
-  `resume from cold_boot: processes restarted (runtime_changed)`.
+- `ServerTiming.coldBootReason` (type `ColdBootReason`): why it booted, `runtime_changed` or `host_lost`
+  **(0.13.1+)**, else null. `formatTiming()` prints `resume from cold_boot: processes restarted (runtime_changed)`.
 - The finished resume operation carries the same in `result` (`memory_restored`, `cold_boot_reason`, and
   `cold_boot` with details such as `files_as_of`; it is informational and not part of the stable contract).
 
+**Workspaces recover by themselves (0.13.1+).** When the machine a workspace runs on fails, the workspace is suspended,
+and its next use (a tool call, `resume()`, `open()`) restores it; there is nothing extra to call.
+`hostLostOf(op)` and `lastTiming.server.hostLost` (`HostLost`) say how:
+
+- `restoredFrom: 'disk'`: it booted from its own disk, files kept (`coldBootReason` `host_lost`, processes restarted).
+- `restoredFrom: 'checkpoint'`: it resumed its newest checkpoint (`restoredCheckpointId`); its state is as of
+  `stateAsOf`. `formatTiming()` adds `restored checkpoint <id> (host_lost; state as of <time>)`.
+- A `suspend()` that finds the machine failed succeeds (`result.durable` true, `hostLost.detectedAt`).
+
 `suspend`, `resume`, `fork`, `snapshot` and `delete` return the lifecycle operation.
 `cloud.workspaces.waitForOperation(id)` waits for it (default timeout 5 minutes): each poll asks the API to hold
 the response until the operation changes (`Prefer: wait`, at most 20 s per request), so completion arrives within
@@ -324,7 +333,15 @@ operation keeps running server side (a queued start until its `deadlineAt`); wai
 same call. `templates.builds.waitForBuild()` waits the same way. A waited `open()` issues the first tool token
 together with the final workspace read, so the first tool call starts at once.
 
-From 0.12.0, Node 26 reuses TLS connections through a private HTTP/1.1 pool. Other runtimes keep native fetch. A supplied `fetch` stays unchanged. `SHARDFLUX_HTTP_KEEPALIVE=0` forces connection closure for diagnosis; `=1` opts into native pooling. No global dispatcher is installed.
+From 0.12.0, Node 26 reuses TLS connections through a private HTTP/1.1 pool. From 0.13.0, an idle connection stays
+reusable for 5 minutes, so the call after an agent's pause between tool calls skips the TCP and TLS handshake, and
+pooled sockets send TCP keepalive probes after 60 s idle. Idle connections never keep a process alive. Other runtimes
+keep native fetch. A supplied `fetch` stays unchanged. `SHARDFLUX_HTTP_KEEPALIVE=0` forces connection closure for
+diagnosis; `=1` opts into native pooling. No global dispatcher is installed.
+
+A request the SDK retries (safe methods, requests with an `Idempotency-Key`, read-only POSTs) whose connection closes
+before the response arrives is sent again at once on a new connection **(0.13.0+)**, without backoff and outside
+`maxRetries`; timing records list it with `delayMs: 0`.
 
 List and look up workspaces:
 
@@ -412,7 +429,7 @@ This resume brought the workspace back from its host's local cache, memory and r
   running, including any time in `capacity_pending`; `ran` is the cell's work (placement, boot or restore, guest
   readiness). `start` / `resume from` and `boot to ready` / `host …` are what the cell reported. A resume that booted
   the saved disk instead of restoring memory reads `resume from cold_boot: processes restarted (runtime_changed)`
-  (0.11.0+).
+  (0.11.0+), or `(host_lost)` (0.13.1+) after the machine the workspace ran on failed.
 - **outside the server** is your total minus the operation's: network, TLS, polling latency, view and token. A large
   value with a small server total points at the connection between you and the API, not at the workspace.
 - **retries** lists transient failures the SDK retried (cause and backoff).

package/dist/cell.d.ts

@@ -174,8 +174,9 @@ export interface CellClientOptions {
     sleep?: (ms: number) => Promise<void>;
     /**
      * Wakes a suspended workspace (resume, or join the active resume/open) within `timeoutMs` and resolves once it runs;
-     * resolves `false` when there was nothing to wake (the API already reports it running), and the refusal then
-     * surfaces. It throws when the wake fails (OperationFailedError) or outlasts `timeoutMs` (OperationTimeoutError).
+     * resolves `false` when there was nothing to wake (the API already reports it running): the refused call is then
+     * retried with a current token, since the transition that refused it ended meanwhile (0.13.1+). It throws when the
+     * wake fails (OperationFailedError) or outlasts `timeoutMs` (OperationTimeoutError).
      * Workspace.cell() supplies `workspace.wake()`. A call refused with `workspace_not_running` wakes the workspace and is
      * retried; a refused call was never executed, so the retry cannot duplicate it. `null` surfaces
      * the refusal instead. A wake that put a new token into this client's manager (the held resume;
@@ -320,8 +321,10 @@ export declare class CellClient {
      * One authorized request. Refreshes the token once on stale_epoch / 401. Lifecycle transitions,
      * bounded in total by `transitionTimeoutMs`: a call refused with `workspace_busy` is retried after `Retry-After`; one
      * refused with `workspace_not_running` (or whose token cannot be minted because the workspace is not running) wakes
-     * the workspace through `wake`, given the time left, and is retried with a fresh token, at most 3 wakes per call.
-     * Refused calls were never executed, so retrying is safe. When the budget is spent the refusal surfaces.
+     * the workspace through `wake`, given the time left, and is retried with a fresh token, at most 3 wakes per call. A
+     * wake that finds the workspace already running (`false`) is retried the same way: the transition that refused the
+     * call ended meanwhile (0.13.1+). Refused calls were never executed, so retrying is safe. When the budget is spent
+     * the refusal surfaces.
      */
     request(method: string, path: string, init?: RequestOptions & TransitionOptions): Promise<Response>;
     /**

package/dist/cell.js

@@ -1,4 +1,4 @@
-import { ExecStartError, NotSupportedForModeError, ShardfluxApiError, ShardfluxProtocolError } from "./errors.js";
+import { ExecStartError, NotSupportedForModeError, ShardfluxApiError, ShardfluxProtocolError, apiError, isErrorBody, isWorkingQuotaRefusal } from "./errors.js";
 import { EXECUTION_ID, ExecutionResult, newExecutionId } from "./executions.js";
 import { HttpClient, defaultSleep, randomId, treeRevisionOf } from "./http.js";
 import { describeFailure, emitTo } from "./progress.js";
@@ -20,8 +20,13 @@ export const DEFAULT_TRANSITION_TIMEOUT_MS = 120_000;
  * capture writes recorded before them (read-your-writes). Workspace.cell() sets it; the capture's own client does not.
  */
 export const CAPTURE_BARRIER = Symbol('shardflux.captureBarrier');
-/** Wakes per call at most: a workspace that keeps being suspended again surfaces the refusal. */
+/**
+ * Wakes per call at most: a workspace that keeps being suspended again, or keeps refusing the call while the API reports
+ * it running, surfaces the refusal.
+ */
 const MAX_WAKES = 3;
+/** The pause before retrying a call refused again while the API reports the workspace running (from the second time). */
+const RUNNING_AGAIN_PAUSE_MS = 500;
 /** exec.cancel grace: the workspace waits 5 s without one and takes at most 60 s. */
 const DEFAULT_CANCEL_GRACE_MS = 5_000;
 const MAX_CANCEL_GRACE_MS = 60_000;
@@ -63,6 +68,30 @@ export async function* ndjson(body) {
         reader.releaseLock();
     }
 }
+/**
+ * A refused attach the gateway had to accept (cell-api.yaml "WebSocket close codes": an allowed Origin cannot read the
+ * HTTP answer to a failed upgrade): its only text frame is exactly the ErrorBody, then a close with 4000 + the HTTP
+ * status (4410 for not_found). A 4429 without a body (its close reason is `{"code","retryable",...}`) is the same
+ * refusal in short. Undefined for any other close.
+ */
+function attachRefusal(code, reason, body) {
+    const status = code === 4410 ? 404 : code !== undefined && code >= 4000 && code < 5000 ? code - 4000 : undefined;
+    if (body) {
+        const hinted = body.error.details?.retry_after_seconds;
+        return apiError(status ?? 502, body, 'cell', typeof hinted === 'number' && hinted >= 0 ? hinted : undefined);
+    }
+    if (code !== 4429)
+        return undefined;
+    let short = {};
+    try {
+        short = JSON.parse(reason ?? '');
+    }
+    catch {
+        // Not the compact JSON: the close code alone says it.
+    }
+    const errCode = typeof short.code === 'string' ? short.code : 'rate_limited';
+    return apiError(429, { error: { code: errCode, message: `The attach was refused (${errCode}); retry shortly.`, request_id: typeof short.request_id === 'string' ? short.request_id : '', retryable: short.retryable !== false } }, 'cell');
+}
 class ByteSink {
     #max;
     #chunks = [];
@@ -108,6 +137,10 @@ const executionRetryDelayMs = (failures, retryAfterSeconds) => retryAfterSeconds
 const attemptTimedOut = (err) => err instanceof Error && err.name === 'TimeoutError';
 /** A transport failure or a retryable 429/5xx: the execution request may be sent again with the same id. */
 function transientExecutionFailure(err) {
+    // The working-at-once refusal was already retried by the HTTP layer within the client's maxRetries: a second loop
+    // here would multiply that budget, so it surfaces.
+    if (isWorkingQuotaRefusal(err))
+        return null;
     if (err instanceof ShardfluxApiError) {
         const status = err.status === 429 || err.status === 502 || err.status === 503 || err.status === 504;
         return status && err.retryable ? { retryAfterSeconds: err.retryAfterSeconds } : null;
@@ -213,8 +246,10 @@ export class CellClient {
      * One authorized request. Refreshes the token once on stale_epoch / 401. Lifecycle transitions,
      * bounded in total by `transitionTimeoutMs`: a call refused with `workspace_busy` is retried after `Retry-After`; one
      * refused with `workspace_not_running` (or whose token cannot be minted because the workspace is not running) wakes
-     * the workspace through `wake`, given the time left, and is retried with a fresh token, at most 3 wakes per call.
-     * Refused calls were never executed, so retrying is safe. When the budget is spent the refusal surfaces.
+     * the workspace through `wake`, given the time left, and is retried with a fresh token, at most 3 wakes per call. A
+     * wake that finds the workspace already running (`false`) is retried the same way: the transition that refused the
+     * call ended meanwhile (0.13.1+). Refused calls were never executed, so retrying is safe. When the budget is spent
+     * the refusal surfaces.
      */
     async request(method, path, init = {}) {
         const closer = this.#closer.signal;
@@ -237,6 +272,7 @@ export class CellClient {
         const retry = (cause, delayMs, attempt) => emit?.({ type: 'retry', retry: { atMs: Math.round((performance.now() - t0) * 10) / 10, request: `${method} ${path}`, attempt, cause, delayMs } });
         let refreshed = false;
         let wakes = 0;
+        let runningAgain = 0;
         let attempts = 0;
         for (;;) {
             try {
@@ -270,8 +306,17 @@ export class CellClient {
                 if (notRunning && wakeAllowed && this.#wake && wakes < MAX_WAKES && left > 0) {
                     wakes += 1;
                     const before = this.tokens.current;
-                    if ((await this.#wake(left, signal)) === false)
-                        throw err; // running per the API: nothing to wait for
+                    if ((await this.#wake(left, signal)) === false) {
+                        // Running per the API: the refusal was answered from a transition that ended before the wake looked (a
+                        // resume or a move committed in between: the token request was refused while the workspace was resuming,
+                        // and the wake found it running). The refused call never ran, so it is retried with a current token. A
+                        // refusal that comes back while the API keeps reporting the workspace running is retried after a pause,
+                        // within the same wakes and time budget, then surfaces.
+                        const pause = runningAgain++ === 0 ? 0 : Math.max(0, Math.min(deadline - Date.now(), RUNNING_AGAIN_PAUSE_MS));
+                        retry(`${err.code === 'conflict' ? 'conflict workspace_not_running' : err.code} (the workspace runs; new token)`, pause, (attempts += 1));
+                        if (pause > 0)
+                            await this.#opts.sleep(pause);
+                    }
                     // A held resume handed this client a token of the woken workspace: use it. Otherwise the
                     // old token is of the previous epoch: fetch a new one.
                     if (this.tokens.current === before)
@@ -744,6 +789,7 @@ export class CellClient {
             let next = opts.offset ?? 0;
             let session = null;
             let exited = false;
+            let refusal;
             await new Promise((resolve, reject) => {
                 const onClose = () => finish(this.#closer.signal.reason instanceof Error ? this.#closer.signal.reason : new Error('closed'));
                 const ws = new WS(url, { headers: { authorization: `Bearer ${token}` } });
@@ -760,8 +806,10 @@ export class CellClient {
                     catch {
                         // already closed
                     }
-                    if (err)
-                        reject(err);
+                    // A refusal body whose close did not arrive within the read: the body alone says it.
+                    const failure = err ?? (refusal ? attachRefusal(undefined, undefined, refusal) : undefined);
+                    if (failure)
+                        reject(failure);
                     else
                         resolve();
                 };
@@ -775,6 +823,12 @@ export class CellClient {
                     if (typeof e.data !== 'string')
                         return;
                     const m = JSON.parse(e.data);
+                    // A refused attach (e.g. 4429 quota_exceeded: every working slot is taken): the bare ErrorBody, then the close.
+                    if (!('type' in m)) {
+                        if (isErrorBody(m))
+                            refusal = m;
+                        return;
+                    }
                     if (m.type === 'output' && m.data) {
                         const bytes = unb64(m.data);
                         sink.push(bytes);
@@ -791,7 +845,7 @@ export class CellClient {
                     }
                 });
                 ws.addEventListener('error', () => finish(new ShardfluxProtocolError('pty attach WebSocket failed', 0, 'cell')));
-                ws.addEventListener('close', () => finish());
+                ws.addEventListener('close', (e) => finish(attachRefusal(e.code, e.reason, refusal)));
                 if (this.closed)
                     onClose();
                 else
@@ -955,7 +1009,8 @@ export class CellClient {
         // Nothing of a file-first workspace sleeps: the cell would answer `resident`.
         if (this.mode === 'file_first')
             return Promise.resolve({ residency: 'resident' });
-        return this.#json('POST', this.#p('/v1/workspaces/{workspace_id}/wake-hint'), { wake: false, busy: false, timeoutMs: 10_000, ...(signal ? { signal } : {}) });
+        // A hint never waits: a 429 quota_exceeded (every working slot is taken) surfaces at once, like its other refusals.
+        return this.#json('POST', this.#p('/v1/workspaces/{workspace_id}/wake-hint'), { wake: false, busy: false, quotaRetry: false, timeoutMs: 10_000, ...(signal ? { signal } : {}) });
     }
     /** Read idle signals without recording activity or waking the workspace. */
     idle(signal) {

package/dist/errors.d.ts

@@ -70,8 +70,19 @@ export type ErrorCode = AppErrorCode | CellErrorCode;
  * 409 conflict read_only_path (a files write under an immutable path, from the cell gateway). The reason
  * `update_policy_not_available` is gone with the update policy. A build that fails on them carries `failure.code`
  * immutable_path_missing (details.path) or immutable_image_too_large.
+ * Host loss (0.13.1; the machine a workspace ran on failed, and the workspace restores itself on its next use): the
+ * operation error `workspace_storage_unavailable` carries details.reason host_lost (and details.workspace_state
+ * `suspended`) when neither the workspace's disk nor a checkpoint could be restored (not retryable). A fork or snapshot
+ * of such a workspace before its resume fails with the operation error `resume_required` (details.reason host_lost;
+ * not retryable): resume the workspace first.
+ * Workspaces working at once (0.14.0): the cell gateway answers a tool call that finds every working slot of the plan
+ * taken with 429 `quota_exceeded` (retryable, Retry-After; details.limit `concurrent_workspaces`, limit_value, current,
+ * retry_after_seconds); the call did not run and the SDK retries it like any transient refusal. The API's 403
+ * `quota_exceeded` (not retryable) names details.limit `concurrent_workspaces` on open, resume and fork, and
+ * `retained_state` (details.limit_value and details.current in GiB) when opening a new key or forking with the plan's
+ * Retained state used up. `details.limit` is not a reason: these are not in this union.
  */
-export type KnownErrorReason = 'invalid_recipe' | 'base_not_layered' | 'language_unavailable' | 'language_conflict' | 'invalid_package' | 'too_many_files' | 'platform_owned_path' | 'upload_required' | 'upload_missing' | 'upload_digest_mismatch' | 'upload_too_large' | 'extra_hosts_without_auto' | 'invalid_settings' | 'services_unsupported' | 'input_required' | 'input_unknown' | 'input_invalid' | 'egress_widening' | 'reserved_session_id' | 'env_collision' | 'reserved_template_slug' | 'package_index_unavailable' | 'package_not_found' | 'startup_failed' | 'service_not_ready' | 'secrets_unavailable' | 'workspace_not_running' | 'operation_in_progress' | 'workspace_deleted' | 'secret_not_available' | 'legacy_disk_layout' | 'not_session' | 'session_lifetime' | 'lifetime_mismatch' | 'not_resettable' | 'template_not_layered' | 'draft_exists' | 'draft_stale' | 'build_in_progress' | 'file_list_unavailable' | 'file_list_indexing' | 'guest_feature_unavailable' | 'confirm_destructive_required' | 'reserved_key_prefix' | 'invalid_defaults' | 'invalid_path' | 'too_many_acknowledged_findings' | 'template_dev_mode_role' | 'draft_not_found' | 'version_not_found' | 'path_not_found' | 'revision_mismatch' | 'edit_not_found' | 'edit_ambiguous' | 'edit_not_text' | 'patch_invalid' | 'host_capacity' | 'wake_failed' | 'workspace_fenced' | 'offline_unavailable' | 'offline_budget' | 'offline_changed' | 'host_feature_unavailable' | 'not_supported_for_mode' | 'mode_mismatch' | 'mode_not_available' | 'layout_unsupported' | 'tree_revision_mismatch' | 'outside_tree_root' | 'execution_in_progress' | 'execution_id_reused' | 'operation_id_reused' | 'no_execution_host' | 'lease_expired' | 'host_unreachable' | 'host_restarted' | 'tree_moved' | 'blob_missing' | 'blob_corrupt' | 'exec_failed_to_start' | 'invalid_cwd' | 'allowance_used' | 'overage_paused' | 'spend_cap_reached' | 'overage_unavailable' | 'spend_cap_required' | 'spend_cap_below_minimum' | 'spend_cap_above_plan_price' | 'spend_cap_below_charges' | 'version_mismatch' | 'allocation_mode_not_available' | 'requires_elastic' | 'exceeds_memory_mib' | 'burst_mode_not_supported' | 'burst_not_supported' | 'burst_size_exceeds_plan' | 'not_available' | 'shared_volumes' | 'fence_not_drained' | 'apply_pending' | 'park_failed' | 'workspace_resumed' | 'interrupted' | 'burst_lost' | 'disk_full' | 'apply_failed' | 'reverted' | 'revert_failed' | 'immutable_path_removed' | 'immutable_paths_unsupported_base' | 'read_only_path';
+export type KnownErrorReason = 'invalid_recipe' | 'base_not_layered' | 'language_unavailable' | 'language_conflict' | 'invalid_package' | 'too_many_files' | 'platform_owned_path' | 'upload_required' | 'upload_missing' | 'upload_digest_mismatch' | 'upload_too_large' | 'extra_hosts_without_auto' | 'invalid_settings' | 'services_unsupported' | 'input_required' | 'input_unknown' | 'input_invalid' | 'egress_widening' | 'reserved_session_id' | 'env_collision' | 'reserved_template_slug' | 'package_index_unavailable' | 'package_not_found' | 'startup_failed' | 'service_not_ready' | 'secrets_unavailable' | 'workspace_not_running' | 'operation_in_progress' | 'workspace_deleted' | 'secret_not_available' | 'legacy_disk_layout' | 'not_session' | 'session_lifetime' | 'lifetime_mismatch' | 'not_resettable' | 'template_not_layered' | 'draft_exists' | 'draft_stale' | 'build_in_progress' | 'file_list_unavailable' | 'file_list_indexing' | 'guest_feature_unavailable' | 'confirm_destructive_required' | 'reserved_key_prefix' | 'invalid_defaults' | 'invalid_path' | 'too_many_acknowledged_findings' | 'template_dev_mode_role' | 'draft_not_found' | 'version_not_found' | 'path_not_found' | 'revision_mismatch' | 'edit_not_found' | 'edit_ambiguous' | 'edit_not_text' | 'patch_invalid' | 'host_capacity' | 'wake_failed' | 'workspace_fenced' | 'offline_unavailable' | 'offline_budget' | 'offline_changed' | 'host_feature_unavailable' | 'not_supported_for_mode' | 'mode_mismatch' | 'mode_not_available' | 'layout_unsupported' | 'tree_revision_mismatch' | 'outside_tree_root' | 'execution_in_progress' | 'execution_id_reused' | 'operation_id_reused' | 'no_execution_host' | 'lease_expired' | 'host_unreachable' | 'host_restarted' | 'tree_moved' | 'blob_missing' | 'blob_corrupt' | 'exec_failed_to_start' | 'invalid_cwd' | 'allowance_used' | 'overage_paused' | 'spend_cap_reached' | 'overage_unavailable' | 'spend_cap_required' | 'spend_cap_below_minimum' | 'spend_cap_above_plan_price' | 'spend_cap_below_charges' | 'version_mismatch' | 'allocation_mode_not_available' | 'requires_elastic' | 'exceeds_memory_mib' | 'burst_mode_not_supported' | 'burst_not_supported' | 'burst_size_exceeds_plan' | 'not_available' | 'shared_volumes' | 'fence_not_drained' | 'apply_pending' | 'park_failed' | 'workspace_resumed' | 'interrupted' | 'burst_lost' | 'disk_full' | 'apply_failed' | 'reverted' | 'revert_failed' | 'immutable_path_removed' | 'immutable_paths_unsupported_base' | 'read_only_path' | 'host_lost';
 /** A known reason, or any other string the server sends (reasons are open-ended). */
 export type ErrorReason = KnownErrorReason | (string & {});
 export interface ErrorBodyLike {
@@ -106,6 +117,13 @@ export declare class ShardfluxApiError extends Error {
     readonly treeRevision: number | undefined;
     constructor(status: number, body: ErrorBodyLike, source: 'api' | 'cell', retryAfterSeconds?: number, treeRevision?: number);
 }
+/**
+ * The cell gateway's working-at-once refusal (0.14.0+): 429 `quota_exceeded`, retryable, with Retry-After. The gateway
+ * refuses the call at admission, before the workspace is touched, so nothing ran and any request (exec start, stdin,
+ * PTY create, keepalive included) may be sent again. The API's 403 `quota_exceeded` is a different refusal (not
+ * retryable) and never matches.
+ */
+export declare function isWorkingQuotaRefusal(err: unknown): err is ShardfluxApiError;
 /** processful (the default: one VM keeps processes, memory and files) or file_first. */
 export type WorkspaceMode = AppComponents['schemas']['WorkspaceMode'];
 /**
@@ -197,6 +215,7 @@ export declare class OperationTimeoutError extends Error {
  * The operation reached `failed` or `canceled`. A suspend-when-idle that found the workspace active (a tool call after
  * the request, an attached stream) is `canceled` with `errorCode` `workspace_active` (`workspaceActive` true, 0.12.0+):
  * nothing changed and the workspace keeps running. `waitUntilReady()` and `wake()` treat it as running, not as a failure.
+ * `resume_required` (0.13.1+): a fork or snapshot of a workspace whose machine failed; resume it, then call again.
  */
 export declare class OperationFailedError extends Error {
     readonly operation: Operation;

package/dist/errors.js

@@ -40,6 +40,15 @@ export class ShardfluxApiError extends Error {
         this.reason = typeof reason === 'string' ? reason : undefined;
     }
 }
+/**
+ * The cell gateway's working-at-once refusal (0.14.0+): 429 `quota_exceeded`, retryable, with Retry-After. The gateway
+ * refuses the call at admission, before the workspace is touched, so nothing ran and any request (exec start, stdin,
+ * PTY create, keepalive included) may be sent again. The API's 403 `quota_exceeded` is a different refusal (not
+ * retryable) and never matches.
+ */
+export function isWorkingQuotaRefusal(err) {
+    return err instanceof ShardfluxApiError && err.source === 'cell' && err.status === 429 && err.code === 'quota_exceeded' && err.retryable;
+}
 /**
  * The call does not exist for the workspace's mode: 409 `conflict` with details.reason
  * `not_supported_for_mode`, `details.mode` (the workspace's mode) and `details.operation`. File-first workspaces have no
@@ -187,6 +196,7 @@ export class OperationTimeoutError extends Error {
  * The operation reached `failed` or `canceled`. A suspend-when-idle that found the workspace active (a tool call after
  * the request, an attached stream) is `canceled` with `errorCode` `workspace_active` (`workspaceActive` true, 0.12.0+):
  * nothing changed and the workspace keeps running. `waitUntilReady()` and `wake()` treat it as running, not as a failure.
+ * `resume_required` (0.13.1+): a fork or snapshot of a workspace whose machine failed; resume it, then call again.
  */
 export class OperationFailedError extends Error {
     operation;

package/dist/generated/app-api.d.ts

@@ -5630,6 +5630,21 @@ export interface components {
                 } & {
                     [key: string]: unknown;
                 };
+                host_lost?: {
+                    /** @description RFC 3339: when the host failure was detected; the workspace was suspended then. */
+                    detected_at: string;
+                    /**
+                     * @description resume and open: disk when the workspace booted from its disk (files kept, processes and memory not); checkpoint when it was restored from its newest checkpoint (see state_as_of). Absent on a suspend.
+                     * @enum {string}
+                     */
+                    restored_from?: "disk" | "checkpoint";
+                    /** @description restored_from checkpoint: the checkpoint this resume restored. */
+                    restored_checkpoint_id?: string;
+                    /** @description restored_from checkpoint, RFC 3339: when that checkpoint committed; the workspace state is as of then. */
+                    state_as_of?: string;
+                } & {
+                    [key: string]: unknown;
+                };
             } & {
                 [key: string]: unknown;
             };
@@ -28007,8 +28022,8 @@ export interface operations {
                             display_name: string;
                             unit: string;
                             meters: ("cpu_seconds" | "memory_gib_seconds" | "storage_gib_seconds" | "egress_bytes" | "ingress_bytes" | "volume_storage_gib_seconds")[];
-                            /** @description hard_cap: exhausting it refuses opens/resumes (402 allowance_exhausted) and running workspaces are suspended; spare_capacity: no allowance cap (runs on reclaimable spare capacity); reported: shown and notified, never enforced by refusing access; egress_block (outbound_transfer_gb, contracts §18): at 100% outbound internet traffic is blocked by an organization egress override until upgrade, purchase or the next period (workspaces keep running, starts are not refused). */
-                            enforcement: "hard_cap" | "spare_capacity" | "reported" | "egress_block";
+                            /** @description hard_cap: exhausting it refuses opens/resumes (402 allowance_exhausted) and running workspaces are suspended; spare_capacity: no allowance cap (runs on reclaimable spare capacity); reported: shown and notified, never enforced by refusing access; egress_block (outbound_transfer_gb, contracts §18): at 100% outbound internet traffic is blocked by an organization egress override until upgrade, purchase or the next period (workspaces keep running, starts are not refused); storage_block (retained_state_gib, contracts §40): at 100% opening a new workspace key and forking are refused (403 quota_exceeded, details.limit retained_state) until retained state is below the allowance (existing workspaces keep running, waking, suspending and resuming; nothing is deleted or charged). */
+                            enforcement: "hard_cap" | "spare_capacity" | "reported" | "egress_block" | "storage_block";
                             included: number | null;
                             used: number;
                             remaining: number | null;
@@ -28016,8 +28031,8 @@ export interface operations {
                             included_meter_units: number | null;
                             used_meter_units: number | null;
                             remaining_meter_units: number | null;
-                            /** @description ok: below 80 %. warning: 80 % or more of the allowance. exhausted: a hard cap is used up and starts are refused (402 allowance_exhausted; see exhausted_reason). overage: a CPU-hours or RAM GiB-hours allowance is used up while opt-in overage is on and below its spend cap, so starts are admitted and the usage past it is charged (spend_cap). over_allowance: a reported allowance is exceeded (never refused). egress_blocked: the outbound transfer allowance is used up and outbound traffic is blocked. uncapped: runs on spare capacity. not_included: the plan does not define it. */
-                            cap_state: "ok" | "warning" | "exhausted" | "overage" | "over_allowance" | "egress_blocked" | "uncapped" | "not_included";
+                            /** @description ok: below 80 %. warning: 80 % or more of the allowance. exhausted: a hard cap is used up and starts are refused (402 allowance_exhausted; see exhausted_reason). overage: a CPU-hours or RAM GiB-hours allowance is used up while opt-in overage is on and below its spend cap, so starts are admitted and the usage past it is charged (spend_cap). over_allowance: a reported allowance is exceeded (never refused). egress_blocked: the outbound transfer allowance is used up and outbound traffic is blocked. storage_blocked: the retained state allowance is used up and new workspaces and forks are refused (403 quota_exceeded). uncapped: runs on spare capacity. not_included: the plan does not define it. */
+                            cap_state: "ok" | "warning" | "exhausted" | "overage" | "over_allowance" | "egress_blocked" | "storage_blocked" | "uncapped" | "not_included";
                         }[];
                         meters: {
                             /** @enum {string} */
@@ -29062,8 +29077,8 @@ export interface operations {
                             display_name: string;
                             unit: string;
                             meters: ("cpu_seconds" | "memory_gib_seconds" | "storage_gib_seconds" | "egress_bytes" | "ingress_bytes" | "volume_storage_gib_seconds")[];
-                            /** @description hard_cap: exhausting it refuses opens/resumes (402 allowance_exhausted) and running workspaces are suspended; spare_capacity: no allowance cap (runs on reclaimable spare capacity); reported: shown and notified, never enforced by refusing access; egress_block (outbound_transfer_gb, contracts §18): at 100% outbound internet traffic is blocked by an organization egress override until upgrade, purchase or the next period (workspaces keep running, starts are not refused). */
-                            enforcement: "hard_cap" | "spare_capacity" | "reported" | "egress_block";
+                            /** @description hard_cap: exhausting it refuses opens/resumes (402 allowance_exhausted) and running workspaces are suspended; spare_capacity: no allowance cap (runs on reclaimable spare capacity); reported: shown and notified, never enforced by refusing access; egress_block (outbound_transfer_gb, contracts §18): at 100% outbound internet traffic is blocked by an organization egress override until upgrade, purchase or the next period (workspaces keep running, starts are not refused); storage_block (retained_state_gib, contracts §40): at 100% opening a new workspace key and forking are refused (403 quota_exceeded, details.limit retained_state) until retained state is below the allowance (existing workspaces keep running, waking, suspending and resuming; nothing is deleted or charged). */
+                            enforcement: "hard_cap" | "spare_capacity" | "reported" | "egress_block" | "storage_block";
                             included: number | null;
                             used: number;
                             remaining: number | null;
@@ -29071,8 +29086,8 @@ export interface operations {
                             included_meter_units: number | null;
                             used_meter_units: number | null;
                             remaining_meter_units: number | null;
-                            /** @description ok: below 80 %. warning: 80 % or more of the allowance. exhausted: a hard cap is used up and starts are refused (402 allowance_exhausted; see exhausted_reason). overage: a CPU-hours or RAM GiB-hours allowance is used up while opt-in overage is on and below its spend cap, so starts are admitted and the usage past it is charged (spend_cap). over_allowance: a reported allowance is exceeded (never refused). egress_blocked: the outbound transfer allowance is used up and outbound traffic is blocked. uncapped: runs on spare capacity. not_included: the plan does not define it. */
-                            cap_state: "ok" | "warning" | "exhausted" | "overage" | "over_allowance" | "egress_blocked" | "uncapped" | "not_included";
+                            /** @description ok: below 80 %. warning: 80 % or more of the allowance. exhausted: a hard cap is used up and starts are refused (402 allowance_exhausted; see exhausted_reason). overage: a CPU-hours or RAM GiB-hours allowance is used up while opt-in overage is on and below its spend cap, so starts are admitted and the usage past it is charged (spend_cap). over_allowance: a reported allowance is exceeded (never refused). egress_blocked: the outbound transfer allowance is used up and outbound traffic is blocked. storage_blocked: the retained state allowance is used up and new workspaces and forks are refused (403 quota_exceeded). uncapped: runs on spare capacity. not_included: the plan does not define it. */
+                            cap_state: "ok" | "warning" | "exhausted" | "overage" | "over_allowance" | "egress_blocked" | "storage_blocked" | "uncapped" | "not_included";
                         }[];
                         meters: {
                             /** @enum {string} */

package/dist/generated/cell-api.d.ts

@@ -191,7 +191,7 @@ export interface paths {
          *     Client messages are JSON `ExecClientMessage`. Close codes: 1000 after
          *     the `exit` event, 1001 on gateway shutdown (reconnect with offsets),
          *     4401 unauthenticated, 4403 forbidden, 4409 stale epoch / workspace
-         *     busy / workspace not running, 4410 workspace gone, 4429 rate limited,
+         *     busy / workspace not running, 4410 workspace gone, 4429 rate limited / quota exceeded,
          *     4500/4503/4504 internal/unavailable/timeout (4000 + HTTP status; see
          *     "WebSocket close codes" above; the close reason is compact JSON with
          *     the error `code`). A refused upgrade from an allowed Origin is
@@ -265,14 +265,18 @@ export interface paths {
         get?: never;
         put?: never;
         /**
-         * SIGTERM the process group, SIGKILL after grace
-         * @description Sends SIGTERM to the session's process group and SIGKILL after
-         *     `grace_ms` (0 to 60000; 0 or omitted is 5000). Answers when the
-         *     command has ended: at once when SIGTERM ends it, otherwise after the
-         *     grace and the SIGKILL. A `grace_ms` outside the range is `422
-         *     validation_failed`
+         * Stop the command and every process it started (SIGTERM, SIGKILL after grace)
+         * @description Sends SIGTERM to every process the command started, background and
+         *     setsid processes included, and SIGKILL after `grace_ms` (0 to 60000;
+         *     0 or omitted is 5000). Answers when they have all exited: at once when
+         *     SIGTERM ends them, otherwise after the grace and the SIGKILL. A lost
+         *     command is stopped the same way; a command that already exited keeps
+         *     its result and what it left running is stopped. `503
+         *     dependency_unavailable` (retryable) while a process cannot exit yet.
+         *     Template versions before python-node-browser 13, ubuntu-24.04 4,
+         *     ubuntu-22.04 2 and debian-12 2 signal the command's process group and
+         *     return an ended session as it is. A `grace_ms` outside the range is `422 validation_failed`
          *     (`details.field` `grace_ms`, `details.minimum`, `details.maximum`).
-         *     An ended session is returned as it is.
          */
         post: operations["execCancel"];
         delete?: never;
@@ -876,7 +880,7 @@ export type webhooks = Record<string, never>;
 export interface components {
     schemas: {
         /** @enum {string} */
-        ErrorCode: "bad_request" | "unauthenticated" | "forbidden" | "not_found" | "conflict" | "stale_epoch" | "workspace_not_running" | "validation_failed" | "payload_too_large" | "range_not_satisfiable" | "rate_limited" | "capacity_pending" | "dependency_unavailable" | "timeout" | "internal_error" | "service_unavailable" | "workspace_busy" | "burst_unavailable" | "burst_apply_failed";
+        ErrorCode: "bad_request" | "unauthenticated" | "forbidden" | "not_found" | "conflict" | "stale_epoch" | "workspace_not_running" | "validation_failed" | "payload_too_large" | "range_not_satisfiable" | "rate_limited" | "capacity_pending" | "dependency_unavailable" | "timeout" | "internal_error" | "service_unavailable" | "workspace_busy" | "burst_unavailable" | "burst_apply_failed" | "quota_exceeded";
         ErrorBody: {
             error: {
                 code: components["schemas"]["ErrorCode"];

package/dist/http.d.ts

@@ -1,5 +1,5 @@
 import type { RetryRecord } from './progress.js';
-export declare const SDK_VERSION = "0.13.0";
+export declare const SDK_VERSION = "0.13.1";
 export interface RequestOptions {
     query?: Record<string, string | number | boolean | undefined | null>;
     json?: unknown;
@@ -10,6 +10,11 @@ export interface RequestOptions {
     idempotencyKey?: string;
     /** The request has no effect a retry could duplicate (a read-only POST such as files/search): retried like a GET. */
     idempotent?: boolean;
+    /**
+     * false: the cell gateway's 429 `quota_exceeded` surfaces at once instead of being retried (the wake hint, which
+     * never waits). Default true.
+     */
+    quotaRetry?: boolean;
     signal?: AbortSignal;
     /** Override the client's default request timeout (ms); 0 disables it (streams). */
     timeoutMs?: number;
@@ -33,13 +38,45 @@ export interface HttpOptions {
     onSuccess?: (() => void) | undefined;
 }
 export declare const defaultSleep: (ms: number) => Promise<void>;
+/**
+ * How long an idle pooled connection stays reusable (ms): 5 minutes, instead of undici's 4 s default, so the request
+ * after an agent's usual pause between tool calls (15 s to a few minutes) skips a new TCP and TLS handshake. Far below
+ * the 3600 s idle timeout of the load balancer in front of the API and the cell endpoints, so the server side never
+ * closes a connection for idleness while the client still treats it as reusable; below the AWS NAT gateway's 350 s idle
+ * timeout; and the same limit Chrome keeps for used idle sockets. Also caps a server's `Keep-Alive: timeout` hint.
+ */
+export declare const KEEP_ALIVE_TIMEOUT_MS = 300000;
+/**
+ * TCP keepalive probes start after a pooled socket has been idle this long (ms), so NATs and firewalls with shorter
+ * idle timeouts than the pool's (Azure SNAT: 4 minutes) keep the connection instead of silently dropping it. Equal to
+ * undici's own default, set explicitly so it is visible and tested.
+ */
+export declare const TCP_KEEPALIVE_INITIAL_DELAY_MS = 60000;
+/** The private pool's undici Agent options (exported for tests). */
+export declare const POOL_OPTIONS: {
+    readonly allowH2: false;
+    readonly pipelining: 1;
+    readonly keepAliveTimeout: 300000;
+    readonly keepAliveMaxTimeout: 300000;
+    readonly connect: {
+        readonly keepAlive: true;
+        readonly keepAliveInitialDelay: 60000;
+    };
+};
 /**
  * Reuses TLS connections. Node 26's bundled undici 8.9 can stall reused connections: use pinned undici with a
- * private HTTP/1.1-only dispatcher. Other runtimes retain native fetch. SHARDFLUX_HTTP_KEEPALIVE=0 forces close
- * for diagnosis; =1 retains the historical opt-in to native pooling. An explicitly supplied fetch is untouched.
+ * private HTTP/1.1-only dispatcher whose idle connections stay reusable for KEEP_ALIVE_TIMEOUT_MS. Other runtimes
+ * retain native fetch. SHARDFLUX_HTTP_KEEPALIVE=0 forces close for diagnosis; =1 retains the historical opt-in to
+ * native pooling. An explicitly supplied fetch is untouched. Idle pooled sockets do not keep the process alive.
  */
 export declare function defaultFetch(env?: Record<string, string | undefined> | undefined, undiciVersion?: string | undefined): typeof fetch;
 export declare function buildUrl(base: string, path: string, query?: RequestOptions['query']): string;
+/**
+ * True when fetch failed because its connection closed or reset before a response arrived: what a request sees when
+ * it went out on a pooled connection the other side closed while it sat idle. Connect failures (ECONNREFUSED,
+ * ENOTFOUND, UND_ERR_CONNECT_TIMEOUT), timeouts and aborts are not this.
+ */
+export declare function isStaleConnectionError(err: unknown): boolean;
 /** `X-Tree-Revision` (file-first workspaces): a non-negative integer, else null. */
 export declare function treeRevisionOf(headers: Headers): number | null;
 /** Turns a non-2xx response into a ShardfluxApiError (or a protocol error for undocumented bodies). */

package/dist/http.js

@@ -4,18 +4,44 @@
  * retries only where a retry cannot duplicate an effect (safe methods,
  * requests carrying an Idempotency-Key, and read-only POSTs marked
  * `idempotent`). A retryable 429/502/503/504 (e.g. 503 `host_capacity`, 503
- * `wake_failed`) waits `Retry-After` (at most 5 s) before the retry.
+ * `wake_failed`) waits `Retry-After` (at most 5 s) before the retry. Such a request whose connection closed before
+ * any response (a pooled connection gone stale while idle) is sent again at once, once, outside `maxRetries`. The cell
+ * gateway's 429 `quota_exceeded` (every working slot of the plan is taken) is retried for any request (0.14.0+): it is
+ * refused at admission, so nothing ran.
  */
-import { ShardfluxApiError, ShardfluxProtocolError, apiError, isErrorBody } from "./errors.js";
+import { ShardfluxApiError, ShardfluxProtocolError, apiError, isErrorBody, isWorkingQuotaRefusal } from "./errors.js";
 import { describeFailure } from "./progress.js";
-export const SDK_VERSION = '0.13.0';
+export const SDK_VERSION = '0.13.1';
 export const defaultSleep = (ms) => new Promise((r) => setTimeout(r, ms));
+/**
+ * How long an idle pooled connection stays reusable (ms): 5 minutes, instead of undici's 4 s default, so the request
+ * after an agent's usual pause between tool calls (15 s to a few minutes) skips a new TCP and TLS handshake. Far below
+ * the 3600 s idle timeout of the load balancer in front of the API and the cell endpoints, so the server side never
+ * closes a connection for idleness while the client still treats it as reusable; below the AWS NAT gateway's 350 s idle
+ * timeout; and the same limit Chrome keeps for used idle sockets. Also caps a server's `Keep-Alive: timeout` hint.
+ */
+export const KEEP_ALIVE_TIMEOUT_MS = 300_000;
+/**
+ * TCP keepalive probes start after a pooled socket has been idle this long (ms), so NATs and firewalls with shorter
+ * idle timeouts than the pool's (Azure SNAT: 4 minutes) keep the connection instead of silently dropping it. Equal to
+ * undici's own default, set explicitly so it is visible and tested.
+ */
+export const TCP_KEEPALIVE_INITIAL_DELAY_MS = 60_000;
+/** The private pool's undici Agent options (exported for tests). */
+export const POOL_OPTIONS = {
+    allowH2: false,
+    pipelining: 1,
+    keepAliveTimeout: KEEP_ALIVE_TIMEOUT_MS,
+    keepAliveMaxTimeout: KEEP_ALIVE_TIMEOUT_MS,
+    connect: { keepAlive: true, keepAliveInitialDelay: TCP_KEEPALIVE_INITIAL_DELAY_MS },
+};
 /** One private HTTP/1.1 pool on Node 26+, shared by SDK clients. No global dispatcher changes. */
 let nodeFetch;
 /**
  * Reuses TLS connections. Node 26's bundled undici 8.9 can stall reused connections: use pinned undici with a
- * private HTTP/1.1-only dispatcher. Other runtimes retain native fetch. SHARDFLUX_HTTP_KEEPALIVE=0 forces close
- * for diagnosis; =1 retains the historical opt-in to native pooling. An explicitly supplied fetch is untouched.
+ * private HTTP/1.1-only dispatcher whose idle connections stay reusable for KEEP_ALIVE_TIMEOUT_MS. Other runtimes
+ * retain native fetch. SHARDFLUX_HTTP_KEEPALIVE=0 forces close for diagnosis; =1 retains the historical opt-in to
+ * native pooling. An explicitly supplied fetch is untouched. Idle pooled sockets do not keep the process alive.
  */
 export function defaultFetch(env = globalThis.process?.env, undiciVersion = globalThis.process?.versions?.undici) {
     // Why a fresh connection on Node 26 (its bundled HTTP client, measured): docs/progress/startup-latency.md.
@@ -30,7 +56,7 @@ export function defaultFetch(env = globalThis.process?.env, undiciVersion = glob
         return base;
     return async (input, init) => {
         nodeFetch ??= import('undici').then(({ Agent, fetch: pooledFetch }) => {
-            const dispatcher = new Agent({ allowH2: false, pipelining: 1 });
+            const dispatcher = new Agent({ ...POOL_OPTIONS, connect: { ...POOL_OPTIONS.connect } });
             return async (input, init) => {
                 const response = await pooledFetch(input, { ...init, dispatcher });
                 // Web-standard runtime shape; undici and DOM iterator declarations differ.
@@ -49,6 +75,25 @@ export function buildUrl(base, path, query) {
     }
     return u.toString();
 }
+/** Codes (on fetch's `cause` chain) of a connection that closed or reset before the response arrived. */
+const STALE_CONNECTION_CODES = new Set(['UND_ERR_SOCKET', 'ECONNRESET', 'EPIPE', 'UND_ERR_CLOSED']);
+/**
+ * True when fetch failed because its connection closed or reset before a response arrived: what a request sees when
+ * it went out on a pooled connection the other side closed while it sat idle. Connect failures (ECONNREFUSED,
+ * ENOTFOUND, UND_ERR_CONNECT_TIMEOUT), timeouts and aborts are not this.
+ */
+export function isStaleConnectionError(err) {
+    if (!(err instanceof TypeError))
+        return false;
+    const seen = new Set();
+    for (let c = err.cause; typeof c === 'object' && c !== null && !seen.has(c); c = c.cause) {
+        seen.add(c);
+        const code = c.code;
+        if (typeof code === 'string' && STALE_CONNECTION_CODES.has(code))
+            return true;
+    }
+    return false;
+}
 function retryAfterSeconds(res) {
     const v = res.headers.get('retry-after');
     if (v === null)
@@ -89,6 +134,9 @@ export class HttpClient {
         const safe = method === 'GET' || method === 'HEAD';
         const retriable = safe || init.idempotencyKey !== undefined || init.idempotent === true;
         const sleep = this.opts.sleep ?? defaultSleep;
+        // Retries counted against maxRetries (they also set the backoff); `attempt` counts every send.
+        let counted = 0;
+        let staleReplayed = false;
         for (let attempt = 0;; attempt += 1) {
             const headers = {
                 accept: init.accept ?? 'application/json',
@@ -122,9 +170,20 @@ export class HttpClient {
             catch (err) {
                 if (init.signal?.aborted)
                     throw err;
-                if (!retriable || attempt >= this.opts.maxRetries)
+                if (!retriable)
+                    throw err;
+                if (!staleReplayed && !signal?.aborted && isStaleConnectionError(err)) {
+                    // The connection closed before any response, typically a pooled one the server or a NAT closed while idle:
+                    // send again at once on a fresh connection, once per request and outside maxRetries (as Go's net/http
+                    // replays a request that failed on a reused connection when it is idempotent or carries an Idempotency-Key).
+                    staleReplayed = true;
+                    init.onRetry?.({ request: `${method} ${path}`, attempt: attempt + 1, cause: describeFailure(err), delayMs: 0 });
+                    continue;
+                }
+                if (counted >= this.opts.maxRetries)
                     throw err;
-                const delayMs = Math.min(2_000, 200 * 2 ** attempt);
+                const delayMs = Math.min(2_000, 200 * 2 ** counted);
+                counted += 1;
                 init.onRetry?.({ request: `${method} ${path}`, attempt: attempt + 1, cause: describeFailure(err, headersAbort?.signal.aborted ? headersMs : timeoutMs), delayMs });
                 await sleep(delayMs);
                 continue;
@@ -146,10 +205,15 @@ export class HttpClient {
             const error = await errorFrom(res, this.opts.source);
             const retryableStatus = res.status === 429 || res.status === 502 || res.status === 503 || res.status === 504;
             const retryableError = error instanceof ShardfluxApiError ? error.retryable && retryableStatus : retryableStatus;
-            if (!retriable || !retryableError || attempt >= this.opts.maxRetries)
+            // The working-at-once quota (cell 429 quota_exceeded): the gateway refuses the call at admission, before the
+            // workspace is touched, so nothing ran and even a non-idempotent request (exec start, stdin, PTY create,
+            // keepalive) is safe to send again. Same Retry-After wait and retry budget as every other retry.
+            const admissionRefusal = init.quotaRetry !== false && isWorkingQuotaRefusal(error);
+            if ((!retriable && !admissionRefusal) || !retryableError || counted >= this.opts.maxRetries)
                 throw error;
             const hinted = error instanceof ShardfluxApiError ? error.retryAfterSeconds : undefined;
-            const delayMs = Math.min(5_000, hinted !== undefined ? hinted * 1000 : 200 * 2 ** attempt);
+            const delayMs = Math.min(5_000, hinted !== undefined ? hinted * 1000 : 200 * 2 ** counted);
+            counted += 1;
             init.onRetry?.({ request: `${method} ${path}`, attempt: attempt + 1, cause: describeFailure(error), delayMs });
             await sleep(delayMs);
         }

package/dist/index.d.ts

@@ -17,8 +17,8 @@ export type { components as CellComponents, paths as CellPaths } from './generat
 export { BillingApi, Shardflux, WorkspacesApi, fetchBillingCatalog, pickByKey } from './client.js';
 export type { AgentSession, AllocationMode, BillingCatalog, BillingSubscription, Caps, CheckoutSession, DiskLayout, Entitlements, FindByKeyOptions, ForkTarget, Invoice, InvoicePage, LifetimeFilter, ListParams, Me, OpenParams, OpenResponse, Operation, Page, PortalSession, PurposeFilter, ResetWorkspaceBody, ResumeAnswer, ResumeRequestOptions, ResumeResponse, ShardfluxOptions, SuspendRequest, SuspendWhenIdleOptions, SuspendWhenIdleResponse, SuspendWhenIdleResult, WaitOptions, WorkspaceInputs, WorkspaceLifetime, WorkspaceMemory, WorkspaceOrigin, WorkspacePurpose, WorkspaceView, } from './client.js';
 export type { FinishedOperation, ForkOptions, LifecycleOptions, ResumeOptions, SuspendOptions, WaitedForkOptions, WaitedLifecycleOptions, WaitedResumeOptions, WaitedSuspendOptions } from './lifecycle.js';
-export { durabilityOf, formatTiming, isDurable, lostSuspendOf } from './progress.js';
-export type { Durability, LostSuspend, LifecycleAction, LifecyclePhase, LifecycleTiming, ProgressEvent, ProgressListener, RetryRecord, ServerTiming, TimingOutcome, TimingPhase, } from './progress.js';
+export { durabilityOf, formatTiming, hostLostOf, isDurable, lostSuspendOf } from './progress.js';
+export type { ColdBootReason, Durability, HostLost, LostSuspend, LifecycleAction, LifecyclePhase, LifecycleTiming, ProgressEvent, ProgressListener, RetryRecord, ServerTiming, TimingOutcome, TimingPhase, } from './progress.js';
 export { FEEDBACK_CATEGORIES, FEEDBACK_MESSAGE_MAX_LENGTH } from './feedback.js';
 export type { AccountFeedbackParams, FeedbackCategory, FeedbackContext, FeedbackReceipt, SendFeedbackParams } from './feedback.js';
 export { UsageApi } from './usage.js';

package/dist/index.js

@@ -1,5 +1,5 @@
 export { BillingApi, Shardflux, WorkspacesApi, fetchBillingCatalog, pickByKey } from "./client.js";
-export { durabilityOf, formatTiming, isDurable, lostSuspendOf } from "./progress.js";
+export { durabilityOf, formatTiming, hostLostOf, isDurable, lostSuspendOf } from "./progress.js";
 export { FEEDBACK_CATEGORIES, FEEDBACK_MESSAGE_MAX_LENGTH } from "./feedback.js";
 export { UsageApi } from "./usage.js";
 export { TemplateBuildTimeoutError, TemplateBuildsApi, TemplateDraftApi, TemplatePackagesApi, TemplateUploadError, TemplateUploadsApi, TemplateVersionTestInstancesApi, TemplateVersionsApi, TemplatesApi, buildSettled, saveAsTemplateBody, } from "./templates.js";

package/dist/progress.d.ts

@@ -74,9 +74,10 @@ export interface ServerTiming {
     warmFallback: string | null;
     /**
      * `result.resume_path`: `local_cache`, `prestaged` (copied to this host ahead of the resume),
-     * `download` (the checkpoint had to be fetched first), `cold_boot` (0.11.0+: the saved disk was booted after a
-     * platform runtime change; processes restarted; see `memoryRestored`) or `reset_blank_layer` (the first start after a
-     * reset).
+     * `download` (the checkpoint had to be fetched first), `cold_boot` (0.11.0+: the saved disk was booted instead of
+     * restoring memory, see `coldBootReason`; processes restarted; see `memoryRestored`), `reset_blank_layer` (the first
+     * start after a reset) or `thaw` (0.13.1+: resumed while its suspend was still being written; the same VM continued
+     * in place).
      */
     resumePath: string | null;
     /**
@@ -87,10 +88,11 @@ export interface ServerTiming {
      */
     memoryRestored?: boolean | null;
     /**
-     * `result.cold_boot_reason` (0.11.0+), with `resumePath` `cold_boot`: why the memory could not be restored, e.g.
-     * `runtime_changed` (the platform's VM runtime changed after the suspend). Null otherwise.
+     * `result.cold_boot_reason` (0.11.0+), with `resumePath` `cold_boot`: why the memory could not be restored:
+     * `runtime_changed` (the platform's VM runtime changed after the suspend) or `host_lost` (0.13.1+: the machine the
+     * workspace ran on failed; it booted from its disk, files kept, see `hostLost`). Null otherwise.
      */
-    coldBootReason?: string | null;
+    coldBootReason?: ColdBootReason | null;
     /**
      * `result.durable` (0.12.0+), suspend and fork: `true` when the capture is in durable storage, `false` while its
      * durable copy is being written (see `durability`). `null` when the result does not say (other kinds, an older API).
@@ -105,6 +107,12 @@ export interface ServerTiming {
      * checkpoint before it (`restoredCheckpointId`, state as of `stateAsOf`). Null otherwise.
      */
     lostSuspend?: LostSuspend | null;
+    /**
+     * `result.host_lost` (0.13.1+): the machine the workspace ran on failed. A resume (or open) that restored the
+     * workspace says from what (`restoredFrom` `disk`, or `checkpoint` with `stateAsOf`); a suspend that found it so
+     * succeeds with only `detectedAt`. Null otherwise.
+     */
+    hostLost?: HostLost | null;
     /** `result.boot_to_ready_ms`: VM start until the guest agent answered. */
     bootToReadyMs: number | null;
     /** `result.host_timings_ms`: the host's own steps (restore: load, after_restore, ready, …). */
@@ -143,6 +151,30 @@ export interface LostSuspend {
     /** The restored state is as of this time (RFC 3339). */
     stateAsOf: string | null;
 }
+/**
+ * `result.cold_boot_reason` (0.11.0+): `runtime_changed` (the platform's VM runtime changed after the suspend) or
+ * `host_lost` (0.13.1+: the machine the workspace ran on failed). Any other string is a reason this version does not
+ * know.
+ */
+export type ColdBootReason = 'runtime_changed' | 'host_lost' | (string & {});
+/**
+ * `result.host_lost` (0.13.1+), camelCased: the machine the workspace ran on failed. The workspace was moved to
+ * `suspended` at that moment, and its next use (a resume, a tool call's wake, an open) restored it. See the lifecycle
+ * reference.
+ */
+export interface HostLost {
+    /** When the failure was detected (RFC 3339). */
+    detectedAt: string | null;
+    /**
+     * What the resume restored: `disk` (the workspace's own disk, files kept; `coldBootReason` `host_lost`, processes
+     * restarted) or `checkpoint` (its newest checkpoint, see `stateAsOf`). Null on a suspend's result.
+     */
+    restoredFrom: 'disk' | 'checkpoint' | null;
+    /** `checkpoint`: the checkpoint the resume restored. */
+    restoredCheckpointId: string | null;
+    /** `checkpoint`: the restored state is as of this time (RFC 3339); changes after it are not in the workspace. */
+    stateAsOf: string | null;
+}
 /**
  * The durable copy of a suspend or fork operation (0.12.0+): `result.durability` camelCased, or null when the result has
  * none (the suspend was stored durably before it completed, or another kind).
@@ -150,6 +182,11 @@ export interface LostSuspend {
 export declare function durabilityOf(op: Pick<Operation, 'result'>): Durability | null;
 /** A resume operation's `result.lost_suspend` camelCased (0.12.0+), or null. */
 export declare function lostSuspendOf(op: Pick<Operation, 'result'>): LostSuspend | null;
+/**
+ * An operation's `result.host_lost` camelCased (0.13.1+), or null: on a resume or open that restored a workspace whose
+ * machine failed, and on a suspend that found it so (`detectedAt` only).
+ */
+export declare function hostLostOf(op: Pick<Operation, 'result'>): HostLost | null;
 /**
  * Whether a suspend or fork operation's capture is in durable storage (0.12.0+): `result.durable`, else `true` for a
  * succeeded suspend or fork whose result predates the field, else null.
@@ -255,7 +292,10 @@ export declare function traced<T>(trace: Trace, fn: () => Promise<T>): Promise<T
  *
  * A call that retried a request adds `retries: <n> (<request>, <cause>, after <delay>)`.
  * A resume that booted the workspace instead of restoring its memory (0.11.0+) says so in the server line:
- * `resume from cold_boot: processes restarted (runtime_changed)`.
+ * `resume from cold_boot: processes restarted (runtime_changed)`, or `(host_lost)` (0.13.1+) when the machine the
+ * workspace ran on failed and it booted from its disk. A resume that restored the newest checkpoint after such a
+ * failure adds `restored checkpoint <id> (host_lost; state as of <time>)`; a suspend that found the machine failed
+ * adds `host_lost (detected <time>)`.
  */
 export declare function formatTiming(t: LifecycleTiming): string;
 export {};

package/dist/progress.js

@@ -40,6 +40,22 @@ export function lostSuspendOf(op) {
         stateAsOf: str(r.state_as_of),
     };
 }
+/**
+ * An operation's `result.host_lost` camelCased (0.13.1+), or null: on a resume or open that restored a workspace whose
+ * machine failed, and on a suspend that found it so (`detectedAt` only).
+ */
+export function hostLostOf(op) {
+    const h = op.result?.host_lost;
+    if (typeof h !== 'object' || h === null || Array.isArray(h))
+        return null;
+    const r = h;
+    return {
+        detectedAt: str(r.detected_at),
+        restoredFrom: r.restored_from === 'disk' || r.restored_from === 'checkpoint' ? r.restored_from : null,
+        restoredCheckpointId: str(r.restored_checkpoint_id),
+        stateAsOf: str(r.state_as_of),
+    };
+}
 /**
  * Whether a suspend or fork operation's capture is in durable storage (0.12.0+): `result.durable`, else `true` for a
  * succeeded suspend or fork whose result predates the field, else null.
@@ -122,6 +138,7 @@ export function serverTiming(op) {
         suspendPath: str(r.suspend_path),
         durability: durabilityOf(op),
         lostSuspend: lostSuspendOf(op),
+        hostLost: hostLostOf(op),
         bootToReadyMs: num(r.boot_to_ready_ms),
         hostTimingsMs,
     };
@@ -275,6 +292,19 @@ export async function traced(trace, fn) {
     }
 }
 const fmt = (ms) => (ms === null ? '?' : ms < 1000 ? `${Math.round(ms)} ms` : `${(ms / 1000).toFixed(2)} s`);
+/**
+ * The server-line part of `result.host_lost` (0.13.1+): the checkpoint a resume restored, or a suspend's detection time.
+ * A resume from the disk says nothing more here: its cold boot reads `processes restarted (host_lost)`.
+ */
+function hostLostText(h) {
+    if (!h)
+        return null;
+    if (h.restoredFrom === 'checkpoint')
+        return `restored checkpoint ${h.restoredCheckpointId ?? '?'} (host_lost; state as of ${h.stateAsOf ?? '?'})`;
+    if (h.restoredFrom === null)
+        return `host_lost${h.detectedAt ? ` (detected ${h.detectedAt})` : ''}`;
+    return null;
+}
 /**
  * A human-readable account of a timing, for logs and bug reports. A resume in production:
  *
@@ -285,7 +315,10 @@ const fmt = (ms) => (ms === null ? '?' : ms < 1000 ? `${Math.round(ms)} ms` : `$
  *
  * A call that retried a request adds `retries: <n> (<request>, <cause>, after <delay>)`.
  * A resume that booted the workspace instead of restoring its memory (0.11.0+) says so in the server line:
- * `resume from cold_boot: processes restarted (runtime_changed)`.
+ * `resume from cold_boot: processes restarted (runtime_changed)`, or `(host_lost)` (0.13.1+) when the machine the
+ * workspace ran on failed and it booted from its disk. A resume that restored the newest checkpoint after such a
+ * failure adds `restored checkpoint <id> (host_lost; state as of <time>)`; a suspend that found the machine failed
+ * adds `host_lost (detected <time>)`.
  */
 export function formatTiming(t) {
     const ids = [t.workspaceId ? `workspace ${t.workspaceId}` : null, t.operationId ? `operation ${t.operationId}` : null].filter(Boolean).join(', ');
@@ -308,6 +341,7 @@ export function formatTiming(t) {
             s.startPath ? `start ${s.startPath}${s.warmFallback ? ` (warm fallback: ${s.warmFallback})` : ''}` : null,
             s.resumePath ? `resume from ${s.resumePath}${s.memoryRestored === false ? `: processes restarted${s.coldBootReason ? ` (${s.coldBootReason})` : ''}` : ''}` : null,
             s.lostSuspend ? `restored ${s.lostSuspend.restoredCheckpointId ?? 'no checkpoint'} (latest suspend ${s.lostSuspend.checkpointId} ${s.lostSuspend.reason ?? 'lost'})` : null,
+            hostLostText(s.hostLost),
             s.suspendPath === 'local_commit' ? 'sealed on host' : null,
             s.durability?.state === 'durable' ? `durable${s.durability.localCommitToDurableMs !== null ? ` ${fmt(s.durability.localCommitToDurableMs)} later` : ''}` : s.durability ? `durable copy ${s.durability.state}` : null,
             s.bootToReadyMs !== null ? `boot to ready ${fmt(s.bootToReadyMs)}` : null,

package/dist/usage.d.ts

@@ -56,7 +56,9 @@ export declare class UsageApi {
     constructor(ctx: () => ClientContext);
     /**
      * Current-period usage per meter, allowances with enforcement and cap state (`overage` while opt-in overage covers
-     * usage past a CPU-hours or RAM GiB-hours allowance), measurement freshness, `exhausted_reason` (the 402
+     * usage past a CPU-hours or RAM GiB-hours allowance; `storage_blocked` (0.14.0+, enforcement `storage_block`) while
+     * the Retained state allowance is used up: opening a new key and forking are refused with 403 `quota_exceeded`,
+     * details.limit `retained_state`), measurement freshness, `exhausted_reason` (the 402
      * allowance_exhausted reason while starts are refused: `allowance_used`, `overage_paused`, `spend_cap_reached`) and
      * `spend_cap` (0.10.0).
      */

package/dist/usage.js

@@ -9,7 +9,9 @@ export class UsageApi {
     }
     /**
      * Current-period usage per meter, allowances with enforcement and cap state (`overage` while opt-in overage covers
-     * usage past a CPU-hours or RAM GiB-hours allowance), measurement freshness, `exhausted_reason` (the 402
+     * usage past a CPU-hours or RAM GiB-hours allowance; `storage_blocked` (0.14.0+, enforcement `storage_block`) while
+     * the Retained state allowance is used up: opening a new key and forking are refused with 403 `quota_exceeded`,
+     * details.limit `retained_state`), measurement freshness, `exhausted_reason` (the 402
      * allowance_exhausted reason while starts are refused: `allowance_used`, `overage_paused`, `spend_cap_reached`) and
      * `spend_cap` (0.10.0).
      */

package/dist/workspace.d.ts

@@ -67,7 +67,9 @@ export declare class Workspace {
      * woke the workspace), or a lifecycle call with `wait`. Null for handles from get()/list() until such a call.
      * `formatTiming(workspace.lastTiming)` prints it. After a resume or wake, `server.memoryRestored === false` (0.11.0+)
      * means the workspace booted from its saved disk instead of restoring its memory (`server.resumePath` `cold_boot`,
-     * reason in `server.coldBootReason`): files are as of the suspend, running processes were restarted.
+     * reason in `server.coldBootReason`): files are as of the suspend, running processes were restarted. Reason
+     * `host_lost` (0.13.1+): the machine the workspace ran on failed and it booted from its disk, files kept;
+     * `server.hostLost` says what the resume restored (`hostLostOf(op)` on the operation).
      */
     get lastTiming(): LifecycleTiming | null;
     get id(): string;
@@ -174,6 +176,8 @@ export declare class Workspace {
      * resolve once it has FINISHED, with `workspace.state` then `suspended`. That is as soon as the workspace is sealed on
      * its host, typically in a few hundred ms. `result.durable` (also `lastTiming.server.durable`, 0.12.0+) turns true
      * when the copy lands in durable storage, typically within a second; `{ durable: true }` resolves only then.
+     * A suspend that finds the machine the workspace ran on failed succeeds at once (0.13.1+): `result.durable` is true
+     * and `result.host_lost` (`hostLostOf(op)`) has `detectedAt`; the next resume restores the workspace.
      *
      *   await workspace.suspend({ wait: true });
      *   await workspace.suspend({ durable: true }); // 0.12.0+: also wait for the durable copy
@@ -205,11 +209,16 @@ export declare class Workspace {
      * (also `lastTiming.server.memoryRestored`, 0.11.0+) is false when the resume booted the saved disk instead
      * (`resume_path` `cold_boot`): files kept, processes restarted. `result.lost_suspend` (also
      * `lastTiming.server.lostSuspend`, `lostSuspendOf(op)`, 0.12.0+) names a suspend this resume could not restore and the
-     * checkpoint it restored instead (see the lifecycle reference).
+     * checkpoint it restored instead (see the lifecycle reference). `result.host_lost` (also `lastTiming.server.hostLost`,
+     * `hostLostOf(op)`, 0.13.1+): the machine the workspace ran on failed and this resume restored it, from its disk
+     * (`cold_boot_reason` `host_lost`) or from its newest checkpoint (`state_as_of`).
      */
     resume(opts: WaitedResumeOptions): Promise<FinishedOperation>;
     resume(opts?: ResumeOptions): Promise<Operation>;
-    /** Takes a snapshot. Resolves when it is REQUESTED; with `{ wait: true }`, once it is taken. */
+    /**
+     * Takes a snapshot. Resolves when it is REQUESTED; with `{ wait: true }`, once it is taken. After the machine the
+     * workspace ran on failed, resume it first: until then the snapshot fails `resume_required` (0.13.1+).
+     */
     snapshot(opts: WaitedLifecycleOptions & {
         label?: string;
     }): Promise<FinishedOperation>;
@@ -218,7 +227,8 @@ export declare class Workspace {
     }): Promise<Operation>;
     /**
      * Forks into a new key. Resolves when the fork is REQUESTED (the copy's handle is returned at once); with
-     * `{ wait: true }`, once the copy exists, with its handle refreshed.
+     * `{ wait: true }`, once the copy exists, with its handle refreshed. After the machine the workspace ran on failed,
+     * resume it first: until then the fork fails `resume_required` (0.13.1+).
      */
     fork(target: ForkTarget, opts: WaitedForkOptions): Promise<{
         operation: FinishedOperation;
@@ -301,8 +311,9 @@ export declare class Workspace {
      * that does not hold the request answers at once; the wake then waits for the operation and reads the view.
      *
      * A wake whose resume could not restore the workspace's memory (0.11.0+; the platform's VM runtime changed after the
-     * suspend) still resolves `true`: the workspace runs from its saved disk, with every process restarted. `lastTiming`
-     * and the `done` progress event say so (`server.memoryRestored === false`, `server.resumePath` `cold_boot`).
+     * suspend, or, 0.13.1+, the machine the workspace ran on failed) still resolves `true`: the workspace runs from its
+     * saved disk, with every process restarted. `lastTiming` and the `done` progress event say so
+     * (`server.memoryRestored === false`, `server.resumePath` `cold_boot`, `server.coldBootReason`).
      */
     wake(opts?: WakeOptions): Promise<boolean>;
     /**

package/dist/workspace.js

@@ -44,7 +44,9 @@ export class Workspace {
      * woke the workspace), or a lifecycle call with `wait`. Null for handles from get()/list() until such a call.
      * `formatTiming(workspace.lastTiming)` prints it. After a resume or wake, `server.memoryRestored === false` (0.11.0+)
      * means the workspace booted from its saved disk instead of restoring its memory (`server.resumePath` `cold_boot`,
-     * reason in `server.coldBootReason`): files are as of the suspend, running processes were restarted.
+     * reason in `server.coldBootReason`): files are as of the suspend, running processes were restarted. Reason
+     * `host_lost` (0.13.1+): the machine the workspace ran on failed and it booted from its disk, files kept;
+     * `server.hostLost` says what the resume restored (`hostLostOf(op)` on the operation).
      */
     get lastTiming() {
         return this.#lastTiming ?? this.#openTrace?.finished ?? null;
@@ -438,8 +440,9 @@ export class Workspace {
      * that does not hold the request answers at once; the wake then waits for the operation and reads the view.
      *
      * A wake whose resume could not restore the workspace's memory (0.11.0+; the platform's VM runtime changed after the
-     * suspend) still resolves `true`: the workspace runs from its saved disk, with every process restarted. `lastTiming`
-     * and the `done` progress event say so (`server.memoryRestored === false`, `server.resumePath` `cold_boot`).
+     * suspend, or, 0.13.1+, the machine the workspace ran on failed) still resolves `true`: the workspace runs from its
+     * saved disk, with every process restarted. `lastTiming` and the `done` progress event say so
+     * (`server.memoryRestored === false`, `server.resumePath` `cold_boot`, `server.coldBootReason`).
      */
     async wake(opts = {}) {
         // A file-first workspace runs from creation and is never suspended: nothing to wake.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@shardflux/sdk",
-  "version": "0.13.0",
+  "version": "0.13.1",
   "type": "module",
   "description": "Shardflux TypeScript SDK: open persistent agent workspaces by key and give your agent workspace tools (exec, files, processes, PTY, git, browser).",
   "license": "Apache-2.0",