@shardflux/cli 0.8.0 → 0.8.1

package/CHANGELOG.md

@@ -3,7 +3,43 @@
 Every command and option the README shows is available from the version named here. Breaking changes ship in minor
 releases and are marked **Breaking**.
 
-## 0.8.0 (not yet published)
+## 0.9.0 (not yet published)
+
+Needs `@shardflux/sdk` 0.14.0 or later.
+
+### Workspaces working at once
+
+- A command whose workspace call finds every workspace the plan allows to work at once busy is sent again after
+  `Retry-After` (the SDK's retries; the call did not run). A refusal that lasts exits 1 with 429 `quota_exceeded`
+  (`details.limit` `concurrent_workspaces`) and the hint `every workspace your plan allows to work at once is busy
+  and the call did not run (it was already retried); retry in 2 s, or upgrade: shard billing plans`.
+- The API's 403 `quota_exceeded` with `details.limit` `retained_state` (opening a new key or forking while the
+  plan's Retained state is used up) hints `delete workspaces you no longer need (shard ws delete <key> --yes), or
+  upgrade: shard billing plans`. Neither suggests `shard feedback`.
+- `--json` errors carry `retry_after_seconds` when the refusal has `Retry-After` (e.g. 429 `quota_exceeded`, 429
+  `rate_limited`).
+- `shard usage` says `retained state  used up: new workspaces and forks are refused (...)` when the Retained state
+  allowance's `cap_state` is `storage_blocked` (`enforcement` `storage_block`).
+
+## 0.8.1 (not yet published)
+
+Needs `@shardflux/sdk` 0.13.1 or later.
+
+### Workspaces recover by themselves
+
+When the machine a workspace runs on fails, the workspace is suspended, and the next command on it resumes it.
+
+- A resume that booted the workspace's own disk (`cold_boot_reason` `host_lost`) prints the cold-boot note with the
+  reason in words: `note: workspace <id> resumed from its saved disk; running processes were restarted (the machine it
+  ran on failed)`.
+- A resume that restored its newest checkpoint prints `note: workspace <id> resumed from checkpoint <id> (state as of
+  <time>) after the machine it ran on failed; changes after that time are not in it` on stderr (also with `--json`,
+  `ws resume --wait`, `operations wait` and a command's wake).
+- `--timing --json`: `server.host_lost` (`detected_at`, `restored_from`, `restored_checkpoint_id`, `state_as_of`).
+  `--timing` prints the SDK's `restored checkpoint <id> (host_lost; state as of <time>)`.
+- `hostLostNotice()` is exported next to `coldBootNotice()`.
+
+## 0.8.0
 
 Needs `@shardflux/sdk` 0.13.0 or later.
 

package/README.md

@@ -607,11 +607,11 @@ without waking it (the state at suspension); `files search` then notes it on std
   refused at once (exit 1, `workspace_not_running`).
 
 **Detecting a cold resume (0.6.0+).** A resume restores memory and running processes. After a platform runtime
-update, a resume can boot from the saved disk instead of restoring memory (a cold boot), also when a command wakes the
-workspace; check `memory_restored`. The command still runs with the files as of the suspend, and processes start
-fresh, as after a reboot: start dev servers and background jobs again. `shard` then prints one line on stderr after
-the output (also with `--json`, except on a failure, whose stderr is only the error document), whether or not you
-pass `--timing`:
+update, or after the machine the workspace ran on failed (0.8.1+), a resume can boot from the saved disk instead of
+restoring memory (a cold boot), also when a command wakes the workspace; check `memory_restored`. The command still runs
+with the files kept, and processes start fresh, as after a reboot: start dev servers and background jobs again. `shard`
+then prints one line on stderr after the output (also with `--json`, except on a failure, whose stderr is only the
+error document), whether or not you pass `--timing`:
 
 ```text
 note: workspace 01a0e5a8-3edd-74ba-b489-d62b8925e342 resumed from its saved disk; running processes were restarted (runtime_changed)
@@ -621,6 +621,12 @@ note: workspace 01a0e5a8-3edd-74ba-b489-d62b8925e342 resumed from its saved disk
 `cold_boot`, `memory_restored` false, `cold_boot_reason`), and `--timing` shows `resume from cold_boot: processes
 restarted (runtime_changed)`. The same applies to `operations wait` of such a resume.
 
+**Workspaces recover by themselves (0.8.1+).** When the machine a workspace runs on fails, the workspace is suspended,
+and the next command on it resumes it; there is nothing extra to run. From its own disk, the note above ends in
+`(the machine it ran on failed)`; from its newest checkpoint, the note names the checkpoint and the time its state is
+from (`state as of <time>`). `--json` carries `result.host_lost` (`detected_at`, `restored_from` `disk` or
+`checkpoint`, `restored_checkpoint_id`, `state_as_of`), and `--timing` adds `host_lost` to `server`.
+
 ## Suspend when idle (0.5.1+)
 
 A running workspace is billed while it is awake. When an agent's turn ends, ask for a suspend once the workspace has
@@ -778,7 +784,7 @@ With `--json`, stdout carries the timing as a `timing` field of the command's JS
 `workspace_id`, `operation_id`, `total_ms`, `phases` (`phase`, `reason`, `operation_id`, `start_ms`, `duration_ms`),
 `retries`, `server` (`queued_ms`, `run_ms`, `total_ms`, `start_path`, `resume_path`, `memory_restored` and
 `cold_boot_reason` (0.6.0+; `memory_restored` is `false` when a resume booted the saved disk, `null` when the API did
-not say), `boot_to_ready_ms`, ...) and `outside_server_ms`. It is `null` when the command made no lifecycle call. On a failure the error document on stderr
+not say), `host_lost` (0.8.1+), `boot_to_ready_ms`, ...) and `outside_server_ms`. It is `null` when the command made no lifecycle call. On a failure the error document on stderr
 carries it: `{"error": {...}, "timing": {...}}`.
 
 For live progress events, use the SDK's `onProgress`.

package/dist/args.d.ts

@@ -1,4 +1,4 @@
-export declare const CLI_VERSION = "0.8.0";
+export declare const CLI_VERSION = "0.8.1";
 export interface OptionSpec {
     type: 'string' | 'boolean';
     short?: string;

package/dist/args.js

@@ -12,7 +12,7 @@
 import { parseArgs } from 'node:util';
 import { ACCOUNT_COMMANDS } from "./account-args.js";
 import { UsageError } from "./errors.js";
-export const CLI_VERSION = '0.8.0';
+export const CLI_VERSION = '0.8.1';
 export const GLOBAL_OPTIONS = {
     'api-url': { type: 'string', value: '<url>', description: 'API base URL (default $SHARDFLUX_API_URL, else https://api.shardflux.dev).' },
     json: { type: 'boolean', description: 'Print machine-readable JSON on stdout (errors as JSON on stderr).' },

package/dist/errors.d.ts

@@ -98,6 +98,8 @@ export interface ErrorInfo {
     operation?: unknown;
     /** A file-first execution the error is about (failed, lost, still running, or left running by Ctrl-C). */
     execution_id?: string;
+    /** The refusal's Retry-After in seconds (0.9.0+), e.g. 429 quota_exceeded from the cell gateway. */
+    retry_after_seconds?: number;
 }
 /**
  * Removes anything shaped like a Shardflux API key or user session token from text, and each of the given secrets

package/dist/errors.js

@@ -147,6 +147,7 @@ export function describeError(err) {
             ...(err.operationId ? { operation_id: err.operationId } : {}),
             ...(err.details ? { details: err.details } : {}),
             source: err.source,
+            ...(err.retryAfterSeconds !== undefined ? { retry_after_seconds: err.retryAfterSeconds } : {}),
         };
     }
     if (err instanceof OperationTimeoutError) {

package/dist/format.js

@@ -396,6 +396,10 @@ export function usageText(summary) {
         const trialEnded = isObject(s.plan) && s.plan.plan_key === 'trial';
         head.push(['allowance exhausted', s.allowance_exhausted ? (trialEnded ? `yes (${reason}the trial ends; the organization is back on free)` : `yes (${reason}new opens/resumes are refused)`) : 'no']);
     }
+    // cap_state storage_blocked (0.9.0): the Retained state allowance is used up; existing workspaces keep working.
+    if (Array.isArray(s.allowances) && s.allowances.some((a) => isObject(a) && a.cap_state === 'storage_blocked')) {
+        head.push(['retained state', 'used up: new workspaces and forks are refused (delete workspaces you no longer need, or upgrade)']);
+    }
     for (const [k, v] of Object.entries(s))
         if (!USAGE_KNOWN.has(k) && (typeof v !== 'object' || v === null))
             head.push([k.replace(/_/g, ' '), scalar(v)]);

package/dist/index.d.ts

@@ -22,4 +22,4 @@ export { openUrl, openerFor } from './browser.js';
 export { EXIT, AuthConfigError, CheckoutEndedError, ExecutionFailedError, ExecutionPendingError, InterruptedError, MfaPendingError, NotFoundError, UsageError, describeError, exitCodeFor, redact } from './errors.js';
 export type { ErrorInfo } from './errors.js';
 export { changedText, errorText, executionJson, executionSummary, feedbackCommand, json, suspendRequestText, suspendWhenIdleText, table, time, usageText, workspaceDetail, workspaceTable } from './format.js';
-export { TimingRecorder, coldBootNotice, lostSuspendNotice, notices, timingJson, timingText } from './timing.js';
+export { TimingRecorder, coldBootNotice, hostLostNotice, lostSuspendNotice, notices, timingJson, timingText } from './timing.js';

package/dist/index.js

@@ -14,4 +14,4 @@ export { CHECK_INTERVAL_MS, CLI_DISTRIBUTION, NOTICE_WAIT_MS, UPDATE_CHECK_FILE,
 export { openUrl, openerFor } from "./browser.js";
 export { EXIT, AuthConfigError, CheckoutEndedError, ExecutionFailedError, ExecutionPendingError, InterruptedError, MfaPendingError, NotFoundError, UsageError, describeError, exitCodeFor, redact } from "./errors.js";
 export { changedText, errorText, executionJson, executionSummary, feedbackCommand, json, suspendRequestText, suspendWhenIdleText, table, time, usageText, workspaceDetail, workspaceTable } from "./format.js";
-export { TimingRecorder, coldBootNotice, lostSuspendNotice, notices, timingJson, timingText } from "./timing.js";
+export { TimingRecorder, coldBootNotice, hostLostNotice, lostSuspendNotice, notices, timingJson, timingText } from "./timing.js";

package/dist/main.js

@@ -76,6 +76,11 @@ function hintFor(info, cred, file) {
         return 'this API has no billing configured: plans cannot be bought here';
     if (info.code === 'allowance_exhausted')
         return allowanceHint(info);
+    if (info.code === 'quota_exceeded') {
+        const quota = quotaHint(info);
+        if (quota)
+            return quota;
+    }
     const policyHint = spendPolicyHint(info);
     if (policyHint)
         return policyHint;
@@ -135,6 +140,21 @@ function allowanceHint(info) {
             return undefined;
     }
 }
+/**
+ * quota_exceeded (0.9.0): the cell gateway's 429 when every workspace the plan allows to work at once is busy (the SDK
+ * already retried it after Retry-After), and the API's 403 when the plan's Retained state is used up (new keys and
+ * forks are refused). Other limits keep the API's message alone.
+ */
+function quotaHint(info) {
+    const limit = info.details?.limit;
+    if (limit === 'concurrent_workspaces' && info.status === 429) {
+        const s = info.retry_after_seconds;
+        return `every workspace your plan allows to work at once is busy and the call did not run (it was already retried); retry ${typeof s === 'number' ? `in ${s} s` : 'in a few seconds'}, or upgrade: shard billing plans`;
+    }
+    if (limit === 'retained_state')
+        return 'delete workspaces you no longer need (shard ws delete <key> --yes), or upgrade: shard billing plans';
+    return undefined;
+}
 /** Dollars for a hint: `9` for 900 minor units, `9.50` for 950. */
 const dollars = (minor) => (typeof minor === 'number' ? (minor % 100 === 0 ? String(minor / 100) : (minor / 100).toFixed(2)) : undefined);
 /**
@@ -300,6 +320,9 @@ const NO_NUDGE_REASONS = new Set([
 function nudges(info, command) {
     if (command === undefined || command === 'feedback')
         return false;
+    // A plan limit whose hint names the next step (0.9.0): busy working slots, or Retained state used up.
+    if (info.code === 'quota_exceeded' && quotaHint(info) !== undefined)
+        return false;
     const reason = reasonOf(info);
     return !NO_NUDGE_CODES.has(info.code) && !(reason !== undefined && NO_NUDGE_REASONS.has(reason));
 }

package/dist/timing.d.ts

@@ -15,6 +15,12 @@ export declare class TimingRecorder {
      * --wait`, an `operations wait` of that resume, or a command's wake. `lostSuspendNotice()` says it.
      */
     lostSuspend: LifecycleTiming | null;
+    /**
+     * (0.8.1+) The last traced resume that restored the workspace's newest checkpoint after the machine it ran on failed
+     * (`host_lost.restored_from` `checkpoint`): a `ws resume --wait`, an `operations wait` of that resume, or a command's
+     * wake. `hostLostNotice()` says it. A restore from the disk is a cold boot (`coldBootNotice()`).
+     */
+    hostLost: LifecycleTiming | null;
     readonly listener: ProgressListener;
     /**
      * The failed call's own timing (`err.timing`) when the SDK attached one, else the last recorded. A refused tool token
@@ -24,15 +30,21 @@ export declare class TimingRecorder {
 }
 /**
  * The one-line stderr notice after a command whose resume booted the workspace's saved disk (0.6.0+), with or without
- * `--timing`: the workspace runs and its files are as of the suspend, but running processes were restarted.
+ * `--timing`: the workspace runs with its files kept, but running processes were restarted. The reason follows in
+ * parentheses: `runtime_changed`, or in words for `host_lost` (0.8.1+).
  */
 export declare function coldBootNotice(t: LifecycleTiming): string;
+/**
+ * The one-line stderr notice after a command whose resume restored the workspace's newest checkpoint after the machine
+ * it ran on failed (0.8.1+; `host_lost.restored_from` `checkpoint`): the state is as of `state_as_of`.
+ */
+export declare function hostLostNotice(t: LifecycleTiming): string;
 /**
  * The one-line stderr notice after a command whose resume restored an earlier checkpoint because the latest suspend
  * could not be kept (0.7.0+; `lost_suspend`): the state the workspace runs with is as of `state_as_of`.
  */
 export declare function lostSuspendNotice(t: LifecycleTiming): string;
-/** The notices a command's traced calls leave on stderr (cold boot, lost suspend), in that order. */
+/** The notices a command's traced calls leave on stderr (cold boot, lost suspend, host loss), in that order. */
 export declare function notices(r: TimingRecorder): string;
 /** The human form (stderr): formatTiming() of the SDK, one block. */
 export declare function timingText(t: LifecycleTiming): string;

package/dist/timing.js

@@ -23,6 +23,12 @@ export class TimingRecorder {
      * --wait`, an `operations wait` of that resume, or a command's wake. `lostSuspendNotice()` says it.
      */
     lostSuspend = null;
+    /**
+     * (0.8.1+) The last traced resume that restored the workspace's newest checkpoint after the machine it ran on failed
+     * (`host_lost.restored_from` `checkpoint`): a `ws resume --wait`, an `operations wait` of that resume, or a command's
+     * wake. `hostLostNotice()` says it. A restore from the disk is a cold boot (`coldBootNotice()`).
+     */
+    hostLost = null;
     listener = (e) => {
         if (e.action === 'token' || e.action === 'tool')
             return;
@@ -35,6 +41,8 @@ export class TimingRecorder {
             this.coldBoot = e.timing;
         if (e.timing.server?.lostSuspend)
             this.lostSuspend = e.timing;
+        if (e.timing.server?.hostLost?.restoredFrom === 'checkpoint')
+            this.hostLost = e.timing;
     };
     /**
      * The failed call's own timing (`err.timing`) when the SDK attached one, else the last recorded. A refused tool token
@@ -49,13 +57,29 @@ export class TimingRecorder {
 function restartedProcesses(t) {
     return t.server?.resumePath === 'cold_boot' && t.server.memoryRestored === false;
 }
+/** Cold boot reasons the notice says in words (0.8.1+); any other reason is printed as the API sent it. */
+const COLD_BOOT_WORDS = {
+    host_lost: 'the machine it ran on failed',
+};
 /**
  * The one-line stderr notice after a command whose resume booted the workspace's saved disk (0.6.0+), with or without
- * `--timing`: the workspace runs and its files are as of the suspend, but running processes were restarted.
+ * `--timing`: the workspace runs with its files kept, but running processes were restarted. The reason follows in
+ * parentheses: `runtime_changed`, or in words for `host_lost` (0.8.1+).
  */
 export function coldBootNotice(t) {
     const reason = t.server?.coldBootReason;
-    return `note: ${t.workspaceId ? `workspace ${t.workspaceId} ` : ''}resumed from its saved disk; running processes were restarted${reason ? ` (${reason})` : ''}\n`;
+    const why = reason ? (COLD_BOOT_WORDS[reason] ?? reason) : null;
+    return `note: ${t.workspaceId ? `workspace ${t.workspaceId} ` : ''}resumed from its saved disk; running processes were restarted${why ? ` (${why})` : ''}\n`;
+}
+/**
+ * The one-line stderr notice after a command whose resume restored the workspace's newest checkpoint after the machine
+ * it ran on failed (0.8.1+; `host_lost.restored_from` `checkpoint`): the state is as of `state_as_of`.
+ */
+export function hostLostNotice(t) {
+    const h = t.server?.hostLost;
+    if (h?.restoredFrom !== 'checkpoint')
+        return '';
+    return `note: ${t.workspaceId ? `workspace ${t.workspaceId} ` : ''}resumed from checkpoint ${h.restoredCheckpointId ?? '-'}${h.stateAsOf ? ` (state as of ${h.stateAsOf})` : ''} after the machine it ran on failed; changes after that time are not in it\n`;
 }
 /**
  * The one-line stderr notice after a command whose resume restored an earlier checkpoint because the latest suspend
@@ -67,9 +91,9 @@ export function lostSuspendNotice(t) {
         return '';
     return `note: ${t.workspaceId ? `workspace ${t.workspaceId} ` : ''}resumed from checkpoint ${l.restoredCheckpointId ?? '-'}${l.stateAsOf ? ` (state as of ${l.stateAsOf})` : ''}; the suspend of ${l.suspendedAt ?? l.checkpointId} was not kept${l.reason ? ` (${l.reason})` : ''}\n`;
 }
-/** The notices a command's traced calls leave on stderr (cold boot, lost suspend), in that order. */
+/** The notices a command's traced calls leave on stderr (cold boot, lost suspend, host loss), in that order. */
 export function notices(r) {
-    return `${r.coldBoot ? coldBootNotice(r.coldBoot) : ''}${r.lostSuspend ? lostSuspendNotice(r.lostSuspend) : ''}`;
+    return `${r.coldBoot ? coldBootNotice(r.coldBoot) : ''}${r.lostSuspend ? lostSuspendNotice(r.lostSuspend) : ''}${r.hostLost ? hostLostNotice(r.hostLost) : ''}`;
 }
 /** The human form (stderr): formatTiming() of the SDK, one block. */
 export function timingText(t) {
@@ -126,6 +150,15 @@ export function timingJson(t) {
                         state_as_of: s.lostSuspend.stateAsOf,
                     }
                     : null,
+                // 0.8.1+: the machine the workspace ran on failed (resume: what it restored; suspend: detected_at only).
+                host_lost: s.hostLost
+                    ? {
+                        detected_at: s.hostLost.detectedAt,
+                        restored_from: s.hostLost.restoredFrom,
+                        restored_checkpoint_id: s.hostLost.restoredCheckpointId,
+                        state_as_of: s.hostLost.stateAsOf,
+                    }
+                    : null,
                 boot_to_ready_ms: s.bootToReadyMs,
                 host_timings_ms: s.hostTimingsMs,
             }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@shardflux/cli",
-  "version": "0.8.0",
+  "version": "0.8.1",
   "type": "module",
   "description": "shard: the Shardflux command line. Create an account, sign in, set up keys and billing, then open, run commands in, move files to, suspend, resume and fork persistent agent workspaces.",
   "license": "Apache-2.0",
@@ -44,7 +44,7 @@
   },
   "dependencies": {
     "yaml": "2.9.1",
-    "@shardflux/sdk": "^0.13.0"
+    "@shardflux/sdk": "^0.13.1"
   },
   "devDependencies": {
     "@eslint/js": "10.0.1",