@beam-network/sdk 0.5.13 → 0.5.15

package/README.md

@@ -98,7 +98,7 @@ For low-level raw transfer configs, use `createRawTransfer`; lifecycle transport
 
 Provider transfers use `transfer-client-control/v6`. Every reply carries Runtime and transport epochs; prepare and every route-stream message carry a UUID route generation. S3, R2, and S3-compatible destinations retain direct multipart UploadPart/ListParts/HEAD handling.
 
-The client signs up to 64 routes concurrently by default and emits 2,048-route logical batches. Encoded MessagePack requests target 4 MiB physical batches under `maxPayloadBytes` (8 MiB by default; explicit positive overrides remain supported). The splitter reserves control-envelope headroom for the live auth token and stable request identity so the final encoded request remains below the configured guard. A single route above the 4 MiB target but within the configured guard publishes alone; a single route larger than the configured limit fails before publication with both sizes in the error. Multipart create and abort requests use a separate `multipartControlConcurrency` limit of 2 by default, with explicit positive overrides supported, and S3-compatible control requests make up to five attempts for transient provider failures. This keeps URL signing throughput independent from provider control-plane pressure.
+The client signs up to 64 routes concurrently by default and emits 2,048-route logical batches. Encoded MessagePack requests target 4 MiB physical batches under `maxPayloadBytes` (8 MiB by default; explicit positive overrides remain supported). The splitter reserves control-envelope headroom for the live auth token and stable request identity so the final encoded request remains below the configured guard. Lifecycle control refreshes cached SDK auth before the final 30 seconds of token lifetime and retries `auth_token_expired` replies with the same request identity and payload but a fresh token, preserving Core request-conflict protection during large route streams. A single route above the 4 MiB target but within the configured guard publishes alone; a single route larger than the configured limit fails before publication with both sizes in the error. Multipart create and abort requests use a separate `multipartControlConcurrency` limit of 2 by default, with explicit positive overrides supported, and S3-compatible control requests make up to five attempts for transient provider failures. This keeps URL signing throughput independent from provider control-plane pressure.
 
 The stream ID derives from the transfer, selected flow, and immutable plan identity, and each ordered batch ID includes its route-coordinate checksum. Manual multi-destination attachment requires `delivery_index` on every route. Lifecycle mutations retry transient failures up to three times with the same request identity. A non-recoverable provider setup or route failure raises `BeamProviderTransferError`, whose `errors`, `transferCancelled`, and `multipartCleanupComplete` fields retain the original provider, cancellation, and cleanup outcomes. `waitForTransfer` subscribes to the API-key-owned, at-most-once terminal signal before its first status read and reconciles every signal through authoritative status; subscription failure degrades to the same jittered 15-to-30-second status fallback. Call `close()` when the client is no longer needed; it also closes outstanding terminal waiters.
 
@@ -110,4 +110,4 @@ NATS requires this guard because the broker rejects messages above its configure
 
 ## Restart Recovery
 
-The client keeps one in-memory recovery lease per active transfer and one `runtime.hello` monitor per active shard. The lease is installed before route-stream begin, and a per-transfer lock coalesces initial streaming with replay. Multipart recovery retains the existing upload IDs and compact group state, then re-signs only the expiring route and commit controls. A Runtime epoch change invalidates cached auth, coalesces one `transfer.resume`, and regenerates routes under a fresh generation; a transport-only epoch change replays only when Runtime reports routes missing or expired. Provider credentials and signing inputs are retained only in memory and released on terminal status or `close()`. Low-level `attachSignedUrls` requires `routeGenerationId`, the plan fingerprint/checksum, and a `recoveryFactory`; signed URLs are never journaled to disk. Foreground cancellation, deadline expiry, and Runtime state-loss responses keep the background lease and retained multipart upload alive.
+The client keeps one in-memory recovery lease per active transfer and one `runtime.hello` monitor per active shard. The lease is installed before route-stream begin, and a per-transfer lock coalesces initial streaming with replay. Multipart recovery retains the existing upload IDs and compact group state, then re-signs only the expiring route and commit controls. Each `resumeProviderTransfer` invocation re-prepares an existing transfer by explicit transfer id with a fresh request and route generation, while transport retries within that invocation reuse the same request. It then reattaches route-recovery signing from the current provider configs without creating duplicate multipart uploads. A Runtime epoch change invalidates cached auth, coalesces one `transfer.resume`, and regenerates routes under a fresh generation; a transport-only epoch change replays only when Runtime reports routes missing or expired. Provider credentials and signing inputs are retained only in memory and released on terminal status or `close()`. Low-level `attachSignedUrls` requires `routeGenerationId`, the plan fingerprint/checksum, and a `recoveryFactory`; signed URLs are never journaled to disk. Foreground cancellation, deadline expiry, and Runtime state-loss responses keep the background lease and retained multipart upload alive.

package/dist/client.d.ts

@@ -1,4 +1,4 @@
-import type { AttachSignedUrlsResponse, BeamClientOptions, DistributeResponse, MultipartGroupManifest, PlanningHttpSource, PreparedDestination, PreparedHttpSource, ProviderTransferCreateInput, RawTransferCreateInput, SignedChunkRoute, SignedUrlFlow, TransferCancelResponse, TransferCreateResponse, TransferPlanResponse, TransferPrepareResponse, TransferStatusInfo, TransferTerminalSignalWaiter } from "./models.js";
+import type { AttachSignedUrlsResponse, BeamClientOptions, DistributeResponse, MultipartGroupManifest, PlanningHttpSource, PreparedDestination, PreparedHttpSource, ProviderTransferCreateInput, ProviderTransferResumeInput, RawTransferCreateInput, SignedChunkRoute, SignedUrlFlow, TransferCancelResponse, TransferCreateResponse, TransferPlanResponse, TransferPrepareResponse, TransferStatusInfo, TransferTerminalSignalWaiter } from "./models.js";
 import { BEAM_DEFAULT_NATS_URL } from "./nats-control.js";
 export declare class BeamRouteRecoveryPendingError extends Error {
     readonly transferId: string;
@@ -24,6 +24,17 @@ export declare class BeamApiError extends Error {
     readonly body: string;
     constructor(message: string, status: number, body: string);
 }
+type TransferPrepareInput = {
+    transferId?: string;
+    sources: PreparedHttpSource[];
+    destinations: PreparedDestination[];
+    name?: string;
+    testMode?: boolean;
+    urlsExpiresAt?: string;
+    signedUrlFlow?: SignedUrlFlow;
+    idempotencyKey?: string;
+    routeGenerationId?: string;
+};
 export declare class BeamClient {
     readonly apiKey: string;
     readonly natsUrl: string;
@@ -40,6 +51,7 @@ export declare class BeamClient {
     openTransferTerminalWaiter(transferId: string): Promise<TransferTerminalSignalWaiter>;
     createRawTransfer(input: RawTransferCreateInput): Promise<TransferCreateResponse>;
     createTransfer(input: ProviderTransferCreateInput): Promise<TransferPrepareResponse>;
+    resumeProviderTransfer(input: ProviderTransferResumeInput): Promise<TransferPrepareResponse>;
     transferStatus(transferId: string): Promise<TransferStatusInfo>;
     distributeTransfer(transferId: string): Promise<DistributeResponse>;
     cancelTransfer(transferId: string): Promise<TransferCancelResponse>;
@@ -51,16 +63,8 @@ export declare class BeamClient {
         urlsExpiresAt?: string;
         signedUrlFlow?: SignedUrlFlow;
     }): Promise<TransferPlanResponse>;
-    prepareTransfer(input: {
-        sources: PreparedHttpSource[];
-        destinations: PreparedDestination[];
-        name?: string;
-        testMode?: boolean;
-        urlsExpiresAt?: string;
-        signedUrlFlow?: SignedUrlFlow;
-        idempotencyKey?: string;
-        routeGenerationId?: string;
-    }): Promise<TransferPrepareResponse>;
+    prepareTransfer(input: TransferPrepareInput): Promise<TransferPrepareResponse>;
+    private prepareTransferWithRequestKey;
     attachSignedUrls(transferId: string, input: {
         chunkRoutes: SignedChunkRoute[];
         multipartGroupManifest: MultipartGroupManifest[];
@@ -79,6 +83,7 @@ export declare class BeamClient {
     prepareProviderTransfer(input: ProviderTransferCreateInput): Promise<TransferPrepareResponse>;
     private signProviderRoute;
     private startProviderRouteRecoverySigner;
+    private recoveryMultipartUploadState;
     private submitIntegrityAuditGrantsIfPresent;
     private submitProviderIntegrityAuditGrants;
     private stopRecoverySigner;

package/dist/client.js

@@ -111,6 +111,65 @@ export class BeamClient {
     createTransfer(input) {
         return this.prepareProviderTransfer(input);
     }
+    async resumeProviderTransfer(input) {
+        validateId(input.transferId, "transferId");
+        const expiresIn = input.expiresIn ?? 3600;
+        const requestedSignedUrlFlow = input.signedUrlFlow ?? "signed_url";
+        const retainedRecoveryInput = {
+            ...input,
+            sources: input.sources.map((source) => ({ ...source })),
+            destinations: input.destinations.map((destination) => ({ ...destination }))
+        };
+        const preparedDestinations = retainedRecoveryInput.destinations.map((destination, index) => prepareProviderDestination(destination, { index }));
+        const destinationsById = new Map();
+        preparedDestinations.forEach((preparedDestination, index) => {
+            const destination = retainedRecoveryInput.destinations[index];
+            if (!destination) {
+                throw new Error(`missing provider destination config for prepared destination ${preparedDestination.destination_id}`);
+            }
+            destinationsById.set(preparedDestination.destination_id, destination);
+        });
+        const preparedSources = await Promise.all(retainedRecoveryInput.sources.map((source, index) => prepareProviderSource(source, { index, expiresIn, fetchImpl: this.fetchImpl })));
+        const resumePrepareRequestKey = `transfer:${input.transferId}:prepare:resume:${randomUuid()}`;
+        const prepared = await this.prepareTransferWithRequestKey({
+            transferId: input.transferId,
+            sources: preparedSources,
+            destinations: preparedDestinations,
+            name: input.name,
+            testMode: input.testMode,
+            signedUrlFlow: requestedSignedUrlFlow
+        }, resumePrepareRequestKey);
+        if (!prepared.success)
+            return prepared;
+        if (prepared.transfer_id !== input.transferId) {
+            throw new Error("resumed provider transfer id mismatch");
+        }
+        const sourcesById = new Map();
+        preparedSources.forEach((preparedSource, index) => {
+            const source = retainedRecoveryInput.sources[index];
+            if (!source) {
+                throw new Error(`missing provider source config for prepared source ${preparedSource.source_id}`);
+            }
+            sourcesById.set(preparedSource.source_id, source);
+        });
+        await this.startProviderRouteRecoverySigner({
+            prepared,
+            sourcesById,
+            destinationsById,
+            multipartUploads: new Map(),
+            expiresIn
+        });
+        this.control.registerRecoveryLease({
+            transferId: prepared.transfer_id,
+            planFingerprint: prepared.plan_fingerprint,
+            coordinateChecksum: prepared.coordinate_checksum,
+            replayRoutes: async () => {
+                throw new Error("route replay is unavailable after provider action resume; route signing remains available");
+            },
+            disposeSecrets: () => clearProviderResumeSecrets(retainedRecoveryInput)
+        });
+        return prepared;
+    }
     async transferStatus(transferId) {
         validateId(transferId, "transferId");
         const status = await this.control.request("transfer.status", { transfer_id: transferId }, { transferId });
@@ -151,9 +210,13 @@ export class BeamClient {
             validateCompactTransferPlan(result.plan_descriptor, result.signed_url_flow);
         return result;
     }
-    async prepareTransfer(input) {
-        const transferId = await transferIdForIdempotencyKey(input.idempotencyKey);
-        const prepareIdempotencyKey = `transfer:${transferId}:prepare`;
+    prepareTransfer(input) {
+        return this.prepareTransferWithRequestKey(input);
+    }
+    async prepareTransferWithRequestKey(input, requestKey) {
+        const transferId = input.transferId ?? await transferIdForIdempotencyKey(input.idempotencyKey);
+        validateId(transferId, "transferId");
+        const prepareIdempotencyKey = requestKey ?? `transfer:${transferId}:prepare`;
         const routeGenerationId = input.routeGenerationId ?? await routeGenerationIdForPrepareIdempotencyKey(prepareIdempotencyKey);
         const result = await this.control.request("transfer.prepare", compact({
             transfer_id: transferId,
@@ -548,15 +611,15 @@ export class BeamClient {
                 const destination = input.destinationsById.get(requested.destination_id);
                 if (!source || !destination)
                     throw new Error("route recovery provider configuration is unavailable");
-                const upload = input.multipartUploads.get(requested.multipart_group_id);
-                if (!upload)
-                    throw new Error(`route recovery multipart upload is unavailable: ${requested.multipart_group_id}`);
-                if (upload.uploadId !== requested.upload_id
-                    || upload.objectKey !== requested.final_object_key
-                    || upload.manifest.source_id !== requested.source_id
-                    || upload.manifest.destination_id !== requested.destination_id) {
-                    throw new Error("route recovery multipart identity mismatch");
-                }
+                const upload = isHippiusDestination(destination)
+                    ? undefined
+                    : await this.recoveryMultipartUploadState({
+                        prepared: input.prepared,
+                        destination,
+                        requested,
+                        multipartUploads: input.multipartUploads,
+                        expiresIn: input.expiresIn
+                    });
                 return this.signProviderRoute({
                     chunk,
                     target: {
@@ -579,7 +642,7 @@ export class BeamClient {
                     source,
                     destination,
                     expiresIn: input.expiresIn,
-                    upload,
+                    ...(upload ? { upload } : {}),
                     transferId: input.prepared.transfer_id,
                     finalObjectKey: requested.final_object_key,
                     signedUrlFlow: "signed_url",
@@ -604,6 +667,63 @@ export class BeamClient {
             });
         });
     }
+    async recoveryMultipartUploadState(input) {
+        const existing = input.multipartUploads.get(input.requested.multipart_group_id);
+        if (existing) {
+            if (existing.uploadId !== input.requested.upload_id
+                || existing.objectKey !== input.requested.final_object_key
+                || existing.manifest.source_id !== input.requested.source_id
+                || existing.manifest.destination_id !== input.requested.destination_id) {
+                throw new Error("route recovery multipart identity mismatch");
+            }
+            return existing;
+        }
+        if (!input.requested.upload_id) {
+            throw new Error("route recovery multipart upload id is required");
+        }
+        const source = input.prepared.plan_descriptor.sources.find((candidate) => candidate.source_id === input.requested.source_id);
+        if (!source) {
+            throw new Error("route recovery source plan is unavailable");
+        }
+        const expectedGroupId = multipartGroupStateKey(input.prepared.transfer_id, input.requested.destination_id, input.requested.source_id, input.requested.final_object_key);
+        if (input.requested.multipart_group_id !== expectedGroupId) {
+            throw new Error("route recovery multipart group identity mismatch");
+        }
+        const finalObjectMetadata = { "beam-transfer-id": input.prepared.transfer_id };
+        const maxPartNumber = multipartMaxPartNumber(source.chunk_count);
+        const listPageMarkers = multipartListPageMarkers(maxPartNumber);
+        const [completeUrl, abortUrl, listPageUrls, finalHeadUrl] = await Promise.all([
+            signCompleteMultipartUpload(input.destination, input.requested.final_object_key, input.requested.upload_id, input.expiresIn),
+            signAbortMultipartUpload(input.destination, input.requested.final_object_key, input.requested.upload_id, input.expiresIn),
+            Promise.all(listPageMarkers.map((partNumberMarker) => signListMultipartUpload(input.destination, input.requested.final_object_key, input.requested.upload_id, input.expiresIn, { maxParts: 1_000, ...(partNumberMarker ? { partNumberMarker } : {}) }))),
+            signFinalObjectHead(input.destination, input.requested.final_object_key, input.expiresIn)
+        ]);
+        const manifest = {
+            multipart_group_id: input.requested.multipart_group_id,
+            source_id: input.requested.source_id,
+            destination_id: input.requested.destination_id,
+            final_object_key: input.requested.final_object_key,
+            upload_id: input.requested.upload_id,
+            expected_object_size: source.size,
+            expected_part_count: source.chunk_count,
+            max_part_number: maxPartNumber,
+            complete_url: completeUrl,
+            abort_url: abortUrl,
+            list_page_urls: listPageUrls,
+            final_head_url: finalHeadUrl,
+            final_object_metadata: finalObjectMetadata,
+            urls_expires_at: expiresAtIso(input.expiresIn)
+        };
+        validateMultipartGroupManifest([manifest], input.prepared.transfer_id);
+        const state = {
+            destination: input.destination,
+            objectKey: input.requested.final_object_key,
+            uploadId: input.requested.upload_id,
+            manifest
+        };
+        input.multipartUploads.set(input.requested.multipart_group_id, state);
+        return state;
+    }
     async submitIntegrityAuditGrantsIfPresent(status) {
         const challenge = status.integrity_audit_challenge;
         if (!challenge)
@@ -1496,3 +1616,13 @@ function clearRecoverySecrets(input) {
         }
     }
 }
+function clearProviderResumeSecrets(input) {
+    const secretKeys = new Set(["access_key_id", "secret_access_key", "session_token", "api_token", "token", "password"]);
+    for (const provider of [...input.sources, ...input.destinations]) {
+        const record = provider;
+        for (const key of secretKeys) {
+            if (key in record)
+                record[key] = "";
+        }
+    }
+}

package/dist/models.d.ts

@@ -171,6 +171,18 @@ export interface TransferStatusInfo {
     };
     integrity_audit_challenge?: IntegrityAuditChallenge;
 }
+export interface ProviderTransferResumeInput {
+    transferId: string;
+    sources: ProviderSourceConfig[];
+    destinations: ProviderDestinationConfig[];
+    name?: string;
+    /**
+     * Maps to BeamCore test_mode. Must match the original prepared transfer.
+     */
+    testMode?: boolean;
+    expiresIn?: number;
+    signedUrlFlow?: SignedUrlFlow;
+}
 export interface TransferTerminalEvent {
     schema_version: "transfer-client-control/v6";
     producer: "transfer-runtime";

package/dist/nats-control.js

@@ -8,6 +8,7 @@ export const BEAM_DEFAULT_NATS_WS_URL = "wss://nats.b1m.ai:443";
 export const BEAM_DEFAULT_MAX_PAYLOAD_BYTES = 8 * 1024 * 1024;
 export const BEAM_ROUTE_TARGET_PAYLOAD_BYTES = 4 * 1024 * 1024;
 const ROUTE_BATCH_AUTH_TOKEN_ESTIMATE_BYTES = 64 * 1024;
+const AUTH_TOKEN_REFRESH_SAFETY_SECONDS = 30;
 export const ROUTE_RECOVERY_SIGN_MESSAGE_TYPE = "transfer.route_recovery.sign";
 export function buildConnectionOptions(options) {
     const connectionOptions = {
@@ -199,32 +200,33 @@ export class BeamTransferControl {
         void this.recoverLease(lease);
     }
     async request(messageType, payload, options = {}) {
-        const authToken = await this.resolveAuthToken();
         const shardId = options.shardId ?? (options.transferId ? transferShardId(options.transferId, this.shardCount) : 0);
         const requestId = options.idempotencyKey
             ? await stableRequestId(`${messageType}:${options.idempotencyKey}`)
             : randomId();
-        const envelope = {
-            message_id: `${TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION}:${this.environment}:${this.keyPrefix}:${messageType}:${requestId}`,
-            schema_version: TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION,
-            environment: this.environment,
-            key_prefix: this.keyPrefix,
-            shard_id: shardId,
-            message_type: messageType,
-            request_id: requestId,
-            auth_token: authToken,
-            occurred_at: new Date().toISOString(),
-            producer: "sdk",
-            payload
-        };
-        const bytes = encode(envelope);
-        if (bytes.byteLength > this.maxPayloadBytes) {
-            throw new Error(`NATS lifecycle request is ${bytes.byteLength} bytes, above maxPayloadBytes=${this.maxPayloadBytes}`);
-        }
+        const occurredAt = new Date().toISOString();
         const subject = this.requestSubject(messageType, shardId);
         let lastError;
         for (let attempt = 0; attempt < LIFECYCLE_REQUEST_MAX_ATTEMPTS; attempt += 1) {
             try {
+                const authToken = await this.resolveAuthToken();
+                const envelope = {
+                    message_id: `${TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION}:${this.environment}:${this.keyPrefix}:${messageType}:${requestId}`,
+                    schema_version: TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION,
+                    environment: this.environment,
+                    key_prefix: this.keyPrefix,
+                    shard_id: shardId,
+                    message_type: messageType,
+                    request_id: requestId,
+                    auth_token: authToken,
+                    occurred_at: occurredAt,
+                    producer: "sdk",
+                    payload
+                };
+                const bytes = encode(envelope);
+                if (bytes.byteLength > this.maxPayloadBytes) {
+                    throw new Error(`NATS lifecycle request is ${bytes.byteLength} bytes, above maxPayloadBytes=${this.maxPayloadBytes}`);
+                }
                 const nc = await this.connection();
                 const response = await nc.request(subject, bytes, { timeout: this.requestTimeoutMs });
                 const decoded = decode(response.data);
@@ -233,6 +235,13 @@ export class BeamTransferControl {
                     return (decoded.payload ?? {});
                 const body = JSON.stringify(decoded.error ?? {});
                 const error = new LifecycleRequestError(decoded.status, `Beam lifecycle request failed with ${decoded.status}: ${body}`);
+                if (isExpiredAuthTokenReply(decoded) && attempt + 1 < LIFECYCLE_REQUEST_MAX_ATTEMPTS) {
+                    this.authToken = null;
+                    this.authTokenExpiresAt = 0;
+                    lastError = error;
+                    await sleepWithJitter(LIFECYCLE_REQUEST_RETRY_DELAYS_MS[attempt] ?? 1_000);
+                    continue;
+                }
                 if (!isRetryableLifecycleStatus(decoded.status) || attempt + 1 >= LIFECYCLE_REQUEST_MAX_ATTEMPTS) {
                     throw error;
                 }
@@ -507,7 +516,7 @@ export class BeamTransferControl {
     }
     async resolveAuthToken() {
         const now = Math.floor(Date.now() / 1000);
-        if (this.authToken && this.authTokenExpiresAt - 5 > now)
+        if (this.authToken && this.authTokenExpiresAt - AUTH_TOKEN_REFRESH_SAFETY_SECONDS > now)
             return this.authToken;
         if (this.authResolvePromise)
             return this.authResolvePromise;
@@ -707,6 +716,9 @@ const LIFECYCLE_REQUEST_RETRY_DELAYS_MS = [150, 500];
 function isRetryableLifecycleStatus(status) {
     return status === 408 || status === 425 || status === 429 || status >= 500;
 }
+function isExpiredAuthTokenReply(reply) {
+    return reply.status === 401 && reply.error?.code === "auth_token_expired";
+}
 export function isRetryableLifecycleError(error) {
     if (error instanceof LifecycleRequestError)
         return isRetryableLifecycleStatus(error.status);

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@beam-network/sdk",
-    "version": "0.5.13",
+    "version": "0.5.15",
     "description": "TypeScript SDK for BEAM transfer creation and management.",
     "type": "module",
     "license": "MIT",