@beam-network/sdk 0.5.21 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,5 +1,10 @@
1
1
  # BEAM SDK for TypeScript
2
2
 
3
+ This release requires Core's `transfer-client-control/v7` contract. Multipart uploads use consecutive parts;
4
+ Core may request retained staging for recovery through the existing SDK signing connection. Workers receive
5
+ ordinary upload URLs. Keep the client connected until the transfer reaches a terminal state so Core can renew
6
+ grants, promote recovered data, and clean up staging. Drain active transfers before upgrading Core and SDK consumers.
7
+
3
8
  Install:
4
9
 
5
10
  ```bash
@@ -149,21 +154,21 @@ For low-level raw transfer configs, use `createRawTransfer`; lifecycle transport
149
154
 
150
155
  ## Route Streaming And Payload Size
151
156
 
152
- Provider transfers use `transfer-client-control/v6`. Every reply carries Runtime and transport epochs; prepare and every route-stream message carry a UUID route generation. S3, R2, and S3-compatible destinations retain direct multipart UploadPart/ListParts/HEAD handling. Hippius and Hugging Face destinations take a plain PUT per chunk instead, so they carry no multipart group manifest.
157
+ Provider transfers use `transfer-client-control/v7`. Every reply carries Runtime and transport epochs; prepare and every route-stream message carry a UUID route generation. S3, R2, and S3-compatible destinations retain direct multipart UploadPart/ListParts/HEAD handling. Hippius and Hugging Face destinations take a plain PUT per chunk instead, so they carry no multipart group manifest.
153
158
 
154
- The client signs up to 64 routes concurrently by default and emits 2,048-route logical batches. Encoded MessagePack requests target 4 MiB physical batches under `maxPayloadBytes` (8 MiB by default; explicit positive overrides remain supported). The splitter reserves control-envelope headroom for the live auth token and stable request identity so the final encoded request remains below the configured guard. Lifecycle control refreshes cached SDK auth before the final 30 seconds of token lifetime and retries `auth_token_expired` replies with the same request identity and payload but a fresh token, preserving Core request-conflict protection during large route streams. A single route above the 4 MiB target but within the configured guard publishes alone; a single route larger than the configured limit fails before publication with both sizes in the error. Multipart create and abort requests use a separate `multipartControlConcurrency` limit of 2 by default, with explicit positive overrides supported, and S3-compatible control requests make up to five attempts for transient provider failures. This keeps URL signing throughput independent from provider control-plane pressure.
159
+ The client signs up to 64 routes concurrently by default and emits 1,024-route logical batches. Encoded MessagePack requests target 4 MiB physical batches under `maxPayloadBytes` (8 MiB by default; explicit positive overrides remain supported). The splitter reserves control-envelope headroom for the live auth token and stable request identity so the final encoded request remains below the configured guard. Lifecycle control refreshes cached SDK auth before the final 30 seconds of token lifetime and retries `auth_token_expired` replies with the same request identity and payload but a fresh token, preserving Core request-conflict protection during large route streams. A single route above the 4 MiB target but within the configured guard publishes alone; a single route larger than the configured limit fails before publication with both sizes in the error. Multipart create and abort requests use a separate `multipartControlConcurrency` limit of 2 by default, with explicit positive overrides supported, and S3-compatible control requests make up to five attempts for transient provider failures. This keeps URL signing throughput independent from provider control-plane pressure.
155
160
 
156
161
  Callers that need durable recovery accounting can provide `onMultipartGroupReady`. The SDK invokes it after creating a multipart upload and before streaming that group’s signed routes. Its payload contains transfer, source, destination, object-key, upload, size, part-count, and expiry identities only; it never includes credentials, request headers, or signed URLs. A rejected callback aborts the newly created upload and fails preparation closed.
157
162
 
158
163
  `resumeProviderTransfer` requires the complete saved callback payloads in `multipartGroups`. It validates every transfer/source/destination/object/size/part coordinate, then uses the same route-stream implementation as creation with those upload IDs. Missing, duplicate, or changed identities fail before new provider uploads or route publication. If a process died before recording an upload identity, treat it as uncertain cleanup instead of inventing a replacement. The `onPrepared` callback also runs during resume, before routes are streamed.
159
164
 
160
- Create and resume accept an ownership `signal`. Aborting it stops initial routes, multipart creation, and recovery signing/replay; it does not cancel the transfer or abort a replacement owner's uploads. User cancellation still uses `cancelTransfer` and verified provider cleanup. `prepareProviderSource` accepts the same signal for metadata requests.
165
+ Create and resume accept an ownership `signal`. Aborting it stops initial routes, multipart creation, and recovery signing/replay; it does not cancel the transfer or abort a replacement owner's uploads. Recovery leases, signers, and background resume/replay are keyed by owner identity: a fenced-off owner never replays into, releases, or stops the signers of a replacement owner registered for the same transfer, and a replacement always gets its own recovery. User cancellation still uses `cancelTransfer` and verified provider cleanup. `prepareProviderSource` accepts the same signal for metadata requests.
161
166
 
162
167
  The stream ID derives from the transfer, selected flow, and immutable plan identity, and each ordered batch ID includes its route-coordinate checksum. Manual multi-destination attachment requires `delivery_index` on every route. Lifecycle mutations retry transient failures up to three times with the same request identity. A non-recoverable provider setup or route failure raises `BeamProviderTransferError`, whose `errors`, `transferCancelled`, and `multipartCleanupComplete` fields retain the original provider, cancellation, and cleanup outcomes. `waitForTransfer` subscribes to the API-key-owned, at-most-once terminal signal before its first status read and reconciles every signal through authoritative status; subscription failure degrades to the same jittered 15-to-30-second status fallback. Call `close()` when the client is no longer needed; it also closes outstanding terminal waiters.
163
168
 
164
169
  During the pre-completion `integrity_check` phase, Runtime may include an `integrity_audit_challenge` in status. The TypeScript client signs exact read-only source and final-destination GET ranges and submits `transfer.integrity_audit_grants`. S3-compatible source and destination grants use the prepared source ETag and Runtime-verified final-object ETag as `If-Match` conditions. Submission failures retry on the next status poll and appear as a URL-safe `integrity_audit_submission_error` on that status response; the transfer completes with a warning if the check cannot finish by its deadline.
165
170
 
166
- Hippius uses canonical non-multipart `signed_url` routes, so its manifest is empty and group-level final HEAD verification is not available from that provider flow.
171
+ Hippius uses canonical non-multipart `signed_url` routes (initial and recovered routes both PUT to the planned `<final>/<plan_nonce>/chunk-NNNNNN` key), so its manifest is empty and group-level final HEAD verification is not available from that provider flow.
167
172
 
168
173
  NATS requires this guard because the broker rejects messages above its configured `max_payload`. This limit applies only to lifecycle/control metadata; transfer file bytes do not flow through NATS.
169
174
 
@@ -184,3 +189,9 @@ Only short-lived range/upload routes go to workers. Multipart creation, verifica
184
189
  Manual releases from `dev` require a committed `-dev.N` version and publish only to npm tag `dev`; the stable tag is unchanged.
185
190
 
186
191
  Multipart control helpers (`createMultipartUpload`, `listMultipartParts`, `completeMultipartUpload`, `inspectDestinationObject`, and `abortMultipartUpload`) accept an optional `AbortSignal` and cancel through the existing provider HTTP transport. Pass it as `signal` in object arguments or as the last positional argument. Cancellation stops waiting and network activity; it cannot undo a provider operation already accepted. After an interrupted completion, inspect the durable upload/object identity before retrying or reporting cleanup complete.
192
+
193
+ ### Preparation diagnostics
194
+
195
+ Route streaming flushes at 1,024 routes, with no timer, and sends the final partial batch at stream completion. Signing overlaps acknowledged publication through bounded buffering; existing encoded-payload limits still apply. A source grant is reused across destinations only within the same signing generation and actual expiry.
196
+
197
+ The optional `onDiagnostics` callback receives bounded preparation measurements and source-grant reuse counts. Callbacks are best effort, may be dropped under load, and do not affect transfer outcomes. Durations may overlap; do not sum them into elapsed transfer time. Transfer status also exposes a typed `performance` summary when supported by Core.
package/dist/client.d.ts CHANGED
@@ -44,8 +44,11 @@ export declare class BeamClient {
44
44
  private readonly routeSigningConcurrency;
45
45
  private readonly routeSigningConcurrencyOverridden;
46
46
  private readonly multipartControlConcurrency;
47
+ private readonly onDiagnostics;
48
+ private diagnosticPending;
47
49
  private readonly recoverySigners;
48
50
  private readonly integrityAuditSigners;
51
+ private readonly integrityGrantCache;
49
52
  private readonly integrityAuditSubmissions;
50
53
  private readonly huggingFaceUploads;
51
54
  constructor(options?: BeamClientOptions);
@@ -105,7 +108,9 @@ export declare class BeamClient {
105
108
  private startProviderRouteRecoverySigner;
106
109
  private recoveryMultipartUploadState;
107
110
  private submitIntegrityAuditGrantsIfPresent;
108
- private submitProviderIntegrityAuditGrants;
111
+ private signedIntegrityGrants;
112
+ private buildProviderIntegrityAuditGrants;
113
+ /** Stop a transfer's recovery and integrity signers; with `owner`, only if that owner installed them. */
109
114
  private stopRecoverySigner;
110
115
  private stopAllRecoverySigners;
111
116
  createAndDistribute(input: RawTransferCreateInput): Promise<TransferCreateResponse>;
package/dist/client.js CHANGED
@@ -1,4 +1,5 @@
1
- import { abortMultipartUpload, createMultipartUpload, expiresAtIso, prepareProviderDestination, prepareProviderSource, signAbortMultipartUpload, signCompleteMultipartUpload, signDestinationReadRange, signDestinationRoute, signFinalObjectHead, signListMultipartUpload, signSourceReadRange, isHuggingFaceProvider } from "./provider-signing.js";
1
+ import { SdkPerformanceCollector } from "./performance.js";
2
+ import { signMultipartRecovery, abortMultipartUpload, createMultipartUpload, expiresAtIso, prepareProviderDestination, prepareProviderSource, releaseProviderClients, signAbortMultipartUpload, signCompleteMultipartUpload, signDestinationReadRange, signDestinationRoute, signFinalObjectHead, signListMultipartUpload, signSourceReadRange, signSourceChunk, boundedGrantExpiry, isHuggingFaceProvider } from "./provider-signing.js";
2
3
  import { describe as describeHuggingFace, hashSourceStream, huggingFaceCommit, huggingFaceCompleteLfsUpload, huggingFaceLfsBatch, huggingFacePreupload, huggingFaceVerifyLfsUpload, readSourceSample } from "./huggingface.js";
3
4
  import { BeamTransferControl, BEAM_DEFAULT_NATS_URL, compactSignedRoutes, isRecoverableRouteStreamError } from "./nats-control.js";
4
5
  export class BeamRouteRecoveryPendingError extends Error {
@@ -53,11 +54,15 @@ export class BeamClient {
53
54
  routeSigningConcurrency;
54
55
  routeSigningConcurrencyOverridden;
55
56
  multipartControlConcurrency;
57
+ onDiagnostics;
58
+ diagnosticPending = false;
56
59
  recoverySigners = new Map();
57
60
  integrityAuditSigners = new Map();
61
+ integrityGrantCache = new Map();
58
62
  integrityAuditSubmissions = new Map();
59
63
  huggingFaceUploads = new Map();
60
64
  constructor(options = {}) {
65
+ this.onDiagnostics = options.onDiagnostics;
61
66
  if (!options.apiKey?.trim()) {
62
67
  throw new Error("apiKey is required.");
63
68
  }
@@ -237,7 +242,7 @@ export class BeamClient {
237
242
  };
238
243
  const routeStreamLock = new AsyncMutex();
239
244
  const releaseInitialStream = await routeStreamLock.acquire();
240
- this.control.registerRecoveryLease({
245
+ const recoveryLease = {
241
246
  transferId,
242
247
  planFingerprint: input.planFingerprint,
243
248
  coordinateChecksum: input.coordinateChecksum,
@@ -249,19 +254,20 @@ export class BeamClient {
249
254
  throw new Error(attached.error ?? attached.message ?? "route replay failed");
250
255
  });
251
256
  }
252
- });
257
+ };
258
+ this.control.registerRecoveryLease(recoveryLease);
253
259
  try {
254
260
  const attached = await streamRoutes(input.chunkRoutes, input.multipartGroupManifest, input.routeGenerationId, input.urlsExpiresAt);
255
261
  if (!attached.success)
256
- this.control.releaseRecoveryLease(transferId);
262
+ this.control.releaseRecoveryLease(transferId, recoveryLease);
257
263
  return attached;
258
264
  }
259
265
  catch (error) {
260
266
  if (isRecoverableRouteStreamError(error)) {
261
- this.control.continueRecoveryLease(transferId);
267
+ this.control.continueRecoveryLease(transferId, recoveryLease);
262
268
  throw new BeamRouteRecoveryPendingError(transferId, error);
263
269
  }
264
- this.control.releaseRecoveryLease(transferId);
270
+ this.control.releaseRecoveryLease(transferId, recoveryLease);
265
271
  throw error;
266
272
  }
267
273
  finally {
@@ -294,8 +300,10 @@ export class BeamClient {
294
300
  destinationsById.set(preparedDestination.destination_id, destination);
295
301
  });
296
302
  await input.throwIfCancelled?.();
303
+ const discoveryStarted = performance.now();
297
304
  const preparedSources = await Promise.all(retainedRecoveryInput.sources.map((source, index) => prepareProviderSource(source, { index, expiresIn, fetchImpl: this.fetchImpl, signal: input.signal })));
298
305
  await input.throwIfCancelled?.();
306
+ const discoveryMs = performance.now() - discoveryStarted;
299
307
  const huggingFace = await this.planHuggingFaceUploads({
300
308
  sources: retainedRecoveryInput.sources,
301
309
  preparedSources,
@@ -346,6 +354,9 @@ export class BeamClient {
346
354
  const routeStreamLock = new AsyncMutex();
347
355
  const releaseInitialStream = await routeStreamLock.acquire();
348
356
  const streamPreparedRoutes = async (routeGenerationId, recoveryReplay) => {
357
+ const telemetry = new SdkPerformanceCollector();
358
+ if (!recoveryReplay)
359
+ telemetry.observe("sdk.discovery", discoveryMs);
349
360
  const pendingRoutes = new Set();
350
361
  let signingConcurrency = this.routeSigningConcurrency;
351
362
  let signedInWindow = 0;
@@ -376,6 +387,15 @@ export class BeamClient {
376
387
  signedUrlFlow: requestedSignedUrlFlow
377
388
  });
378
389
  routeStream = new RouteStreamSender(this.control, {
390
+ telemetry,
391
+ onDiagnostics: (summary) => {
392
+ if (!this.onDiagnostics || this.diagnosticPending)
393
+ return;
394
+ this.diagnosticPending = true;
395
+ setTimeout(() => {
396
+ void Promise.resolve().then(() => this.onDiagnostics?.(summary)).catch(() => { }).finally(() => { this.diagnosticPending = false; });
397
+ }, 0);
398
+ },
379
399
  streamId,
380
400
  transferId: prepared.transfer_id,
381
401
  routeGenerationId,
@@ -388,7 +408,7 @@ export class BeamClient {
388
408
  routeStreamBeginAttempted = true;
389
409
  await routeStream.begin();
390
410
  multipartGroupWaiters = createMultipartGroupWaiters(prepared, destinationsById);
391
- multipartManifestTask = createMultipartGroupManifest({
411
+ multipartManifestTask = telemetry.measure("sdk.multipart_create", () => createMultipartGroupManifest({
392
412
  prepared,
393
413
  destinationsById,
394
414
  multipartUploads,
@@ -405,7 +425,7 @@ export class BeamClient {
405
425
  multipartGroupWaiters.get(state.manifest.multipart_group_id)?.resolve(state);
406
426
  },
407
427
  onGroupFailed: (groupId, error) => multipartGroupWaiters.get(groupId)?.reject(error)
408
- });
428
+ }));
409
429
  const streamNextCompletedRoute = async () => {
410
430
  const settled = await Promise.race([...pendingRoutes].map((pending) => pending.then((route) => ({ pending, route }))));
411
431
  pendingRoutes.delete(settled.pending);
@@ -423,6 +443,12 @@ export class BeamClient {
423
443
  };
424
444
  for (const chunk of materializePlanChunks(prepared.plan_descriptor, prepared.transfer_id)) {
425
445
  await throwIfCancelled?.(prepared.transfer_id);
446
+ // This promise belongs to this chunk and signing generation only. Its
447
+ // immutable credential/range identity and actual expiry cannot be renewed.
448
+ const sourceGrant = signSourceChunk({ source: sourcesById.get(chunk.source_id), fallbackUrl: chunk.source_url, chunk, expiresIn, fetchImpl: this.fetchImpl });
449
+ void sourceGrant.catch(() => { });
450
+ telemetry.counters.source_signatures++;
451
+ telemetry.counters.source_reuses += Math.max(0, chunk.destinations.length - 1);
426
452
  for (const target of chunk.destinations) {
427
453
  const pendingRoute = (async () => {
428
454
  await throwIfCancelled?.(prepared.transfer_id);
@@ -442,10 +468,11 @@ export class BeamClient {
442
468
  : await multipartGroupWaiters.get(multipartGroupId)?.promise;
443
469
  if (!isDirectPutDestination(destination) && !upload)
444
470
  throw new Error(`multipart group manifest is missing for ${chunk.source_id}:${target.destination_id}`);
445
- return this.signProviderRoute({
471
+ return telemetry.measure("sdk.signing", () => this.signProviderRoute({
446
472
  chunk,
447
473
  target,
448
474
  source,
475
+ sourceGrant,
449
476
  destination,
450
477
  expiresIn,
451
478
  ...(upload ? { upload } : {}),
@@ -456,7 +483,7 @@ export class BeamClient {
456
483
  partNumber: typeof metadata.part_number === "number"
457
484
  ? metadata.part_number
458
485
  : multipartPartNumber(chunk.source_chunk_index),
459
- });
486
+ }));
460
487
  })();
461
488
  pendingRoutes.add(pendingRoute);
462
489
  if (pendingRoutes.size >= signingConcurrency)
@@ -475,8 +502,8 @@ export class BeamClient {
475
502
  }
476
503
  catch (error) {
477
504
  if (input.signal?.aborted) {
478
- this.stopRecoverySigner(prepared.transfer_id);
479
- this.control.releaseRecoveryLease(prepared.transfer_id);
505
+ this.stopRecoverySigner(prepared.transfer_id, recoveryLease);
506
+ this.control.releaseRecoveryLease(prepared.transfer_id, recoveryLease);
480
507
  await Promise.allSettled(pendingRoutes);
481
508
  if (multipartManifestTask)
482
509
  await Promise.allSettled([multipartManifestTask]);
@@ -489,10 +516,10 @@ export class BeamClient {
489
516
  await Promise.allSettled([multipartManifestTask]);
490
517
  if (!recoveryReplay) {
491
518
  if (foregroundCancelled) {
492
- this.control.continueRecoveryLease(prepared.transfer_id);
519
+ this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
493
520
  }
494
521
  else if (isRecoverableRouteStreamError(error)) {
495
- this.control.continueRecoveryLease(prepared.transfer_id);
522
+ this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
496
523
  throw new BeamRouteRecoveryPendingError(prepared.transfer_id, error);
497
524
  }
498
525
  else {
@@ -510,27 +537,16 @@ export class BeamClient {
510
537
  // The retained recovery input owns the same destination objects used by
511
538
  // multipart cleanup. Releasing it earlier scrubs their scoped credentials
512
539
  // before abortMultipartUpload can clean up already-created uploads.
513
- this.control.releaseRecoveryLease(prepared.transfer_id);
540
+ this.control.releaseRecoveryLease(prepared.transfer_id, recoveryLease);
514
541
  }
515
542
  }
516
543
  }
517
544
  throw error;
518
545
  }
519
546
  };
520
- await this.startProviderRouteRecoverySigner({
521
- prepared,
522
- sourcesById,
523
- destinationsById,
524
- multipartUploads: recoveryMultipartUploads,
525
- signal: input.signal,
526
- expiresIn
527
- });
528
- const stopOwnedRecovery = () => {
529
- this.stopRecoverySigner(prepared.transfer_id);
530
- this.control.releaseRecoveryLease(prepared.transfer_id);
531
- };
532
- input.signal?.addEventListener("abort", stopOwnedRecovery, { once: true });
533
- this.control.registerRecoveryLease({
547
+ // The lease object is this owner's identity: releases, recovery requests, and
548
+ // signer shutdowns below only act while it is still the registered lease.
549
+ const recoveryLease = {
534
550
  transferId: prepared.transfer_id,
535
551
  planFingerprint: prepared.plan_fingerprint,
536
552
  coordinateChecksum: prepared.coordinate_checksum,
@@ -539,23 +555,39 @@ export class BeamClient {
539
555
  },
540
556
  disposeSecrets: () => {
541
557
  input.signal?.removeEventListener("abort", stopOwnedRecovery);
558
+ releaseProviderClients([...retainedRecoveryInput.sources, ...retainedRecoveryInput.destinations]);
542
559
  clearRecoverySecrets(retainedRecoveryInput);
543
560
  }
561
+ };
562
+ await this.startProviderRouteRecoverySigner({
563
+ owner: recoveryLease,
564
+ prepared,
565
+ sourcesById,
566
+ destinationsById,
567
+ multipartUploads: recoveryMultipartUploads,
568
+ signal: input.signal,
569
+ expiresIn
544
570
  });
571
+ const stopOwnedRecovery = () => {
572
+ this.stopRecoverySigner(prepared.transfer_id, recoveryLease);
573
+ this.control.releaseRecoveryLease(prepared.transfer_id, recoveryLease);
574
+ };
575
+ input.signal?.addEventListener("abort", stopOwnedRecovery, { once: true });
576
+ this.control.registerRecoveryLease(recoveryLease);
545
577
  try {
546
578
  try {
547
579
  assertOwnership();
548
580
  await input.onPrepared?.(prepared);
549
581
  }
550
582
  catch (error) {
551
- this.control.continueRecoveryLease(prepared.transfer_id);
583
+ this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
552
584
  throw error;
553
585
  }
554
586
  try {
555
587
  await input.throwIfCancelled?.(prepared.transfer_id);
556
588
  }
557
589
  catch (error) {
558
- this.control.continueRecoveryLease(prepared.transfer_id);
590
+ this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
559
591
  throw error;
560
592
  }
561
593
  await streamPreparedRoutes(prepared.route_generation_id, false);
@@ -740,6 +772,7 @@ export class BeamClient {
740
772
  metadata: directPutRouteMetadata(input.target.metadata)
741
773
  },
742
774
  source: input.source,
775
+ sourceGrant: input.sourceGrant,
743
776
  destination: input.destination,
744
777
  expiresIn: input.expiresIn,
745
778
  destUrl,
@@ -751,6 +784,7 @@ export class BeamClient {
751
784
  chunk: input.chunk,
752
785
  target: { ...input.target, metadata: partRouteMetadata(input.target.metadata) },
753
786
  source: input.source,
787
+ sourceGrant: input.sourceGrant,
754
788
  destination: input.destination,
755
789
  expiresIn: input.expiresIn,
756
790
  fetchImpl: this.fetchImpl
@@ -768,6 +802,7 @@ export class BeamClient {
768
802
  metadata: partRouteMetadata(input.target.metadata)
769
803
  },
770
804
  source: input.source,
805
+ sourceGrant: input.sourceGrant,
771
806
  destination: input.destination,
772
807
  expiresIn: input.expiresIn,
773
808
  uploadId: upload.uploadId,
@@ -791,6 +826,7 @@ export class BeamClient {
791
826
  input.signal?.throwIfAborted();
792
827
  if (payload.route_generation_id.length === 0)
793
828
  throw new Error("route recovery generation is required");
829
+ const sourceGrants = new Map();
794
830
  const routes = await mapOrderedWithConcurrency(payload.chunks, this.routeSigningConcurrency, async (requested) => {
795
831
  input.signal?.throwIfAborted();
796
832
  if (requested.route_generation_id !== payload.route_generation_id) {
@@ -821,11 +857,22 @@ export class BeamClient {
821
857
  multipartUploads: input.multipartUploads,
822
858
  expiresIn: input.expiresIn
823
859
  });
824
- return this.signProviderRoute({
860
+ const sourceKey = JSON.stringify([chunk.source_id, chunk.chunk_index]);
861
+ let sourceGrant = sourceGrants.get(sourceKey);
862
+ if (!sourceGrant) {
863
+ sourceGrant = signSourceChunk({ source, chunk, fallbackUrl: chunk.source_url, expiresIn: input.expiresIn, fetchImpl: this.fetchImpl });
864
+ sourceGrants.set(sourceKey, sourceGrant);
865
+ void sourceGrant.catch(() => { });
866
+ }
867
+ const route = await this.signProviderRoute({
868
+ sourceGrant,
825
869
  chunk,
826
870
  target: {
827
871
  ...target,
828
- object_key: requested.final_object_key,
872
+ // Direct-PUT destinations (Hippius, Hugging Face) keep the planned
873
+ // per-chunk key exactly as initial materialization does; only
874
+ // multipart destinations sign parts against the final object key.
875
+ object_key: isDirectPutDestination(destination) ? target.object_key : requested.final_object_key,
829
876
  metadata: {
830
877
  ...(target.metadata ?? {}),
831
878
  ...(requested.destination_metadata ?? {}),
@@ -852,6 +899,7 @@ export class BeamClient {
852
899
  signedUrlFlow: "signed_url",
853
900
  partNumber: requested.part_number
854
901
  });
902
+ return signMultipartRecovery({ destination, transferId: input.prepared.transfer_id, requested, route, expiresIn: input.expiresIn });
855
903
  });
856
904
  return {
857
905
  transfer_id: payload.transfer_id,
@@ -860,16 +908,33 @@ export class BeamClient {
860
908
  chunk_routes: routes
861
909
  };
862
910
  });
863
- this.recoverySigners.set(input.prepared.transfer_id, stop);
864
- this.integrityAuditSigners.set(input.prepared.transfer_id, async (challenge) => {
865
- await this.submitProviderIntegrityAuditGrants({
866
- challenge,
867
- prepared: input.prepared,
868
- sourcesById: input.sourcesById,
869
- destinationsById: input.destinationsById,
870
- expiresIn: input.expiresIn
911
+ if (input.signal?.aborted) {
912
+ stop();
913
+ return;
914
+ }
915
+ this.stopRecoverySigner(input.prepared.transfer_id);
916
+ this.integrityAuditSigners.set(input.prepared.transfer_id, (challenge) => this.buildProviderIntegrityAuditGrants({
917
+ challenge, prepared: input.prepared, sourcesById: input.sourcesById, destinationsById: input.destinationsById, expiresIn: input.expiresIn
918
+ }));
919
+ const registration = { owner: input.owner, stop };
920
+ this.recoverySigners.set(input.prepared.transfer_id, registration);
921
+ try {
922
+ const stopIntegrity = await this.control.serveIntegritySigner(input.prepared.transfer_id, challenge => {
923
+ input.signal?.throwIfAborted();
924
+ return this.signedIntegrityGrants(challenge);
871
925
  });
872
- });
926
+ if (input.signal?.aborted || this.recoverySigners.get(input.prepared.transfer_id) !== registration) {
927
+ stop();
928
+ stopIntegrity();
929
+ return;
930
+ }
931
+ registration.stop = () => { stop(); stopIntegrity(); };
932
+ }
933
+ catch {
934
+ // Status-driven signing remains available if the optional subscription fails.
935
+ if (input.signal?.aborted)
936
+ this.stopRecoverySigner(input.prepared.transfer_id, input.owner);
937
+ }
873
938
  }
874
939
  async recoveryMultipartUploadState(input) {
875
940
  const existing = input.multipartUploads.get(input.requested.multipart_group_id);
@@ -936,21 +1001,56 @@ export class BeamClient {
936
1001
  if (!signer)
937
1002
  throw new Error("integrity audit signer unavailable");
938
1003
  const existing = this.integrityAuditSubmissions.get(challenge.audit_id);
939
- if (existing) {
1004
+ const cached = this.integrityGrantCache.get(challenge.audit_id);
1005
+ if (cached && cached.fingerprint !== JSON.stringify(challenge))
1006
+ throw new Error("integrity audit identity changed");
1007
+ if (existing && cached && cached.expiresAt > Date.now() + 30_000) {
940
1008
  await existing;
941
1009
  return;
942
1010
  }
943
- const submission = signer(challenge);
1011
+ const submission = this.signedIntegrityGrants(challenge).then(async (payload) => {
1012
+ const result = await this.control.request("transfer.integrity_audit_grants", payload, {
1013
+ transferId: challenge.transfer_id, idempotencyKey: `transfer:${challenge.transfer_id}:integrity-audit:${challenge.audit_id}:${payload.submitted_at}`
1014
+ });
1015
+ if (result.published !== true)
1016
+ throw new Error("integrity audit delivery unavailable");
1017
+ });
944
1018
  this.integrityAuditSubmissions.set(challenge.audit_id, submission);
945
1019
  try {
946
1020
  await submission;
947
1021
  }
948
1022
  catch (error) {
949
- this.integrityAuditSubmissions.delete(challenge.audit_id);
1023
+ if (this.integrityAuditSubmissions.get(challenge.audit_id) === submission)
1024
+ this.integrityAuditSubmissions.delete(challenge.audit_id);
950
1025
  throw error;
951
1026
  }
952
1027
  }
953
- async submitProviderIntegrityAuditGrants(input) {
1028
+ signedIntegrityGrants(challenge) {
1029
+ const fingerprint = JSON.stringify(challenge);
1030
+ const cached = this.integrityGrantCache.get(challenge.audit_id);
1031
+ if (cached && cached.fingerprint !== fingerprint)
1032
+ return Promise.reject(new Error("integrity audit identity changed"));
1033
+ if (cached && cached.expiresAt > Date.now() + 30_000)
1034
+ return cached.payload;
1035
+ if (!cached && this.integrityGrantCache.size >= 1024)
1036
+ return Promise.reject(new Error("integrity signer capacity unavailable"));
1037
+ const signer = this.integrityAuditSigners.get(challenge.transfer_id);
1038
+ if (!signer)
1039
+ return Promise.reject(new Error("integrity audit signer unavailable"));
1040
+ this.integrityAuditSubmissions.delete(challenge.audit_id);
1041
+ const entry = { transferId: challenge.transfer_id, expiresAt: 0, fingerprint, payload: Promise.resolve({}) };
1042
+ entry.payload = signer(challenge).then(payload => {
1043
+ const chunks = payload.chunks;
1044
+ entry.expiresAt = Math.min(...chunks.flatMap(chunk => [Date.parse(chunk.source.expires_at), Date.parse(chunk.destination.expires_at)]));
1045
+ return payload;
1046
+ }).catch(error => { if (this.integrityGrantCache.get(challenge.audit_id) === entry)
1047
+ this.integrityGrantCache.delete(challenge.audit_id); throw error; });
1048
+ // Pending grants are shared too. Actual expiry replaces this sentinel on completion.
1049
+ entry.expiresAt = Infinity;
1050
+ this.integrityGrantCache.set(challenge.audit_id, entry);
1051
+ return entry.payload;
1052
+ }
1053
+ async buildProviderIntegrityAuditGrants(input) {
954
1054
  if (input.challenge.transfer_id !== input.prepared.transfer_id) {
955
1055
  throw new Error("integrity audit challenge transfer mismatch");
956
1056
  }
@@ -977,6 +1077,7 @@ export class BeamClient {
977
1077
  }
978
1078
  const sourceEtag = plannedSource?.metadata?.etag;
979
1079
  const sourceVersionId = plannedSource?.metadata?.version_id;
1080
+ const grantExpiresAt = new Date(Math.floor(Date.now() / 1000) * 1000 + input.expiresIn * 1000).toISOString();
980
1081
  const [sourceGrant, destinationGrant] = await Promise.all([
981
1082
  signSourceReadRange({
982
1083
  source,
@@ -997,31 +1098,30 @@ export class BeamClient {
997
1098
  fetchImpl: this.fetchImpl
998
1099
  })
999
1100
  ]);
1000
- const grantExpiresAt = expiresAtIso(input.expiresIn);
1001
1101
  const { orchestrator_id: _orchestratorId, orchestrator_hotkey: _orchestratorHotkey, worker_id: _workerId, ...grantChunk } = chunk;
1002
1102
  return {
1003
1103
  ...grantChunk,
1004
- source: { ...sourceGrant, expires_at: grantExpiresAt },
1005
- destination: { ...destinationGrant, expires_at: grantExpiresAt }
1104
+ source: { ...sourceGrant, expires_at: boundedGrantExpiry(sourceGrant.url, grantExpiresAt) },
1105
+ destination: { ...destinationGrant, expires_at: boundedGrantExpiry(destinationGrant.url, grantExpiresAt) }
1006
1106
  };
1007
1107
  });
1008
- await this.control.request("transfer.integrity_audit_grants", {
1009
- transfer_id: input.challenge.transfer_id,
1010
- audit_id: input.challenge.audit_id,
1011
- submitted_at: submittedAt,
1012
- chunks
1013
- }, {
1014
- transferId: input.challenge.transfer_id,
1015
- idempotencyKey: `transfer:${input.challenge.transfer_id}:integrity-audit:${input.challenge.audit_id}`
1016
- });
1108
+ return { transfer_id: input.challenge.transfer_id, audit_id: input.challenge.audit_id, submitted_at: submittedAt, chunks };
1017
1109
  }
1018
- stopRecoverySigner(transferId) {
1019
- const stop = this.recoverySigners.get(transferId);
1020
- if (stop) {
1021
- stop();
1110
+ /** Stop a transfer's recovery and integrity signers; with `owner`, only if that owner installed them. */
1111
+ stopRecoverySigner(transferId, owner) {
1112
+ const signer = this.recoverySigners.get(transferId);
1113
+ if (owner && signer?.owner !== owner)
1114
+ return;
1115
+ if (signer) {
1116
+ signer.stop();
1022
1117
  this.recoverySigners.delete(transferId);
1023
1118
  }
1024
1119
  this.integrityAuditSigners.delete(transferId);
1120
+ for (const [auditId, entry] of this.integrityGrantCache)
1121
+ if (entry.transferId === transferId) {
1122
+ this.integrityGrantCache.delete(auditId);
1123
+ this.integrityAuditSubmissions.delete(auditId);
1124
+ }
1025
1125
  }
1026
1126
  stopAllRecoverySigners() {
1027
1127
  for (const transferId of [...this.recoverySigners.keys()])
@@ -1117,7 +1217,7 @@ export class BeamClient {
1117
1217
  }
1118
1218
  }
1119
1219
  }
1120
- const ROUTE_STREAM_BATCH_ROUTES = 2_048;
1220
+ const ROUTE_STREAM_BATCH_ROUTES = 1_024;
1121
1221
  class AsyncMutex {
1122
1222
  tail = Promise.resolve();
1123
1223
  async acquire() {
@@ -1142,6 +1242,8 @@ class AsyncMutex {
1142
1242
  class RouteStreamSender {
1143
1243
  control;
1144
1244
  options;
1245
+ started = performance.now();
1246
+ telemetry;
1145
1247
  streamId;
1146
1248
  checksum = new RouteKeysChecksum();
1147
1249
  batch = [];
@@ -1154,6 +1256,7 @@ class RouteStreamSender {
1154
1256
  this.control = control;
1155
1257
  this.options = options;
1156
1258
  this.streamId = options.streamId;
1259
+ this.telemetry = options.telemetry ?? new SdkPerformanceCollector();
1157
1260
  }
1158
1261
  async begin() {
1159
1262
  await this.control.request("transfer.route_stream.begin", compact({
@@ -1215,7 +1318,11 @@ class RouteStreamSender {
1215
1318
  await this.sendTail;
1216
1319
  if (this.sendError)
1217
1320
  throw this.sendError;
1321
+ this.telemetry.observe("sdk.preparation", performance.now() - this.started);
1322
+ const sdkPerformance = this.telemetry.snapshot();
1323
+ this.options.onDiagnostics?.(sdkPerformance);
1218
1324
  return this.control.request("transfer.route_stream.complete", {
1325
+ sdk_performance: sdkPerformance,
1219
1326
  transfer_id: this.options.transferId,
1220
1327
  route_generation_id: this.options.routeGenerationId,
1221
1328
  stream_id: this.streamId,
@@ -1230,7 +1337,7 @@ class RouteStreamSender {
1230
1337
  async enqueueFlush() {
1231
1338
  if (!this.batch.length)
1232
1339
  return;
1233
- await this.sendTail;
1340
+ await this.telemetry.measure("sdk.buffer_wait", () => this.sendTail);
1234
1341
  if (this.sendError)
1235
1342
  throw this.sendError;
1236
1343
  const routes = this.batch.splice(0, this.batch.length);
@@ -1254,7 +1361,8 @@ class RouteStreamSender {
1254
1361
  const routeChecksum = chunkChecksum.value();
1255
1362
  const coordinateChecksum = await routeCoordinateChecksum(chunk);
1256
1363
  const batchId = `${this.streamId}:${batchIndex}:${coordinateChecksum}`;
1257
- await this.control.request("transfer.route_stream.batch", {
1364
+ this.telemetry.counters.route_batches++;
1365
+ await this.telemetry.measure("sdk.batch_ack", () => this.control.request("transfer.route_stream.batch", {
1258
1366
  transfer_id: this.options.transferId,
1259
1367
  route_generation_id: this.options.routeGenerationId,
1260
1368
  stream_id: this.streamId,
@@ -1266,7 +1374,7 @@ class RouteStreamSender {
1266
1374
  }, {
1267
1375
  transferId: this.options.transferId,
1268
1376
  idempotencyKey: `transfer:${this.options.transferId}:route-stream:${this.streamId}:batch:${batchIndex}:${coordinateChecksum}`
1269
- });
1377
+ }));
1270
1378
  }
1271
1379
  catch (error) {
1272
1380
  this.sendError = error;
@@ -1510,6 +1618,9 @@ function createDeferred() {
1510
1618
  resolve = resolvePromise;
1511
1619
  reject = rejectPromise;
1512
1620
  });
1621
+ // A waiter may be rejected after its route loop already stopped (for example when
1622
+ // ownership is fenced mid-stream); awaiting callers still observe the rejection.
1623
+ promise.catch(() => undefined);
1513
1624
  return { promise, resolve, reject };
1514
1625
  }
1515
1626
  function createMultipartGroupWaiters(prepared, destinationsById) {
@@ -1661,7 +1772,7 @@ function multipartListPageMarkers(maxPartNumber) {
1661
1772
  return markers;
1662
1773
  }
1663
1774
  function multipartMaxPartNumber(chunkCount) {
1664
- return multipartPartNumber(positiveInteger(chunkCount, "source chunk_count") - 1, 2);
1775
+ return multipartPartNumber(positiveInteger(chunkCount, "source chunk_count") - 1);
1665
1776
  }
1666
1777
  function validateMultipartPartNumber(partNumber, manifest) {
1667
1778
  if (!Number.isInteger(partNumber) || partNumber < 1 || partNumber > manifest.max_part_number) {
@@ -1715,7 +1826,7 @@ const COMPACT_TRANSFER_PLAN_KEYS = new Set([
1715
1826
  const COMPACT_TRANSFER_PLAN_FORMULAS = {
1716
1827
  source_offset: "source_chunk_index * chunk_size",
1717
1828
  delivery_index: "chunk_index * destination_count + destination_index",
1718
- part_number: "source_chunk_index * 3 + attempt_slot + 1",
1829
+ part_number: "source_chunk_index + 1",
1719
1830
  route_generation_id: "initial-{chunk_index}-{destination_id}"
1720
1831
  };
1721
1832
  function validateCompactTransferPlan(descriptor, signedUrlFlow) {
@@ -1729,7 +1840,7 @@ function validateCompactTransferPlan(descriptor, signedUrlFlow) {
1729
1840
  if (descriptor.version !== "compact-transfer-plan/v1") {
1730
1841
  throw new Error(`BeamCore returned unsupported plan version: ${descriptor.version}`);
1731
1842
  }
1732
- if (descriptor.multipart_attempt_slots !== 3) {
1843
+ if (descriptor.multipart_attempt_slots !== 1) {
1733
1844
  throw new Error("BeamCore returned unsupported multipart attempt slot count");
1734
1845
  }
1735
1846
  const formulaKeys = Object.keys(descriptor.formulas);
@@ -1782,7 +1893,7 @@ function validateMultipartGroupManifest(manifest, transferId) {
1782
1893
  if (!Number.isInteger(group.expected_part_count) || group.expected_part_count < 1 || group.expected_part_count > 10_000) {
1783
1894
  throw new Error(`multipart group ${group.multipart_group_id} has invalid expected_part_count`);
1784
1895
  }
1785
- if (!Number.isInteger(group.max_part_number) || group.max_part_number < group.expected_part_count || group.max_part_number > 10_000) {
1896
+ if (!Number.isInteger(group.max_part_number) || group.max_part_number !== group.expected_part_count || group.max_part_number > 10_000) {
1786
1897
  throw new Error(`multipart group ${group.multipart_group_id} has invalid max_part_number`);
1787
1898
  }
1788
1899
  if (!Number.isSafeInteger(group.expected_object_size) || group.expected_object_size <= 0) {
@@ -1810,7 +1921,8 @@ const PART_ROUTE_METADATA_KEYS = new Set([
1810
1921
  "logical_attempt_index",
1811
1922
  "attempt_slot",
1812
1923
  "route_generation_id",
1813
- "delivery_index"
1924
+ "delivery_index",
1925
+ "etag_required"
1814
1926
  ]);
1815
1927
  function partRouteMetadata(metadata) {
1816
1928
  return Object.fromEntries(Object.entries(metadata ?? {}).filter(([key]) => PART_ROUTE_METADATA_KEYS.has(key)));