@beam-network/sdk 0.5.21 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -4
- package/dist/client.d.ts +6 -1
- package/dist/client.js +183 -71
- package/dist/models.d.ts +46 -3
- package/dist/multipart-limits.d.ts +1 -1
- package/dist/multipart-limits.js +3 -3
- package/dist/nats-control.d.ts +21 -4
- package/dist/nats-control.js +106 -31
- package/dist/performance.d.ts +12 -0
- package/dist/performance.js +25 -0
- package/dist/provider-signing.d.ts +29 -0
- package/dist/provider-signing.js +105 -9
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,5 +1,10 @@
|
|
|
1
1
|
# BEAM SDK for TypeScript
|
|
2
2
|
|
|
3
|
+
This release requires Core's `transfer-client-control/v7` contract. Multipart uploads use consecutive parts;
|
|
4
|
+
Core may request retained staging for recovery through the existing SDK signing connection. Workers receive
|
|
5
|
+
ordinary upload URLs. Keep the client connected until the transfer reaches a terminal state so Core can renew
|
|
6
|
+
grants, promote recovered data, and clean up staging. Drain active transfers before upgrading Core and SDK consumers.
|
|
7
|
+
|
|
3
8
|
Install:
|
|
4
9
|
|
|
5
10
|
```bash
|
|
@@ -149,21 +154,21 @@ For low-level raw transfer configs, use `createRawTransfer`; lifecycle transport
|
|
|
149
154
|
|
|
150
155
|
## Route Streaming And Payload Size
|
|
151
156
|
|
|
152
|
-
Provider transfers use `transfer-client-control/
|
|
157
|
+
Provider transfers use `transfer-client-control/v7`. Every reply carries Runtime and transport epochs; prepare and every route-stream message carry a UUID route generation. S3, R2, and S3-compatible destinations retain direct multipart UploadPart/ListParts/HEAD handling. Hippius and Hugging Face destinations take a plain PUT per chunk instead, so they carry no multipart group manifest.
|
|
153
158
|
|
|
154
|
-
The client signs up to 64 routes concurrently by default and emits
|
|
159
|
+
The client signs up to 64 routes concurrently by default and emits 1,024-route logical batches. Encoded MessagePack requests target 4 MiB physical batches under `maxPayloadBytes` (8 MiB by default; explicit positive overrides remain supported). The splitter reserves control-envelope headroom for the live auth token and stable request identity so the final encoded request remains below the configured guard. Lifecycle control refreshes cached SDK auth before the final 30 seconds of token lifetime and retries `auth_token_expired` replies with the same request identity and payload but a fresh token, preserving Core request-conflict protection during large route streams. A single route above the 4 MiB target but within the configured guard publishes alone; a single route larger than the configured limit fails before publication with both sizes in the error. Multipart create and abort requests use a separate `multipartControlConcurrency` limit of 2 by default, with explicit positive overrides supported, and S3-compatible control requests make up to five attempts for transient provider failures. This keeps URL signing throughput independent from provider control-plane pressure.
|
|
155
160
|
|
|
156
161
|
Callers that need durable recovery accounting can provide `onMultipartGroupReady`. The SDK invokes it after creating a multipart upload and before streaming that group’s signed routes. Its payload contains transfer, source, destination, object-key, upload, size, part-count, and expiry identities only; it never includes credentials, request headers, or signed URLs. A rejected callback aborts the newly created upload and fails preparation closed.
|
|
157
162
|
|
|
158
163
|
`resumeProviderTransfer` requires the complete saved callback payloads in `multipartGroups`. It validates every transfer/source/destination/object/size/part coordinate, then uses the same route-stream implementation as creation with those upload IDs. Missing, duplicate, or changed identities fail before new provider uploads or route publication. If a process died before recording an upload identity, treat it as uncertain cleanup instead of inventing a replacement. The `onPrepared` callback also runs during resume, before routes are streamed.
|
|
159
164
|
|
|
160
|
-
Create and resume accept an ownership `signal`. Aborting it stops initial routes, multipart creation, and recovery signing/replay; it does not cancel the transfer or abort a replacement owner's uploads. User cancellation still uses `cancelTransfer` and verified provider cleanup. `prepareProviderSource` accepts the same signal for metadata requests.
|
|
165
|
+
Create and resume accept an ownership `signal`. Aborting it stops initial routes, multipart creation, and recovery signing/replay; it does not cancel the transfer or abort a replacement owner's uploads. Recovery leases, signers, and background resume/replay are keyed by owner identity: a fenced-off owner never replays into, releases, or stops the signers of a replacement owner registered for the same transfer, and a replacement always gets its own recovery. User cancellation still uses `cancelTransfer` and verified provider cleanup. `prepareProviderSource` accepts the same signal for metadata requests.
|
|
161
166
|
|
|
162
167
|
The stream ID derives from the transfer, selected flow, and immutable plan identity, and each ordered batch ID includes its route-coordinate checksum. Manual multi-destination attachment requires `delivery_index` on every route. Lifecycle mutations retry transient failures up to three times with the same request identity. A non-recoverable provider setup or route failure raises `BeamProviderTransferError`, whose `errors`, `transferCancelled`, and `multipartCleanupComplete` fields retain the original provider, cancellation, and cleanup outcomes. `waitForTransfer` subscribes to the API-key-owned, at-most-once terminal signal before its first status read and reconciles every signal through authoritative status; subscription failure degrades to the same jittered 15-to-30-second status fallback. Call `close()` when the client is no longer needed; it also closes outstanding terminal waiters.
|
|
163
168
|
|
|
164
169
|
During the pre-completion `integrity_check` phase, Runtime may include an `integrity_audit_challenge` in status. The TypeScript client signs exact read-only source and final-destination GET ranges and submits `transfer.integrity_audit_grants`. S3-compatible source and destination grants use the prepared source ETag and Runtime-verified final-object ETag as `If-Match` conditions. Submission failures retry on the next status poll and appear as a URL-safe `integrity_audit_submission_error` on that status response; the transfer completes with a warning if the check cannot finish by its deadline.
|
|
165
170
|
|
|
166
|
-
Hippius uses canonical non-multipart `signed_url` routes, so its manifest is empty and group-level final HEAD verification is not available from that provider flow.
|
|
171
|
+
Hippius uses canonical non-multipart `signed_url` routes (initial and recovered routes both PUT to the planned `<final>/<plan_nonce>/chunk-NNNNNN` key), so its manifest is empty and group-level final HEAD verification is not available from that provider flow.
|
|
167
172
|
|
|
168
173
|
NATS requires this guard because the broker rejects messages above its configured `max_payload`. This limit applies only to lifecycle/control metadata; transfer file bytes do not flow through NATS.
|
|
169
174
|
|
|
@@ -184,3 +189,9 @@ Only short-lived range/upload routes go to workers. Multipart creation, verifica
|
|
|
184
189
|
Manual releases from `dev` require a committed `-dev.N` version and publish only to npm tag `dev`; the stable tag is unchanged.
|
|
185
190
|
|
|
186
191
|
Multipart control helpers (`createMultipartUpload`, `listMultipartParts`, `completeMultipartUpload`, `inspectDestinationObject`, and `abortMultipartUpload`) accept an optional `AbortSignal` and cancel through the existing provider HTTP transport. Pass it as `signal` in object arguments or as the last positional argument. Cancellation stops waiting and network activity; it cannot undo a provider operation already accepted. After an interrupted completion, inspect the durable upload/object identity before retrying or reporting cleanup complete.
|
|
192
|
+
|
|
193
|
+
### Preparation diagnostics
|
|
194
|
+
|
|
195
|
+
Route streaming flushes at 1,024 routes, with no timer, and sends the final partial batch at stream completion. Signing overlaps acknowledged publication through bounded buffering; existing encoded-payload limits still apply. A source grant is reused across destinations only within the same signing generation and actual expiry.
|
|
196
|
+
|
|
197
|
+
The optional `onDiagnostics` callback receives bounded preparation measurements and source-grant reuse counts. Callbacks are best effort, may be dropped under load, and do not affect transfer outcomes. Durations may overlap; do not sum them into elapsed transfer time. Transfer status also exposes a typed `performance` summary when supported by Core.
|
package/dist/client.d.ts
CHANGED
|
@@ -44,8 +44,11 @@ export declare class BeamClient {
|
|
|
44
44
|
private readonly routeSigningConcurrency;
|
|
45
45
|
private readonly routeSigningConcurrencyOverridden;
|
|
46
46
|
private readonly multipartControlConcurrency;
|
|
47
|
+
private readonly onDiagnostics;
|
|
48
|
+
private diagnosticPending;
|
|
47
49
|
private readonly recoverySigners;
|
|
48
50
|
private readonly integrityAuditSigners;
|
|
51
|
+
private readonly integrityGrantCache;
|
|
49
52
|
private readonly integrityAuditSubmissions;
|
|
50
53
|
private readonly huggingFaceUploads;
|
|
51
54
|
constructor(options?: BeamClientOptions);
|
|
@@ -105,7 +108,9 @@ export declare class BeamClient {
|
|
|
105
108
|
private startProviderRouteRecoverySigner;
|
|
106
109
|
private recoveryMultipartUploadState;
|
|
107
110
|
private submitIntegrityAuditGrantsIfPresent;
|
|
108
|
-
private
|
|
111
|
+
private signedIntegrityGrants;
|
|
112
|
+
private buildProviderIntegrityAuditGrants;
|
|
113
|
+
/** Stop a transfer's recovery and integrity signers; with `owner`, only if that owner installed them. */
|
|
109
114
|
private stopRecoverySigner;
|
|
110
115
|
private stopAllRecoverySigners;
|
|
111
116
|
createAndDistribute(input: RawTransferCreateInput): Promise<TransferCreateResponse>;
|
package/dist/client.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
|
-
import {
|
|
1
|
+
import { SdkPerformanceCollector } from "./performance.js";
|
|
2
|
+
import { signMultipartRecovery, abortMultipartUpload, createMultipartUpload, expiresAtIso, prepareProviderDestination, prepareProviderSource, releaseProviderClients, signAbortMultipartUpload, signCompleteMultipartUpload, signDestinationReadRange, signDestinationRoute, signFinalObjectHead, signListMultipartUpload, signSourceReadRange, signSourceChunk, boundedGrantExpiry, isHuggingFaceProvider } from "./provider-signing.js";
|
|
2
3
|
import { describe as describeHuggingFace, hashSourceStream, huggingFaceCommit, huggingFaceCompleteLfsUpload, huggingFaceLfsBatch, huggingFacePreupload, huggingFaceVerifyLfsUpload, readSourceSample } from "./huggingface.js";
|
|
3
4
|
import { BeamTransferControl, BEAM_DEFAULT_NATS_URL, compactSignedRoutes, isRecoverableRouteStreamError } from "./nats-control.js";
|
|
4
5
|
export class BeamRouteRecoveryPendingError extends Error {
|
|
@@ -53,11 +54,15 @@ export class BeamClient {
|
|
|
53
54
|
routeSigningConcurrency;
|
|
54
55
|
routeSigningConcurrencyOverridden;
|
|
55
56
|
multipartControlConcurrency;
|
|
57
|
+
onDiagnostics;
|
|
58
|
+
diagnosticPending = false;
|
|
56
59
|
recoverySigners = new Map();
|
|
57
60
|
integrityAuditSigners = new Map();
|
|
61
|
+
integrityGrantCache = new Map();
|
|
58
62
|
integrityAuditSubmissions = new Map();
|
|
59
63
|
huggingFaceUploads = new Map();
|
|
60
64
|
constructor(options = {}) {
|
|
65
|
+
this.onDiagnostics = options.onDiagnostics;
|
|
61
66
|
if (!options.apiKey?.trim()) {
|
|
62
67
|
throw new Error("apiKey is required.");
|
|
63
68
|
}
|
|
@@ -237,7 +242,7 @@ export class BeamClient {
|
|
|
237
242
|
};
|
|
238
243
|
const routeStreamLock = new AsyncMutex();
|
|
239
244
|
const releaseInitialStream = await routeStreamLock.acquire();
|
|
240
|
-
|
|
245
|
+
const recoveryLease = {
|
|
241
246
|
transferId,
|
|
242
247
|
planFingerprint: input.planFingerprint,
|
|
243
248
|
coordinateChecksum: input.coordinateChecksum,
|
|
@@ -249,19 +254,20 @@ export class BeamClient {
|
|
|
249
254
|
throw new Error(attached.error ?? attached.message ?? "route replay failed");
|
|
250
255
|
});
|
|
251
256
|
}
|
|
252
|
-
}
|
|
257
|
+
};
|
|
258
|
+
this.control.registerRecoveryLease(recoveryLease);
|
|
253
259
|
try {
|
|
254
260
|
const attached = await streamRoutes(input.chunkRoutes, input.multipartGroupManifest, input.routeGenerationId, input.urlsExpiresAt);
|
|
255
261
|
if (!attached.success)
|
|
256
|
-
this.control.releaseRecoveryLease(transferId);
|
|
262
|
+
this.control.releaseRecoveryLease(transferId, recoveryLease);
|
|
257
263
|
return attached;
|
|
258
264
|
}
|
|
259
265
|
catch (error) {
|
|
260
266
|
if (isRecoverableRouteStreamError(error)) {
|
|
261
|
-
this.control.continueRecoveryLease(transferId);
|
|
267
|
+
this.control.continueRecoveryLease(transferId, recoveryLease);
|
|
262
268
|
throw new BeamRouteRecoveryPendingError(transferId, error);
|
|
263
269
|
}
|
|
264
|
-
this.control.releaseRecoveryLease(transferId);
|
|
270
|
+
this.control.releaseRecoveryLease(transferId, recoveryLease);
|
|
265
271
|
throw error;
|
|
266
272
|
}
|
|
267
273
|
finally {
|
|
@@ -294,8 +300,10 @@ export class BeamClient {
|
|
|
294
300
|
destinationsById.set(preparedDestination.destination_id, destination);
|
|
295
301
|
});
|
|
296
302
|
await input.throwIfCancelled?.();
|
|
303
|
+
const discoveryStarted = performance.now();
|
|
297
304
|
const preparedSources = await Promise.all(retainedRecoveryInput.sources.map((source, index) => prepareProviderSource(source, { index, expiresIn, fetchImpl: this.fetchImpl, signal: input.signal })));
|
|
298
305
|
await input.throwIfCancelled?.();
|
|
306
|
+
const discoveryMs = performance.now() - discoveryStarted;
|
|
299
307
|
const huggingFace = await this.planHuggingFaceUploads({
|
|
300
308
|
sources: retainedRecoveryInput.sources,
|
|
301
309
|
preparedSources,
|
|
@@ -346,6 +354,9 @@ export class BeamClient {
|
|
|
346
354
|
const routeStreamLock = new AsyncMutex();
|
|
347
355
|
const releaseInitialStream = await routeStreamLock.acquire();
|
|
348
356
|
const streamPreparedRoutes = async (routeGenerationId, recoveryReplay) => {
|
|
357
|
+
const telemetry = new SdkPerformanceCollector();
|
|
358
|
+
if (!recoveryReplay)
|
|
359
|
+
telemetry.observe("sdk.discovery", discoveryMs);
|
|
349
360
|
const pendingRoutes = new Set();
|
|
350
361
|
let signingConcurrency = this.routeSigningConcurrency;
|
|
351
362
|
let signedInWindow = 0;
|
|
@@ -376,6 +387,15 @@ export class BeamClient {
|
|
|
376
387
|
signedUrlFlow: requestedSignedUrlFlow
|
|
377
388
|
});
|
|
378
389
|
routeStream = new RouteStreamSender(this.control, {
|
|
390
|
+
telemetry,
|
|
391
|
+
onDiagnostics: (summary) => {
|
|
392
|
+
if (!this.onDiagnostics || this.diagnosticPending)
|
|
393
|
+
return;
|
|
394
|
+
this.diagnosticPending = true;
|
|
395
|
+
setTimeout(() => {
|
|
396
|
+
void Promise.resolve().then(() => this.onDiagnostics?.(summary)).catch(() => { }).finally(() => { this.diagnosticPending = false; });
|
|
397
|
+
}, 0);
|
|
398
|
+
},
|
|
379
399
|
streamId,
|
|
380
400
|
transferId: prepared.transfer_id,
|
|
381
401
|
routeGenerationId,
|
|
@@ -388,7 +408,7 @@ export class BeamClient {
|
|
|
388
408
|
routeStreamBeginAttempted = true;
|
|
389
409
|
await routeStream.begin();
|
|
390
410
|
multipartGroupWaiters = createMultipartGroupWaiters(prepared, destinationsById);
|
|
391
|
-
multipartManifestTask = createMultipartGroupManifest({
|
|
411
|
+
multipartManifestTask = telemetry.measure("sdk.multipart_create", () => createMultipartGroupManifest({
|
|
392
412
|
prepared,
|
|
393
413
|
destinationsById,
|
|
394
414
|
multipartUploads,
|
|
@@ -405,7 +425,7 @@ export class BeamClient {
|
|
|
405
425
|
multipartGroupWaiters.get(state.manifest.multipart_group_id)?.resolve(state);
|
|
406
426
|
},
|
|
407
427
|
onGroupFailed: (groupId, error) => multipartGroupWaiters.get(groupId)?.reject(error)
|
|
408
|
-
});
|
|
428
|
+
}));
|
|
409
429
|
const streamNextCompletedRoute = async () => {
|
|
410
430
|
const settled = await Promise.race([...pendingRoutes].map((pending) => pending.then((route) => ({ pending, route }))));
|
|
411
431
|
pendingRoutes.delete(settled.pending);
|
|
@@ -423,6 +443,12 @@ export class BeamClient {
|
|
|
423
443
|
};
|
|
424
444
|
for (const chunk of materializePlanChunks(prepared.plan_descriptor, prepared.transfer_id)) {
|
|
425
445
|
await throwIfCancelled?.(prepared.transfer_id);
|
|
446
|
+
// This promise belongs to this chunk and signing generation only. Its
|
|
447
|
+
// immutable credential/range identity and actual expiry cannot be renewed.
|
|
448
|
+
const sourceGrant = signSourceChunk({ source: sourcesById.get(chunk.source_id), fallbackUrl: chunk.source_url, chunk, expiresIn, fetchImpl: this.fetchImpl });
|
|
449
|
+
void sourceGrant.catch(() => { });
|
|
450
|
+
telemetry.counters.source_signatures++;
|
|
451
|
+
telemetry.counters.source_reuses += Math.max(0, chunk.destinations.length - 1);
|
|
426
452
|
for (const target of chunk.destinations) {
|
|
427
453
|
const pendingRoute = (async () => {
|
|
428
454
|
await throwIfCancelled?.(prepared.transfer_id);
|
|
@@ -442,10 +468,11 @@ export class BeamClient {
|
|
|
442
468
|
: await multipartGroupWaiters.get(multipartGroupId)?.promise;
|
|
443
469
|
if (!isDirectPutDestination(destination) && !upload)
|
|
444
470
|
throw new Error(`multipart group manifest is missing for ${chunk.source_id}:${target.destination_id}`);
|
|
445
|
-
return this.signProviderRoute({
|
|
471
|
+
return telemetry.measure("sdk.signing", () => this.signProviderRoute({
|
|
446
472
|
chunk,
|
|
447
473
|
target,
|
|
448
474
|
source,
|
|
475
|
+
sourceGrant,
|
|
449
476
|
destination,
|
|
450
477
|
expiresIn,
|
|
451
478
|
...(upload ? { upload } : {}),
|
|
@@ -456,7 +483,7 @@ export class BeamClient {
|
|
|
456
483
|
partNumber: typeof metadata.part_number === "number"
|
|
457
484
|
? metadata.part_number
|
|
458
485
|
: multipartPartNumber(chunk.source_chunk_index),
|
|
459
|
-
});
|
|
486
|
+
}));
|
|
460
487
|
})();
|
|
461
488
|
pendingRoutes.add(pendingRoute);
|
|
462
489
|
if (pendingRoutes.size >= signingConcurrency)
|
|
@@ -475,8 +502,8 @@ export class BeamClient {
|
|
|
475
502
|
}
|
|
476
503
|
catch (error) {
|
|
477
504
|
if (input.signal?.aborted) {
|
|
478
|
-
this.stopRecoverySigner(prepared.transfer_id);
|
|
479
|
-
this.control.releaseRecoveryLease(prepared.transfer_id);
|
|
505
|
+
this.stopRecoverySigner(prepared.transfer_id, recoveryLease);
|
|
506
|
+
this.control.releaseRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
480
507
|
await Promise.allSettled(pendingRoutes);
|
|
481
508
|
if (multipartManifestTask)
|
|
482
509
|
await Promise.allSettled([multipartManifestTask]);
|
|
@@ -489,10 +516,10 @@ export class BeamClient {
|
|
|
489
516
|
await Promise.allSettled([multipartManifestTask]);
|
|
490
517
|
if (!recoveryReplay) {
|
|
491
518
|
if (foregroundCancelled) {
|
|
492
|
-
this.control.continueRecoveryLease(prepared.transfer_id);
|
|
519
|
+
this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
493
520
|
}
|
|
494
521
|
else if (isRecoverableRouteStreamError(error)) {
|
|
495
|
-
this.control.continueRecoveryLease(prepared.transfer_id);
|
|
522
|
+
this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
496
523
|
throw new BeamRouteRecoveryPendingError(prepared.transfer_id, error);
|
|
497
524
|
}
|
|
498
525
|
else {
|
|
@@ -510,27 +537,16 @@ export class BeamClient {
|
|
|
510
537
|
// The retained recovery input owns the same destination objects used by
|
|
511
538
|
// multipart cleanup. Releasing it earlier scrubs their scoped credentials
|
|
512
539
|
// before abortMultipartUpload can clean up already-created uploads.
|
|
513
|
-
this.control.releaseRecoveryLease(prepared.transfer_id);
|
|
540
|
+
this.control.releaseRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
514
541
|
}
|
|
515
542
|
}
|
|
516
543
|
}
|
|
517
544
|
throw error;
|
|
518
545
|
}
|
|
519
546
|
};
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
destinationsById,
|
|
524
|
-
multipartUploads: recoveryMultipartUploads,
|
|
525
|
-
signal: input.signal,
|
|
526
|
-
expiresIn
|
|
527
|
-
});
|
|
528
|
-
const stopOwnedRecovery = () => {
|
|
529
|
-
this.stopRecoverySigner(prepared.transfer_id);
|
|
530
|
-
this.control.releaseRecoveryLease(prepared.transfer_id);
|
|
531
|
-
};
|
|
532
|
-
input.signal?.addEventListener("abort", stopOwnedRecovery, { once: true });
|
|
533
|
-
this.control.registerRecoveryLease({
|
|
547
|
+
// The lease object is this owner's identity: releases, recovery requests, and
|
|
548
|
+
// signer shutdowns below only act while it is still the registered lease.
|
|
549
|
+
const recoveryLease = {
|
|
534
550
|
transferId: prepared.transfer_id,
|
|
535
551
|
planFingerprint: prepared.plan_fingerprint,
|
|
536
552
|
coordinateChecksum: prepared.coordinate_checksum,
|
|
@@ -539,23 +555,39 @@ export class BeamClient {
|
|
|
539
555
|
},
|
|
540
556
|
disposeSecrets: () => {
|
|
541
557
|
input.signal?.removeEventListener("abort", stopOwnedRecovery);
|
|
558
|
+
releaseProviderClients([...retainedRecoveryInput.sources, ...retainedRecoveryInput.destinations]);
|
|
542
559
|
clearRecoverySecrets(retainedRecoveryInput);
|
|
543
560
|
}
|
|
561
|
+
};
|
|
562
|
+
await this.startProviderRouteRecoverySigner({
|
|
563
|
+
owner: recoveryLease,
|
|
564
|
+
prepared,
|
|
565
|
+
sourcesById,
|
|
566
|
+
destinationsById,
|
|
567
|
+
multipartUploads: recoveryMultipartUploads,
|
|
568
|
+
signal: input.signal,
|
|
569
|
+
expiresIn
|
|
544
570
|
});
|
|
571
|
+
const stopOwnedRecovery = () => {
|
|
572
|
+
this.stopRecoverySigner(prepared.transfer_id, recoveryLease);
|
|
573
|
+
this.control.releaseRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
574
|
+
};
|
|
575
|
+
input.signal?.addEventListener("abort", stopOwnedRecovery, { once: true });
|
|
576
|
+
this.control.registerRecoveryLease(recoveryLease);
|
|
545
577
|
try {
|
|
546
578
|
try {
|
|
547
579
|
assertOwnership();
|
|
548
580
|
await input.onPrepared?.(prepared);
|
|
549
581
|
}
|
|
550
582
|
catch (error) {
|
|
551
|
-
this.control.continueRecoveryLease(prepared.transfer_id);
|
|
583
|
+
this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
552
584
|
throw error;
|
|
553
585
|
}
|
|
554
586
|
try {
|
|
555
587
|
await input.throwIfCancelled?.(prepared.transfer_id);
|
|
556
588
|
}
|
|
557
589
|
catch (error) {
|
|
558
|
-
this.control.continueRecoveryLease(prepared.transfer_id);
|
|
590
|
+
this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
559
591
|
throw error;
|
|
560
592
|
}
|
|
561
593
|
await streamPreparedRoutes(prepared.route_generation_id, false);
|
|
@@ -740,6 +772,7 @@ export class BeamClient {
|
|
|
740
772
|
metadata: directPutRouteMetadata(input.target.metadata)
|
|
741
773
|
},
|
|
742
774
|
source: input.source,
|
|
775
|
+
sourceGrant: input.sourceGrant,
|
|
743
776
|
destination: input.destination,
|
|
744
777
|
expiresIn: input.expiresIn,
|
|
745
778
|
destUrl,
|
|
@@ -751,6 +784,7 @@ export class BeamClient {
|
|
|
751
784
|
chunk: input.chunk,
|
|
752
785
|
target: { ...input.target, metadata: partRouteMetadata(input.target.metadata) },
|
|
753
786
|
source: input.source,
|
|
787
|
+
sourceGrant: input.sourceGrant,
|
|
754
788
|
destination: input.destination,
|
|
755
789
|
expiresIn: input.expiresIn,
|
|
756
790
|
fetchImpl: this.fetchImpl
|
|
@@ -768,6 +802,7 @@ export class BeamClient {
|
|
|
768
802
|
metadata: partRouteMetadata(input.target.metadata)
|
|
769
803
|
},
|
|
770
804
|
source: input.source,
|
|
805
|
+
sourceGrant: input.sourceGrant,
|
|
771
806
|
destination: input.destination,
|
|
772
807
|
expiresIn: input.expiresIn,
|
|
773
808
|
uploadId: upload.uploadId,
|
|
@@ -791,6 +826,7 @@ export class BeamClient {
|
|
|
791
826
|
input.signal?.throwIfAborted();
|
|
792
827
|
if (payload.route_generation_id.length === 0)
|
|
793
828
|
throw new Error("route recovery generation is required");
|
|
829
|
+
const sourceGrants = new Map();
|
|
794
830
|
const routes = await mapOrderedWithConcurrency(payload.chunks, this.routeSigningConcurrency, async (requested) => {
|
|
795
831
|
input.signal?.throwIfAborted();
|
|
796
832
|
if (requested.route_generation_id !== payload.route_generation_id) {
|
|
@@ -821,11 +857,22 @@ export class BeamClient {
|
|
|
821
857
|
multipartUploads: input.multipartUploads,
|
|
822
858
|
expiresIn: input.expiresIn
|
|
823
859
|
});
|
|
824
|
-
|
|
860
|
+
const sourceKey = JSON.stringify([chunk.source_id, chunk.chunk_index]);
|
|
861
|
+
let sourceGrant = sourceGrants.get(sourceKey);
|
|
862
|
+
if (!sourceGrant) {
|
|
863
|
+
sourceGrant = signSourceChunk({ source, chunk, fallbackUrl: chunk.source_url, expiresIn: input.expiresIn, fetchImpl: this.fetchImpl });
|
|
864
|
+
sourceGrants.set(sourceKey, sourceGrant);
|
|
865
|
+
void sourceGrant.catch(() => { });
|
|
866
|
+
}
|
|
867
|
+
const route = await this.signProviderRoute({
|
|
868
|
+
sourceGrant,
|
|
825
869
|
chunk,
|
|
826
870
|
target: {
|
|
827
871
|
...target,
|
|
828
|
-
|
|
872
|
+
// Direct-PUT destinations (Hippius, Hugging Face) keep the planned
|
|
873
|
+
// per-chunk key exactly as initial materialization does; only
|
|
874
|
+
// multipart destinations sign parts against the final object key.
|
|
875
|
+
object_key: isDirectPutDestination(destination) ? target.object_key : requested.final_object_key,
|
|
829
876
|
metadata: {
|
|
830
877
|
...(target.metadata ?? {}),
|
|
831
878
|
...(requested.destination_metadata ?? {}),
|
|
@@ -852,6 +899,7 @@ export class BeamClient {
|
|
|
852
899
|
signedUrlFlow: "signed_url",
|
|
853
900
|
partNumber: requested.part_number
|
|
854
901
|
});
|
|
902
|
+
return signMultipartRecovery({ destination, transferId: input.prepared.transfer_id, requested, route, expiresIn: input.expiresIn });
|
|
855
903
|
});
|
|
856
904
|
return {
|
|
857
905
|
transfer_id: payload.transfer_id,
|
|
@@ -860,16 +908,33 @@ export class BeamClient {
|
|
|
860
908
|
chunk_routes: routes
|
|
861
909
|
};
|
|
862
910
|
});
|
|
863
|
-
|
|
864
|
-
|
|
865
|
-
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
869
|
-
|
|
870
|
-
|
|
911
|
+
if (input.signal?.aborted) {
|
|
912
|
+
stop();
|
|
913
|
+
return;
|
|
914
|
+
}
|
|
915
|
+
this.stopRecoverySigner(input.prepared.transfer_id);
|
|
916
|
+
this.integrityAuditSigners.set(input.prepared.transfer_id, (challenge) => this.buildProviderIntegrityAuditGrants({
|
|
917
|
+
challenge, prepared: input.prepared, sourcesById: input.sourcesById, destinationsById: input.destinationsById, expiresIn: input.expiresIn
|
|
918
|
+
}));
|
|
919
|
+
const registration = { owner: input.owner, stop };
|
|
920
|
+
this.recoverySigners.set(input.prepared.transfer_id, registration);
|
|
921
|
+
try {
|
|
922
|
+
const stopIntegrity = await this.control.serveIntegritySigner(input.prepared.transfer_id, challenge => {
|
|
923
|
+
input.signal?.throwIfAborted();
|
|
924
|
+
return this.signedIntegrityGrants(challenge);
|
|
871
925
|
});
|
|
872
|
-
|
|
926
|
+
if (input.signal?.aborted || this.recoverySigners.get(input.prepared.transfer_id) !== registration) {
|
|
927
|
+
stop();
|
|
928
|
+
stopIntegrity();
|
|
929
|
+
return;
|
|
930
|
+
}
|
|
931
|
+
registration.stop = () => { stop(); stopIntegrity(); };
|
|
932
|
+
}
|
|
933
|
+
catch {
|
|
934
|
+
// Status-driven signing remains available if the optional subscription fails.
|
|
935
|
+
if (input.signal?.aborted)
|
|
936
|
+
this.stopRecoverySigner(input.prepared.transfer_id, input.owner);
|
|
937
|
+
}
|
|
873
938
|
}
|
|
874
939
|
async recoveryMultipartUploadState(input) {
|
|
875
940
|
const existing = input.multipartUploads.get(input.requested.multipart_group_id);
|
|
@@ -936,21 +1001,56 @@ export class BeamClient {
|
|
|
936
1001
|
if (!signer)
|
|
937
1002
|
throw new Error("integrity audit signer unavailable");
|
|
938
1003
|
const existing = this.integrityAuditSubmissions.get(challenge.audit_id);
|
|
939
|
-
|
|
1004
|
+
const cached = this.integrityGrantCache.get(challenge.audit_id);
|
|
1005
|
+
if (cached && cached.fingerprint !== JSON.stringify(challenge))
|
|
1006
|
+
throw new Error("integrity audit identity changed");
|
|
1007
|
+
if (existing && cached && cached.expiresAt > Date.now() + 30_000) {
|
|
940
1008
|
await existing;
|
|
941
1009
|
return;
|
|
942
1010
|
}
|
|
943
|
-
const submission =
|
|
1011
|
+
const submission = this.signedIntegrityGrants(challenge).then(async (payload) => {
|
|
1012
|
+
const result = await this.control.request("transfer.integrity_audit_grants", payload, {
|
|
1013
|
+
transferId: challenge.transfer_id, idempotencyKey: `transfer:${challenge.transfer_id}:integrity-audit:${challenge.audit_id}:${payload.submitted_at}`
|
|
1014
|
+
});
|
|
1015
|
+
if (result.published !== true)
|
|
1016
|
+
throw new Error("integrity audit delivery unavailable");
|
|
1017
|
+
});
|
|
944
1018
|
this.integrityAuditSubmissions.set(challenge.audit_id, submission);
|
|
945
1019
|
try {
|
|
946
1020
|
await submission;
|
|
947
1021
|
}
|
|
948
1022
|
catch (error) {
|
|
949
|
-
this.integrityAuditSubmissions.
|
|
1023
|
+
if (this.integrityAuditSubmissions.get(challenge.audit_id) === submission)
|
|
1024
|
+
this.integrityAuditSubmissions.delete(challenge.audit_id);
|
|
950
1025
|
throw error;
|
|
951
1026
|
}
|
|
952
1027
|
}
|
|
953
|
-
|
|
1028
|
+
signedIntegrityGrants(challenge) {
|
|
1029
|
+
const fingerprint = JSON.stringify(challenge);
|
|
1030
|
+
const cached = this.integrityGrantCache.get(challenge.audit_id);
|
|
1031
|
+
if (cached && cached.fingerprint !== fingerprint)
|
|
1032
|
+
return Promise.reject(new Error("integrity audit identity changed"));
|
|
1033
|
+
if (cached && cached.expiresAt > Date.now() + 30_000)
|
|
1034
|
+
return cached.payload;
|
|
1035
|
+
if (!cached && this.integrityGrantCache.size >= 1024)
|
|
1036
|
+
return Promise.reject(new Error("integrity signer capacity unavailable"));
|
|
1037
|
+
const signer = this.integrityAuditSigners.get(challenge.transfer_id);
|
|
1038
|
+
if (!signer)
|
|
1039
|
+
return Promise.reject(new Error("integrity audit signer unavailable"));
|
|
1040
|
+
this.integrityAuditSubmissions.delete(challenge.audit_id);
|
|
1041
|
+
const entry = { transferId: challenge.transfer_id, expiresAt: 0, fingerprint, payload: Promise.resolve({}) };
|
|
1042
|
+
entry.payload = signer(challenge).then(payload => {
|
|
1043
|
+
const chunks = payload.chunks;
|
|
1044
|
+
entry.expiresAt = Math.min(...chunks.flatMap(chunk => [Date.parse(chunk.source.expires_at), Date.parse(chunk.destination.expires_at)]));
|
|
1045
|
+
return payload;
|
|
1046
|
+
}).catch(error => { if (this.integrityGrantCache.get(challenge.audit_id) === entry)
|
|
1047
|
+
this.integrityGrantCache.delete(challenge.audit_id); throw error; });
|
|
1048
|
+
// Pending grants are shared too. Actual expiry replaces this sentinel on completion.
|
|
1049
|
+
entry.expiresAt = Infinity;
|
|
1050
|
+
this.integrityGrantCache.set(challenge.audit_id, entry);
|
|
1051
|
+
return entry.payload;
|
|
1052
|
+
}
|
|
1053
|
+
async buildProviderIntegrityAuditGrants(input) {
|
|
954
1054
|
if (input.challenge.transfer_id !== input.prepared.transfer_id) {
|
|
955
1055
|
throw new Error("integrity audit challenge transfer mismatch");
|
|
956
1056
|
}
|
|
@@ -977,6 +1077,7 @@ export class BeamClient {
|
|
|
977
1077
|
}
|
|
978
1078
|
const sourceEtag = plannedSource?.metadata?.etag;
|
|
979
1079
|
const sourceVersionId = plannedSource?.metadata?.version_id;
|
|
1080
|
+
const grantExpiresAt = new Date(Math.floor(Date.now() / 1000) * 1000 + input.expiresIn * 1000).toISOString();
|
|
980
1081
|
const [sourceGrant, destinationGrant] = await Promise.all([
|
|
981
1082
|
signSourceReadRange({
|
|
982
1083
|
source,
|
|
@@ -997,31 +1098,30 @@ export class BeamClient {
|
|
|
997
1098
|
fetchImpl: this.fetchImpl
|
|
998
1099
|
})
|
|
999
1100
|
]);
|
|
1000
|
-
const grantExpiresAt = expiresAtIso(input.expiresIn);
|
|
1001
1101
|
const { orchestrator_id: _orchestratorId, orchestrator_hotkey: _orchestratorHotkey, worker_id: _workerId, ...grantChunk } = chunk;
|
|
1002
1102
|
return {
|
|
1003
1103
|
...grantChunk,
|
|
1004
|
-
source: { ...sourceGrant, expires_at: grantExpiresAt },
|
|
1005
|
-
destination: { ...destinationGrant, expires_at: grantExpiresAt }
|
|
1104
|
+
source: { ...sourceGrant, expires_at: boundedGrantExpiry(sourceGrant.url, grantExpiresAt) },
|
|
1105
|
+
destination: { ...destinationGrant, expires_at: boundedGrantExpiry(destinationGrant.url, grantExpiresAt) }
|
|
1006
1106
|
};
|
|
1007
1107
|
});
|
|
1008
|
-
|
|
1009
|
-
transfer_id: input.challenge.transfer_id,
|
|
1010
|
-
audit_id: input.challenge.audit_id,
|
|
1011
|
-
submitted_at: submittedAt,
|
|
1012
|
-
chunks
|
|
1013
|
-
}, {
|
|
1014
|
-
transferId: input.challenge.transfer_id,
|
|
1015
|
-
idempotencyKey: `transfer:${input.challenge.transfer_id}:integrity-audit:${input.challenge.audit_id}`
|
|
1016
|
-
});
|
|
1108
|
+
return { transfer_id: input.challenge.transfer_id, audit_id: input.challenge.audit_id, submitted_at: submittedAt, chunks };
|
|
1017
1109
|
}
|
|
1018
|
-
|
|
1019
|
-
|
|
1020
|
-
|
|
1021
|
-
|
|
1110
|
+
/** Stop a transfer's recovery and integrity signers; with `owner`, only if that owner installed them. */
|
|
1111
|
+
stopRecoverySigner(transferId, owner) {
|
|
1112
|
+
const signer = this.recoverySigners.get(transferId);
|
|
1113
|
+
if (owner && signer?.owner !== owner)
|
|
1114
|
+
return;
|
|
1115
|
+
if (signer) {
|
|
1116
|
+
signer.stop();
|
|
1022
1117
|
this.recoverySigners.delete(transferId);
|
|
1023
1118
|
}
|
|
1024
1119
|
this.integrityAuditSigners.delete(transferId);
|
|
1120
|
+
for (const [auditId, entry] of this.integrityGrantCache)
|
|
1121
|
+
if (entry.transferId === transferId) {
|
|
1122
|
+
this.integrityGrantCache.delete(auditId);
|
|
1123
|
+
this.integrityAuditSubmissions.delete(auditId);
|
|
1124
|
+
}
|
|
1025
1125
|
}
|
|
1026
1126
|
stopAllRecoverySigners() {
|
|
1027
1127
|
for (const transferId of [...this.recoverySigners.keys()])
|
|
@@ -1117,7 +1217,7 @@ export class BeamClient {
|
|
|
1117
1217
|
}
|
|
1118
1218
|
}
|
|
1119
1219
|
}
|
|
1120
|
-
const ROUTE_STREAM_BATCH_ROUTES =
|
|
1220
|
+
const ROUTE_STREAM_BATCH_ROUTES = 1_024;
|
|
1121
1221
|
class AsyncMutex {
|
|
1122
1222
|
tail = Promise.resolve();
|
|
1123
1223
|
async acquire() {
|
|
@@ -1142,6 +1242,8 @@ class AsyncMutex {
|
|
|
1142
1242
|
class RouteStreamSender {
|
|
1143
1243
|
control;
|
|
1144
1244
|
options;
|
|
1245
|
+
started = performance.now();
|
|
1246
|
+
telemetry;
|
|
1145
1247
|
streamId;
|
|
1146
1248
|
checksum = new RouteKeysChecksum();
|
|
1147
1249
|
batch = [];
|
|
@@ -1154,6 +1256,7 @@ class RouteStreamSender {
|
|
|
1154
1256
|
this.control = control;
|
|
1155
1257
|
this.options = options;
|
|
1156
1258
|
this.streamId = options.streamId;
|
|
1259
|
+
this.telemetry = options.telemetry ?? new SdkPerformanceCollector();
|
|
1157
1260
|
}
|
|
1158
1261
|
async begin() {
|
|
1159
1262
|
await this.control.request("transfer.route_stream.begin", compact({
|
|
@@ -1215,7 +1318,11 @@ class RouteStreamSender {
|
|
|
1215
1318
|
await this.sendTail;
|
|
1216
1319
|
if (this.sendError)
|
|
1217
1320
|
throw this.sendError;
|
|
1321
|
+
this.telemetry.observe("sdk.preparation", performance.now() - this.started);
|
|
1322
|
+
const sdkPerformance = this.telemetry.snapshot();
|
|
1323
|
+
this.options.onDiagnostics?.(sdkPerformance);
|
|
1218
1324
|
return this.control.request("transfer.route_stream.complete", {
|
|
1325
|
+
sdk_performance: sdkPerformance,
|
|
1219
1326
|
transfer_id: this.options.transferId,
|
|
1220
1327
|
route_generation_id: this.options.routeGenerationId,
|
|
1221
1328
|
stream_id: this.streamId,
|
|
@@ -1230,7 +1337,7 @@ class RouteStreamSender {
|
|
|
1230
1337
|
async enqueueFlush() {
|
|
1231
1338
|
if (!this.batch.length)
|
|
1232
1339
|
return;
|
|
1233
|
-
await this.sendTail;
|
|
1340
|
+
await this.telemetry.measure("sdk.buffer_wait", () => this.sendTail);
|
|
1234
1341
|
if (this.sendError)
|
|
1235
1342
|
throw this.sendError;
|
|
1236
1343
|
const routes = this.batch.splice(0, this.batch.length);
|
|
@@ -1254,7 +1361,8 @@ class RouteStreamSender {
|
|
|
1254
1361
|
const routeChecksum = chunkChecksum.value();
|
|
1255
1362
|
const coordinateChecksum = await routeCoordinateChecksum(chunk);
|
|
1256
1363
|
const batchId = `${this.streamId}:${batchIndex}:${coordinateChecksum}`;
|
|
1257
|
-
|
|
1364
|
+
this.telemetry.counters.route_batches++;
|
|
1365
|
+
await this.telemetry.measure("sdk.batch_ack", () => this.control.request("transfer.route_stream.batch", {
|
|
1258
1366
|
transfer_id: this.options.transferId,
|
|
1259
1367
|
route_generation_id: this.options.routeGenerationId,
|
|
1260
1368
|
stream_id: this.streamId,
|
|
@@ -1266,7 +1374,7 @@ class RouteStreamSender {
|
|
|
1266
1374
|
}, {
|
|
1267
1375
|
transferId: this.options.transferId,
|
|
1268
1376
|
idempotencyKey: `transfer:${this.options.transferId}:route-stream:${this.streamId}:batch:${batchIndex}:${coordinateChecksum}`
|
|
1269
|
-
});
|
|
1377
|
+
}));
|
|
1270
1378
|
}
|
|
1271
1379
|
catch (error) {
|
|
1272
1380
|
this.sendError = error;
|
|
@@ -1510,6 +1618,9 @@ function createDeferred() {
|
|
|
1510
1618
|
resolve = resolvePromise;
|
|
1511
1619
|
reject = rejectPromise;
|
|
1512
1620
|
});
|
|
1621
|
+
// A waiter may be rejected after its route loop already stopped (for example when
|
|
1622
|
+
// ownership is fenced mid-stream); awaiting callers still observe the rejection.
|
|
1623
|
+
promise.catch(() => undefined);
|
|
1513
1624
|
return { promise, resolve, reject };
|
|
1514
1625
|
}
|
|
1515
1626
|
function createMultipartGroupWaiters(prepared, destinationsById) {
|
|
@@ -1661,7 +1772,7 @@ function multipartListPageMarkers(maxPartNumber) {
|
|
|
1661
1772
|
return markers;
|
|
1662
1773
|
}
|
|
1663
1774
|
function multipartMaxPartNumber(chunkCount) {
|
|
1664
|
-
return multipartPartNumber(positiveInteger(chunkCount, "source chunk_count") - 1
|
|
1775
|
+
return multipartPartNumber(positiveInteger(chunkCount, "source chunk_count") - 1);
|
|
1665
1776
|
}
|
|
1666
1777
|
function validateMultipartPartNumber(partNumber, manifest) {
|
|
1667
1778
|
if (!Number.isInteger(partNumber) || partNumber < 1 || partNumber > manifest.max_part_number) {
|
|
@@ -1715,7 +1826,7 @@ const COMPACT_TRANSFER_PLAN_KEYS = new Set([
|
|
|
1715
1826
|
const COMPACT_TRANSFER_PLAN_FORMULAS = {
|
|
1716
1827
|
source_offset: "source_chunk_index * chunk_size",
|
|
1717
1828
|
delivery_index: "chunk_index * destination_count + destination_index",
|
|
1718
|
-
part_number: "source_chunk_index
|
|
1829
|
+
part_number: "source_chunk_index + 1",
|
|
1719
1830
|
route_generation_id: "initial-{chunk_index}-{destination_id}"
|
|
1720
1831
|
};
|
|
1721
1832
|
function validateCompactTransferPlan(descriptor, signedUrlFlow) {
|
|
@@ -1729,7 +1840,7 @@ function validateCompactTransferPlan(descriptor, signedUrlFlow) {
|
|
|
1729
1840
|
if (descriptor.version !== "compact-transfer-plan/v1") {
|
|
1730
1841
|
throw new Error(`BeamCore returned unsupported plan version: ${descriptor.version}`);
|
|
1731
1842
|
}
|
|
1732
|
-
if (descriptor.multipart_attempt_slots !==
|
|
1843
|
+
if (descriptor.multipart_attempt_slots !== 1) {
|
|
1733
1844
|
throw new Error("BeamCore returned unsupported multipart attempt slot count");
|
|
1734
1845
|
}
|
|
1735
1846
|
const formulaKeys = Object.keys(descriptor.formulas);
|
|
@@ -1782,7 +1893,7 @@ function validateMultipartGroupManifest(manifest, transferId) {
|
|
|
1782
1893
|
if (!Number.isInteger(group.expected_part_count) || group.expected_part_count < 1 || group.expected_part_count > 10_000) {
|
|
1783
1894
|
throw new Error(`multipart group ${group.multipart_group_id} has invalid expected_part_count`);
|
|
1784
1895
|
}
|
|
1785
|
-
if (!Number.isInteger(group.max_part_number) || group.max_part_number
|
|
1896
|
+
if (!Number.isInteger(group.max_part_number) || group.max_part_number !== group.expected_part_count || group.max_part_number > 10_000) {
|
|
1786
1897
|
throw new Error(`multipart group ${group.multipart_group_id} has invalid max_part_number`);
|
|
1787
1898
|
}
|
|
1788
1899
|
if (!Number.isSafeInteger(group.expected_object_size) || group.expected_object_size <= 0) {
|
|
@@ -1810,7 +1921,8 @@ const PART_ROUTE_METADATA_KEYS = new Set([
|
|
|
1810
1921
|
"logical_attempt_index",
|
|
1811
1922
|
"attempt_slot",
|
|
1812
1923
|
"route_generation_id",
|
|
1813
|
-
"delivery_index"
|
|
1924
|
+
"delivery_index",
|
|
1925
|
+
"etag_required"
|
|
1814
1926
|
]);
|
|
1815
1927
|
function partRouteMetadata(metadata) {
|
|
1816
1928
|
return Object.fromEntries(Object.entries(metadata ?? {}).filter(([key]) => PART_ROUTE_METADATA_KEYS.has(key)));
|