@beam-network/sdk 0.6.0 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +9 -3
- package/dist/client.d.ts +6 -1
- package/dist/client.js +175 -65
- package/dist/models.d.ts +43 -0
- package/dist/nats-control.d.ts +11 -2
- package/dist/nats-control.js +104 -30
- package/dist/performance.d.ts +12 -0
- package/dist/performance.js +25 -0
- package/dist/provider-signing.d.ts +20 -0
- package/dist/provider-signing.js +41 -9
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -156,19 +156,19 @@ For low-level raw transfer configs, use `createRawTransfer`; lifecycle transport
|
|
|
156
156
|
|
|
157
157
|
Provider transfers use `transfer-client-control/v7`. Every reply carries Runtime and transport epochs; prepare and every route-stream message carry a UUID route generation. S3, R2, and S3-compatible destinations retain direct multipart UploadPart/ListParts/HEAD handling. Hippius and Hugging Face destinations take a plain PUT per chunk instead, so they carry no multipart group manifest.
|
|
158
158
|
|
|
159
|
-
The client signs up to 64 routes concurrently by default and emits
|
|
159
|
+
The client signs up to 64 routes concurrently by default and emits 1,024-route logical batches. Encoded MessagePack requests target 4 MiB physical batches under `maxPayloadBytes` (8 MiB by default; explicit positive overrides remain supported). The splitter reserves control-envelope headroom for the live auth token and stable request identity so the final encoded request remains below the configured guard. Lifecycle control refreshes cached SDK auth before the final 30 seconds of token lifetime and retries `auth_token_expired` replies with the same request identity and payload but a fresh token, preserving Core request-conflict protection during large route streams. A single route above the 4 MiB target but within the configured guard publishes alone; a single route larger than the configured limit fails before publication with both sizes in the error. Multipart create and abort requests use a separate `multipartControlConcurrency` limit of 2 by default, with explicit positive overrides supported, and S3-compatible control requests make up to five attempts for transient provider failures. This keeps URL signing throughput independent from provider control-plane pressure.
|
|
160
160
|
|
|
161
161
|
Callers that need durable recovery accounting can provide `onMultipartGroupReady`. The SDK invokes it after creating a multipart upload and before streaming that group’s signed routes. Its payload contains transfer, source, destination, object-key, upload, size, part-count, and expiry identities only; it never includes credentials, request headers, or signed URLs. A rejected callback aborts the newly created upload and fails preparation closed.
|
|
162
162
|
|
|
163
163
|
`resumeProviderTransfer` requires the complete saved callback payloads in `multipartGroups`. It validates every transfer/source/destination/object/size/part coordinate, then uses the same route-stream implementation as creation with those upload IDs. Missing, duplicate, or changed identities fail before new provider uploads or route publication. If a process died before recording an upload identity, treat it as uncertain cleanup instead of inventing a replacement. The `onPrepared` callback also runs during resume, before routes are streamed.
|
|
164
164
|
|
|
165
|
-
Create and resume accept an ownership `signal`. Aborting it stops initial routes, multipart creation, and recovery signing/replay; it does not cancel the transfer or abort a replacement owner's uploads. User cancellation still uses `cancelTransfer` and verified provider cleanup. `prepareProviderSource` accepts the same signal for metadata requests.
|
|
165
|
+
Create and resume accept an ownership `signal`. Aborting it stops initial routes, multipart creation, and recovery signing/replay; it does not cancel the transfer or abort a replacement owner's uploads. Recovery leases, signers, and background resume/replay are keyed by owner identity: a fenced-off owner never replays into, releases, or stops the signers of a replacement owner registered for the same transfer, and a replacement always gets its own recovery. User cancellation still uses `cancelTransfer` and verified provider cleanup. `prepareProviderSource` accepts the same signal for metadata requests.
|
|
166
166
|
|
|
167
167
|
The stream ID derives from the transfer, selected flow, and immutable plan identity, and each ordered batch ID includes its route-coordinate checksum. Manual multi-destination attachment requires `delivery_index` on every route. Lifecycle mutations retry transient failures up to three times with the same request identity. A non-recoverable provider setup or route failure raises `BeamProviderTransferError`, whose `errors`, `transferCancelled`, and `multipartCleanupComplete` fields retain the original provider, cancellation, and cleanup outcomes. `waitForTransfer` subscribes to the API-key-owned, at-most-once terminal signal before its first status read and reconciles every signal through authoritative status; subscription failure degrades to the same jittered 15-to-30-second status fallback. Call `close()` when the client is no longer needed; it also closes outstanding terminal waiters.
|
|
168
168
|
|
|
169
169
|
During the pre-completion `integrity_check` phase, Runtime may include an `integrity_audit_challenge` in status. The TypeScript client signs exact read-only source and final-destination GET ranges and submits `transfer.integrity_audit_grants`. S3-compatible source and destination grants use the prepared source ETag and Runtime-verified final-object ETag as `If-Match` conditions. Submission failures retry on the next status poll and appear as a URL-safe `integrity_audit_submission_error` on that status response; the transfer completes with a warning if the check cannot finish by its deadline.
|
|
170
170
|
|
|
171
|
-
Hippius uses canonical non-multipart `signed_url` routes, so its manifest is empty and group-level final HEAD verification is not available from that provider flow.
|
|
171
|
+
Hippius uses canonical non-multipart `signed_url` routes (initial and recovered routes both PUT to the planned `<final>/<plan_nonce>/chunk-NNNNNN` key), so its manifest is empty and group-level final HEAD verification is not available from that provider flow.
|
|
172
172
|
|
|
173
173
|
NATS requires this guard because the broker rejects messages above its configured `max_payload`. This limit applies only to lifecycle/control metadata; transfer file bytes do not flow through NATS.
|
|
174
174
|
|
|
@@ -189,3 +189,9 @@ Only short-lived range/upload routes go to workers. Multipart creation, verifica
|
|
|
189
189
|
Manual releases from `dev` require a committed `-dev.N` version and publish only to npm tag `dev`; the stable tag is unchanged.
|
|
190
190
|
|
|
191
191
|
Multipart control helpers (`createMultipartUpload`, `listMultipartParts`, `completeMultipartUpload`, `inspectDestinationObject`, and `abortMultipartUpload`) accept an optional `AbortSignal` and cancel through the existing provider HTTP transport. Pass it as `signal` in object arguments or as the last positional argument. Cancellation stops waiting and network activity; it cannot undo a provider operation already accepted. After an interrupted completion, inspect the durable upload/object identity before retrying or reporting cleanup complete.
|
|
192
|
+
|
|
193
|
+
### Preparation diagnostics
|
|
194
|
+
|
|
195
|
+
Route streaming flushes at 1,024 routes, with no timer, and sends the final partial batch at stream completion. Signing overlaps acknowledged publication through bounded buffering; existing encoded-payload limits still apply. A source grant is reused across destinations only within the same signing generation and actual expiry.
|
|
196
|
+
|
|
197
|
+
The optional `onDiagnostics` callback receives bounded preparation measurements and source-grant reuse counts. Callbacks are best effort, may be dropped under load, and do not affect transfer outcomes. Durations may overlap; do not sum them into elapsed transfer time. Transfer status also exposes a typed `performance` summary when supported by Core.
|
package/dist/client.d.ts
CHANGED
|
@@ -44,8 +44,11 @@ export declare class BeamClient {
|
|
|
44
44
|
private readonly routeSigningConcurrency;
|
|
45
45
|
private readonly routeSigningConcurrencyOverridden;
|
|
46
46
|
private readonly multipartControlConcurrency;
|
|
47
|
+
private readonly onDiagnostics;
|
|
48
|
+
private diagnosticPending;
|
|
47
49
|
private readonly recoverySigners;
|
|
48
50
|
private readonly integrityAuditSigners;
|
|
51
|
+
private readonly integrityGrantCache;
|
|
49
52
|
private readonly integrityAuditSubmissions;
|
|
50
53
|
private readonly huggingFaceUploads;
|
|
51
54
|
constructor(options?: BeamClientOptions);
|
|
@@ -105,7 +108,9 @@ export declare class BeamClient {
|
|
|
105
108
|
private startProviderRouteRecoverySigner;
|
|
106
109
|
private recoveryMultipartUploadState;
|
|
107
110
|
private submitIntegrityAuditGrantsIfPresent;
|
|
108
|
-
private
|
|
111
|
+
private signedIntegrityGrants;
|
|
112
|
+
private buildProviderIntegrityAuditGrants;
|
|
113
|
+
/** Stop a transfer's recovery and integrity signers; with `owner`, only if that owner installed them. */
|
|
109
114
|
private stopRecoverySigner;
|
|
110
115
|
private stopAllRecoverySigners;
|
|
111
116
|
createAndDistribute(input: RawTransferCreateInput): Promise<TransferCreateResponse>;
|
package/dist/client.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
|
-
import {
|
|
1
|
+
import { SdkPerformanceCollector } from "./performance.js";
|
|
2
|
+
import { signMultipartRecovery, abortMultipartUpload, createMultipartUpload, expiresAtIso, prepareProviderDestination, prepareProviderSource, releaseProviderClients, signAbortMultipartUpload, signCompleteMultipartUpload, signDestinationReadRange, signDestinationRoute, signFinalObjectHead, signListMultipartUpload, signSourceReadRange, signSourceChunk, boundedGrantExpiry, isHuggingFaceProvider } from "./provider-signing.js";
|
|
2
3
|
import { describe as describeHuggingFace, hashSourceStream, huggingFaceCommit, huggingFaceCompleteLfsUpload, huggingFaceLfsBatch, huggingFacePreupload, huggingFaceVerifyLfsUpload, readSourceSample } from "./huggingface.js";
|
|
3
4
|
import { BeamTransferControl, BEAM_DEFAULT_NATS_URL, compactSignedRoutes, isRecoverableRouteStreamError } from "./nats-control.js";
|
|
4
5
|
export class BeamRouteRecoveryPendingError extends Error {
|
|
@@ -53,11 +54,15 @@ export class BeamClient {
|
|
|
53
54
|
routeSigningConcurrency;
|
|
54
55
|
routeSigningConcurrencyOverridden;
|
|
55
56
|
multipartControlConcurrency;
|
|
57
|
+
onDiagnostics;
|
|
58
|
+
diagnosticPending = false;
|
|
56
59
|
recoverySigners = new Map();
|
|
57
60
|
integrityAuditSigners = new Map();
|
|
61
|
+
integrityGrantCache = new Map();
|
|
58
62
|
integrityAuditSubmissions = new Map();
|
|
59
63
|
huggingFaceUploads = new Map();
|
|
60
64
|
constructor(options = {}) {
|
|
65
|
+
this.onDiagnostics = options.onDiagnostics;
|
|
61
66
|
if (!options.apiKey?.trim()) {
|
|
62
67
|
throw new Error("apiKey is required.");
|
|
63
68
|
}
|
|
@@ -237,7 +242,7 @@ export class BeamClient {
|
|
|
237
242
|
};
|
|
238
243
|
const routeStreamLock = new AsyncMutex();
|
|
239
244
|
const releaseInitialStream = await routeStreamLock.acquire();
|
|
240
|
-
|
|
245
|
+
const recoveryLease = {
|
|
241
246
|
transferId,
|
|
242
247
|
planFingerprint: input.planFingerprint,
|
|
243
248
|
coordinateChecksum: input.coordinateChecksum,
|
|
@@ -249,19 +254,20 @@ export class BeamClient {
|
|
|
249
254
|
throw new Error(attached.error ?? attached.message ?? "route replay failed");
|
|
250
255
|
});
|
|
251
256
|
}
|
|
252
|
-
}
|
|
257
|
+
};
|
|
258
|
+
this.control.registerRecoveryLease(recoveryLease);
|
|
253
259
|
try {
|
|
254
260
|
const attached = await streamRoutes(input.chunkRoutes, input.multipartGroupManifest, input.routeGenerationId, input.urlsExpiresAt);
|
|
255
261
|
if (!attached.success)
|
|
256
|
-
this.control.releaseRecoveryLease(transferId);
|
|
262
|
+
this.control.releaseRecoveryLease(transferId, recoveryLease);
|
|
257
263
|
return attached;
|
|
258
264
|
}
|
|
259
265
|
catch (error) {
|
|
260
266
|
if (isRecoverableRouteStreamError(error)) {
|
|
261
|
-
this.control.continueRecoveryLease(transferId);
|
|
267
|
+
this.control.continueRecoveryLease(transferId, recoveryLease);
|
|
262
268
|
throw new BeamRouteRecoveryPendingError(transferId, error);
|
|
263
269
|
}
|
|
264
|
-
this.control.releaseRecoveryLease(transferId);
|
|
270
|
+
this.control.releaseRecoveryLease(transferId, recoveryLease);
|
|
265
271
|
throw error;
|
|
266
272
|
}
|
|
267
273
|
finally {
|
|
@@ -294,8 +300,10 @@ export class BeamClient {
|
|
|
294
300
|
destinationsById.set(preparedDestination.destination_id, destination);
|
|
295
301
|
});
|
|
296
302
|
await input.throwIfCancelled?.();
|
|
303
|
+
const discoveryStarted = performance.now();
|
|
297
304
|
const preparedSources = await Promise.all(retainedRecoveryInput.sources.map((source, index) => prepareProviderSource(source, { index, expiresIn, fetchImpl: this.fetchImpl, signal: input.signal })));
|
|
298
305
|
await input.throwIfCancelled?.();
|
|
306
|
+
const discoveryMs = performance.now() - discoveryStarted;
|
|
299
307
|
const huggingFace = await this.planHuggingFaceUploads({
|
|
300
308
|
sources: retainedRecoveryInput.sources,
|
|
301
309
|
preparedSources,
|
|
@@ -346,6 +354,9 @@ export class BeamClient {
|
|
|
346
354
|
const routeStreamLock = new AsyncMutex();
|
|
347
355
|
const releaseInitialStream = await routeStreamLock.acquire();
|
|
348
356
|
const streamPreparedRoutes = async (routeGenerationId, recoveryReplay) => {
|
|
357
|
+
const telemetry = new SdkPerformanceCollector();
|
|
358
|
+
if (!recoveryReplay)
|
|
359
|
+
telemetry.observe("sdk.discovery", discoveryMs);
|
|
349
360
|
const pendingRoutes = new Set();
|
|
350
361
|
let signingConcurrency = this.routeSigningConcurrency;
|
|
351
362
|
let signedInWindow = 0;
|
|
@@ -376,6 +387,15 @@ export class BeamClient {
|
|
|
376
387
|
signedUrlFlow: requestedSignedUrlFlow
|
|
377
388
|
});
|
|
378
389
|
routeStream = new RouteStreamSender(this.control, {
|
|
390
|
+
telemetry,
|
|
391
|
+
onDiagnostics: (summary) => {
|
|
392
|
+
if (!this.onDiagnostics || this.diagnosticPending)
|
|
393
|
+
return;
|
|
394
|
+
this.diagnosticPending = true;
|
|
395
|
+
setTimeout(() => {
|
|
396
|
+
void Promise.resolve().then(() => this.onDiagnostics?.(summary)).catch(() => { }).finally(() => { this.diagnosticPending = false; });
|
|
397
|
+
}, 0);
|
|
398
|
+
},
|
|
379
399
|
streamId,
|
|
380
400
|
transferId: prepared.transfer_id,
|
|
381
401
|
routeGenerationId,
|
|
@@ -388,7 +408,7 @@ export class BeamClient {
|
|
|
388
408
|
routeStreamBeginAttempted = true;
|
|
389
409
|
await routeStream.begin();
|
|
390
410
|
multipartGroupWaiters = createMultipartGroupWaiters(prepared, destinationsById);
|
|
391
|
-
multipartManifestTask = createMultipartGroupManifest({
|
|
411
|
+
multipartManifestTask = telemetry.measure("sdk.multipart_create", () => createMultipartGroupManifest({
|
|
392
412
|
prepared,
|
|
393
413
|
destinationsById,
|
|
394
414
|
multipartUploads,
|
|
@@ -405,7 +425,7 @@ export class BeamClient {
|
|
|
405
425
|
multipartGroupWaiters.get(state.manifest.multipart_group_id)?.resolve(state);
|
|
406
426
|
},
|
|
407
427
|
onGroupFailed: (groupId, error) => multipartGroupWaiters.get(groupId)?.reject(error)
|
|
408
|
-
});
|
|
428
|
+
}));
|
|
409
429
|
const streamNextCompletedRoute = async () => {
|
|
410
430
|
const settled = await Promise.race([...pendingRoutes].map((pending) => pending.then((route) => ({ pending, route }))));
|
|
411
431
|
pendingRoutes.delete(settled.pending);
|
|
@@ -423,6 +443,12 @@ export class BeamClient {
|
|
|
423
443
|
};
|
|
424
444
|
for (const chunk of materializePlanChunks(prepared.plan_descriptor, prepared.transfer_id)) {
|
|
425
445
|
await throwIfCancelled?.(prepared.transfer_id);
|
|
446
|
+
// This promise belongs to this chunk and signing generation only. Its
|
|
447
|
+
// immutable credential/range identity and actual expiry cannot be renewed.
|
|
448
|
+
const sourceGrant = signSourceChunk({ source: sourcesById.get(chunk.source_id), fallbackUrl: chunk.source_url, chunk, expiresIn, fetchImpl: this.fetchImpl });
|
|
449
|
+
void sourceGrant.catch(() => { });
|
|
450
|
+
telemetry.counters.source_signatures++;
|
|
451
|
+
telemetry.counters.source_reuses += Math.max(0, chunk.destinations.length - 1);
|
|
426
452
|
for (const target of chunk.destinations) {
|
|
427
453
|
const pendingRoute = (async () => {
|
|
428
454
|
await throwIfCancelled?.(prepared.transfer_id);
|
|
@@ -442,10 +468,11 @@ export class BeamClient {
|
|
|
442
468
|
: await multipartGroupWaiters.get(multipartGroupId)?.promise;
|
|
443
469
|
if (!isDirectPutDestination(destination) && !upload)
|
|
444
470
|
throw new Error(`multipart group manifest is missing for ${chunk.source_id}:${target.destination_id}`);
|
|
445
|
-
return this.signProviderRoute({
|
|
471
|
+
return telemetry.measure("sdk.signing", () => this.signProviderRoute({
|
|
446
472
|
chunk,
|
|
447
473
|
target,
|
|
448
474
|
source,
|
|
475
|
+
sourceGrant,
|
|
449
476
|
destination,
|
|
450
477
|
expiresIn,
|
|
451
478
|
...(upload ? { upload } : {}),
|
|
@@ -456,7 +483,7 @@ export class BeamClient {
|
|
|
456
483
|
partNumber: typeof metadata.part_number === "number"
|
|
457
484
|
? metadata.part_number
|
|
458
485
|
: multipartPartNumber(chunk.source_chunk_index),
|
|
459
|
-
});
|
|
486
|
+
}));
|
|
460
487
|
})();
|
|
461
488
|
pendingRoutes.add(pendingRoute);
|
|
462
489
|
if (pendingRoutes.size >= signingConcurrency)
|
|
@@ -475,8 +502,8 @@ export class BeamClient {
|
|
|
475
502
|
}
|
|
476
503
|
catch (error) {
|
|
477
504
|
if (input.signal?.aborted) {
|
|
478
|
-
this.stopRecoverySigner(prepared.transfer_id);
|
|
479
|
-
this.control.releaseRecoveryLease(prepared.transfer_id);
|
|
505
|
+
this.stopRecoverySigner(prepared.transfer_id, recoveryLease);
|
|
506
|
+
this.control.releaseRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
480
507
|
await Promise.allSettled(pendingRoutes);
|
|
481
508
|
if (multipartManifestTask)
|
|
482
509
|
await Promise.allSettled([multipartManifestTask]);
|
|
@@ -489,10 +516,10 @@ export class BeamClient {
|
|
|
489
516
|
await Promise.allSettled([multipartManifestTask]);
|
|
490
517
|
if (!recoveryReplay) {
|
|
491
518
|
if (foregroundCancelled) {
|
|
492
|
-
this.control.continueRecoveryLease(prepared.transfer_id);
|
|
519
|
+
this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
493
520
|
}
|
|
494
521
|
else if (isRecoverableRouteStreamError(error)) {
|
|
495
|
-
this.control.continueRecoveryLease(prepared.transfer_id);
|
|
522
|
+
this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
496
523
|
throw new BeamRouteRecoveryPendingError(prepared.transfer_id, error);
|
|
497
524
|
}
|
|
498
525
|
else {
|
|
@@ -510,27 +537,16 @@ export class BeamClient {
|
|
|
510
537
|
// The retained recovery input owns the same destination objects used by
|
|
511
538
|
// multipart cleanup. Releasing it earlier scrubs their scoped credentials
|
|
512
539
|
// before abortMultipartUpload can clean up already-created uploads.
|
|
513
|
-
this.control.releaseRecoveryLease(prepared.transfer_id);
|
|
540
|
+
this.control.releaseRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
514
541
|
}
|
|
515
542
|
}
|
|
516
543
|
}
|
|
517
544
|
throw error;
|
|
518
545
|
}
|
|
519
546
|
};
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
destinationsById,
|
|
524
|
-
multipartUploads: recoveryMultipartUploads,
|
|
525
|
-
signal: input.signal,
|
|
526
|
-
expiresIn
|
|
527
|
-
});
|
|
528
|
-
const stopOwnedRecovery = () => {
|
|
529
|
-
this.stopRecoverySigner(prepared.transfer_id);
|
|
530
|
-
this.control.releaseRecoveryLease(prepared.transfer_id);
|
|
531
|
-
};
|
|
532
|
-
input.signal?.addEventListener("abort", stopOwnedRecovery, { once: true });
|
|
533
|
-
this.control.registerRecoveryLease({
|
|
547
|
+
// The lease object is this owner's identity: releases, recovery requests, and
|
|
548
|
+
// signer shutdowns below only act while it is still the registered lease.
|
|
549
|
+
const recoveryLease = {
|
|
534
550
|
transferId: prepared.transfer_id,
|
|
535
551
|
planFingerprint: prepared.plan_fingerprint,
|
|
536
552
|
coordinateChecksum: prepared.coordinate_checksum,
|
|
@@ -539,23 +555,39 @@ export class BeamClient {
|
|
|
539
555
|
},
|
|
540
556
|
disposeSecrets: () => {
|
|
541
557
|
input.signal?.removeEventListener("abort", stopOwnedRecovery);
|
|
558
|
+
releaseProviderClients([...retainedRecoveryInput.sources, ...retainedRecoveryInput.destinations]);
|
|
542
559
|
clearRecoverySecrets(retainedRecoveryInput);
|
|
543
560
|
}
|
|
561
|
+
};
|
|
562
|
+
await this.startProviderRouteRecoverySigner({
|
|
563
|
+
owner: recoveryLease,
|
|
564
|
+
prepared,
|
|
565
|
+
sourcesById,
|
|
566
|
+
destinationsById,
|
|
567
|
+
multipartUploads: recoveryMultipartUploads,
|
|
568
|
+
signal: input.signal,
|
|
569
|
+
expiresIn
|
|
544
570
|
});
|
|
571
|
+
const stopOwnedRecovery = () => {
|
|
572
|
+
this.stopRecoverySigner(prepared.transfer_id, recoveryLease);
|
|
573
|
+
this.control.releaseRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
574
|
+
};
|
|
575
|
+
input.signal?.addEventListener("abort", stopOwnedRecovery, { once: true });
|
|
576
|
+
this.control.registerRecoveryLease(recoveryLease);
|
|
545
577
|
try {
|
|
546
578
|
try {
|
|
547
579
|
assertOwnership();
|
|
548
580
|
await input.onPrepared?.(prepared);
|
|
549
581
|
}
|
|
550
582
|
catch (error) {
|
|
551
|
-
this.control.continueRecoveryLease(prepared.transfer_id);
|
|
583
|
+
this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
552
584
|
throw error;
|
|
553
585
|
}
|
|
554
586
|
try {
|
|
555
587
|
await input.throwIfCancelled?.(prepared.transfer_id);
|
|
556
588
|
}
|
|
557
589
|
catch (error) {
|
|
558
|
-
this.control.continueRecoveryLease(prepared.transfer_id);
|
|
590
|
+
this.control.continueRecoveryLease(prepared.transfer_id, recoveryLease);
|
|
559
591
|
throw error;
|
|
560
592
|
}
|
|
561
593
|
await streamPreparedRoutes(prepared.route_generation_id, false);
|
|
@@ -740,6 +772,7 @@ export class BeamClient {
|
|
|
740
772
|
metadata: directPutRouteMetadata(input.target.metadata)
|
|
741
773
|
},
|
|
742
774
|
source: input.source,
|
|
775
|
+
sourceGrant: input.sourceGrant,
|
|
743
776
|
destination: input.destination,
|
|
744
777
|
expiresIn: input.expiresIn,
|
|
745
778
|
destUrl,
|
|
@@ -751,6 +784,7 @@ export class BeamClient {
|
|
|
751
784
|
chunk: input.chunk,
|
|
752
785
|
target: { ...input.target, metadata: partRouteMetadata(input.target.metadata) },
|
|
753
786
|
source: input.source,
|
|
787
|
+
sourceGrant: input.sourceGrant,
|
|
754
788
|
destination: input.destination,
|
|
755
789
|
expiresIn: input.expiresIn,
|
|
756
790
|
fetchImpl: this.fetchImpl
|
|
@@ -768,6 +802,7 @@ export class BeamClient {
|
|
|
768
802
|
metadata: partRouteMetadata(input.target.metadata)
|
|
769
803
|
},
|
|
770
804
|
source: input.source,
|
|
805
|
+
sourceGrant: input.sourceGrant,
|
|
771
806
|
destination: input.destination,
|
|
772
807
|
expiresIn: input.expiresIn,
|
|
773
808
|
uploadId: upload.uploadId,
|
|
@@ -791,6 +826,7 @@ export class BeamClient {
|
|
|
791
826
|
input.signal?.throwIfAborted();
|
|
792
827
|
if (payload.route_generation_id.length === 0)
|
|
793
828
|
throw new Error("route recovery generation is required");
|
|
829
|
+
const sourceGrants = new Map();
|
|
794
830
|
const routes = await mapOrderedWithConcurrency(payload.chunks, this.routeSigningConcurrency, async (requested) => {
|
|
795
831
|
input.signal?.throwIfAborted();
|
|
796
832
|
if (requested.route_generation_id !== payload.route_generation_id) {
|
|
@@ -821,11 +857,22 @@ export class BeamClient {
|
|
|
821
857
|
multipartUploads: input.multipartUploads,
|
|
822
858
|
expiresIn: input.expiresIn
|
|
823
859
|
});
|
|
860
|
+
const sourceKey = JSON.stringify([chunk.source_id, chunk.chunk_index]);
|
|
861
|
+
let sourceGrant = sourceGrants.get(sourceKey);
|
|
862
|
+
if (!sourceGrant) {
|
|
863
|
+
sourceGrant = signSourceChunk({ source, chunk, fallbackUrl: chunk.source_url, expiresIn: input.expiresIn, fetchImpl: this.fetchImpl });
|
|
864
|
+
sourceGrants.set(sourceKey, sourceGrant);
|
|
865
|
+
void sourceGrant.catch(() => { });
|
|
866
|
+
}
|
|
824
867
|
const route = await this.signProviderRoute({
|
|
868
|
+
sourceGrant,
|
|
825
869
|
chunk,
|
|
826
870
|
target: {
|
|
827
871
|
...target,
|
|
828
|
-
|
|
872
|
+
// Direct-PUT destinations (Hippius, Hugging Face) keep the planned
|
|
873
|
+
// per-chunk key exactly as initial materialization does; only
|
|
874
|
+
// multipart destinations sign parts against the final object key.
|
|
875
|
+
object_key: isDirectPutDestination(destination) ? target.object_key : requested.final_object_key,
|
|
829
876
|
metadata: {
|
|
830
877
|
...(target.metadata ?? {}),
|
|
831
878
|
...(requested.destination_metadata ?? {}),
|
|
@@ -861,16 +908,33 @@ export class BeamClient {
|
|
|
861
908
|
chunk_routes: routes
|
|
862
909
|
};
|
|
863
910
|
});
|
|
864
|
-
|
|
865
|
-
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
869
|
-
|
|
870
|
-
|
|
871
|
-
|
|
911
|
+
if (input.signal?.aborted) {
|
|
912
|
+
stop();
|
|
913
|
+
return;
|
|
914
|
+
}
|
|
915
|
+
this.stopRecoverySigner(input.prepared.transfer_id);
|
|
916
|
+
this.integrityAuditSigners.set(input.prepared.transfer_id, (challenge) => this.buildProviderIntegrityAuditGrants({
|
|
917
|
+
challenge, prepared: input.prepared, sourcesById: input.sourcesById, destinationsById: input.destinationsById, expiresIn: input.expiresIn
|
|
918
|
+
}));
|
|
919
|
+
const registration = { owner: input.owner, stop };
|
|
920
|
+
this.recoverySigners.set(input.prepared.transfer_id, registration);
|
|
921
|
+
try {
|
|
922
|
+
const stopIntegrity = await this.control.serveIntegritySigner(input.prepared.transfer_id, challenge => {
|
|
923
|
+
input.signal?.throwIfAborted();
|
|
924
|
+
return this.signedIntegrityGrants(challenge);
|
|
872
925
|
});
|
|
873
|
-
|
|
926
|
+
if (input.signal?.aborted || this.recoverySigners.get(input.prepared.transfer_id) !== registration) {
|
|
927
|
+
stop();
|
|
928
|
+
stopIntegrity();
|
|
929
|
+
return;
|
|
930
|
+
}
|
|
931
|
+
registration.stop = () => { stop(); stopIntegrity(); };
|
|
932
|
+
}
|
|
933
|
+
catch {
|
|
934
|
+
// Status-driven signing remains available if the optional subscription fails.
|
|
935
|
+
if (input.signal?.aborted)
|
|
936
|
+
this.stopRecoverySigner(input.prepared.transfer_id, input.owner);
|
|
937
|
+
}
|
|
874
938
|
}
|
|
875
939
|
async recoveryMultipartUploadState(input) {
|
|
876
940
|
const existing = input.multipartUploads.get(input.requested.multipart_group_id);
|
|
@@ -937,21 +1001,56 @@ export class BeamClient {
|
|
|
937
1001
|
if (!signer)
|
|
938
1002
|
throw new Error("integrity audit signer unavailable");
|
|
939
1003
|
const existing = this.integrityAuditSubmissions.get(challenge.audit_id);
|
|
940
|
-
|
|
1004
|
+
const cached = this.integrityGrantCache.get(challenge.audit_id);
|
|
1005
|
+
if (cached && cached.fingerprint !== JSON.stringify(challenge))
|
|
1006
|
+
throw new Error("integrity audit identity changed");
|
|
1007
|
+
if (existing && cached && cached.expiresAt > Date.now() + 30_000) {
|
|
941
1008
|
await existing;
|
|
942
1009
|
return;
|
|
943
1010
|
}
|
|
944
|
-
const submission =
|
|
1011
|
+
const submission = this.signedIntegrityGrants(challenge).then(async (payload) => {
|
|
1012
|
+
const result = await this.control.request("transfer.integrity_audit_grants", payload, {
|
|
1013
|
+
transferId: challenge.transfer_id, idempotencyKey: `transfer:${challenge.transfer_id}:integrity-audit:${challenge.audit_id}:${payload.submitted_at}`
|
|
1014
|
+
});
|
|
1015
|
+
if (result.published !== true)
|
|
1016
|
+
throw new Error("integrity audit delivery unavailable");
|
|
1017
|
+
});
|
|
945
1018
|
this.integrityAuditSubmissions.set(challenge.audit_id, submission);
|
|
946
1019
|
try {
|
|
947
1020
|
await submission;
|
|
948
1021
|
}
|
|
949
1022
|
catch (error) {
|
|
950
|
-
this.integrityAuditSubmissions.
|
|
1023
|
+
if (this.integrityAuditSubmissions.get(challenge.audit_id) === submission)
|
|
1024
|
+
this.integrityAuditSubmissions.delete(challenge.audit_id);
|
|
951
1025
|
throw error;
|
|
952
1026
|
}
|
|
953
1027
|
}
|
|
954
|
-
|
|
1028
|
+
signedIntegrityGrants(challenge) {
|
|
1029
|
+
const fingerprint = JSON.stringify(challenge);
|
|
1030
|
+
const cached = this.integrityGrantCache.get(challenge.audit_id);
|
|
1031
|
+
if (cached && cached.fingerprint !== fingerprint)
|
|
1032
|
+
return Promise.reject(new Error("integrity audit identity changed"));
|
|
1033
|
+
if (cached && cached.expiresAt > Date.now() + 30_000)
|
|
1034
|
+
return cached.payload;
|
|
1035
|
+
if (!cached && this.integrityGrantCache.size >= 1024)
|
|
1036
|
+
return Promise.reject(new Error("integrity signer capacity unavailable"));
|
|
1037
|
+
const signer = this.integrityAuditSigners.get(challenge.transfer_id);
|
|
1038
|
+
if (!signer)
|
|
1039
|
+
return Promise.reject(new Error("integrity audit signer unavailable"));
|
|
1040
|
+
this.integrityAuditSubmissions.delete(challenge.audit_id);
|
|
1041
|
+
const entry = { transferId: challenge.transfer_id, expiresAt: 0, fingerprint, payload: Promise.resolve({}) };
|
|
1042
|
+
entry.payload = signer(challenge).then(payload => {
|
|
1043
|
+
const chunks = payload.chunks;
|
|
1044
|
+
entry.expiresAt = Math.min(...chunks.flatMap(chunk => [Date.parse(chunk.source.expires_at), Date.parse(chunk.destination.expires_at)]));
|
|
1045
|
+
return payload;
|
|
1046
|
+
}).catch(error => { if (this.integrityGrantCache.get(challenge.audit_id) === entry)
|
|
1047
|
+
this.integrityGrantCache.delete(challenge.audit_id); throw error; });
|
|
1048
|
+
// Pending grants are shared too. Actual expiry replaces this sentinel on completion.
|
|
1049
|
+
entry.expiresAt = Infinity;
|
|
1050
|
+
this.integrityGrantCache.set(challenge.audit_id, entry);
|
|
1051
|
+
return entry.payload;
|
|
1052
|
+
}
|
|
1053
|
+
async buildProviderIntegrityAuditGrants(input) {
|
|
955
1054
|
if (input.challenge.transfer_id !== input.prepared.transfer_id) {
|
|
956
1055
|
throw new Error("integrity audit challenge transfer mismatch");
|
|
957
1056
|
}
|
|
@@ -978,6 +1077,7 @@ export class BeamClient {
|
|
|
978
1077
|
}
|
|
979
1078
|
const sourceEtag = plannedSource?.metadata?.etag;
|
|
980
1079
|
const sourceVersionId = plannedSource?.metadata?.version_id;
|
|
1080
|
+
const grantExpiresAt = new Date(Math.floor(Date.now() / 1000) * 1000 + input.expiresIn * 1000).toISOString();
|
|
981
1081
|
const [sourceGrant, destinationGrant] = await Promise.all([
|
|
982
1082
|
signSourceReadRange({
|
|
983
1083
|
source,
|
|
@@ -998,31 +1098,30 @@ export class BeamClient {
|
|
|
998
1098
|
fetchImpl: this.fetchImpl
|
|
999
1099
|
})
|
|
1000
1100
|
]);
|
|
1001
|
-
const grantExpiresAt = expiresAtIso(input.expiresIn);
|
|
1002
1101
|
const { orchestrator_id: _orchestratorId, orchestrator_hotkey: _orchestratorHotkey, worker_id: _workerId, ...grantChunk } = chunk;
|
|
1003
1102
|
return {
|
|
1004
1103
|
...grantChunk,
|
|
1005
|
-
source: { ...sourceGrant, expires_at: grantExpiresAt },
|
|
1006
|
-
destination: { ...destinationGrant, expires_at: grantExpiresAt }
|
|
1104
|
+
source: { ...sourceGrant, expires_at: boundedGrantExpiry(sourceGrant.url, grantExpiresAt) },
|
|
1105
|
+
destination: { ...destinationGrant, expires_at: boundedGrantExpiry(destinationGrant.url, grantExpiresAt) }
|
|
1007
1106
|
};
|
|
1008
1107
|
});
|
|
1009
|
-
|
|
1010
|
-
transfer_id: input.challenge.transfer_id,
|
|
1011
|
-
audit_id: input.challenge.audit_id,
|
|
1012
|
-
submitted_at: submittedAt,
|
|
1013
|
-
chunks
|
|
1014
|
-
}, {
|
|
1015
|
-
transferId: input.challenge.transfer_id,
|
|
1016
|
-
idempotencyKey: `transfer:${input.challenge.transfer_id}:integrity-audit:${input.challenge.audit_id}`
|
|
1017
|
-
});
|
|
1108
|
+
return { transfer_id: input.challenge.transfer_id, audit_id: input.challenge.audit_id, submitted_at: submittedAt, chunks };
|
|
1018
1109
|
}
|
|
1019
|
-
|
|
1020
|
-
|
|
1021
|
-
|
|
1022
|
-
|
|
1110
|
+
/** Stop a transfer's recovery and integrity signers; with `owner`, only if that owner installed them. */
|
|
1111
|
+
stopRecoverySigner(transferId, owner) {
|
|
1112
|
+
const signer = this.recoverySigners.get(transferId);
|
|
1113
|
+
if (owner && signer?.owner !== owner)
|
|
1114
|
+
return;
|
|
1115
|
+
if (signer) {
|
|
1116
|
+
signer.stop();
|
|
1023
1117
|
this.recoverySigners.delete(transferId);
|
|
1024
1118
|
}
|
|
1025
1119
|
this.integrityAuditSigners.delete(transferId);
|
|
1120
|
+
for (const [auditId, entry] of this.integrityGrantCache)
|
|
1121
|
+
if (entry.transferId === transferId) {
|
|
1122
|
+
this.integrityGrantCache.delete(auditId);
|
|
1123
|
+
this.integrityAuditSubmissions.delete(auditId);
|
|
1124
|
+
}
|
|
1026
1125
|
}
|
|
1027
1126
|
stopAllRecoverySigners() {
|
|
1028
1127
|
for (const transferId of [...this.recoverySigners.keys()])
|
|
@@ -1118,7 +1217,7 @@ export class BeamClient {
|
|
|
1118
1217
|
}
|
|
1119
1218
|
}
|
|
1120
1219
|
}
|
|
1121
|
-
const ROUTE_STREAM_BATCH_ROUTES =
|
|
1220
|
+
const ROUTE_STREAM_BATCH_ROUTES = 1_024;
|
|
1122
1221
|
class AsyncMutex {
|
|
1123
1222
|
tail = Promise.resolve();
|
|
1124
1223
|
async acquire() {
|
|
@@ -1143,6 +1242,8 @@ class AsyncMutex {
|
|
|
1143
1242
|
class RouteStreamSender {
|
|
1144
1243
|
control;
|
|
1145
1244
|
options;
|
|
1245
|
+
started = performance.now();
|
|
1246
|
+
telemetry;
|
|
1146
1247
|
streamId;
|
|
1147
1248
|
checksum = new RouteKeysChecksum();
|
|
1148
1249
|
batch = [];
|
|
@@ -1155,6 +1256,7 @@ class RouteStreamSender {
|
|
|
1155
1256
|
this.control = control;
|
|
1156
1257
|
this.options = options;
|
|
1157
1258
|
this.streamId = options.streamId;
|
|
1259
|
+
this.telemetry = options.telemetry ?? new SdkPerformanceCollector();
|
|
1158
1260
|
}
|
|
1159
1261
|
async begin() {
|
|
1160
1262
|
await this.control.request("transfer.route_stream.begin", compact({
|
|
@@ -1216,7 +1318,11 @@ class RouteStreamSender {
|
|
|
1216
1318
|
await this.sendTail;
|
|
1217
1319
|
if (this.sendError)
|
|
1218
1320
|
throw this.sendError;
|
|
1321
|
+
this.telemetry.observe("sdk.preparation", performance.now() - this.started);
|
|
1322
|
+
const sdkPerformance = this.telemetry.snapshot();
|
|
1323
|
+
this.options.onDiagnostics?.(sdkPerformance);
|
|
1219
1324
|
return this.control.request("transfer.route_stream.complete", {
|
|
1325
|
+
sdk_performance: sdkPerformance,
|
|
1220
1326
|
transfer_id: this.options.transferId,
|
|
1221
1327
|
route_generation_id: this.options.routeGenerationId,
|
|
1222
1328
|
stream_id: this.streamId,
|
|
@@ -1231,7 +1337,7 @@ class RouteStreamSender {
|
|
|
1231
1337
|
async enqueueFlush() {
|
|
1232
1338
|
if (!this.batch.length)
|
|
1233
1339
|
return;
|
|
1234
|
-
await this.sendTail;
|
|
1340
|
+
await this.telemetry.measure("sdk.buffer_wait", () => this.sendTail);
|
|
1235
1341
|
if (this.sendError)
|
|
1236
1342
|
throw this.sendError;
|
|
1237
1343
|
const routes = this.batch.splice(0, this.batch.length);
|
|
@@ -1255,7 +1361,8 @@ class RouteStreamSender {
|
|
|
1255
1361
|
const routeChecksum = chunkChecksum.value();
|
|
1256
1362
|
const coordinateChecksum = await routeCoordinateChecksum(chunk);
|
|
1257
1363
|
const batchId = `${this.streamId}:${batchIndex}:${coordinateChecksum}`;
|
|
1258
|
-
|
|
1364
|
+
this.telemetry.counters.route_batches++;
|
|
1365
|
+
await this.telemetry.measure("sdk.batch_ack", () => this.control.request("transfer.route_stream.batch", {
|
|
1259
1366
|
transfer_id: this.options.transferId,
|
|
1260
1367
|
route_generation_id: this.options.routeGenerationId,
|
|
1261
1368
|
stream_id: this.streamId,
|
|
@@ -1267,7 +1374,7 @@ class RouteStreamSender {
|
|
|
1267
1374
|
}, {
|
|
1268
1375
|
transferId: this.options.transferId,
|
|
1269
1376
|
idempotencyKey: `transfer:${this.options.transferId}:route-stream:${this.streamId}:batch:${batchIndex}:${coordinateChecksum}`
|
|
1270
|
-
});
|
|
1377
|
+
}));
|
|
1271
1378
|
}
|
|
1272
1379
|
catch (error) {
|
|
1273
1380
|
this.sendError = error;
|
|
@@ -1511,6 +1618,9 @@ function createDeferred() {
|
|
|
1511
1618
|
resolve = resolvePromise;
|
|
1512
1619
|
reject = rejectPromise;
|
|
1513
1620
|
});
|
|
1621
|
+
// A waiter may be rejected after its route loop already stopped (for example when
|
|
1622
|
+
// ownership is fenced mid-stream); awaiting callers still observe the rejection.
|
|
1623
|
+
promise.catch(() => undefined);
|
|
1514
1624
|
return { promise, resolve, reject };
|
|
1515
1625
|
}
|
|
1516
1626
|
function createMultipartGroupWaiters(prepared, destinationsById) {
|
package/dist/models.d.ts
CHANGED
|
@@ -1,5 +1,21 @@
|
|
|
1
1
|
export type SignedUrlFlow = "signed_url";
|
|
2
|
+
export interface SdkPerformanceSummary {
|
|
3
|
+
schema_version: "sdk-performance/v1";
|
|
4
|
+
measurements: Array<{
|
|
5
|
+
name: "sdk.discovery" | "sdk.multipart_create" | "sdk.signing" | "sdk.batch_ack" | "sdk.buffer_wait" | "sdk.preparation";
|
|
6
|
+
count: number;
|
|
7
|
+
work_ms: number;
|
|
8
|
+
max_ms: number;
|
|
9
|
+
}>;
|
|
10
|
+
counters: {
|
|
11
|
+
source_signatures: number;
|
|
12
|
+
source_reuses: number;
|
|
13
|
+
route_batches: number;
|
|
14
|
+
};
|
|
15
|
+
}
|
|
2
16
|
export interface BeamClientOptions {
|
|
17
|
+
/** Best effort, at most one bounded summary per route stream. Never awaited. */
|
|
18
|
+
onDiagnostics?: (summary: SdkPerformanceSummary) => void | Promise<void>;
|
|
3
19
|
apiKey?: string;
|
|
4
20
|
natsUrl?: string;
|
|
5
21
|
natsWsUrl?: string;
|
|
@@ -141,7 +157,34 @@ export interface IntegrityAuditChallenge {
|
|
|
141
157
|
range_bytes: number;
|
|
142
158
|
chunks: IntegrityAuditChallengeChunk[];
|
|
143
159
|
}
|
|
160
|
+
export interface TransferPerformance {
|
|
161
|
+
schema_version: "transfer-performance/v1";
|
|
162
|
+
runtime_epoch: string;
|
|
163
|
+
coverage: "complete" | "restarted";
|
|
164
|
+
elapsed_ms: number;
|
|
165
|
+
/** Work can overlap; these durations must not be summed into elapsed time. */
|
|
166
|
+
measurements: Record<string, {
|
|
167
|
+
count: number;
|
|
168
|
+
work_ms: number;
|
|
169
|
+
max_ms: number;
|
|
170
|
+
}>;
|
|
171
|
+
events: Record<string, {
|
|
172
|
+
count: number;
|
|
173
|
+
first_ms: number;
|
|
174
|
+
last_ms: number;
|
|
175
|
+
}>;
|
|
176
|
+
sdk_counters: {
|
|
177
|
+
source_signatures: number;
|
|
178
|
+
source_reuses: number;
|
|
179
|
+
route_batches: number;
|
|
180
|
+
};
|
|
181
|
+
detail_dropped: number;
|
|
182
|
+
unmeasured?: string[];
|
|
183
|
+
measurement_clocks?: Record<string, "runtime" | "sdk" | "fraud-service">;
|
|
184
|
+
sdk_report_received?: boolean;
|
|
185
|
+
}
|
|
144
186
|
export interface TransferStatusInfo {
|
|
187
|
+
performance?: TransferPerformance | null;
|
|
145
188
|
transfer_id: string;
|
|
146
189
|
name?: string;
|
|
147
190
|
status: string;
|
package/dist/nats-control.d.ts
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import type { IntegrityAuditChallenge } from "./models.js";
|
|
1
2
|
import { type ConnectionOptions } from "nats";
|
|
2
3
|
import type { MultipartGroupManifest, SignedChunkRoute, TransferTerminalSignalWaiter } from "./models.js";
|
|
3
4
|
export declare const TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION = "transfer-client-control/v7";
|
|
@@ -137,10 +138,18 @@ export declare class BeamTransferControl {
|
|
|
137
138
|
private helloTimer;
|
|
138
139
|
constructor(options: TransferControlOptions);
|
|
139
140
|
close(): Promise<void>;
|
|
141
|
+
serveIntegritySigner(transferId: string, handler: (challenge: IntegrityAuditChallenge) => Promise<Record<string, unknown>>): Promise<() => void>;
|
|
140
142
|
serveRouteRecoverySigner(transferId: string, handler: (payload: RouteRecoverySignRequestPayload) => Promise<RouteRecoverySignReplyPayload>): Promise<() => void>;
|
|
141
143
|
registerRecoveryLease(lease: TransferRecoveryLease): () => void;
|
|
142
|
-
|
|
143
|
-
|
|
144
|
+
/**
|
|
145
|
+
* Release a transfer's recovery lease, disposing its retained secrets.
|
|
146
|
+
*
|
|
147
|
+
* With `owner`, release only when that exact lease is still registered, so a
|
|
148
|
+
* fenced-off owner can never release a replacement owner's lease.
|
|
149
|
+
*/
|
|
150
|
+
releaseRecoveryLease(transferId: string, owner?: TransferRecoveryLease): void;
|
|
151
|
+
/** Request recovery for the registered lease; with `owner`, only if that lease is still registered. */
|
|
152
|
+
continueRecoveryLease(transferId: string, owner?: TransferRecoveryLease): void;
|
|
144
153
|
request<T>(messageType: TransferClientMessageType, payload: Record<string, unknown>, options?: LifecycleRequestOptions): Promise<T>;
|
|
145
154
|
openTerminalSignalWaiter(transferId: string): Promise<TransferTerminalSignalWaiter>;
|
|
146
155
|
splitRoutesForPayload(messageType: TransferClientMessageType, basePayload: Record<string, unknown>, routes: SignedChunkRoute[]): SignedChunkRoute[][];
|
package/dist/nats-control.js
CHANGED
|
@@ -60,6 +60,8 @@ export class BeamTransferControl {
|
|
|
60
60
|
terminalWaitCancellations = new Set();
|
|
61
61
|
routeRecoverySignerCancellations = new Set();
|
|
62
62
|
recoveryLeases = new Map();
|
|
63
|
+
// Keyed by lease identity: a replacement owner's recovery must never be
|
|
64
|
+
// coalesced into (and silently dropped behind) a fenced-off owner's loop.
|
|
63
65
|
recoveryOperations = new Map();
|
|
64
66
|
recoveryRequestedEpoch = new Map();
|
|
65
67
|
runtimeEpochs = new Map();
|
|
@@ -106,6 +108,58 @@ export class BeamTransferControl {
|
|
|
106
108
|
if (nc && !nc.isClosed())
|
|
107
109
|
await nc.drain();
|
|
108
110
|
}
|
|
111
|
+
async serveIntegritySigner(transferId, handler) {
|
|
112
|
+
const nc = await this.connection();
|
|
113
|
+
const subject = this.routeRecoverySignSubject(transferId).replace(/route_recovery_sign$/, "integrity_sign");
|
|
114
|
+
let active = true;
|
|
115
|
+
let pending = null;
|
|
116
|
+
let auditId = "";
|
|
117
|
+
let challengeFingerprint = "";
|
|
118
|
+
let awaiting = 0;
|
|
119
|
+
const subscription = nc.subscribe(subject, { callback: (error, message) => {
|
|
120
|
+
if (!active || error || !message.reply)
|
|
121
|
+
return;
|
|
122
|
+
if (awaiting >= 32) {
|
|
123
|
+
message.respond(new TextEncoder().encode('{"retry":true}'));
|
|
124
|
+
return;
|
|
125
|
+
}
|
|
126
|
+
awaiting++;
|
|
127
|
+
void (async () => {
|
|
128
|
+
try {
|
|
129
|
+
const request = JSON.parse(new TextDecoder().decode(message.data));
|
|
130
|
+
if (request.schema_version !== TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION || request.capability !== "integrity-signing/v1"
|
|
131
|
+
|| request.producer !== "transfer-runtime" || request.environment !== this.environment || request.key_prefix !== this.keyPrefix
|
|
132
|
+
|| request.transfer_id !== transferId || request.challenge?.transfer_id !== transferId || typeof request.request_id !== "string")
|
|
133
|
+
throw new Error("integrity_sign_scope_mismatch");
|
|
134
|
+
const fingerprint = JSON.stringify(request.challenge);
|
|
135
|
+
if (pending && (auditId !== request.challenge.audit_id || challengeFingerprint !== fingerprint))
|
|
136
|
+
throw new Error("integrity_sign_conflicting_audit");
|
|
137
|
+
if (!pending) {
|
|
138
|
+
auditId = request.challenge.audit_id;
|
|
139
|
+
challengeFingerprint = fingerprint;
|
|
140
|
+
pending = handler(request.challenge).finally(() => { pending = null; });
|
|
141
|
+
}
|
|
142
|
+
const payload = await pending;
|
|
143
|
+
if (active)
|
|
144
|
+
message.respond(new TextEncoder().encode(JSON.stringify({ schema_version: TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION,
|
|
145
|
+
capability: "integrity-signing/v1", environment: this.environment, key_prefix: this.keyPrefix,
|
|
146
|
+
transfer_id: transferId, request_id: request.request_id, payload })));
|
|
147
|
+
}
|
|
148
|
+
catch {
|
|
149
|
+
message.respond(new TextEncoder().encode('{"retry":true}'));
|
|
150
|
+
}
|
|
151
|
+
finally {
|
|
152
|
+
awaiting--;
|
|
153
|
+
}
|
|
154
|
+
})();
|
|
155
|
+
} });
|
|
156
|
+
await nc.flush();
|
|
157
|
+
const stop = () => { if (!active)
|
|
158
|
+
return; active = false; subscription.unsubscribe(); this.routeRecoverySignerCancellations.delete(stop); };
|
|
159
|
+
this.routeRecoverySignerCancellations.add(stop);
|
|
160
|
+
void this.pollRuntimeHello(transferShardId(transferId, this.shardCount));
|
|
161
|
+
return stop;
|
|
162
|
+
}
|
|
109
163
|
async serveRouteRecoverySigner(transferId, handler) {
|
|
110
164
|
const nc = await this.connection();
|
|
111
165
|
const subject = this.routeRecoverySignSubject(transferId);
|
|
@@ -165,23 +219,20 @@ export class BeamTransferControl {
|
|
|
165
219
|
this.ensureHelloMonitor();
|
|
166
220
|
const shardId = transferShardId(lease.transferId, this.shardCount);
|
|
167
221
|
void this.pollRuntimeHello(shardId);
|
|
168
|
-
return () =>
|
|
169
|
-
const current = this.recoveryLeases.get(lease.transferId);
|
|
170
|
-
if (current !== lease)
|
|
171
|
-
return;
|
|
172
|
-
this.recoveryLeases.delete(lease.transferId);
|
|
173
|
-
this.recoveryRequestedEpoch.delete(lease.transferId);
|
|
174
|
-
lease.disposeSecrets?.();
|
|
175
|
-
if (!this.recoveryLeases.size && this.helloTimer) {
|
|
176
|
-
clearInterval(this.helloTimer);
|
|
177
|
-
this.helloTimer = null;
|
|
178
|
-
}
|
|
179
|
-
};
|
|
222
|
+
return () => this.releaseRecoveryLease(lease.transferId, lease);
|
|
180
223
|
}
|
|
181
|
-
|
|
224
|
+
/**
|
|
225
|
+
* Release a transfer's recovery lease, disposing its retained secrets.
|
|
226
|
+
*
|
|
227
|
+
* With `owner`, release only when that exact lease is still registered, so a
|
|
228
|
+
* fenced-off owner can never release a replacement owner's lease.
|
|
229
|
+
*/
|
|
230
|
+
releaseRecoveryLease(transferId, owner) {
|
|
182
231
|
const lease = this.recoveryLeases.get(transferId);
|
|
183
232
|
if (!lease)
|
|
184
233
|
return;
|
|
234
|
+
if (owner && lease !== owner)
|
|
235
|
+
return;
|
|
185
236
|
this.recoveryLeases.delete(transferId);
|
|
186
237
|
this.recoveryRequestedEpoch.delete(transferId);
|
|
187
238
|
lease.disposeSecrets?.();
|
|
@@ -190,10 +241,13 @@ export class BeamTransferControl {
|
|
|
190
241
|
this.helloTimer = null;
|
|
191
242
|
}
|
|
192
243
|
}
|
|
193
|
-
|
|
244
|
+
/** Request recovery for the registered lease; with `owner`, only if that lease is still registered. */
|
|
245
|
+
continueRecoveryLease(transferId, owner) {
|
|
194
246
|
const lease = this.recoveryLeases.get(transferId);
|
|
195
247
|
if (!lease || this.closed)
|
|
196
248
|
return;
|
|
249
|
+
if (owner && lease !== owner)
|
|
250
|
+
return;
|
|
197
251
|
const shardId = transferShardId(transferId, this.shardCount);
|
|
198
252
|
const epoch = this.runtimeEpochs.get(shardId);
|
|
199
253
|
this.recoveryRequestedEpoch.set(transferId, epoch ? `${epoch.runtimeEpoch}:${epoch.transportEpoch}` : `foreground:${randomId()}`);
|
|
@@ -436,7 +490,12 @@ export class BeamTransferControl {
|
|
|
436
490
|
}
|
|
437
491
|
async pollRuntimeHello(shardId) {
|
|
438
492
|
try {
|
|
439
|
-
|
|
493
|
+
try {
|
|
494
|
+
await this.request("runtime.hello", { capabilities: ["integrity-signing/v1"] }, { idempotencyKey: `runtime:hello:${shardId}:integrity-signing`, shardId });
|
|
495
|
+
}
|
|
496
|
+
catch {
|
|
497
|
+
await this.request("runtime.hello", {}, { idempotencyKey: `runtime:hello:${shardId}`, shardId });
|
|
498
|
+
}
|
|
440
499
|
}
|
|
441
500
|
catch {
|
|
442
501
|
// Infinite NATS reconnect continues in the background; the next hello reconciles epochs.
|
|
@@ -463,13 +522,14 @@ export class BeamTransferControl {
|
|
|
463
522
|
}
|
|
464
523
|
}
|
|
465
524
|
recoverLease(lease) {
|
|
466
|
-
const existing = this.recoveryOperations.get(lease
|
|
525
|
+
const existing = this.recoveryOperations.get(lease);
|
|
467
526
|
if (existing)
|
|
468
527
|
return existing;
|
|
469
|
-
|
|
470
|
-
|
|
528
|
+
const owns = () => !this.closed && this.recoveryLeases.get(lease.transferId) === lease;
|
|
529
|
+
let replaced = false;
|
|
530
|
+
const operation = (async () => {
|
|
471
531
|
let attempt = 0;
|
|
472
|
-
while (
|
|
532
|
+
while (owns()) {
|
|
473
533
|
const requestedEpoch = this.recoveryRequestedEpoch.get(lease.transferId);
|
|
474
534
|
const routeGenerationId = randomId();
|
|
475
535
|
try {
|
|
@@ -479,39 +539,53 @@ export class BeamTransferControl {
|
|
|
479
539
|
coordinate_checksum: lease.coordinateChecksum,
|
|
480
540
|
route_generation_id: routeGenerationId
|
|
481
541
|
}, { transferId: lease.transferId, idempotencyKey: `transfer:${lease.transferId}:resume:${routeGenerationId}` });
|
|
542
|
+
if (!owns()) {
|
|
543
|
+
// A fenced-off owner must neither act on nor release a replacement's lease.
|
|
544
|
+
replaced = true;
|
|
545
|
+
return;
|
|
546
|
+
}
|
|
482
547
|
if (response.recovery === "terminal") {
|
|
483
|
-
this.releaseRecoveryLease(lease.transferId);
|
|
484
|
-
if (this.recoveryOperations.get(lease.transferId) === operation) {
|
|
485
|
-
this.recoveryOperations.delete(lease.transferId);
|
|
486
|
-
}
|
|
548
|
+
this.releaseRecoveryLease(lease.transferId, lease);
|
|
487
549
|
return;
|
|
488
550
|
}
|
|
489
551
|
if (response.route_replay_required || response.recovery === "route_replay_required") {
|
|
490
552
|
await lease.replayRoutes(routeGenerationId);
|
|
553
|
+
if (!owns()) {
|
|
554
|
+
replaced = true;
|
|
555
|
+
return;
|
|
556
|
+
}
|
|
491
557
|
}
|
|
492
558
|
if (this.recoveryRequestedEpoch.get(lease.transferId) !== requestedEpoch) {
|
|
493
559
|
attempt = 0;
|
|
494
560
|
continue;
|
|
495
561
|
}
|
|
496
|
-
if (this.recoveryOperations.get(lease.transferId) === operation) {
|
|
497
|
-
this.recoveryOperations.delete(lease.transferId);
|
|
498
|
-
}
|
|
499
562
|
return;
|
|
500
563
|
}
|
|
501
564
|
catch (error) {
|
|
565
|
+
if (!owns()) {
|
|
566
|
+
replaced = true;
|
|
567
|
+
return;
|
|
568
|
+
}
|
|
502
569
|
if (!isRetryableLifecycleError(error)) {
|
|
503
|
-
this.releaseRecoveryLease(lease.transferId);
|
|
570
|
+
this.releaseRecoveryLease(lease.transferId, lease);
|
|
504
571
|
return;
|
|
505
572
|
}
|
|
506
573
|
attempt += 1;
|
|
507
574
|
await sleepWithJitter(Math.min(30_000, 500 * (2 ** Math.min(attempt, 6))));
|
|
508
575
|
}
|
|
509
576
|
}
|
|
577
|
+
replaced = !this.closed;
|
|
510
578
|
})().finally(() => {
|
|
511
|
-
if (this.recoveryOperations.get(lease
|
|
512
|
-
this.recoveryOperations.delete(lease
|
|
579
|
+
if (this.recoveryOperations.get(lease) === operation)
|
|
580
|
+
this.recoveryOperations.delete(lease);
|
|
581
|
+
if (!replaced || this.closed)
|
|
582
|
+
return;
|
|
583
|
+
// The recovery this loop was driving still matters: hand it to the replacement owner.
|
|
584
|
+
const current = this.recoveryLeases.get(lease.transferId);
|
|
585
|
+
if (current && current !== lease)
|
|
586
|
+
void this.recoverLease(current);
|
|
513
587
|
});
|
|
514
|
-
this.recoveryOperations.set(lease
|
|
588
|
+
this.recoveryOperations.set(lease, operation);
|
|
515
589
|
return operation;
|
|
516
590
|
}
|
|
517
591
|
async resolveAuthToken() {
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
import type { SdkPerformanceSummary } from './models.js';
|
|
2
|
+
export declare class SdkPerformanceCollector {
|
|
3
|
+
private readonly measurements;
|
|
4
|
+
readonly counters: {
|
|
5
|
+
source_signatures: number;
|
|
6
|
+
source_reuses: number;
|
|
7
|
+
route_batches: number;
|
|
8
|
+
};
|
|
9
|
+
observe(name: SdkPerformanceSummary['measurements'][number]['name'], milliseconds: number): void;
|
|
10
|
+
measure<T>(name: SdkPerformanceSummary['measurements'][number]['name'], work: () => Promise<T>): Promise<T>;
|
|
11
|
+
snapshot(): SdkPerformanceSummary;
|
|
12
|
+
}
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
export class SdkPerformanceCollector {
|
|
2
|
+
measurements = new Map();
|
|
3
|
+
counters = { source_signatures: 0, source_reuses: 0, route_batches: 0 };
|
|
4
|
+
observe(name, milliseconds) {
|
|
5
|
+
if (!Number.isFinite(milliseconds) || milliseconds < 0)
|
|
6
|
+
return;
|
|
7
|
+
const value = this.measurements.get(name) ?? { count: 0, work_ms: 0, max_ms: 0 };
|
|
8
|
+
value.count++;
|
|
9
|
+
value.work_ms += milliseconds;
|
|
10
|
+
value.max_ms = Math.max(value.max_ms, milliseconds);
|
|
11
|
+
this.measurements.set(name, value);
|
|
12
|
+
}
|
|
13
|
+
async measure(name, work) {
|
|
14
|
+
const start = performance.now();
|
|
15
|
+
try {
|
|
16
|
+
return await work();
|
|
17
|
+
}
|
|
18
|
+
finally {
|
|
19
|
+
this.observe(name, performance.now() - start);
|
|
20
|
+
}
|
|
21
|
+
}
|
|
22
|
+
snapshot() {
|
|
23
|
+
return { schema_version: 'sdk-performance/v1', measurements: [...this.measurements].map(([name, value]) => ({ name: name, ...value })), counters: { ...this.counters } };
|
|
24
|
+
}
|
|
25
|
+
}
|
|
@@ -63,7 +63,15 @@ export declare function inspectDestinationObject(destination: ProviderDestinatio
|
|
|
63
63
|
versionId: string | undefined;
|
|
64
64
|
metadata: Record<string, string>;
|
|
65
65
|
}>;
|
|
66
|
+
export declare function boundedGrantExpiry(url: string, upperBound: string): string;
|
|
67
|
+
export type SourceRouteGrant = {
|
|
68
|
+
url: string;
|
|
69
|
+
headers: Record<string, string>;
|
|
70
|
+
expiresAt: string;
|
|
71
|
+
};
|
|
72
|
+
export declare function signSourceChunk(input: Parameters<typeof signSourceRoute>[0]): Promise<SourceRouteGrant>;
|
|
66
73
|
export declare function signDestinationRoute(input: {
|
|
74
|
+
sourceGrant?: Promise<SourceRouteGrant>;
|
|
67
75
|
chunk: ChunkSigningPlanItem;
|
|
68
76
|
target: ChunkDestinationSigningTarget;
|
|
69
77
|
source?: ProviderSourceConfig;
|
|
@@ -109,6 +117,16 @@ export declare function signDestinationReadRange(input: {
|
|
|
109
117
|
url: string;
|
|
110
118
|
headers: Record<string, string>;
|
|
111
119
|
}>;
|
|
120
|
+
declare function signSourceRoute(input: {
|
|
121
|
+
source?: ProviderSourceConfig;
|
|
122
|
+
fallbackUrl: string;
|
|
123
|
+
chunk: ChunkSigningPlanItem;
|
|
124
|
+
expiresIn: number;
|
|
125
|
+
fetchImpl?: typeof fetch;
|
|
126
|
+
}): Promise<{
|
|
127
|
+
url: string;
|
|
128
|
+
headers: Record<string, string>;
|
|
129
|
+
}>;
|
|
112
130
|
export declare function signDestinationUrl(input: {
|
|
113
131
|
contentMd5?: string;
|
|
114
132
|
destination: ProviderDestinationConfig;
|
|
@@ -118,8 +136,10 @@ export declare function signDestinationUrl(input: {
|
|
|
118
136
|
expiresIn: number;
|
|
119
137
|
fetchImpl?: typeof fetch;
|
|
120
138
|
}): Promise<string>;
|
|
139
|
+
export declare function releaseProviderClients(configs: Array<ProviderSourceConfig | ProviderDestinationConfig>): void;
|
|
121
140
|
export declare function isS3CompatibleProvider(config: ProviderSourceConfig | ProviderDestinationConfig): config is AnyS3CompatibleProviderConfig;
|
|
122
141
|
export declare function s3CompatibleEndpoint(source: AnyS3CompatibleProviderConfig): string | undefined;
|
|
123
142
|
export declare function s3CompatibleRegion(source: AnyS3CompatibleProviderConfig): string;
|
|
124
143
|
export declare function s3CompatibleForcePathStyle(source: AnyS3CompatibleProviderConfig, endpoint?: string | undefined): boolean | undefined;
|
|
125
144
|
export declare function isHuggingFaceProvider(config: ProviderSourceConfig | ProviderDestinationConfig): config is HuggingFaceProviderConfig;
|
|
145
|
+
export {};
|
package/dist/provider-signing.js
CHANGED
|
@@ -69,7 +69,7 @@ export function nowIso() {
|
|
|
69
69
|
return new Date().toISOString();
|
|
70
70
|
}
|
|
71
71
|
export function expiresAtIso(expiresIn) {
|
|
72
|
-
return new Date(Date.now() + expiresIn * 1000).toISOString();
|
|
72
|
+
return new Date((Math.floor(Date.now() / 1000) + expiresIn) * 1000).toISOString();
|
|
73
73
|
}
|
|
74
74
|
export async function prepareProviderSource(source, options = {}) {
|
|
75
75
|
const index = options.index ?? 0;
|
|
@@ -307,11 +307,34 @@ export async function inspectDestinationObject(destination, objectKey, signal) {
|
|
|
307
307
|
const result = await createS3CompatibleClient(destination).send(new HeadObjectCommand({ Bucket: destination.bucket, Key: objectKey }), { abortSignal: signal });
|
|
308
308
|
return { size: result.ContentLength, etag: result.ETag, versionId: result.VersionId, metadata: result.Metadata ?? {} };
|
|
309
309
|
}
|
|
310
|
+
// Never advertise a renewed lifetime for a previously issued provider URL.
|
|
311
|
+
export function boundedGrantExpiry(url, upperBound) {
|
|
312
|
+
let expiry = Date.parse(upperBound);
|
|
313
|
+
const query = new URL(url).searchParams;
|
|
314
|
+
for (const prefix of ["X-Amz", "X-Goog"]) {
|
|
315
|
+
const date = query.get(`${prefix}-Date`), duration = Number(query.get(`${prefix}-Expires`));
|
|
316
|
+
if (date && /^\d{8}T\d{6}Z$/.test(date) && Number.isFinite(duration) && duration > 0) {
|
|
317
|
+
const start = Date.parse(`${date.slice(0, 4)}-${date.slice(4, 6)}-${date.slice(6, 8)}T${date.slice(9, 11)}:${date.slice(11, 13)}:${date.slice(13, 15)}Z`);
|
|
318
|
+
if (Number.isFinite(start))
|
|
319
|
+
expiry = Math.min(expiry, start + duration * 1000);
|
|
320
|
+
}
|
|
321
|
+
}
|
|
322
|
+
const epoch = Number(query.get("Expires"));
|
|
323
|
+
if (Number.isFinite(epoch) && epoch > 0)
|
|
324
|
+
expiry = Math.min(expiry, epoch * 1000);
|
|
325
|
+
return new Date(expiry).toISOString();
|
|
326
|
+
}
|
|
327
|
+
export async function signSourceChunk(input) {
|
|
328
|
+
const expiresAt = expiresAtIso(input.expiresIn);
|
|
329
|
+
const grant = await signSourceRoute(input);
|
|
330
|
+
return { ...grant, expiresAt: boundedGrantExpiry(grant.url, expiresAt) };
|
|
331
|
+
}
|
|
310
332
|
export async function signDestinationRoute(input) {
|
|
311
333
|
const targetObjectKey = input.target.object_key;
|
|
312
334
|
if (!targetObjectKey) {
|
|
313
335
|
throw new Error("destination signing target is missing object_key");
|
|
314
336
|
}
|
|
337
|
+
const destinationExpiresAt = expiresAtIso(input.expiresIn);
|
|
315
338
|
const [destUrl, sourceRoute] = await Promise.all([
|
|
316
339
|
input.destUrl ?? signDestinationUrl({
|
|
317
340
|
destination: input.destination,
|
|
@@ -321,7 +344,7 @@ export async function signDestinationRoute(input) {
|
|
|
321
344
|
expiresIn: input.expiresIn,
|
|
322
345
|
fetchImpl: input.fetchImpl,
|
|
323
346
|
}),
|
|
324
|
-
|
|
347
|
+
input.sourceGrant ?? signSourceChunk({
|
|
325
348
|
source: input.source,
|
|
326
349
|
fallbackUrl: input.chunk.source_url,
|
|
327
350
|
chunk: input.chunk,
|
|
@@ -337,8 +360,8 @@ export async function signDestinationRoute(input) {
|
|
|
337
360
|
dest_url: destUrl,
|
|
338
361
|
source_offset: input.chunk.source_offset,
|
|
339
362
|
chunk_size: input.chunk.chunk_size,
|
|
340
|
-
expires_at:
|
|
341
|
-
headers: sourceRoute.headers,
|
|
363
|
+
expires_at: boundedGrantExpiry(destUrl, new Date(Math.min(Date.parse(destinationExpiresAt), Date.parse(sourceRoute.expiresAt))).toISOString()),
|
|
364
|
+
headers: { ...sourceRoute.headers },
|
|
342
365
|
metadata: {
|
|
343
366
|
...(input.target.metadata ?? {}),
|
|
344
367
|
...(input.multipartGroupId ? { multipart_group_id: input.multipartGroupId } : {}),
|
|
@@ -475,7 +498,16 @@ export async function signDestinationUrl(input) {
|
|
|
475
498
|
}
|
|
476
499
|
return unsupportedProviderConfig(input.destination);
|
|
477
500
|
}
|
|
478
|
-
const s3ClientCache = new
|
|
501
|
+
const s3ClientCache = new WeakMap();
|
|
502
|
+
export function releaseProviderClients(configs) {
|
|
503
|
+
for (const config of configs) {
|
|
504
|
+
if (!isS3CompatibleProvider(config))
|
|
505
|
+
continue;
|
|
506
|
+
const cached = s3ClientCache.get(config);
|
|
507
|
+
s3ClientCache.delete(config);
|
|
508
|
+
cached?.client.destroy();
|
|
509
|
+
}
|
|
510
|
+
}
|
|
479
511
|
function s3ClientCacheKey(source, endpoint) {
|
|
480
512
|
return JSON.stringify({
|
|
481
513
|
provider: source.provider,
|
|
@@ -489,9 +521,9 @@ function s3ClientCacheKey(source, endpoint) {
|
|
|
489
521
|
}
|
|
490
522
|
function createS3CompatibleClient(source, endpoint = s3CompatibleEndpoint(source)) {
|
|
491
523
|
const cacheKey = s3ClientCacheKey(source, endpoint);
|
|
492
|
-
const cached = s3ClientCache.get(
|
|
493
|
-
if (cached)
|
|
494
|
-
return cached;
|
|
524
|
+
const cached = s3ClientCache.get(source);
|
|
525
|
+
if (cached?.identity === cacheKey)
|
|
526
|
+
return cached.client;
|
|
495
527
|
const client = new S3Client({
|
|
496
528
|
region: s3CompatibleRegion(source),
|
|
497
529
|
endpoint,
|
|
@@ -503,7 +535,7 @@ function createS3CompatibleClient(source, endpoint = s3CompatibleEndpoint(source
|
|
|
503
535
|
sessionToken: "session_token" in source ? source.session_token : undefined
|
|
504
536
|
}
|
|
505
537
|
});
|
|
506
|
-
s3ClientCache.set(cacheKey, client);
|
|
538
|
+
s3ClientCache.set(source, { identity: cacheKey, client });
|
|
507
539
|
return client;
|
|
508
540
|
}
|
|
509
541
|
export function isS3CompatibleProvider(config) {
|