@beam-network/sdk 0.5.14 → 0.5.16

package/README.md

@@ -43,7 +43,7 @@ console.log(status.status);
 await beam.close();
 ```
 
-The main `createTransfer` API is provider-aware and strictly typed for S3, R2, S3-compatible, and Hippius configs.
+The main `createTransfer` API is provider-aware and strictly typed for S3, R2, S3-compatible, Hippius, and Hugging Face configs.
 Use `testMode: true` to create a BeamCore test-mode transfer.
 
 ## S3-Compatible Providers
@@ -92,13 +92,68 @@ Use `S3ProviderConfig` for ordinary AWS S3 if preferred, and keep using
 `S3CompatibleProviderConfig` when a provider needs a custom S3-compatible
 endpoint.
 
+## Hugging Face Hub
+
+Use `HuggingFaceProviderConfig` to pull a file out of a Hub repo or push one into
+one. The token stays in this process: a source resolves to the presigned CDN URL
+the Hub redirects to, and a destination uses the presigned part URLs the Hub's
+LFS batch endpoint issues.
+
+```ts
+import { BeamClient, HuggingFaceProviderConfig, S3ProviderConfig } from "@beam-network/sdk";
+
+const beam = new BeamClient({ apiKey: process.env.BEAM_API_KEY! });
+
+const transfer = await beam.prepareProviderTransfer({
+  sources: [
+    HuggingFaceProviderConfig.create({
+      repo_id: "org/dataset",
+      path: "data/train.parquet",
+      repo_type: "dataset",
+      token: process.env.HF_TOKEN!
+    })
+  ],
+  destinations: [
+    S3ProviderConfig.create({
+      bucket: "archive",
+      key: "datasets/train.parquet",
+      access_key_id: process.env.AWS_ACCESS_KEY_ID!,
+      secret_access_key: process.env.AWS_SECRET_ACCESS_KEY!
+    })
+  ],
+  name: "Hugging Face pull"
+});
+
+// waitForTransfer commits any Hugging Face destination once the parts land.
+await beam.waitForTransfer(transfer.transfer_id);
+```
+
+Constraints, all of them the Hub's rather than Beam's — see
+[`HUGGINGFACE_PROVIDER.md`](../../HUGGINGFACE_PROVIDER.md) for the full protocol:
+
+- **Sources are large-file (LFS or Xet) content.** Only those get the
+  credential-free CDN redirect; a small regular file is served inline and is
+  rejected with that reason. Hub **buckets** work as sources too, but cannot be
+  written to — they expose no LFS endpoint.
+- **A destination needs the object's sha256 up front**, because the LFS batch
+  endpoint will not issue upload URLs without it. When the source is not itself a
+  Hub file, set `allow_source_rehash: true` to let the SDK read the source once to
+  compute it.
+- **One source per Hugging Face destination**, because the Hub dictates the part
+  size and a plan carries a single chunk size.
+- If you do not call `waitForTransfer`, call `finalizeHuggingFaceUploads(transferId)`
+  yourself once the transfer completes — without it the parts are uploaded but no
+  commit is made and the file does not appear in the repo.
+
 For low-level raw transfer configs, use `createRawTransfer`; lifecycle transport still goes through NATS.
 
 ## Route Streaming And Payload Size
 
-Provider transfers use `transfer-client-control/v6`. Every reply carries Runtime and transport epochs; prepare and every route-stream message carry a UUID route generation. S3, R2, and S3-compatible destinations retain direct multipart UploadPart/ListParts/HEAD handling.
+Provider transfers use `transfer-client-control/v6`. Every reply carries Runtime and transport epochs; prepare and every route-stream message carry a UUID route generation. S3, R2, and S3-compatible destinations retain direct multipart UploadPart/ListParts/HEAD handling. Hippius and Hugging Face destinations take a plain PUT per chunk instead, so they carry no multipart group manifest.
+
+The client signs up to 64 routes concurrently by default and emits 2,048-route logical batches. Encoded MessagePack requests target 4 MiB physical batches under `maxPayloadBytes` (8 MiB by default; explicit positive overrides remain supported). The splitter reserves control-envelope headroom for the live auth token and stable request identity so the final encoded request remains below the configured guard. Lifecycle control refreshes cached SDK auth before the final 30 seconds of token lifetime and retries `auth_token_expired` replies with the same request identity and payload but a fresh token, preserving Core request-conflict protection during large route streams. A single route above the 4 MiB target but within the configured guard publishes alone; a single route larger than the configured limit fails before publication with both sizes in the error. Multipart create and abort requests use a separate `multipartControlConcurrency` limit of 2 by default, with explicit positive overrides supported, and S3-compatible control requests make up to five attempts for transient provider failures. This keeps URL signing throughput independent from provider control-plane pressure.
 
-The client signs up to 64 routes concurrently by default and emits 2,048-route logical batches. Encoded MessagePack requests target 4 MiB physical batches under `maxPayloadBytes` (8 MiB by default; explicit positive overrides remain supported). The splitter reserves control-envelope headroom for the live auth token and stable request identity so the final encoded request remains below the configured guard. A single route above the 4 MiB target but within the configured guard publishes alone; a single route larger than the configured limit fails before publication with both sizes in the error. Multipart create and abort requests use a separate `multipartControlConcurrency` limit of 2 by default, with explicit positive overrides supported, and S3-compatible control requests make up to five attempts for transient provider failures. This keeps URL signing throughput independent from provider control-plane pressure.
+Callers that need durable recovery accounting can provide `onMultipartGroupReady`. The SDK invokes it after creating a multipart upload and before streaming that group’s signed routes. Its payload contains transfer, source, destination, object-key, upload, size, part-count, and expiry identities only; it never includes credentials, request headers, or signed URLs. A rejected callback aborts the newly created upload and fails preparation closed.
 
 The stream ID derives from the transfer, selected flow, and immutable plan identity, and each ordered batch ID includes its route-coordinate checksum. Manual multi-destination attachment requires `delivery_index` on every route. Lifecycle mutations retry transient failures up to three times with the same request identity. A non-recoverable provider setup or route failure raises `BeamProviderTransferError`, whose `errors`, `transferCancelled`, and `multipartCleanupComplete` fields retain the original provider, cancellation, and cleanup outcomes. `waitForTransfer` subscribes to the API-key-owned, at-most-once terminal signal before its first status read and reconciles every signal through authoritative status; subscription failure degrades to the same jittered 15-to-30-second status fallback. Call `close()` when the client is no longer needed; it also closes outstanding terminal waiters.
 

package/dist/client.d.ts

@@ -34,6 +34,7 @@ type TransferPrepareInput = {
     signedUrlFlow?: SignedUrlFlow;
     idempotencyKey?: string;
     routeGenerationId?: string;
+    chunkSize?: number;
 };
 export declare class BeamClient {
     readonly apiKey: string;
@@ -46,6 +47,7 @@ export declare class BeamClient {
     private readonly recoverySigners;
     private readonly integrityAuditSigners;
     private readonly integrityAuditSubmissions;
+    private readonly huggingFaceUploads;
     constructor(options?: BeamClientOptions);
     close(): Promise<void>;
     openTransferTerminalWaiter(transferId: string): Promise<TransferTerminalSignalWaiter>;
@@ -62,6 +64,7 @@ export declare class BeamClient {
         testMode?: boolean;
         urlsExpiresAt?: string;
         signedUrlFlow?: SignedUrlFlow;
+        chunkSize?: number;
     }): Promise<TransferPlanResponse>;
     prepareTransfer(input: TransferPrepareInput): Promise<TransferPrepareResponse>;
     private prepareTransferWithRequestKey;
@@ -81,6 +84,21 @@ export declare class BeamClient {
         autoDistribute?: boolean;
     }): Promise<AttachSignedUrlsResponse>;
     prepareProviderTransfer(input: ProviderTransferCreateInput): Promise<TransferPrepareResponse>;
+    /**
+     * Negotiate every Hugging Face destination before the plan exists.
+     *
+     * The Hub will not issue upload URLs without the object's sha256, and it chooses the part
+     * size itself, so this runs first and the plan is then requested at the Hub's chunk size.
+     */
+    private planHuggingFaceUploads;
+    /** Fail before any byte moves if BeamCore did not adopt the Hub's part layout. */
+    private assertHuggingFacePlan;
+    /**
+     * Close every Hugging Face upload for a transfer: complete the LFS multipart, verify it, and
+     * commit the blob so the file appears in the repo. Runs only after the transfer is complete,
+     * so it never sits in the transfer's progression path.
+     */
+    finalizeHuggingFaceUploads(transferId: string): Promise<void>;
     private signProviderRoute;
     private startProviderRouteRecoverySigner;
     private recoveryMultipartUploadState;

package/dist/client.js

@@ -1,4 +1,5 @@
-import { abortMultipartUpload, createMultipartUpload, expiresAtIso, prepareProviderDestination, prepareProviderSource, signAbortMultipartUpload, signCompleteMultipartUpload, signDestinationReadRange, signDestinationRoute, signFinalObjectHead, signListMultipartUpload, signSourceReadRange } from "./provider-signing.js";
+import { abortMultipartUpload, createMultipartUpload, expiresAtIso, prepareProviderDestination, prepareProviderSource, signAbortMultipartUpload, signCompleteMultipartUpload, signDestinationReadRange, signDestinationRoute, signFinalObjectHead, signListMultipartUpload, signSourceReadRange, isHuggingFaceProvider } from "./provider-signing.js";
+import { describe as describeHuggingFace, hashSourceStream, huggingFaceCommit, huggingFaceCompleteLfsUpload, huggingFaceLfsBatch, huggingFacePreupload, huggingFaceVerifyLfsUpload, readSourceSample } from "./huggingface.js";
 import { BeamTransferControl, BEAM_DEFAULT_NATS_URL, compactSignedRoutes, isRecoverableRouteStreamError } from "./nats-control.js";
 export class BeamRouteRecoveryPendingError extends Error {
     transferId;
@@ -55,6 +56,7 @@ export class BeamClient {
     recoverySigners = new Map();
     integrityAuditSigners = new Map();
     integrityAuditSubmissions = new Map();
+    huggingFaceUploads = new Map();
     constructor(options = {}) {
         if (!options.apiKey?.trim()) {
             throw new Error("apiKey is required.");
@@ -203,6 +205,7 @@ export class BeamClient {
             destinations: input.destinations,
             name: input.name,
             test_mode: input.testMode || undefined,
+            chunk_size: input.chunkSize,
             urls_expires_at: input.urlsExpiresAt,
             signed_url_flow: input.signedUrlFlow ?? "signed_url"
         }));
@@ -225,6 +228,7 @@ export class BeamClient {
             destinations: input.destinations,
             name: input.name,
             test_mode: input.testMode || undefined,
+            chunk_size: input.chunkSize,
             urls_expires_at: input.urlsExpiresAt,
             signed_url_flow: input.signedUrlFlow ?? "signed_url"
         }), {
@@ -318,6 +322,7 @@ export class BeamClient {
             destinations: input.destinations.map((destination) => ({ ...destination })),
             onBeforeTransferPrepare: undefined,
             onPrepared: undefined,
+            onMultipartGroupReady: undefined,
             throwIfCancelled: undefined
         };
         const preparedDestinations = retainedRecoveryInput.destinations.map((destination, index) => prepareProviderDestination(destination, { index }));
@@ -332,12 +337,20 @@ export class BeamClient {
         await input.throwIfCancelled?.();
         const preparedSources = await Promise.all(retainedRecoveryInput.sources.map((source, index) => prepareProviderSource(source, { index, expiresIn, fetchImpl: this.fetchImpl })));
         await input.throwIfCancelled?.();
+        const huggingFace = await this.planHuggingFaceUploads({
+            sources: retainedRecoveryInput.sources,
+            preparedSources,
+            destinations: retainedRecoveryInput.destinations,
+            preparedDestinations
+        });
+        await input.throwIfCancelled?.();
         await input.onBeforeTransferPrepare?.();
         const prepared = await this.prepareTransfer({
             sources: preparedSources,
             destinations: preparedDestinations,
             name: input.name,
             testMode: input.testMode,
+            chunkSize: huggingFace.chunkSize ?? input.chunkSize,
             signedUrlFlow: requestedSignedUrlFlow,
             idempotencyKey: input.idempotencyKey,
             routeGenerationId: input.routeGenerationId
@@ -345,6 +358,11 @@ export class BeamClient {
         if (!prepared.success) {
             return prepared;
         }
+        if (huggingFace.states.length) {
+            this.assertHuggingFacePlan(prepared, huggingFace.states);
+            this.huggingFaceUploads.set(prepared.transfer_id, huggingFace.states);
+        }
+        const huggingFaceByDestination = new Map(huggingFace.states.map((state) => [state.destinationId, state]));
         const sourcesById = new Map();
         preparedSources.forEach((preparedSource, index) => {
             const source = retainedRecoveryInput.sources[index];
@@ -409,6 +427,7 @@ export class BeamClient {
                     onGroupReady: async (state) => {
                         recoveryMultipartUploads.set(state.manifest.multipart_group_id, state);
                         validateMultipartGroupManifest([state.manifest], prepared.transfer_id);
+                        await input.onMultipartGroupReady?.(multipartGroupIdentity(prepared.transfer_id, state.manifest));
                         await routeStream.addManifestGroups([state.manifest]);
                         multipartGroupWaiters.get(state.manifest.multipart_group_id)?.resolve(state);
                     },
@@ -444,10 +463,10 @@ export class BeamClient {
                             if (!finalObjectKey)
                                 throw new Error("destination signing target is missing object_key");
                             const multipartGroupId = multipartGroupStateKey(prepared.transfer_id, target.destination_id, chunk.source_id, finalObjectKey);
-                            const upload = isHippiusDestination(destination)
+                            const upload = isDirectPutDestination(destination)
                                 ? undefined
                                 : await multipartGroupWaiters.get(multipartGroupId)?.promise;
-                            if (!isHippiusDestination(destination) && !upload)
+                            if (!isDirectPutDestination(destination) && !upload)
                                 throw new Error(`multipart group manifest is missing for ${chunk.source_id}:${target.destination_id}`);
                             return this.signProviderRoute({
                                 chunk,
@@ -456,6 +475,7 @@ export class BeamClient {
                                 destination,
                                 expiresIn,
                                 ...(upload ? { upload } : {}),
+                                huggingFaceUpload: huggingFaceByDestination.get(target.destination_id),
                                 transferId: prepared.transfer_id,
                                 finalObjectKey,
                                 signedUrlFlow: requestedSignedUrlFlow,
@@ -546,7 +566,186 @@ export class BeamClient {
         }
         return prepared;
     }
+    /**
+     * Negotiate every Hugging Face destination before the plan exists.
+     *
+     * The Hub will not issue upload URLs without the object's sha256, and it chooses the part
+     * size itself, so this runs first and the plan is then requested at the Hub's chunk size.
+     */
+    async planHuggingFaceUploads(input) {
+        const targets = input.destinations
+            .map((destination, index) => ({ destination, index }))
+            .filter((entry) => isHuggingFaceProvider(entry.destination));
+        if (!targets.length)
+            return { states: [] };
+        if (input.preparedSources.length !== 1) {
+            throw new Error("a huggingface destination requires exactly one source: the Hub dictates the part size "
+                + `and the plan carries a single chunk size, but ${input.preparedSources.length} sources were given`);
+        }
+        const preparedSource = input.preparedSources[0];
+        const sourceConfig = input.sources[0];
+        // For an LFS source the Hub already published the sha256 as the linked ETag.
+        const publishedSha256 = isHuggingFaceProvider(sourceConfig)
+            ? typeof preparedSource.metadata?.sha256 === "string" ? preparedSource.metadata.sha256 : undefined
+            : undefined;
+        const states = [];
+        let chunkSize;
+        for (const { destination, index } of targets) {
+            const preparedDestination = input.preparedDestinations[index];
+            if ((destination.repo_type ?? "model") === "bucket") {
+                throw new Error(`${destination.repo_id} is a Hugging Face bucket, which Beam cannot write to. Buckets `
+                    + "expose no LFS batch endpoint; their only upload path is the Hub's Xet CAS client, "
+                    + "which cannot be expressed as presigned URLs for Beam's workers. Buckets do work as "
+                    + "a transfer source. Use `hf sync` to write to a bucket.");
+            }
+            const path = huggingFaceTargetPath(destination, preparedSource.filename);
+            const config = { ...destination, path };
+            let oid = publishedSha256;
+            if (!oid) {
+                if (!destination.allow_source_rehash) {
+                    throw new Error(`uploading to ${describeHuggingFace(config)} needs the source sha256, which the Hub requires `
+                        + "before it issues upload URLs. The SDK must read the source once to compute it; set "
+                        + "allow_source_rehash: true to opt in.");
+                }
+                const hashed = await hashSourceStream(this.fetchImpl, preparedSource.url);
+                oid = hashed.sha256;
+            }
+            const sample = await readSourceSample(this.fetchImpl, preparedSource.url);
+            const preupload = await huggingFacePreupload(this.fetchImpl, config, {
+                size: preparedSource.size,
+                sample
+            });
+            if (preupload.shouldIgnore) {
+                throw new Error(`${describeHuggingFace(config)} is excluded by the repo's .gitignore`);
+            }
+            if (preupload.uploadMode !== "lfs") {
+                throw new Error(`${describeHuggingFace(config)} would be committed as a regular git blob, not an LFS blob. `
+                    + "Beam uploads through the LFS protocol only; add the path to .gitattributes as LFS.");
+            }
+            const plan = await huggingFaceLfsBatch(this.fetchImpl, config, {
+                oid,
+                size: preparedSource.size
+            });
+            if (plan.upload?.chunkSize !== undefined) {
+                if (chunkSize !== undefined && chunkSize !== plan.upload.chunkSize) {
+                    throw new Error(`huggingface destinations disagree on part size (${chunkSize} vs ${plan.upload.chunkSize}); `
+                        + "the plan carries a single chunk size");
+                }
+                chunkSize = plan.upload.chunkSize;
+            }
+            states.push({
+                destination: config,
+                destinationId: preparedDestination.destination_id,
+                sourceId: preparedSource.source_id,
+                oid,
+                size: preparedSource.size,
+                chunkSize: plan.upload?.chunkSize,
+                partUrls: plan.upload?.partUrls ?? [],
+                uploadHref: plan.upload?.href,
+                verifyHref: plan.verifyHref,
+                // Runs alongside the transfer; the ETags are only needed at completion time.
+                partEtags: plan.upload?.chunkSize === undefined
+                    ? Promise.resolve([])
+                    : hashSourceStream(this.fetchImpl, preparedSource.url, {
+                        partSize: plan.upload.chunkSize,
+                        sha256: false
+                    }).then((hashed) => hashed.partEtags)
+            });
+        }
+        return { states, chunkSize };
+    }
+    /** Fail before any byte moves if BeamCore did not adopt the Hub's part layout. */
+    assertHuggingFacePlan(prepared, states) {
+        for (const state of states) {
+            const planDestination = prepared.plan_descriptor.destinations.find((candidate) => candidate.destination_id === state.destinationId);
+            const planSource = prepared.plan_descriptor.sources.find((candidate) => candidate.source_id === state.sourceId);
+            if (!planDestination || !planSource) {
+                throw new Error(`BeamCore plan is missing the huggingface coordinate ${state.sourceId}:${state.destinationId}`);
+            }
+            const finalObjectKey = planDestination.final_object_keys[state.sourceId];
+            if (finalObjectKey !== state.destination.path) {
+                throw new Error(`BeamCore planned ${finalObjectKey} but the Hub upload was negotiated for ${state.destination.path}`);
+            }
+            if (state.chunkSize === undefined) {
+                if (planSource.chunk_count !== 1) {
+                    throw new Error(`${describeHuggingFace(state.destination)} was issued a single-part upload, but the plan has `
+                        + `${planSource.chunk_count} chunks`);
+                }
+                continue;
+            }
+            if (prepared.plan_descriptor.chunk_size !== state.chunkSize) {
+                throw new Error(`the Hub requires ${state.chunkSize}-byte parts for ${describeHuggingFace(state.destination)}, `
+                    + `but BeamCore planned ${prepared.plan_descriptor.chunk_size}-byte chunks`);
+            }
+            if (planSource.chunk_count !== state.partUrls.length) {
+                throw new Error(`the Hub issued ${state.partUrls.length} part URLs for ${describeHuggingFace(state.destination)}, `
+                    + `but the plan has ${planSource.chunk_count} chunks`);
+            }
+        }
+    }
+    /**
+     * Close every Hugging Face upload for a transfer: complete the LFS multipart, verify it, and
+     * commit the blob so the file appears in the repo. Runs only after the transfer is complete,
+     * so it never sits in the transfer's progression path.
+     */
+    async finalizeHuggingFaceUploads(transferId) {
+        const states = this.huggingFaceUploads.get(transferId);
+        if (!states?.length)
+            return;
+        this.huggingFaceUploads.delete(transferId);
+        for (const state of states) {
+            if (state.uploadHref && state.chunkSize !== undefined) {
+                const etags = await state.partEtags;
+                if (etags.length !== state.partUrls.length) {
+                    throw new Error(`computed ${etags.length} part ETags for ${describeHuggingFace(state.destination)}, `
+                        + `expected ${state.partUrls.length}`);
+                }
+                await huggingFaceCompleteLfsUpload(this.fetchImpl, state.destination, {
+                    href: state.uploadHref,
+                    oid: state.oid,
+                    etags
+                });
+            }
+            if (state.verifyHref) {
+                await huggingFaceVerifyLfsUpload(this.fetchImpl, state.destination, {
+                    href: state.verifyHref,
+                    oid: state.oid,
+                    size: state.size
+                });
+            }
+            await huggingFaceCommit(this.fetchImpl, state.destination, {
+                oid: state.oid,
+                size: state.size
+            });
+        }
+    }
     async signProviderRoute(input) {
+        if (isHuggingFaceProvider(input.destination)) {
+            const state = input.huggingFaceUpload;
+            if (!state)
+                throw new Error(`huggingface upload state is missing for ${input.target.destination_id}`);
+            // The Hub presigns its own part targets; chunk N carries the URL for part N + 1.
+            const destUrl = state.chunkSize === undefined
+                ? state.uploadHref
+                : state.partUrls[input.chunk.source_chunk_index];
+            if (!destUrl) {
+                throw new Error(`the Hub issued no upload URL for chunk ${input.chunk.source_chunk_index} of `
+                    + describeHuggingFace(state.destination));
+            }
+            return signDestinationRoute({
+                chunk: input.chunk,
+                target: {
+                    ...input.target,
+                    object_key: input.finalObjectKey,
+                    metadata: directPutRouteMetadata(input.target.metadata)
+                },
+                source: input.source,
+                destination: input.destination,
+                expiresIn: input.expiresIn,
+                destUrl,
+                fetchImpl: this.fetchImpl
+            });
+        }
         if (isHippiusDestination(input.destination)) {
             return signDestinationRoute({
                 chunk: input.chunk,
@@ -611,7 +810,7 @@ export class BeamClient {
                 const destination = input.destinationsById.get(requested.destination_id);
                 if (!source || !destination)
                     throw new Error("route recovery provider configuration is unavailable");
-                const upload = isHippiusDestination(destination)
+                const upload = isDirectPutDestination(destination)
                     ? undefined
                     : await this.recoveryMultipartUploadState({
                         prepared: input.prepared,
@@ -643,6 +842,9 @@ export class BeamClient {
                     destination,
                     expiresIn: input.expiresIn,
                     ...(upload ? { upload } : {}),
+                    huggingFaceUpload: this.huggingFaceUploads
+                        .get(input.prepared.transfer_id)
+                        ?.find((state) => state.destinationId === requested.destination_id),
                     transferId: input.prepared.transfer_id,
                     finalObjectKey: requested.final_object_key,
                     signedUrlFlow: "signed_url",
@@ -845,6 +1047,8 @@ export class BeamClient {
             while (true) {
                 const status = await this.transferStatus(transferId);
                 if (status.status === "completed") {
+                    // The parts have landed; publish them as a Hub commit before reporting success.
+                    await this.finalizeHuggingFaceUploads(transferId);
                     return status;
                 }
                 if (status.status === "failed") {
@@ -1263,6 +1467,19 @@ function safeErrorCode(error) {
         .find((value) => Number.isInteger(value) && Number(value) >= 100 && Number(value) <= 599);
     return status === undefined ? code : `${code}:status=${status}`;
 }
+function multipartGroupIdentity(transferId, manifest) {
+    return {
+        transferId,
+        multipartGroupId: manifest.multipart_group_id,
+        sourceId: manifest.source_id,
+        destinationId: manifest.destination_id,
+        objectKey: manifest.final_object_key,
+        uploadId: manifest.upload_id,
+        expectedObjectSize: manifest.expected_object_size,
+        expectedPartCount: manifest.expected_part_count,
+        expiresAt: manifest.urls_expires_at
+    };
+}
 function createDeferred() {
     let resolve;
     let reject;
@@ -1277,7 +1494,7 @@ function createMultipartGroupWaiters(prepared, destinationsById) {
     for (const source of prepared.plan_descriptor.sources) {
         for (const destination of prepared.plan_descriptor.destinations) {
             const destinationConfig = destinationsById.get(destination.destination_id);
-            if (!destinationConfig || isHippiusDestination(destinationConfig))
+            if (!destinationConfig || isDirectPutDestination(destinationConfig))
                 continue;
             const finalObjectKey = destination.final_object_keys[source.source_id];
             if (!finalObjectKey)
@@ -1297,7 +1514,7 @@ async function createMultipartGroupManifest(input) {
             if (!destinationConfig) {
                 throw new Error(`BeamCore returned unknown destination_id: ${destination.destination_id}`);
             }
-            if (isHippiusDestination(destinationConfig)) {
+            if (isDirectPutDestination(destinationConfig)) {
                 return { ok: true, state: null };
             }
             const finalObjectKey = destination.final_object_keys[source.source_id];
@@ -1542,6 +1759,14 @@ const PART_ROUTE_METADATA_KEYS = new Set([
 function partRouteMetadata(metadata) {
     return Object.fromEntries(Object.entries(metadata ?? {}).filter(([key]) => PART_ROUTE_METADATA_KEYS.has(key)));
 }
+/**
+ * Route metadata for a destination BeamCore does not treat as an S3 multipart target. It
+ * rejects `part_number` there as an unexpected multipart signal, so drop it.
+ */
+function directPutRouteMetadata(metadata) {
+    const { part_number: _partNumber, ...rest } = partRouteMetadata(metadata);
+    return rest;
+}
 function randomUuid() {
     const cryptoApi = globalThis.crypto;
     if (cryptoApi?.randomUUID)
@@ -1603,6 +1828,23 @@ async function abortCreatedUploads(uploads, concurrency) {
 function isHippiusDestination(destination) {
     return destination.provider === "hippius" && "api_token" in destination;
 }
+/**
+ * Destinations that take a plain PUT per chunk instead of an S3 multipart upload. BeamCore
+ * rejects multipart route metadata for these, so they never get a multipart group manifest.
+ */
+function isDirectPutDestination(destination) {
+    return isHippiusDestination(destination) || isHuggingFaceProvider(destination);
+}
+/** Resolve the path a Hugging Face upload commits to, expanding a trailing-slash prefix. */
+function huggingFaceTargetPath(destination, sourceFilename) {
+    const path = destination.path.replace(/^\/+/, "");
+    if (!path.endsWith("/"))
+        return path;
+    if (!sourceFilename) {
+        throw new Error(`huggingface path ${destination.path} is a folder and the source has no filename`);
+    }
+    return `${path}${sourceFilename}`;
+}
 function sleep(ms) {
     return new Promise((resolve) => setTimeout(resolve, ms));
 }

package/dist/huggingface.d.ts

@@ -0,0 +1,101 @@
+import type { HuggingFaceProviderConfig, HuggingFaceRepoType } from "./models.js";
+export declare const HUGGINGFACE_DEFAULT_ENDPOINT = "https://huggingface.co";
+export declare const HUGGINGFACE_DEFAULT_REVISION = "main";
+export declare const HUGGINGFACE_DEFAULT_REPO_TYPE: HuggingFaceRepoType;
+export interface HuggingFaceFileMetadata {
+    /** Credential-free presigned CDN URL the workers read from. */
+    url: string;
+    size: number;
+    /** sha256 for an LFS blob, git sha1 otherwise. */
+    etag?: string;
+    commitHash?: string;
+}
+export interface HuggingFaceLfsUploadPlan {
+    oid: string;
+    size: number;
+    /** Absent when the Hub already stores this content and no upload is needed. */
+    upload?: {
+        /** Completion endpoint for multipart, or the single-part PUT target. */
+        href: string;
+        /** Part size the Hub requires. Absent for a single-part upload. */
+        chunkSize?: number;
+        /** Presigned part PUT URLs, ordered by part number. Empty for a single-part upload. */
+        partUrls: string[];
+    };
+    verifyHref?: string;
+}
+export declare function huggingFaceEndpoint(config: HuggingFaceProviderConfig): string;
+export declare function huggingFaceRepoType(config: HuggingFaceProviderConfig): HuggingFaceRepoType;
+export declare function huggingFaceRevision(config: HuggingFaceProviderConfig): string;
+/**
+ * `{endpoint}/{prefix}{repo_id}/resolve/{revision}/{path}`, as built by `hf_hub_url`.
+ *
+ * Buckets are unversioned and take no revision segment, and the Hub escapes their whole key
+ * as one component — see `HfApi.get_bucket_file_metadata`.
+ */
+export declare function huggingFaceResolveUrl(config: HuggingFaceProviderConfig): string;
+/** `{endpoint}/api/{repo_type}s/{repo_id}`. */
+export declare function huggingFaceApiBase(config: HuggingFaceProviderConfig): string;
+/** `{endpoint}/{prefix}{repo_id}.git/info/lfs/objects/batch`. */
+export declare function huggingFaceLfsBatchUrl(config: HuggingFaceProviderConfig): string;
+/**
+ * HEAD the resolve URL and require the Hub to redirect to its CDN. The redirect target is
+ * presigned and carries no credential, so it is the only form of this URL that may be handed
+ * to BeamCore and the workers.
+ */
+export declare function huggingFaceFileMetadata(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig): Promise<HuggingFaceFileMetadata>;
+/**
+ * Ask the Hub whether a path is stored as an LFS blob or as a regular git blob.
+ * `sample` is the base64 of the first 512 bytes, exactly as `_fetch_upload_modes` sends it.
+ */
+export declare function huggingFacePreupload(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig, input: {
+    size: number;
+    sample: string;
+}): Promise<{
+    uploadMode: "lfs" | "regular";
+    shouldIgnore: boolean;
+    oid?: string;
+}>;
+/**
+ * Request upload instructions for one object. The Hub answers with either a single-part PUT
+ * target or a completion endpoint plus one presigned PUT URL per part.
+ */
+export declare function huggingFaceLfsBatch(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig, input: {
+    oid: string;
+    size: number;
+}): Promise<HuggingFaceLfsUploadPlan>;
+/** Close a multipart LFS upload: `{oid, parts:[{partNumber, etag}]}` to the completion href. */
+export declare function huggingFaceCompleteLfsUpload(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig, input: {
+    href: string;
+    oid: string;
+    etags: string[];
+}): Promise<void>;
+/** Optional server-side check that the object landed intact. */
+export declare function huggingFaceVerifyLfsUpload(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig, input: {
+    href: string;
+    oid: string;
+    size: number;
+}): Promise<void>;
+/** Publish the uploaded blob as a commit. The body is NDJSON: a header line then one file line. */
+export declare function huggingFaceCommit(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig, input: {
+    oid: string;
+    size: number;
+}): Promise<{
+    commitOid?: string;
+    pullRequestUrl?: string;
+}>;
+/** Read the first `length` bytes of a URL and return them base64-encoded, for `preupload`. */
+export declare function readSourceSample(fetchImpl: typeof fetch, url: string, length?: number): Promise<string>;
+/**
+ * Stream a URL once and return the sha256 of the whole body plus, when `partSize` is given,
+ * the hex MD5 of every part. The Hub will not issue upload URLs without the sha256, and the
+ * per-part MD5 is the ETag the completion payload has to quote.
+ */
+export declare function hashSourceStream(fetchImpl: typeof fetch, url: string, options?: {
+    partSize?: number;
+    sha256?: boolean;
+}): Promise<{
+    sha256?: string;
+    partEtags: string[];
+}>;
+export declare function describe(config: HuggingFaceProviderConfig): string;

package/dist/huggingface.js

@@ -0,0 +1,298 @@
+export const HUGGINGFACE_DEFAULT_ENDPOINT = "https://huggingface.co";
+export const HUGGINGFACE_DEFAULT_REVISION = "main";
+export const HUGGINGFACE_DEFAULT_REPO_TYPE = "model";
+const LFS_CONTENT_TYPE = "application/vnd.git-lfs+json";
+/** Mirrors `constants.REPO_TYPES_URL_PREFIXES`; models carry no prefix. */
+const REPO_TYPE_URL_PREFIXES = {
+    model: "",
+    dataset: "datasets/",
+    space: "spaces/",
+    kernel: "kernels/",
+    bucket: "buckets/"
+};
+export function huggingFaceEndpoint(config) {
+    return (config.endpoint ?? HUGGINGFACE_DEFAULT_ENDPOINT).replace(/\/+$/, "");
+}
+export function huggingFaceRepoType(config) {
+    return config.repo_type ?? HUGGINGFACE_DEFAULT_REPO_TYPE;
+}
+export function huggingFaceRevision(config) {
+    return config.revision ?? HUGGINGFACE_DEFAULT_REVISION;
+}
+/**
+ * `{endpoint}/{prefix}{repo_id}/resolve/{revision}/{path}`, as built by `hf_hub_url`.
+ *
+ * Buckets are unversioned and take no revision segment, and the Hub escapes their whole key
+ * as one component — see `HfApi.get_bucket_file_metadata`.
+ */
+export function huggingFaceResolveUrl(config) {
+    const repoType = huggingFaceRepoType(config);
+    const prefix = REPO_TYPE_URL_PREFIXES[repoType];
+    if (repoType === "bucket") {
+        return `${huggingFaceEndpoint(config)}/${prefix}${config.repo_id}/resolve/${encodeURIComponent(config.path)}`;
+    }
+    const revision = encodeURIComponent(huggingFaceRevision(config));
+    return `${huggingFaceEndpoint(config)}/${prefix}${config.repo_id}/resolve/${revision}/${encodePath(config.path)}`;
+}
+/** `{endpoint}/api/{repo_type}s/{repo_id}`. */
+export function huggingFaceApiBase(config) {
+    return `${huggingFaceEndpoint(config)}/api/${huggingFaceRepoType(config)}s/${config.repo_id}`;
+}
+/** `{endpoint}/{prefix}{repo_id}.git/info/lfs/objects/batch`. */
+export function huggingFaceLfsBatchUrl(config) {
+    const prefix = REPO_TYPE_URL_PREFIXES[huggingFaceRepoType(config)];
+    return `${huggingFaceEndpoint(config)}/${prefix}${config.repo_id}.git/info/lfs/objects/batch`;
+}
+/**
+ * HEAD the resolve URL and require the Hub to redirect to its CDN. The redirect target is
+ * presigned and carries no credential, so it is the only form of this URL that may be handed
+ * to BeamCore and the workers.
+ */
+export async function huggingFaceFileMetadata(fetchImpl, config) {
+    const resolveUrl = huggingFaceResolveUrl(config);
+    const response = await fetchImpl(resolveUrl, {
+        method: "HEAD",
+        redirect: "manual",
+        headers: {
+            Authorization: `Bearer ${config.token}`,
+            // Compression would report a transformed length instead of the real object size.
+            "Accept-Encoding": "identity"
+        }
+    });
+    if (response.status >= 400) {
+        throw new Error(`Hugging Face file lookup failed status=${response.status} for ${describe(config)}`);
+    }
+    const location = response.headers.get("location");
+    if (!location || sameHost(resolveUrl, location)) {
+        throw new Error(`Hugging Face did not redirect ${describe(config)} to a presigned CDN URL. `
+            + "Beam reads this file over plain HTTP without forwarding your token, and the Hub only "
+            + "redirects for large-file (LFS or Xet) content. A small regular file is served inline "
+            + "from the Hub instead and cannot be transferred.");
+    }
+    const linkedSize = response.headers.get("x-linked-size");
+    const size = Number(linkedSize ?? response.headers.get("content-length") ?? Number.NaN);
+    if (!Number.isFinite(size) || size <= 0) {
+        throw new Error(`Hugging Face did not report a size for ${describe(config)}`);
+    }
+    return {
+        url: location,
+        size,
+        etag: normalizeEtag(response.headers.get("x-linked-etag") ?? response.headers.get("etag")),
+        commitHash: response.headers.get("x-repo-commit") ?? undefined
+    };
+}
+/**
+ * Ask the Hub whether a path is stored as an LFS blob or as a regular git blob.
+ * `sample` is the base64 of the first 512 bytes, exactly as `_fetch_upload_modes` sends it.
+ */
+export async function huggingFacePreupload(fetchImpl, config, input) {
+    const url = new URL(`${huggingFaceApiBase(config)}/preupload/${encodeURIComponent(huggingFaceRevision(config))}`);
+    if (config.create_pr)
+        url.searchParams.set("create_pr", "1");
+    const response = await fetchImpl(url, {
+        method: "POST",
+        headers: jsonHeaders(config),
+        body: JSON.stringify({ files: [{ path: config.path, sample: input.sample, size: input.size }] })
+    });
+    const payload = await readJson(response, "preupload", config);
+    const file = Array.isArray(payload.files) ? payload.files[0] : undefined;
+    if (!file || (file.uploadMode !== "lfs" && file.uploadMode !== "regular")) {
+        throw new Error(`Hugging Face preupload returned a malformed response for ${describe(config)}`);
+    }
+    return {
+        uploadMode: file.uploadMode,
+        shouldIgnore: Boolean(file.shouldIgnore),
+        oid: typeof file.oid === "string" ? file.oid : undefined
+    };
+}
+/**
+ * Request upload instructions for one object. The Hub answers with either a single-part PUT
+ * target or a completion endpoint plus one presigned PUT URL per part.
+ */
+export async function huggingFaceLfsBatch(fetchImpl, config, input) {
+    const response = await fetchImpl(huggingFaceLfsBatchUrl(config), {
+        method: "POST",
+        headers: {
+            Authorization: `Bearer ${config.token}`,
+            Accept: LFS_CONTENT_TYPE,
+            "Content-Type": LFS_CONTENT_TYPE
+        },
+        body: JSON.stringify({
+            operation: "upload",
+            transfers: ["basic", "multipart"],
+            hash_algo: "sha256",
+            ref: { name: huggingFaceRevision(config) },
+            objects: [{ oid: input.oid, size: input.size }]
+        })
+    });
+    const payload = await readJson(response, "LFS batch", config);
+    const object = Array.isArray(payload.objects) ? payload.objects[0] : undefined;
+    if (!object || typeof object.oid !== "string") {
+        throw new Error(`Hugging Face LFS batch returned a malformed response for ${describe(config)}`);
+    }
+    if (object.error) {
+        throw new Error(`Hugging Face LFS batch rejected ${describe(config)}: ${object.error.message ?? "unknown error"}`);
+    }
+    const uploadAction = object.actions?.upload;
+    if (!uploadAction) {
+        // No actions means the Hub already stores this content; only the commit is left to do.
+        return { oid: object.oid, size: input.size };
+    }
+    if (typeof uploadAction.href !== "string") {
+        throw new Error(`Hugging Face LFS batch returned no upload href for ${describe(config)}`);
+    }
+    const header = uploadAction.header ?? {};
+    const rawChunkSize = header.chunk_size;
+    const chunkSize = rawChunkSize === undefined ? undefined : Number(rawChunkSize);
+    if (chunkSize !== undefined && (!Number.isInteger(chunkSize) || chunkSize <= 0)) {
+        throw new Error(`Hugging Face LFS batch returned a malformed chunk_size '${rawChunkSize}' for ${describe(config)}`);
+    }
+    const partUrls = Object.entries(header)
+        .filter(([key]) => /^\d+$/.test(key))
+        .sort(([left], [right]) => Number(left) - Number(right))
+        .map(([, value]) => value);
+    if (chunkSize !== undefined && partUrls.length !== Math.ceil(input.size / chunkSize)) {
+        throw new Error(`Hugging Face returned ${partUrls.length} part URLs for ${describe(config)}, `
+            + `expected ${Math.ceil(input.size / chunkSize)} at chunk_size ${chunkSize}`);
+    }
+    return {
+        oid: object.oid,
+        size: input.size,
+        upload: { href: uploadAction.href, chunkSize, partUrls },
+        verifyHref: typeof object.actions?.verify?.href === "string" ? object.actions.verify.href : undefined
+    };
+}
+/** Close a multipart LFS upload: `{oid, parts:[{partNumber, etag}]}` to the completion href. */
+export async function huggingFaceCompleteLfsUpload(fetchImpl, config, input) {
+    const response = await fetchImpl(input.href, {
+        method: "POST",
+        headers: { Accept: LFS_CONTENT_TYPE, "Content-Type": LFS_CONTENT_TYPE },
+        body: JSON.stringify({
+            oid: input.oid,
+            parts: input.etags.map((etag, index) => ({ partNumber: index + 1, etag }))
+        })
+    });
+    await assertOk(response, "LFS completion", config);
+}
+/** Optional server-side check that the object landed intact. */
+export async function huggingFaceVerifyLfsUpload(fetchImpl, config, input) {
+    const response = await fetchImpl(input.href, {
+        method: "POST",
+        headers: jsonHeaders(config),
+        body: JSON.stringify({ oid: input.oid, size: input.size })
+    });
+    await assertOk(response, "LFS verify", config);
+}
+/** Publish the uploaded blob as a commit. The body is NDJSON: a header line then one file line. */
+export async function huggingFaceCommit(fetchImpl, config, input) {
+    const url = new URL(`${huggingFaceApiBase(config)}/commit/${encodeURIComponent(huggingFaceRevision(config))}`);
+    if (config.create_pr)
+        url.searchParams.set("create_pr", "1");
+    const lines = [
+        {
+            key: "header",
+            value: {
+                summary: config.commit_message ?? `Upload ${config.path} with Beam`,
+                description: config.commit_description ?? ""
+            }
+        },
+        {
+            key: "lfsFile",
+            value: { path: config.path, algo: "sha256", oid: input.oid, size: input.size }
+        }
+    ];
+    const response = await fetchImpl(url, {
+        method: "POST",
+        headers: { Authorization: `Bearer ${config.token}`, "Content-Type": "application/x-ndjson" },
+        body: lines.map((line) => JSON.stringify(line)).join("\n")
+    });
+    const payload = await readJson(response, "commit", config);
+    return {
+        commitOid: typeof payload.commitOid === "string" ? payload.commitOid : undefined,
+        pullRequestUrl: typeof payload.pullRequestUrl === "string" ? payload.pullRequestUrl : undefined
+    };
+}
+/** Read the first `length` bytes of a URL and return them base64-encoded, for `preupload`. */
+export async function readSourceSample(fetchImpl, url, length = 512) {
+    const response = await fetchImpl(url, { headers: { Range: `bytes=0-${length - 1}` } });
+    if (!response.ok) {
+        throw new Error(`source sample read failed status=${response.status}`);
+    }
+    return base64(new Uint8Array(await response.arrayBuffer()));
+}
+/**
+ * Stream a URL once and return the sha256 of the whole body plus, when `partSize` is given,
+ * the hex MD5 of every part. The Hub will not issue upload URLs without the sha256, and the
+ * per-part MD5 is the ETag the completion payload has to quote.
+ */
+export async function hashSourceStream(fetchImpl, url, options = {}) {
+    const { createHash } = await import("node:crypto");
+    const response = await fetchImpl(url);
+    if (!response.ok || !response.body) {
+        throw new Error(`source hash read failed status=${response.status}`);
+    }
+    const whole = options.sha256 === false ? null : createHash("sha256");
+    const partSize = options.partSize;
+    const partEtags = [];
+    let part = partSize ? createHash("md5") : null;
+    let partBytes = 0;
+    for await (const rawChunk of response.body) {
+        let chunk = rawChunk;
+        whole?.update(chunk);
+        while (partSize && part && chunk.length > 0) {
+            const room = partSize - partBytes;
+            if (chunk.length < room) {
+                part.update(chunk);
+                partBytes += chunk.length;
+                break;
+            }
+            part.update(chunk.subarray(0, room));
+            partEtags.push(part.digest("hex"));
+            part = createHash("md5");
+            partBytes = 0;
+            chunk = chunk.subarray(room);
+        }
+    }
+    if (partSize && part && partBytes > 0) {
+        partEtags.push(part.digest("hex"));
+    }
+    return { sha256: whole?.digest("hex"), partEtags };
+}
+function encodePath(path) {
+    return path.split("/").map(encodeURIComponent).join("/");
+}
+function sameHost(left, right) {
+    try {
+        return new URL(left).host === new URL(right, left).host;
+    }
+    catch {
+        return true;
+    }
+}
+function normalizeEtag(value) {
+    if (!value)
+        return undefined;
+    return value.replace(/^W\//, "").replace(/^"|"$/g, "");
+}
+function jsonHeaders(config) {
+    return { Authorization: `Bearer ${config.token}`, "Content-Type": "application/json" };
+}
+async function readJson(response, step, config) {
+    await assertOk(response, step, config);
+    return (await response.json());
+}
+async function assertOk(response, step, config) {
+    if (response.ok)
+        return;
+    const detail = (await response.text().catch(() => "")).slice(0, 512);
+    throw new Error(`Hugging Face ${step} failed status=${response.status} for ${describe(config)}${detail ? `: ${detail}` : ""}`);
+}
+function base64(bytes) {
+    let binary = "";
+    for (const byte of bytes)
+        binary += String.fromCharCode(byte);
+    return typeof btoa === "function" ? btoa(binary) : Buffer.from(bytes).toString("base64");
+}
+export function describe(config) {
+    return `${huggingFaceRepoType(config)} ${config.repo_id}@${huggingFaceRevision(config)}/${config.path}`;
+}

package/dist/models.d.ts

@@ -182,6 +182,8 @@ export interface ProviderTransferResumeInput {
     testMode?: boolean;
     expiresIn?: number;
     signedUrlFlow?: SignedUrlFlow;
+    /** Requested plan chunk size. BeamCore may raise it; the response carries the effective value. */
+    chunkSize?: number;
 }
 export interface TransferTerminalEvent {
     schema_version: "transfer-client-control/v6";
@@ -237,13 +239,41 @@ export interface HippiusProviderConfig {
     api_token: string;
     base_url?: string;
 }
-export type ProviderConfig = S3ProviderConfig | R2ProviderConfig | S3CompatibleProviderConfig | HippiusProviderConfig;
+export type HuggingFaceRepoType = "model" | "dataset" | "space" | "kernel" | "bucket";
+export interface HuggingFaceProviderConfig {
+    provider: "huggingface";
+    id?: string;
+    /** Namespace and repo or bucket name separated by a slash, for example `org/dataset`. */
+    repo_id: string;
+    /** Path of the file inside the repo. */
+    path: string;
+    /** Defaults to `model`. */
+    repo_type?: HuggingFaceRepoType;
+    /** Branch, tag or commit. Defaults to `main`. Buckets are unversioned and ignore it. */
+    revision?: string;
+    token: string;
+    /** Hub endpoint. Defaults to `https://huggingface.co`. */
+    endpoint?: string;
+    /** Destination only. Commit summary for the upload. */
+    commit_message?: string;
+    /** Destination only. Commit description for the upload. */
+    commit_description?: string;
+    /** Destination only. Open the upload as a pull request instead of committing to `revision`. */
+    create_pr?: boolean;
+    /**
+     * Destination only. The Hub issues LFS upload URLs only for a known sha256, so the SDK
+     * must read every source byte once to hash it. Opt in explicitly.
+     */
+    allow_source_rehash?: boolean;
+}
+export type ProviderConfig = S3ProviderConfig | R2ProviderConfig | S3CompatibleProviderConfig | HippiusProviderConfig | HuggingFaceProviderConfig;
 export type ProviderSourceConfig = ProviderConfig;
 export type ProviderDestinationConfig = ProviderConfig;
 export type S3ProviderConfigInput = Omit<S3ProviderConfig, "provider">;
 export type R2ProviderConfigInput = Omit<R2ProviderConfig, "provider">;
 export type S3CompatibleProviderConfigInput = Omit<S3CompatibleProviderConfig, "driver">;
 export type HippiusProviderConfigInput = Omit<HippiusProviderConfig, "provider">;
+export type HuggingFaceProviderConfigInput = Omit<HuggingFaceProviderConfig, "provider">;
 export declare const S3ProviderConfig: Readonly<{
     create(input: S3ProviderConfigInput): S3ProviderConfig;
 }>;
@@ -256,6 +286,9 @@ export declare const S3CompatibleProviderConfig: Readonly<{
 export declare const HippiusProviderConfig: Readonly<{
     create(input: HippiusProviderConfigInput): HippiusProviderConfig;
 }>;
+export declare const HuggingFaceProviderConfig: Readonly<{
+    create(input: HuggingFaceProviderConfigInput): HuggingFaceProviderConfig;
+}>;
 export interface ProviderTransferCreateInput {
     sources: ProviderSourceConfig[];
     destinations: ProviderDestinationConfig[];
@@ -271,12 +304,31 @@ export interface ProviderTransferCreateInput {
     distribute?: boolean;
     onBeforeTransferPrepare?: () => void | Promise<void>;
     onPrepared?: (prepared: TransferPrepareResponse) => void | Promise<void>;
+    /**
+     * Called after the SDK creates a multipart upload and before its signed
+     * routes are streamed. The payload contains durable identity only;
+     * credentials, headers, and signed URLs are never exposed.
+     */
+    onMultipartGroupReady?: (group: ProviderMultipartGroupIdentity) => void | Promise<void>;
     throwIfCancelled?: (transferId?: string) => void | Promise<void>;
     signedUrlFlow?: SignedUrlFlow;
     /** Stable caller identity used to derive the transfer and lifecycle request ids. */
     idempotencyKey?: string;
     /** Internal recovery generation; callers normally omit this. */
     routeGenerationId?: string;
+    /** Requested plan chunk size. BeamCore may raise it; the response carries the effective value. */
+    chunkSize?: number;
+}
+export interface ProviderMultipartGroupIdentity {
+    transferId: string;
+    multipartGroupId: string;
+    sourceId: string;
+    destinationId: string;
+    objectKey: string;
+    uploadId: string;
+    expectedObjectSize: number;
+    expectedPartCount: number;
+    expiresAt: string;
 }
 export interface PlanningHttpSource {
     source_id: string;

package/dist/models.js

@@ -38,6 +38,25 @@ export const HippiusProviderConfig = Object.freeze({
         return { provider: "hippius", ...input };
     }
 });
+const HUGGINGFACE_REPO_TYPES = [
+    "model",
+    "dataset",
+    "space",
+    "kernel",
+    "bucket"
+];
+export const HuggingFaceProviderConfig = Object.freeze({
+    create(input) {
+        requireFields("huggingface", input, ["repo_id", "path", "token"]);
+        if (input.repo_type !== undefined && !HUGGINGFACE_REPO_TYPES.includes(input.repo_type)) {
+            throw new Error(`huggingface config repo_type must be one of ${HUGGINGFACE_REPO_TYPES.join(", ")}.`);
+        }
+        if (!input.repo_id.includes("/")) {
+            throw new Error("huggingface config repo_id must be `namespace/name`.");
+        }
+        return { provider: "huggingface", ...input };
+    }
+});
 function requireFields(provider, input, fields) {
     for (const field of fields) {
         if (!hasText(input[field])) {

package/dist/nats-control.d.ts

@@ -1,8 +1,8 @@
 import { type ConnectionOptions } from "nats";
 import type { MultipartGroupManifest, SignedChunkRoute, TransferTerminalSignalWaiter } from "./models.js";
 export declare const TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION = "transfer-client-control/v6";
-export declare const BEAM_DEFAULT_NATS_URL = "tls://nats.b1m.ai:4222";
-export declare const BEAM_DEFAULT_NATS_WS_URL = "wss://nats.b1m.ai:443";
+export declare const BEAM_DEFAULT_NATS_URL = "tls://orch-gateway.b1m.ai:4222";
+export declare const BEAM_DEFAULT_NATS_WS_URL = "wss://orch-gateway.b1m.ai:443";
 export declare const BEAM_DEFAULT_MAX_PAYLOAD_BYTES: number;
 export declare const BEAM_ROUTE_TARGET_PAYLOAD_BYTES: number;
 export declare const ROUTE_RECOVERY_SIGN_MESSAGE_TYPE: "transfer.route_recovery.sign";

package/dist/nats-control.js

@@ -1,13 +1,14 @@
 import { encode, decode } from "@msgpack/msgpack";
 import { connect } from "nats";
 export const TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION = "transfer-client-control/v6";
-export const BEAM_DEFAULT_NATS_URL = "tls://nats.b1m.ai:4222";
-export const BEAM_DEFAULT_NATS_WS_URL = "wss://nats.b1m.ai:443";
+export const BEAM_DEFAULT_NATS_URL = "tls://orch-gateway.b1m.ai:4222";
+export const BEAM_DEFAULT_NATS_WS_URL = "wss://orch-gateway.b1m.ai:443";
 // NATS enforces max_payload per message. This guard splits signed-route control
 // messages before the broker rejects them; transfer bytes never flow through NATS.
 export const BEAM_DEFAULT_MAX_PAYLOAD_BYTES = 8 * 1024 * 1024;
 export const BEAM_ROUTE_TARGET_PAYLOAD_BYTES = 4 * 1024 * 1024;
 const ROUTE_BATCH_AUTH_TOKEN_ESTIMATE_BYTES = 64 * 1024;
+const AUTH_TOKEN_REFRESH_SAFETY_SECONDS = 30;
 export const ROUTE_RECOVERY_SIGN_MESSAGE_TYPE = "transfer.route_recovery.sign";
 export function buildConnectionOptions(options) {
     const connectionOptions = {
@@ -199,32 +200,33 @@ export class BeamTransferControl {
         void this.recoverLease(lease);
     }
     async request(messageType, payload, options = {}) {
-        const authToken = await this.resolveAuthToken();
         const shardId = options.shardId ?? (options.transferId ? transferShardId(options.transferId, this.shardCount) : 0);
         const requestId = options.idempotencyKey
             ? await stableRequestId(`${messageType}:${options.idempotencyKey}`)
             : randomId();
-        const envelope = {
-            message_id: `${TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION}:${this.environment}:${this.keyPrefix}:${messageType}:${requestId}`,
-            schema_version: TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION,
-            environment: this.environment,
-            key_prefix: this.keyPrefix,
-            shard_id: shardId,
-            message_type: messageType,
-            request_id: requestId,
-            auth_token: authToken,
-            occurred_at: new Date().toISOString(),
-            producer: "sdk",
-            payload
-        };
-        const bytes = encode(envelope);
-        if (bytes.byteLength > this.maxPayloadBytes) {
-            throw new Error(`NATS lifecycle request is ${bytes.byteLength} bytes, above maxPayloadBytes=${this.maxPayloadBytes}`);
-        }
+        const occurredAt = new Date().toISOString();
         const subject = this.requestSubject(messageType, shardId);
         let lastError;
         for (let attempt = 0; attempt < LIFECYCLE_REQUEST_MAX_ATTEMPTS; attempt += 1) {
             try {
+                const authToken = await this.resolveAuthToken();
+                const envelope = {
+                    message_id: `${TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION}:${this.environment}:${this.keyPrefix}:${messageType}:${requestId}`,
+                    schema_version: TRANSFER_CLIENT_CONTROL_SCHEMA_VERSION,
+                    environment: this.environment,
+                    key_prefix: this.keyPrefix,
+                    shard_id: shardId,
+                    message_type: messageType,
+                    request_id: requestId,
+                    auth_token: authToken,
+                    occurred_at: occurredAt,
+                    producer: "sdk",
+                    payload
+                };
+                const bytes = encode(envelope);
+                if (bytes.byteLength > this.maxPayloadBytes) {
+                    throw new Error(`NATS lifecycle request is ${bytes.byteLength} bytes, above maxPayloadBytes=${this.maxPayloadBytes}`);
+                }
                 const nc = await this.connection();
                 const response = await nc.request(subject, bytes, { timeout: this.requestTimeoutMs });
                 const decoded = decode(response.data);
@@ -233,6 +235,13 @@ export class BeamTransferControl {
                     return (decoded.payload ?? {});
                 const body = JSON.stringify(decoded.error ?? {});
                 const error = new LifecycleRequestError(decoded.status, `Beam lifecycle request failed with ${decoded.status}: ${body}`);
+                if (isExpiredAuthTokenReply(decoded) && attempt + 1 < LIFECYCLE_REQUEST_MAX_ATTEMPTS) {
+                    this.authToken = null;
+                    this.authTokenExpiresAt = 0;
+                    lastError = error;
+                    await sleepWithJitter(LIFECYCLE_REQUEST_RETRY_DELAYS_MS[attempt] ?? 1_000);
+                    continue;
+                }
                 if (!isRetryableLifecycleStatus(decoded.status) || attempt + 1 >= LIFECYCLE_REQUEST_MAX_ATTEMPTS) {
                     throw error;
                 }
@@ -507,7 +516,7 @@ export class BeamTransferControl {
     }
     async resolveAuthToken() {
         const now = Math.floor(Date.now() / 1000);
-        if (this.authToken && this.authTokenExpiresAt - 5 > now)
+        if (this.authToken && this.authTokenExpiresAt - AUTH_TOKEN_REFRESH_SAFETY_SECONDS > now)
             return this.authToken;
         if (this.authResolvePromise)
             return this.authResolvePromise;
@@ -707,6 +716,9 @@ const LIFECYCLE_REQUEST_RETRY_DELAYS_MS = [150, 500];
 function isRetryableLifecycleStatus(status) {
     return status === 408 || status === 425 || status === 429 || status >= 500;
 }
+function isExpiredAuthTokenReply(reply) {
+    return reply.status === 401 && reply.error?.code === "auth_token_expired";
+}
 export function isRetryableLifecycleError(error) {
     if (error instanceof LifecycleRequestError)
         return isRetryableLifecycleStatus(error.status);

package/dist/provider-signing.d.ts

@@ -1,4 +1,4 @@
-import type { ChunkDestinationSigningTarget, ChunkSigningPlanItem, PlanningHttpSource, PreparedDestination, PreparedHttpSource, ProviderDestinationConfig, ProviderSourceConfig, R2ProviderConfig, S3CompatibleProviderConfig, S3ProviderConfig, SignedChunkRoute } from "./models.js";
+import type { ChunkDestinationSigningTarget, ChunkSigningPlanItem, HuggingFaceProviderConfig, PlanningHttpSource, PreparedDestination, PreparedHttpSource, ProviderDestinationConfig, ProviderSourceConfig, R2ProviderConfig, S3CompatibleProviderConfig, S3ProviderConfig, SignedChunkRoute } from "./models.js";
 export type AnyS3CompatibleProviderConfig = S3ProviderConfig | R2ProviderConfig | S3CompatibleProviderConfig;
 export declare function nowIso(): string;
 export declare function expiresAtIso(expiresIn: number): string;
@@ -45,6 +45,8 @@ export declare function signDestinationRoute(input: {
     maxPartNumber?: number;
     finalObjectMetadata?: Record<string, string>;
     multipartGroupId?: string;
+    /** Pre-issued destination URL, used by providers that sign their own upload targets. */
+    destUrl?: string;
     fetchImpl?: typeof fetch;
 }): Promise<SignedChunkRoute>;
 export declare function signSourceReadRange(input: {
@@ -72,3 +74,4 @@ export declare function isS3CompatibleProvider(config: ProviderSourceConfig | Pr
 export declare function s3CompatibleEndpoint(source: AnyS3CompatibleProviderConfig): string | undefined;
 export declare function s3CompatibleRegion(source: AnyS3CompatibleProviderConfig): string;
 export declare function s3CompatibleForcePathStyle(source: AnyS3CompatibleProviderConfig, endpoint?: string | undefined): boolean | undefined;
+export declare function isHuggingFaceProvider(config: ProviderSourceConfig | ProviderDestinationConfig): config is HuggingFaceProviderConfig;

package/dist/provider-signing.js

@@ -1,5 +1,6 @@
 import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, GetObjectCommand, HeadObjectCommand, ListPartsCommand, PutObjectCommand, S3Client, UploadPartCommand } from "@aws-sdk/client-s3";
 import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
+import { huggingFaceEndpoint, huggingFaceFileMetadata, huggingFaceRepoType, huggingFaceRevision } from "./huggingface.js";
 export function nowIso() {
     return new Date().toISOString();
 }
@@ -47,6 +48,18 @@ export async function prepareProviderSource(source, options = {}) {
             metadata: { bucket: source.bucket, key: source.key, base_url: baseUrl }
         };
     }
+    if (isHuggingFaceProvider(source)) {
+        const metadata = await huggingFaceFileMetadata(options.fetchImpl ?? globalThis.fetch, source);
+        return {
+            source_id: source.id ?? `src_${index}`,
+            type: "http",
+            provider: "huggingface",
+            url: metadata.url,
+            size: metadata.size,
+            filename: filename(source.path),
+            metadata: huggingFaceMetadata(source, metadata.etag, metadata.commitHash)
+        };
+    }
     return unsupportedProviderConfig(source);
 }
 export async function prepareProviderSourceForPlan(source, options = {}) {
@@ -83,6 +96,17 @@ export async function prepareProviderSourceForPlan(source, options = {}) {
             metadata: { bucket: source.bucket, key: source.key, base_url: baseUrl }
         };
     }
+    if (isHuggingFaceProvider(source)) {
+        const metadata = await huggingFaceFileMetadata(options.fetchImpl ?? globalThis.fetch, source);
+        return {
+            source_id: source.id ?? `src_${index}`,
+            type: "http",
+            provider: "huggingface",
+            size: metadata.size,
+            filename: filename(source.path),
+            metadata: huggingFaceMetadata(source, metadata.etag, metadata.commitHash)
+        };
+    }
     return unsupportedProviderConfig(source);
 }
 export function prepareProviderDestination(destination, options = {}) {
@@ -111,6 +135,14 @@ export function prepareProviderDestination(destination, options = {}) {
             }
         };
     }
+    if (isHuggingFaceProvider(destination)) {
+        return {
+            destination_id: destinationId,
+            provider: "huggingface",
+            logical_prefix: destination.path,
+            metadata: huggingFaceMetadata(destination)
+        };
+    }
     return unsupportedProviderConfig(destination);
 }
 export async function createMultipartUpload(input) {
@@ -175,7 +207,7 @@ export async function signDestinationRoute(input) {
         throw new Error("destination signing target is missing object_key");
     }
     const [destUrl, sourceRoute] = await Promise.all([
-        signDestinationUrl({
+        input.destUrl ?? signDestinationUrl({
             destination: input.destination,
             objectKey: targetObjectKey,
             uploadId: input.uploadId,
@@ -245,6 +277,10 @@ export async function signSourceReadRange(input) {
             headers: { Range: range }
         };
     }
+    if (isHuggingFaceProvider(input.source)) {
+        const metadata = await huggingFaceFileMetadata(input.fetchImpl ?? globalThis.fetch, input.source);
+        return { url: metadata.url, headers: { Range: range } };
+    }
     return unsupportedProviderConfig(input.source);
 }
 export async function signDestinationReadRange(input) {
@@ -266,6 +302,15 @@ export async function signDestinationReadRange(input) {
             headers: { Range: range }
         };
     }
+    if (isHuggingFaceProvider(input.destination)) {
+        // Read-back resolves the committed file. Before the commit lands the bytes exist only as
+        // uncommitted LFS parts, which the Hub does not expose.
+        const metadata = await huggingFaceFileMetadata(input.fetchImpl ?? globalThis.fetch, {
+            ...input.destination,
+            path: input.objectKey
+        });
+        return { url: metadata.url, headers: { Range: range } };
+    }
     return unsupportedProviderConfig(input.destination);
 }
 async function signSourceRoute(input) {
@@ -290,6 +335,10 @@ async function signSourceRoute(input) {
             headers: { Range: range }
         };
     }
+    if (isHuggingFaceProvider(input.source)) {
+        const metadata = await huggingFaceFileMetadata(input.fetchImpl ?? globalThis.fetch, input.source);
+        return { url: metadata.url, headers: { Range: range } };
+    }
     return unsupportedProviderConfig(input.source);
 }
 async function signDestinationUrl(input) {
@@ -396,6 +445,22 @@ function s3CompatibleMetadata(source, endpoint = s3CompatibleEndpoint(source)) {
 function isHippiusProvider(config) {
     return config.provider === "hippius" && "api_token" in config;
 }
+export function isHuggingFaceProvider(config) {
+    return config.provider === "huggingface" && "repo_id" in config;
+}
+function huggingFaceMetadata(config, etag, commitHash) {
+    return compactMetadata({
+        driver: "huggingface",
+        repo_id: config.repo_id,
+        repo_type: huggingFaceRepoType(config),
+        revision: huggingFaceRevision(config),
+        path: config.path,
+        endpoint: huggingFaceEndpoint(config),
+        // For an LFS blob the Hub's linked ETag is the object's sha256.
+        sha256: etag,
+        commit_hash: commitHash
+    });
+}
 async function hippiusPresign(fetchImpl, baseUrl, token, bucket, key, action, expiresIn) {
     const url = new URL(`/api/objectstore/buckets/${bucket}/presigned-url/`, baseUrl);
     url.searchParams.set("key", key);

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@beam-network/sdk",
-    "version": "0.5.14",
+    "version": "0.5.16",
     "description": "TypeScript SDK for BEAM transfer creation and management.",
     "type": "module",
     "license": "MIT",