@beam-network/sdk 0.5.15 → 0.5.17-dev.0

package/README.md

@@ -43,7 +43,7 @@ console.log(status.status);
 await beam.close();
 ```
 
-The main `createTransfer` API is provider-aware and strictly typed for S3, R2, S3-compatible, and Hippius configs.
+The main `createTransfer` API is provider-aware and strictly typed for S3, R2, S3-compatible, Hippius, and Hugging Face configs.
 Use `testMode: true` to create a BeamCore test-mode transfer.
 
 ## S3-Compatible Providers
@@ -92,14 +92,69 @@ Use `S3ProviderConfig` for ordinary AWS S3 if preferred, and keep using
 `S3CompatibleProviderConfig` when a provider needs a custom S3-compatible
 endpoint.
 
+## Hugging Face Hub
+
+Use `HuggingFaceProviderConfig` to pull a file out of a Hub repo or push one into
+one. The token stays in this process: a source resolves to the presigned CDN URL
+the Hub redirects to, and a destination uses the presigned part URLs the Hub's
+LFS batch endpoint issues.
+
+```ts
+import { BeamClient, HuggingFaceProviderConfig, S3ProviderConfig } from "@beam-network/sdk";
+
+const beam = new BeamClient({ apiKey: process.env.BEAM_API_KEY! });
+
+const transfer = await beam.prepareProviderTransfer({
+  sources: [
+    HuggingFaceProviderConfig.create({
+      repo_id: "org/dataset",
+      path: "data/train.parquet",
+      repo_type: "dataset",
+      token: process.env.HF_TOKEN!
+    })
+  ],
+  destinations: [
+    S3ProviderConfig.create({
+      bucket: "archive",
+      key: "datasets/train.parquet",
+      access_key_id: process.env.AWS_ACCESS_KEY_ID!,
+      secret_access_key: process.env.AWS_SECRET_ACCESS_KEY!
+    })
+  ],
+  name: "Hugging Face pull"
+});
+
+// waitForTransfer commits any Hugging Face destination once the parts land.
+await beam.waitForTransfer(transfer.transfer_id);
+```
+
+Constraints, all of them the Hub's rather than Beam's — see
+[`HUGGINGFACE_PROVIDER.md`](../../HUGGINGFACE_PROVIDER.md) for the full protocol:
+
+- **Sources are large-file (LFS or Xet) content.** Only those get the
+  credential-free CDN redirect; a small regular file is served inline and is
+  rejected with that reason. Hub **buckets** work as sources too, but cannot be
+  written to — they expose no LFS endpoint.
+- **A destination needs the object's sha256 up front**, because the LFS batch
+  endpoint will not issue upload URLs without it. When the source is not itself a
+  Hub file, set `allow_source_rehash: true` to let the SDK read the source once to
+  compute it.
+- **One source per Hugging Face destination**, because the Hub dictates the part
+  size and a plan carries a single chunk size.
+- If you do not call `waitForTransfer`, call `finalizeHuggingFaceUploads(transferId)`
+  yourself once the transfer completes — without it the parts are uploaded but no
+  commit is made and the file does not appear in the repo.
+
 For low-level raw transfer configs, use `createRawTransfer`; lifecycle transport still goes through NATS.
 
 ## Route Streaming And Payload Size
 
-Provider transfers use `transfer-client-control/v6`. Every reply carries Runtime and transport epochs; prepare and every route-stream message carry a UUID route generation. S3, R2, and S3-compatible destinations retain direct multipart UploadPart/ListParts/HEAD handling.
+Provider transfers use `transfer-client-control/v6`. Every reply carries Runtime and transport epochs; prepare and every route-stream message carry a UUID route generation. S3, R2, and S3-compatible destinations retain direct multipart UploadPart/ListParts/HEAD handling. Hippius and Hugging Face destinations take a plain PUT per chunk instead, so they carry no multipart group manifest.
 
 The client signs up to 64 routes concurrently by default and emits 2,048-route logical batches. Encoded MessagePack requests target 4 MiB physical batches under `maxPayloadBytes` (8 MiB by default; explicit positive overrides remain supported). The splitter reserves control-envelope headroom for the live auth token and stable request identity so the final encoded request remains below the configured guard. Lifecycle control refreshes cached SDK auth before the final 30 seconds of token lifetime and retries `auth_token_expired` replies with the same request identity and payload but a fresh token, preserving Core request-conflict protection during large route streams. A single route above the 4 MiB target but within the configured guard publishes alone; a single route larger than the configured limit fails before publication with both sizes in the error. Multipart create and abort requests use a separate `multipartControlConcurrency` limit of 2 by default, with explicit positive overrides supported, and S3-compatible control requests make up to five attempts for transient provider failures. This keeps URL signing throughput independent from provider control-plane pressure.
 
+Callers that need durable recovery accounting can provide `onMultipartGroupReady`. The SDK invokes it after creating a multipart upload and before streaming that group’s signed routes. Its payload contains transfer, source, destination, object-key, upload, size, part-count, and expiry identities only; it never includes credentials, request headers, or signed URLs. A rejected callback aborts the newly created upload and fails preparation closed.
+
 The stream ID derives from the transfer, selected flow, and immutable plan identity, and each ordered batch ID includes its route-coordinate checksum. Manual multi-destination attachment requires `delivery_index` on every route. Lifecycle mutations retry transient failures up to three times with the same request identity. A non-recoverable provider setup or route failure raises `BeamProviderTransferError`, whose `errors`, `transferCancelled`, and `multipartCleanupComplete` fields retain the original provider, cancellation, and cleanup outcomes. `waitForTransfer` subscribes to the API-key-owned, at-most-once terminal signal before its first status read and reconciles every signal through authoritative status; subscription failure degrades to the same jittered 15-to-30-second status fallback. Call `close()` when the client is no longer needed; it also closes outstanding terminal waiters.
 
 On completed provider transfers, Runtime may include an `integrity_audit_challenge` in status. Before releasing provider signing state, the TypeScript client signs exact read-only source and final-destination GET ranges and submits `transfer.integrity_audit_grants`; audit submission failures are best-effort and do not change transfer completion.
@@ -111,3 +166,15 @@ NATS requires this guard because the broker rejects messages above its configure
 ## Restart Recovery
 
 The client keeps one in-memory recovery lease per active transfer and one `runtime.hello` monitor per active shard. The lease is installed before route-stream begin, and a per-transfer lock coalesces initial streaming with replay. Multipart recovery retains the existing upload IDs and compact group state, then re-signs only the expiring route and commit controls. Each `resumeProviderTransfer` invocation re-prepares an existing transfer by explicit transfer id with a fresh request and route generation, while transport retries within that invocation reuse the same request. It then reattaches route-recovery signing from the current provider configs without creating duplicate multipart uploads. A Runtime epoch change invalidates cached auth, coalesces one `transfer.resume`, and regenerates routes under a fresh generation; a transport-only epoch change replays only when Runtime reports routes missing or expired. Provider credentials and signing inputs are retained only in memory and released on terminal status or `close()`. Low-level `attachSignedUrls` requires `routeGenerationId`, the plan fingerprint/checksum, and a `recoveryFactory`; signed URLs are never journaled to disk. Foreground cancellation, deadline expiry, and Runtime state-loss responses keep the background lease and retained multipart upload alive.
+
+## Hybrid endpoint signing
+
+Credential adapters can use `signDestinationUrl` independently of `signDestinationRoute`, using the same provider configuration and multipart signer. Destination-only signing requires no synthetic source descriptor and reads no source data. Optional `contentMd5` binds the worker-computed checksum to the provider upload without a separate source checksum scan.
+
+`signSourceReadRange` accepts `ifMatch` and `versionId` for frozen S3-compatible sources. Replay the returned headers unchanged. The ETag condition is signed and the version is included in the signed request. Unsupported non-S3 credential modes reject these conditions instead of dropping them. These helpers support the existing S3-compatible profiles used for R2, Hippius S3 and Hugging Face Storage Buckets; Hub repository tokens are a different provider mode.
+
+Only short-lived range/upload routes go to workers. Multipart creation, verification, completion and abort remain in the existing control path. A worker upload response alone is not final-object completion.
+
+`listMultipartParts`, `completeMultipartUpload`, and `inspectDestinationObject` reuse the same provider client and return metadata only. ListParts walks every page; completion submits the selected provider-verified parts. Adapters must durably retain the upload and verified manifest before completion so restart recovery cannot infer delivery from size alone.
+
+Manual releases from `dev` require a committed `-dev.N` version and publish only to npm tag `dev`; the stable tag is unchanged.

package/dist/client.d.ts

@@ -34,6 +34,7 @@ type TransferPrepareInput = {
     signedUrlFlow?: SignedUrlFlow;
     idempotencyKey?: string;
     routeGenerationId?: string;
+    chunkSize?: number;
 };
 export declare class BeamClient {
     readonly apiKey: string;
@@ -46,6 +47,7 @@ export declare class BeamClient {
     private readonly recoverySigners;
     private readonly integrityAuditSigners;
     private readonly integrityAuditSubmissions;
+    private readonly huggingFaceUploads;
     constructor(options?: BeamClientOptions);
     close(): Promise<void>;
     openTransferTerminalWaiter(transferId: string): Promise<TransferTerminalSignalWaiter>;
@@ -62,6 +64,7 @@ export declare class BeamClient {
         testMode?: boolean;
         urlsExpiresAt?: string;
         signedUrlFlow?: SignedUrlFlow;
+        chunkSize?: number;
     }): Promise<TransferPlanResponse>;
     prepareTransfer(input: TransferPrepareInput): Promise<TransferPrepareResponse>;
     private prepareTransferWithRequestKey;
@@ -81,6 +84,21 @@ export declare class BeamClient {
         autoDistribute?: boolean;
     }): Promise<AttachSignedUrlsResponse>;
     prepareProviderTransfer(input: ProviderTransferCreateInput): Promise<TransferPrepareResponse>;
+    /**
+     * Negotiate every Hugging Face destination before the plan exists.
+     *
+     * The Hub will not issue upload URLs without the object's sha256, and it chooses the part
+     * size itself, so this runs first and the plan is then requested at the Hub's chunk size.
+     */
+    private planHuggingFaceUploads;
+    /** Fail before any byte moves if BeamCore did not adopt the Hub's part layout. */
+    private assertHuggingFacePlan;
+    /**
+     * Close every Hugging Face upload for a transfer: complete the LFS multipart, verify it, and
+     * commit the blob so the file appears in the repo. Runs only after the transfer is complete,
+     * so it never sits in the transfer's progression path.
+     */
+    finalizeHuggingFaceUploads(transferId: string): Promise<void>;
     private signProviderRoute;
     private startProviderRouteRecoverySigner;
     private recoveryMultipartUploadState;

package/dist/client.js

@@ -1,4 +1,5 @@
-import { abortMultipartUpload, createMultipartUpload, expiresAtIso, prepareProviderDestination, prepareProviderSource, signAbortMultipartUpload, signCompleteMultipartUpload, signDestinationReadRange, signDestinationRoute, signFinalObjectHead, signListMultipartUpload, signSourceReadRange } from "./provider-signing.js";
+import { abortMultipartUpload, createMultipartUpload, expiresAtIso, prepareProviderDestination, prepareProviderSource, signAbortMultipartUpload, signCompleteMultipartUpload, signDestinationReadRange, signDestinationRoute, signFinalObjectHead, signListMultipartUpload, signSourceReadRange, isHuggingFaceProvider } from "./provider-signing.js";
+import { describe as describeHuggingFace, hashSourceStream, huggingFaceCommit, huggingFaceCompleteLfsUpload, huggingFaceLfsBatch, huggingFacePreupload, huggingFaceVerifyLfsUpload, readSourceSample } from "./huggingface.js";
 import { BeamTransferControl, BEAM_DEFAULT_NATS_URL, compactSignedRoutes, isRecoverableRouteStreamError } from "./nats-control.js";
 export class BeamRouteRecoveryPendingError extends Error {
     transferId;
@@ -55,6 +56,7 @@ export class BeamClient {
     recoverySigners = new Map();
     integrityAuditSigners = new Map();
     integrityAuditSubmissions = new Map();
+    huggingFaceUploads = new Map();
     constructor(options = {}) {
         if (!options.apiKey?.trim()) {
             throw new Error("apiKey is required.");
@@ -203,6 +205,7 @@ export class BeamClient {
             destinations: input.destinations,
             name: input.name,
             test_mode: input.testMode || undefined,
+            chunk_size: input.chunkSize,
             urls_expires_at: input.urlsExpiresAt,
             signed_url_flow: input.signedUrlFlow ?? "signed_url"
         }));
@@ -225,6 +228,7 @@ export class BeamClient {
             destinations: input.destinations,
             name: input.name,
             test_mode: input.testMode || undefined,
+            chunk_size: input.chunkSize,
             urls_expires_at: input.urlsExpiresAt,
             signed_url_flow: input.signedUrlFlow ?? "signed_url"
         }), {
@@ -318,6 +322,7 @@ export class BeamClient {
             destinations: input.destinations.map((destination) => ({ ...destination })),
             onBeforeTransferPrepare: undefined,
             onPrepared: undefined,
+            onMultipartGroupReady: undefined,
             throwIfCancelled: undefined
         };
         const preparedDestinations = retainedRecoveryInput.destinations.map((destination, index) => prepareProviderDestination(destination, { index }));
@@ -332,12 +337,20 @@ export class BeamClient {
         await input.throwIfCancelled?.();
         const preparedSources = await Promise.all(retainedRecoveryInput.sources.map((source, index) => prepareProviderSource(source, { index, expiresIn, fetchImpl: this.fetchImpl })));
         await input.throwIfCancelled?.();
+        const huggingFace = await this.planHuggingFaceUploads({
+            sources: retainedRecoveryInput.sources,
+            preparedSources,
+            destinations: retainedRecoveryInput.destinations,
+            preparedDestinations
+        });
+        await input.throwIfCancelled?.();
         await input.onBeforeTransferPrepare?.();
         const prepared = await this.prepareTransfer({
             sources: preparedSources,
             destinations: preparedDestinations,
             name: input.name,
             testMode: input.testMode,
+            chunkSize: huggingFace.chunkSize ?? input.chunkSize,
             signedUrlFlow: requestedSignedUrlFlow,
             idempotencyKey: input.idempotencyKey,
             routeGenerationId: input.routeGenerationId
@@ -345,6 +358,11 @@ export class BeamClient {
         if (!prepared.success) {
             return prepared;
         }
+        if (huggingFace.states.length) {
+            this.assertHuggingFacePlan(prepared, huggingFace.states);
+            this.huggingFaceUploads.set(prepared.transfer_id, huggingFace.states);
+        }
+        const huggingFaceByDestination = new Map(huggingFace.states.map((state) => [state.destinationId, state]));
         const sourcesById = new Map();
         preparedSources.forEach((preparedSource, index) => {
             const source = retainedRecoveryInput.sources[index];
@@ -409,6 +427,7 @@ export class BeamClient {
                     onGroupReady: async (state) => {
                         recoveryMultipartUploads.set(state.manifest.multipart_group_id, state);
                         validateMultipartGroupManifest([state.manifest], prepared.transfer_id);
+                        await input.onMultipartGroupReady?.(multipartGroupIdentity(prepared.transfer_id, state.manifest));
                         await routeStream.addManifestGroups([state.manifest]);
                         multipartGroupWaiters.get(state.manifest.multipart_group_id)?.resolve(state);
                     },
@@ -444,10 +463,10 @@ export class BeamClient {
                             if (!finalObjectKey)
                                 throw new Error("destination signing target is missing object_key");
                             const multipartGroupId = multipartGroupStateKey(prepared.transfer_id, target.destination_id, chunk.source_id, finalObjectKey);
-                            const upload = isHippiusDestination(destination)
+                            const upload = isDirectPutDestination(destination)
                                 ? undefined
                                 : await multipartGroupWaiters.get(multipartGroupId)?.promise;
-                            if (!isHippiusDestination(destination) && !upload)
+                            if (!isDirectPutDestination(destination) && !upload)
                                 throw new Error(`multipart group manifest is missing for ${chunk.source_id}:${target.destination_id}`);
                             return this.signProviderRoute({
                                 chunk,
@@ -456,6 +475,7 @@ export class BeamClient {
                                 destination,
                                 expiresIn,
                                 ...(upload ? { upload } : {}),
+                                huggingFaceUpload: huggingFaceByDestination.get(target.destination_id),
                                 transferId: prepared.transfer_id,
                                 finalObjectKey,
                                 signedUrlFlow: requestedSignedUrlFlow,
@@ -546,7 +566,186 @@ export class BeamClient {
         }
         return prepared;
     }
+    /**
+     * Negotiate every Hugging Face destination before the plan exists.
+     *
+     * The Hub will not issue upload URLs without the object's sha256, and it chooses the part
+     * size itself, so this runs first and the plan is then requested at the Hub's chunk size.
+     */
+    async planHuggingFaceUploads(input) {
+        const targets = input.destinations
+            .map((destination, index) => ({ destination, index }))
+            .filter((entry) => isHuggingFaceProvider(entry.destination));
+        if (!targets.length)
+            return { states: [] };
+        if (input.preparedSources.length !== 1) {
+            throw new Error("a huggingface destination requires exactly one source: the Hub dictates the part size "
+                + `and the plan carries a single chunk size, but ${input.preparedSources.length} sources were given`);
+        }
+        const preparedSource = input.preparedSources[0];
+        const sourceConfig = input.sources[0];
+        // For an LFS source the Hub already published the sha256 as the linked ETag.
+        const publishedSha256 = isHuggingFaceProvider(sourceConfig)
+            ? typeof preparedSource.metadata?.sha256 === "string" ? preparedSource.metadata.sha256 : undefined
+            : undefined;
+        const states = [];
+        let chunkSize;
+        for (const { destination, index } of targets) {
+            const preparedDestination = input.preparedDestinations[index];
+            if ((destination.repo_type ?? "model") === "bucket") {
+                throw new Error(`${destination.repo_id} is a Hugging Face bucket, which Beam cannot write to. Buckets `
+                    + "expose no LFS batch endpoint; their only upload path is the Hub's Xet CAS client, "
+                    + "which cannot be expressed as presigned URLs for Beam's workers. Buckets do work as "
+                    + "a transfer source. Use `hf sync` to write to a bucket.");
+            }
+            const path = huggingFaceTargetPath(destination, preparedSource.filename);
+            const config = { ...destination, path };
+            let oid = publishedSha256;
+            if (!oid) {
+                if (!destination.allow_source_rehash) {
+                    throw new Error(`uploading to ${describeHuggingFace(config)} needs the source sha256, which the Hub requires `
+                        + "before it issues upload URLs. The SDK must read the source once to compute it; set "
+                        + "allow_source_rehash: true to opt in.");
+                }
+                const hashed = await hashSourceStream(this.fetchImpl, preparedSource.url);
+                oid = hashed.sha256;
+            }
+            const sample = await readSourceSample(this.fetchImpl, preparedSource.url);
+            const preupload = await huggingFacePreupload(this.fetchImpl, config, {
+                size: preparedSource.size,
+                sample
+            });
+            if (preupload.shouldIgnore) {
+                throw new Error(`${describeHuggingFace(config)} is excluded by the repo's .gitignore`);
+            }
+            if (preupload.uploadMode !== "lfs") {
+                throw new Error(`${describeHuggingFace(config)} would be committed as a regular git blob, not an LFS blob. `
+                    + "Beam uploads through the LFS protocol only; add the path to .gitattributes as LFS.");
+            }
+            const plan = await huggingFaceLfsBatch(this.fetchImpl, config, {
+                oid,
+                size: preparedSource.size
+            });
+            if (plan.upload?.chunkSize !== undefined) {
+                if (chunkSize !== undefined && chunkSize !== plan.upload.chunkSize) {
+                    throw new Error(`huggingface destinations disagree on part size (${chunkSize} vs ${plan.upload.chunkSize}); `
+                        + "the plan carries a single chunk size");
+                }
+                chunkSize = plan.upload.chunkSize;
+            }
+            states.push({
+                destination: config,
+                destinationId: preparedDestination.destination_id,
+                sourceId: preparedSource.source_id,
+                oid,
+                size: preparedSource.size,
+                chunkSize: plan.upload?.chunkSize,
+                partUrls: plan.upload?.partUrls ?? [],
+                uploadHref: plan.upload?.href,
+                verifyHref: plan.verifyHref,
+                // Runs alongside the transfer; the ETags are only needed at completion time.
+                partEtags: plan.upload?.chunkSize === undefined
+                    ? Promise.resolve([])
+                    : hashSourceStream(this.fetchImpl, preparedSource.url, {
+                        partSize: plan.upload.chunkSize,
+                        sha256: false
+                    }).then((hashed) => hashed.partEtags)
+            });
+        }
+        return { states, chunkSize };
+    }
+    /** Fail before any byte moves if BeamCore did not adopt the Hub's part layout. */
+    assertHuggingFacePlan(prepared, states) {
+        for (const state of states) {
+            const planDestination = prepared.plan_descriptor.destinations.find((candidate) => candidate.destination_id === state.destinationId);
+            const planSource = prepared.plan_descriptor.sources.find((candidate) => candidate.source_id === state.sourceId);
+            if (!planDestination || !planSource) {
+                throw new Error(`BeamCore plan is missing the huggingface coordinate ${state.sourceId}:${state.destinationId}`);
+            }
+            const finalObjectKey = planDestination.final_object_keys[state.sourceId];
+            if (finalObjectKey !== state.destination.path) {
+                throw new Error(`BeamCore planned ${finalObjectKey} but the Hub upload was negotiated for ${state.destination.path}`);
+            }
+            if (state.chunkSize === undefined) {
+                if (planSource.chunk_count !== 1) {
+                    throw new Error(`${describeHuggingFace(state.destination)} was issued a single-part upload, but the plan has `
+                        + `${planSource.chunk_count} chunks`);
+                }
+                continue;
+            }
+            if (prepared.plan_descriptor.chunk_size !== state.chunkSize) {
+                throw new Error(`the Hub requires ${state.chunkSize}-byte parts for ${describeHuggingFace(state.destination)}, `
+                    + `but BeamCore planned ${prepared.plan_descriptor.chunk_size}-byte chunks`);
+            }
+            if (planSource.chunk_count !== state.partUrls.length) {
+                throw new Error(`the Hub issued ${state.partUrls.length} part URLs for ${describeHuggingFace(state.destination)}, `
+                    + `but the plan has ${planSource.chunk_count} chunks`);
+            }
+        }
+    }
+    /**
+     * Close every Hugging Face upload for a transfer: complete the LFS multipart, verify it, and
+     * commit the blob so the file appears in the repo. Runs only after the transfer is complete,
+     * so it never sits in the transfer's progression path.
+     */
+    async finalizeHuggingFaceUploads(transferId) {
+        const states = this.huggingFaceUploads.get(transferId);
+        if (!states?.length)
+            return;
+        this.huggingFaceUploads.delete(transferId);
+        for (const state of states) {
+            if (state.uploadHref && state.chunkSize !== undefined) {
+                const etags = await state.partEtags;
+                if (etags.length !== state.partUrls.length) {
+                    throw new Error(`computed ${etags.length} part ETags for ${describeHuggingFace(state.destination)}, `
+                        + `expected ${state.partUrls.length}`);
+                }
+                await huggingFaceCompleteLfsUpload(this.fetchImpl, state.destination, {
+                    href: state.uploadHref,
+                    oid: state.oid,
+                    etags
+                });
+            }
+            if (state.verifyHref) {
+                await huggingFaceVerifyLfsUpload(this.fetchImpl, state.destination, {
+                    href: state.verifyHref,
+                    oid: state.oid,
+                    size: state.size
+                });
+            }
+            await huggingFaceCommit(this.fetchImpl, state.destination, {
+                oid: state.oid,
+                size: state.size
+            });
+        }
+    }
     async signProviderRoute(input) {
+        if (isHuggingFaceProvider(input.destination)) {
+            const state = input.huggingFaceUpload;
+            if (!state)
+                throw new Error(`huggingface upload state is missing for ${input.target.destination_id}`);
+            // The Hub presigns its own part targets; chunk N carries the URL for part N + 1.
+            const destUrl = state.chunkSize === undefined
+                ? state.uploadHref
+                : state.partUrls[input.chunk.source_chunk_index];
+            if (!destUrl) {
+                throw new Error(`the Hub issued no upload URL for chunk ${input.chunk.source_chunk_index} of `
+                    + describeHuggingFace(state.destination));
+            }
+            return signDestinationRoute({
+                chunk: input.chunk,
+                target: {
+                    ...input.target,
+                    object_key: input.finalObjectKey,
+                    metadata: directPutRouteMetadata(input.target.metadata)
+                },
+                source: input.source,
+                destination: input.destination,
+                expiresIn: input.expiresIn,
+                destUrl,
+                fetchImpl: this.fetchImpl
+            });
+        }
         if (isHippiusDestination(input.destination)) {
             return signDestinationRoute({
                 chunk: input.chunk,
@@ -611,7 +810,7 @@ export class BeamClient {
                 const destination = input.destinationsById.get(requested.destination_id);
                 if (!source || !destination)
                     throw new Error("route recovery provider configuration is unavailable");
-                const upload = isHippiusDestination(destination)
+                const upload = isDirectPutDestination(destination)
                     ? undefined
                     : await this.recoveryMultipartUploadState({
                         prepared: input.prepared,
@@ -643,6 +842,9 @@ export class BeamClient {
                     destination,
                     expiresIn: input.expiresIn,
                     ...(upload ? { upload } : {}),
+                    huggingFaceUpload: this.huggingFaceUploads
+                        .get(input.prepared.transfer_id)
+                        ?.find((state) => state.destinationId === requested.destination_id),
                     transferId: input.prepared.transfer_id,
                     finalObjectKey: requested.final_object_key,
                     signedUrlFlow: "signed_url",
@@ -845,6 +1047,8 @@ export class BeamClient {
             while (true) {
                 const status = await this.transferStatus(transferId);
                 if (status.status === "completed") {
+                    // The parts have landed; publish them as a Hub commit before reporting success.
+                    await this.finalizeHuggingFaceUploads(transferId);
                     return status;
                 }
                 if (status.status === "failed") {
@@ -1263,6 +1467,19 @@ function safeErrorCode(error) {
         .find((value) => Number.isInteger(value) && Number(value) >= 100 && Number(value) <= 599);
     return status === undefined ? code : `${code}:status=${status}`;
 }
+function multipartGroupIdentity(transferId, manifest) {
+    return {
+        transferId,
+        multipartGroupId: manifest.multipart_group_id,
+        sourceId: manifest.source_id,
+        destinationId: manifest.destination_id,
+        objectKey: manifest.final_object_key,
+        uploadId: manifest.upload_id,
+        expectedObjectSize: manifest.expected_object_size,
+        expectedPartCount: manifest.expected_part_count,
+        expiresAt: manifest.urls_expires_at
+    };
+}
 function createDeferred() {
     let resolve;
     let reject;
@@ -1277,7 +1494,7 @@ function createMultipartGroupWaiters(prepared, destinationsById) {
     for (const source of prepared.plan_descriptor.sources) {
         for (const destination of prepared.plan_descriptor.destinations) {
             const destinationConfig = destinationsById.get(destination.destination_id);
-            if (!destinationConfig || isHippiusDestination(destinationConfig))
+            if (!destinationConfig || isDirectPutDestination(destinationConfig))
                 continue;
             const finalObjectKey = destination.final_object_keys[source.source_id];
             if (!finalObjectKey)
@@ -1297,7 +1514,7 @@ async function createMultipartGroupManifest(input) {
             if (!destinationConfig) {
                 throw new Error(`BeamCore returned unknown destination_id: ${destination.destination_id}`);
             }
-            if (isHippiusDestination(destinationConfig)) {
+            if (isDirectPutDestination(destinationConfig)) {
                 return { ok: true, state: null };
             }
             const finalObjectKey = destination.final_object_keys[source.source_id];
@@ -1542,6 +1759,14 @@ const PART_ROUTE_METADATA_KEYS = new Set([
 function partRouteMetadata(metadata) {
     return Object.fromEntries(Object.entries(metadata ?? {}).filter(([key]) => PART_ROUTE_METADATA_KEYS.has(key)));
 }
+/**
+ * Route metadata for a destination BeamCore does not treat as an S3 multipart target. It
+ * rejects `part_number` there as an unexpected multipart signal, so drop it.
+ */
+function directPutRouteMetadata(metadata) {
+    const { part_number: _partNumber, ...rest } = partRouteMetadata(metadata);
+    return rest;
+}
 function randomUuid() {
     const cryptoApi = globalThis.crypto;
     if (cryptoApi?.randomUUID)
@@ -1603,6 +1828,23 @@ async function abortCreatedUploads(uploads, concurrency) {
 function isHippiusDestination(destination) {
     return destination.provider === "hippius" && "api_token" in destination;
 }
+/**
+ * Destinations that take a plain PUT per chunk instead of an S3 multipart upload. BeamCore
+ * rejects multipart route metadata for these, so they never get a multipart group manifest.
+ */
+function isDirectPutDestination(destination) {
+    return isHippiusDestination(destination) || isHuggingFaceProvider(destination);
+}
+/** Resolve the path a Hugging Face upload commits to, expanding a trailing-slash prefix. */
+function huggingFaceTargetPath(destination, sourceFilename) {
+    const path = destination.path.replace(/^\/+/, "");
+    if (!path.endsWith("/"))
+        return path;
+    if (!sourceFilename) {
+        throw new Error(`huggingface path ${destination.path} is a folder and the source has no filename`);
+    }
+    return `${path}${sourceFilename}`;
+}
 function sleep(ms) {
     return new Promise((resolve) => setTimeout(resolve, ms));
 }

package/dist/huggingface.d.ts

@@ -0,0 +1,101 @@
+import type { HuggingFaceProviderConfig, HuggingFaceRepoType } from "./models.js";
+export declare const HUGGINGFACE_DEFAULT_ENDPOINT = "https://huggingface.co";
+export declare const HUGGINGFACE_DEFAULT_REVISION = "main";
+export declare const HUGGINGFACE_DEFAULT_REPO_TYPE: HuggingFaceRepoType;
+export interface HuggingFaceFileMetadata {
+    /** Credential-free presigned CDN URL the workers read from. */
+    url: string;
+    size: number;
+    /** sha256 for an LFS blob, git sha1 otherwise. */
+    etag?: string;
+    commitHash?: string;
+}
+export interface HuggingFaceLfsUploadPlan {
+    oid: string;
+    size: number;
+    /** Absent when the Hub already stores this content and no upload is needed. */
+    upload?: {
+        /** Completion endpoint for multipart, or the single-part PUT target. */
+        href: string;
+        /** Part size the Hub requires. Absent for a single-part upload. */
+        chunkSize?: number;
+        /** Presigned part PUT URLs, ordered by part number. Empty for a single-part upload. */
+        partUrls: string[];
+    };
+    verifyHref?: string;
+}
+export declare function huggingFaceEndpoint(config: HuggingFaceProviderConfig): string;
+export declare function huggingFaceRepoType(config: HuggingFaceProviderConfig): HuggingFaceRepoType;
+export declare function huggingFaceRevision(config: HuggingFaceProviderConfig): string;
+/**
+ * `{endpoint}/{prefix}{repo_id}/resolve/{revision}/{path}`, as built by `hf_hub_url`.
+ *
+ * Buckets are unversioned and take no revision segment, and the Hub escapes their whole key
+ * as one component — see `HfApi.get_bucket_file_metadata`.
+ */
+export declare function huggingFaceResolveUrl(config: HuggingFaceProviderConfig): string;
+/** `{endpoint}/api/{repo_type}s/{repo_id}`. */
+export declare function huggingFaceApiBase(config: HuggingFaceProviderConfig): string;
+/** `{endpoint}/{prefix}{repo_id}.git/info/lfs/objects/batch`. */
+export declare function huggingFaceLfsBatchUrl(config: HuggingFaceProviderConfig): string;
+/**
+ * HEAD the resolve URL and require the Hub to redirect to its CDN. The redirect target is
+ * presigned and carries no credential, so it is the only form of this URL that may be handed
+ * to BeamCore and the workers.
+ */
+export declare function huggingFaceFileMetadata(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig): Promise<HuggingFaceFileMetadata>;
+/**
+ * Ask the Hub whether a path is stored as an LFS blob or as a regular git blob.
+ * `sample` is the base64 of the first 512 bytes, exactly as `_fetch_upload_modes` sends it.
+ */
+export declare function huggingFacePreupload(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig, input: {
+    size: number;
+    sample: string;
+}): Promise<{
+    uploadMode: "lfs" | "regular";
+    shouldIgnore: boolean;
+    oid?: string;
+}>;
+/**
+ * Request upload instructions for one object. The Hub answers with either a single-part PUT
+ * target or a completion endpoint plus one presigned PUT URL per part.
+ */
+export declare function huggingFaceLfsBatch(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig, input: {
+    oid: string;
+    size: number;
+}): Promise<HuggingFaceLfsUploadPlan>;
+/** Close a multipart LFS upload: `{oid, parts:[{partNumber, etag}]}` to the completion href. */
+export declare function huggingFaceCompleteLfsUpload(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig, input: {
+    href: string;
+    oid: string;
+    etags: string[];
+}): Promise<void>;
+/** Optional server-side check that the object landed intact. */
+export declare function huggingFaceVerifyLfsUpload(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig, input: {
+    href: string;
+    oid: string;
+    size: number;
+}): Promise<void>;
+/** Publish the uploaded blob as a commit. The body is NDJSON: a header line then one file line. */
+export declare function huggingFaceCommit(fetchImpl: typeof fetch, config: HuggingFaceProviderConfig, input: {
+    oid: string;
+    size: number;
+}): Promise<{
+    commitOid?: string;
+    pullRequestUrl?: string;
+}>;
+/** Read the first `length` bytes of a URL and return them base64-encoded, for `preupload`. */
+export declare function readSourceSample(fetchImpl: typeof fetch, url: string, length?: number): Promise<string>;
+/**
+ * Stream a URL once and return the sha256 of the whole body plus, when `partSize` is given,
+ * the hex MD5 of every part. The Hub will not issue upload URLs without the sha256, and the
+ * per-part MD5 is the ETag the completion payload has to quote.
+ */
+export declare function hashSourceStream(fetchImpl: typeof fetch, url: string, options?: {
+    partSize?: number;
+    sha256?: boolean;
+}): Promise<{
+    sha256?: string;
+    partEtags: string[];
+}>;
+export declare function describe(config: HuggingFaceProviderConfig): string;