@chidchanun/bcp 0.1.25 → 0.1.27

package/docs/s3-storage.md

@@ -0,0 +1,433 @@
+# S3-Compatible Storage
+
+BCP Framework `0.1.26` introduced an S3-compatible streaming `StorageAdapter`. BCP `0.1.27` extends the same `createS3Storage()` API with object listing, native copy/move, user metadata, bulk deletion and presigned read/write URLs.
+
+> `0.1.27` behavior is an unreleased development target until RC validation and publication complete.
+
+The adapter is designed for AWS S3 and services that implement the S3 API, including common S3-compatible providers such as Cloudflare R2 and MinIO. Provider-specific behavior can differ, so production deployments should verify their chosen service with representative object operations.
+
+## Create an adapter
+
+```ts
+import {
+    createS3Storage,
+} from "bcp/server";
+
+export const storage =
+    createS3Storage({
+        bucket:
+            process.env.S3_BUCKET!,
+        region:
+            process.env.S3_REGION!,
+        endpoint:
+            process.env.S3_ENDPOINT,
+        accessKeyId:
+            process.env.S3_ACCESS_KEY_ID,
+        secretAccessKey:
+            process.env.S3_SECRET_ACCESS_KEY,
+    });
+```
+
+When `accessKeyId` / `secretAccessKey` are omitted, the underlying AWS SDK client can use its normal server-side credential provider chain.
+
+Do not expose S3 credentials through `BCP_PUBLIC_*` environment variables.
+
+## MinIO / path-style endpoints
+
+Some S3-compatible endpoints require path-style addressing:
+
+```ts
+const storage =
+    createS3Storage({
+        bucket:
+            "uploads",
+        region:
+            "us-east-1",
+        endpoint:
+            "http://127.0.0.1:9000",
+        forcePathStyle:
+            true,
+        accessKeyId:
+            process.env.MINIO_ACCESS_KEY,
+        secretAccessKey:
+            process.env.MINIO_SECRET_KEY,
+    });
+```
+
+## Prefix objects
+
+A deployment can reserve a logical provider prefix without changing application-facing keys:
+
+```ts
+const storage =
+    createS3Storage({
+        bucket:
+            "app-data",
+        region:
+            "auto",
+        prefix:
+            "production",
+    });
+
+await storage.put(
+    "avatars/user-101.webp",
+    bytes
+);
+```
+
+Application key:
+
+```text
+avatars/user-101.webp
+```
+
+Provider key:
+
+```text
+production/avatars/user-101.webp
+```
+
+All `0.1.27` list/copy/move/delete/signing operations preserve this logical/provider-key boundary.
+
+## Standard operations
+
+```ts
+await storage.put(
+    "documents/report.pdf",
+    bytes,
+    {
+        contentType:
+            "application/pdf",
+    }
+);
+
+const metadata =
+    await storage.stat(
+        "documents/report.pdf"
+    );
+
+const bytes =
+    await storage.read(
+        "documents/report.pdf"
+    );
+
+await storage.exists(
+    "documents/report.pdf"
+);
+
+await storage.delete(
+    "documents/report.pdf"
+);
+```
+
+Buffered `put()` uses a conditional S3 write when `overwrite` is not enabled. Existing objects therefore produce `StorageError` with code `OBJECT_EXISTS` instead of being silently replaced.
+
+## Streaming upload
+
+`createS3Storage()` supports `putStream()` and works with `putStorageStream()`:
+
+```ts
+import {
+    putStorageStream,
+} from "bcp/server";
+
+await putStorageStream(
+    storage,
+    "media/video.mp4",
+    request.body!,
+    {
+        contentType:
+            "video/mp4",
+        maxBytes:
+            500 * 1024 * 1024,
+    }
+);
+```
+
+BCP uses AWS SDK multipart upload for streams whose total size may not be known when upload begins. Multipart cleanup remains enabled on failure.
+
+## Multipart tuning
+
+```ts
+const storage =
+    createS3Storage({
+        bucket:
+            "uploads",
+        region:
+            "us-east-1",
+        multipart: {
+            partSize:
+                8 * 1024 * 1024,
+            queueSize:
+                2,
+        },
+    });
+```
+
+`partSize` must be at least 5 MiB.
+
+## Ranged reads
+
+```ts
+const bytes =
+    await storage.read(
+        "media/video.mp4",
+        {
+            start: 0,
+            end: 1023,
+        }
+    );
+```
+
+`readStream()` also supports ranges and is used automatically by `createStorageResponse()`.
+
+## Listing
+
+BCP `0.1.27` enables native S3 `ListObjectsV2` through the generic helper:
+
+```ts
+import {
+    listStorageObjects,
+} from "bcp/server";
+
+const page =
+    await listStorageObjects(
+        storage,
+        {
+            prefix:
+                "documents/",
+            limit:
+                100,
+        }
+    );
+```
+
+When S3 returns another page, `page.cursor` contains its continuation token. Treat this cursor as opaque and pass it back unchanged.
+
+S3 list responses do not include object content type, so listed `StorageObjectMetadata` uses `application/octet-stream` for `contentType`. Use `stat()` when exact object headers are required.
+
+## Native copy and move
+
+```ts
+import {
+    copyStorageObject,
+    moveStorageObject,
+} from "bcp/server";
+
+await copyStorageObject(
+    storage,
+    "incoming/report.pdf",
+    "archive/report.pdf"
+);
+
+await moveStorageObject(
+    storage,
+    "incoming/avatar.webp",
+    "users/42/avatar.webp"
+);
+```
+
+The S3 adapter uses native `CopyObject`. User metadata and BCP checksum metadata are preserved by normal copies.
+
+Move is copy followed by source deletion. It is not a distributed transaction; applications needing business-level atomicity should coordinate storage state with their database.
+
+## User metadata
+
+```ts
+await storage.put(
+    "documents/report.pdf",
+    bytes,
+    {
+        contentType:
+            "application/pdf",
+        metadata: {
+            owner:
+                "user-42",
+            category:
+                "report",
+        },
+    }
+);
+```
+
+Read or replace metadata:
+
+```ts
+import {
+    getStorageMetadata,
+    setStorageMetadata,
+} from "bcp/server";
+
+const metadata =
+    await getStorageMetadata(
+        storage,
+        "documents/report.pdf"
+    );
+
+await setStorageMetadata(
+    storage,
+    "documents/report.pdf",
+    {
+        owner:
+            "user-42",
+        state:
+            "approved",
+    }
+);
+```
+
+BCP maps application metadata to S3 user metadata and reserves `bcp_sha256` for framework checksum data.
+
+Replacing S3 metadata uses provider copy-to-self semantics with `MetadataDirective: REPLACE` while retaining BCP checksum metadata.
+
+## Bulk deletion
+
+```ts
+import {
+    deleteStorageObjects,
+} from "bcp/server";
+
+const result =
+    await deleteStorageObjects(
+        storage,
+        keys
+    );
+```
+
+BCP batches native S3 multi-delete requests at up to 1000 keys per request.
+
+S3 delete semantics are idempotent: deleting a missing key is normally reported as successful. Therefore S3 results commonly place such keys in `deleted` rather than `missing`.
+
+## Presigned read URLs
+
+```ts
+import {
+    createStorageSignedReadUrl,
+} from "bcp/server";
+
+const url =
+    await createStorageSignedReadUrl(
+        storage,
+        "videos/demo.mp4",
+        {
+            expiresIn:
+                300,
+        }
+    );
+```
+
+The default expiry is 15 minutes. The maximum accepted expiry is seven days, matching the SigV4 presigning boundary.
+
+## Presigned write URLs
+
+```ts
+import {
+    createStorageSignedWriteUrl,
+} from "bcp/server";
+
+const url =
+    await createStorageSignedWriteUrl(
+        storage,
+        "uploads/video.mp4",
+        {
+            expiresIn:
+                300,
+            contentType:
+                "video/mp4",
+            metadata: {
+                owner:
+                    "user-42",
+            },
+        }
+    );
+```
+
+The browser or another client can then upload directly to S3-compatible storage instead of proxying the file bytes through BCP.
+
+When content type or metadata are included in the signed operation, the direct uploader must send the headers required by the generated signature/provider.
+
+Never issue a signed URL before application authorization. Signed URLs should be treated as temporary credentials.
+
+## Capabilities
+
+Use the original capability helper for compatibility:
+
+```ts
+getStorageCapabilities(storage)
+```
+
+For the `0.1.27` S3 wrapper it reports:
+
+```text
+streamingRead: true
+streamingWrite: true
+ranges: true
+signedUrls: true
+listing: true
+```
+
+For the complete ecosystem surface use:
+
+```ts
+getStorageEcosystemCapabilities(storage)
+```
+
+which additionally reports:
+
+```text
+signedReadUrls: true
+signedWriteUrls: true
+copy: true
+move: true
+metadata: true
+bulkDelete: true
+```
+
+## Overwrite semantics
+
+Buffered `put()` retains the `0.1.26` conditional create-only behavior when `overwrite` is false.
+
+Streaming multipart writes still use a preflight existence check because portable S3 multipart APIs do not expose the same atomic create-only condition across every compatible provider.
+
+Copy/move destinations also reject existing objects by default unless `overwrite: true` is supplied.
+
+## Metadata and ETags
+
+Do not assume an S3 ETag is a plain MD5 checksum. Multipart uploads and provider-specific implementations can use different ETag formats.
+
+Buffered BCP writes persist a SHA-256 value in reserved S3 user metadata. Multipart streams may not expose a portable whole-object SHA-256 through the generic contract.
+
+## Client lifecycle
+
+Adapters that create their own AWS SDK client expose:
+
+```ts
+storage.destroy();
+```
+
+Call it during application shutdown when appropriate.
+
+When an existing `S3Client` is injected through `client`, BCP does not destroy that externally owned client.
+
+## Security
+
+Storage keys are not authorization. Authenticate and authorize before exposing object operations.
+
+Recommended production practices:
+
+- keep buckets private by default,
+- use scoped credentials / IAM policies,
+- do not expose access keys to browser bundles,
+- use short presigned URL expirations,
+- avoid logging full signed query strings,
+- validate uploaded content independently of MIME metadata,
+- configure encryption/retention policies when required,
+- keep application upload limits below proxy/infrastructure limits,
+- configure CORS only for the browser origins and methods that need direct object access.
+
+## Related documentation
+
+- [Storage and File Delivery](storage.md)
+- [Storage Ecosystem](storage-ecosystem.md)
+- [File Upload](file-upload.md)
+- [BCP Framework 0.1.27](releases/0.1.27.md)