@xenterprises/fastify-xstorage 1.0.0

package/README.md

@@ -0,0 +1,492 @@
+# xStorage
+
+> Fastify v5 plugin providing a simple, intuitive library of methods for S3-compatible storage.
+
+Manage file uploads, downloads, and storage operations with Digital Ocean Spaces, AWS S3, Cloudflare R2, and any S3-compatible storage service. **For image processing, use [@xenterprises/fastify-ximagepipeline](https://github.com/x-enterprises/fastify-plugins/tree/main/fastify-ximagepipeline).**
+
+## Requirements
+
+- **Fastify v5.0.0+**
+- **Node.js v20+**
+
+## Features
+
+- 📦 **S3-Compatible Storage** - Works with Digital Ocean Spaces, AWS S3, Cloudflare R2
+- 🔒 **Secure by Default** - Private ACL with signed URLs for access control
+- 🔗 **Signed URLs** - Generate temporary access URLs for private files
+- 📊 **File Operations** - Upload, download, delete, list, copy, and metadata retrieval
+- ⚡ **Concurrent Operations** - Batch uploads and deletes for efficiency
+- 🎯 **Simple API** - Intuitive methods decorated on Fastify instance
+
+## Installation
+
+```bash
+npm install @xenterprises/fastify-xstorage @aws-sdk/client-s3 @aws-sdk/s3-request-presigner fastify@5
+```
+
+For file uploads via HTTP, also install:
+
+```bash
+npm install @fastify/multipart@9
+```
+
+## Quick Start
+
+```javascript
+import Fastify from "fastify";
+import xStorage from "@xenterprises/fastify-xstorage";
+
+const fastify = Fastify({ logger: true });
+
+// Register xStorage
+await fastify.register(xStorage, {
+  endpoint: "https://nyc3.digitaloceanspaces.com", // Digital Ocean Spaces
+  region: "us-east-1",
+  accessKeyId: process.env.STORAGE_ACCESS_KEY_ID,
+  secretAccessKey: process.env.STORAGE_SECRET_ACCESS_KEY,
+  bucket: "your-bucket-name",
+  publicUrl: "https://your-bucket-name.nyc3.digitaloceanspaces.com",
+  // Default ACL is "private" - use signed URLs for secure access
+});
+
+// Upload a file programmatically
+const buffer = await fs.readFile("path/to/file.pdf");
+const result = await fastify.xStorage.upload(buffer, "file.pdf", {
+  folder: "documents",
+});
+
+console.log(result);
+// {
+//   key: "documents/file-a1b2c3d4.pdf",
+//   url: "https://your-bucket-name.nyc3.digitaloceanspaces.com/documents/file-a1b2c3d4.pdf",
+//   size: 123456,
+//   contentType: "application/pdf"
+// }
+
+// Generate a signed URL for temporary private file access
+const signedUrl = await fastify.xStorage.getSignedUrl(
+  result.key,
+  3600 // Expires in 1 hour
+);
+
+// Use in your application
+await fastify.prisma.document.create({
+  data: {
+    filename: "file.pdf",
+    storageKey: result.key,
+    size: result.size,
+  },
+});
+
+await fastify.listen({ port: 3000 });
+```
+
+## Configuration
+
+### Digital Ocean Spaces
+
+```javascript
+await fastify.register(xStorage, {
+  endpoint: "https://nyc3.digitaloceanspaces.com",
+  region: "nyc3",
+  accessKeyId: process.env.DO_SPACES_KEY,
+  secretAccessKey: process.env.DO_SPACES_SECRET,
+  bucket: "your-bucket",
+  publicUrl: "https://your-bucket.nyc3.digitaloceanspaces.com",
+  forcePathStyle: true,
+  // acl: "private", // Default - use signed URLs for access
+});
+```
+
+### AWS S3
+
+```javascript
+await fastify.register(xStorage, {
+  region: "us-east-1",
+  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
+  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
+  bucket: "your-bucket",
+  publicUrl: "https://your-bucket.s3.us-east-1.amazonaws.com",
+  forcePathStyle: false,
+  // acl: "private", // Default - use signed URLs for access
+});
+```
+
+### Cloudflare R2
+
+```javascript
+await fastify.register(xStorage, {
+  endpoint: "https://your-account-id.r2.cloudflarestorage.com",
+  region: "auto",
+  accessKeyId: process.env.R2_ACCESS_KEY_ID,
+  secretAccessKey: process.env.R2_SECRET_ACCESS_KEY,
+  bucket: "your-bucket",
+  publicUrl: "https://your-custom-domain.com",
+  forcePathStyle: true,
+  // acl: "private", // Default - use signed URLs for access
+});
+```
+
+## Core Storage API
+
+All methods are available on the `fastify.xStorage` namespace.
+
+### `fastify.xStorage.upload(file, filename, options)`
+
+Upload a file to storage.
+
+```javascript
+const result = await fastify.xStorage.upload(buffer, "document.pdf", {
+  folder: "documents",
+  useRandomName: true,
+});
+
+console.log(result);
+// {
+//   key: "documents/document-a1b2c3d4.pdf",
+//   url: "https://your-bucket.nyc3.digitaloceanspaces.com/documents/document-a1b2c3d4.pdf",
+//   size: 245678,
+//   contentType: "application/pdf"
+// }
+
+// Store in database
+await db.documents.create({
+  data: {
+    filename: "document.pdf",
+    storageKey: result.key,
+    size: result.size,
+  },
+});
+```
+
+**Options:**
+- `folder` - Folder path (e.g., "documents", "docs/2024")
+- `key` - Custom storage key (overrides folder/filename)
+- `contentType` - MIME type (auto-detected if not provided)
+- `metadata` - Custom metadata object
+- `useRandomName` - Add random ID to filename (default: true)
+- `acl` - File ACL override (default: uses plugin's configured ACL)
+  - `"private"` - Only accessible via signed URLs
+  - `"public-read"` - Publicly accessible
+
+**Example with per-file ACL:**
+```javascript
+// Private file (default)
+await fastify.xStorage.upload(buffer, "private.pdf", {
+  folder: "documents",
+});
+
+// Public file
+await fastify.xStorage.upload(buffer, "public.pdf", {
+  folder: "documents",
+  acl: "public-read",
+});
+```
+
+### `fastify.xStorage.uploadMultiple(files, options)`
+
+Upload multiple files at once with optional per-file ACL control.
+
+```javascript
+const files = [
+  { file: buffer1, filename: "private.pdf" },
+  { file: buffer2, filename: "public.pdf", acl: "public-read" },
+];
+
+const results = await fastify.xStorage.uploadMultiple(files, {
+  folder: "documents",
+  acl: "private", // Default ACL for all files
+});
+
+// Returns array of upload results
+// Files can override batch ACL with per-file acl property
+```
+
+### `fastify.xStorage.delete(key)`
+
+Delete a file.
+
+```javascript
+await fastify.xStorage.delete("documents/document-a1b2c3d4.pdf");
+```
+
+### `fastify.xStorage.deleteMultiple(keys)`
+
+Delete multiple files at once.
+
+```javascript
+const keys = ["documents/doc1.pdf", "documents/doc2.pdf"];
+await fastify.xStorage.deleteMultiple(keys);
+```
+
+### `fastify.xStorage.download(key)`
+
+Download a file as a buffer.
+
+```javascript
+const buffer = await fastify.xStorage.download("documents/document-a1b2c3d4.pdf");
+```
+
+### `fastify.xStorage.list(prefix, maxKeys)`
+
+List files in a folder.
+
+```javascript
+const files = await fastify.xStorage.list("documents/", 100);
+
+console.log(files);
+// [
+//   {
+//     key: "documents/doc1.pdf",
+//     url: "https://...",
+//     size: 123456,
+//     lastModified: Date,
+//     etag: "..."
+//   }
+// ]
+```
+
+### `fastify.xStorage.getSignedUrl(key, expiresIn)`
+
+Generate a temporary signed URL for private file access.
+
+```javascript
+const url = await fastify.xStorage.getSignedUrl("documents/document-a1b2c3d4.pdf", 3600); // 1 hour
+```
+
+### `fastify.xStorage.getPublicUrl(key)`
+
+Get public URL for a file. Note: This only works if file ACL is set to public.
+
+```javascript
+const url = fastify.xStorage.getPublicUrl("documents/document.pdf");
+// Returns: "https://your-bucket.nyc3.digitaloceanspaces.com/documents/document.pdf"
+```
+
+### `fastify.xStorage.copy(sourceKey, destinationKey)`
+
+Copy a file to a new location.
+
+```javascript
+await fastify.xStorage.copy("documents/doc1.pdf", "documents/backup/doc1.pdf");
+```
+
+### `fastify.xStorage.exists(key)`
+
+Check if a file exists.
+
+```javascript
+const exists = await fastify.xStorage.exists("documents/document.pdf");
+```
+
+### `fastify.xStorage.getMetadata(key)`
+
+Get file metadata.
+
+```javascript
+const metadata = await fastify.xStorage.getMetadata("documents/document.pdf");
+// Returns: { size, contentType, lastModified, etag, etc }
+```
+
+## Image Processing
+
+For image processing, optimization, resizing, thumbnail generation, and format conversion, use **[@xenterprises/fastify-ximagepipeline](https://github.com/x-enterprises/fastify-plugins/tree/main/fastify-ximagepipeline)**.
+
+xImagePipeline integrates with xStorage and provides:
+- 🖼️ Image optimization and format conversion
+- 🎯 Multiple variant generation (webp, avif, etc)
+- 📐 Intelligent resizing and cropping
+- 🔍 EXIF metadata extraction and stripping
+- 💫 Blur hash generation for progressive loading
+- 📊 Compressed original image storage
+
+```javascript
+import Fastify from "fastify";
+import xImagePipeline from "@xenterprises/fastify-ximagepipeline";
+import xStorage from "@xenterprises/fastify-xstorage";
+
+const fastify = Fastify();
+
+// Register xStorage first
+await fastify.register(xStorage, { /* config */ });
+
+// Register xImagePipeline
+await fastify.register(xImagePipeline, { /* config */ });
+
+// Use for image processing
+const result = await fastify.ximagepipeline.processImage(buffer, "photo.jpg", {
+  sourceType: "avatar", // Uses configured variants for avatar
+});
+```
+
+## Helper Utilities
+
+```javascript
+import { helpers } from "@xenterprises/fastify-xstorage";
+
+// Format file size
+helpers.formatFileSize(1234567); // "1.18 MB"
+
+// Check file types
+helpers.isImage("photo.jpg"); // true
+helpers.isPdf("document.pdf"); // true
+helpers.isVideo("movie.mp4"); // true
+
+// Sanitize filename
+helpers.sanitizeFilename("My File (2024).jpg"); // "my_file_2024.jpg"
+
+// Calculate dimensions
+helpers.calculateFitDimensions(4000, 3000, 1920, 1080);
+// { width: 1440, height: 1080 }
+
+// Generate responsive sizes
+helpers.generateResponsiveSizes(1920, 1080);
+// [
+//   { width: 320, height: 180, name: "w320" },
+//   { width: 640, height: 360, name: "w640" },
+//   // ...
+// ]
+```
+
+## Usage Examples
+
+### Document Upload
+
+```javascript
+import multipart from "@fastify/multipart";
+
+// Register multipart for file uploads
+await fastify.register(multipart);
+
+// HTTP endpoint for document upload
+fastify.post("/documents", async (request, reply) => {
+  const data = await request.file();
+
+  if (!data) {
+    return reply.code(400).send({ error: "No file uploaded" });
+  }
+
+  const buffer = await data.toBuffer();
+
+  // Upload file
+  const result = await fastify.xStorage.upload(buffer, data.filename, {
+    folder: "documents",
+    useRandomName: true,
+  });
+
+  // Save to database
+  await fastify.db.document.create({
+    data: {
+      filename: data.filename,
+      storageKey: result.key,
+      size: result.size,
+      contentType: result.contentType,
+    },
+  });
+
+  return { success: true, file: result };
+});
+
+// Download document with signed URL
+fastify.get("/documents/:id", async (request, reply) => {
+  const { id } = request.params;
+
+  const document = await fastify.db.document.findUnique({ where: { id } });
+
+  // Generate signed URL valid for 1 hour
+  const signedUrl = await fastify.xStorage.getSignedUrl(document.storageKey, 3600);
+
+  return { download: signedUrl };
+});
+```
+
+### Batch File Operations
+
+```javascript
+// Upload multiple files
+fastify.post("/batch-upload", async (request, reply) => {
+  const parts = request.parts();
+  const files = [];
+
+  for await (const part of parts) {
+    if (part.type === "file") {
+      const buffer = await part.toBuffer();
+      files.push({ file: buffer, filename: part.filename });
+    }
+  }
+
+  const results = await fastify.xStorage.uploadMultiple(files, {
+    folder: "batch-uploads",
+  });
+
+  return { success: true, files: results };
+});
+
+// Delete multiple files
+fastify.post("/batch-delete", async (request, reply) => {
+  const { keys } = request.body;
+
+  await fastify.xStorage.deleteMultiple(keys);
+
+  return { success: true, deleted: keys.length };
+});
+```
+
+### File Organization
+
+```javascript
+// List files in a folder
+fastify.get("/files/:folder", async (request, reply) => {
+  const { folder } = request.params;
+
+  const files = await fastify.xStorage.list(`${folder}/`, 100);
+
+  return { folder, files };
+});
+
+// Move file (copy then delete)
+fastify.post("/files/move", async (request, reply) => {
+  const { sourceKey, destinationKey } = request.body;
+
+  await fastify.xStorage.copy(sourceKey, destinationKey);
+  await fastify.xStorage.delete(sourceKey);
+
+  return { success: true, newLocation: destinationKey };
+});
+```
+
+## Best Practices
+
+1. **Use signed URLs by default** - Default ACL is private; always use signed URLs for file access
+2. **Validate file types** - Check file types before accepting uploads
+3. **Use random filenames** - Prevents accidental overwrites of existing files
+4. **Store storage keys in database** - Keep reference to the storage key, not just the URL
+5. **Organize with folders** - Use logical folder structure (e.g., "documents/2024/january")
+6. **Set reasonable expiration times** - Use appropriate TTL for signed URLs (shorter for sensitive data)
+7. **Handle errors gracefully** - Files might be deleted externally; implement proper error handling
+8. **Batch operations for efficiency** - Use uploadMultiple/deleteMultiple for better performance
+
+## Plugin Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `endpoint` | string | - | S3 endpoint URL (required for non-AWS) |
+| `region` | string | `"us-east-1"` | AWS region |
+| `accessKeyId` | string | - | Access key ID (required) |
+| `secretAccessKey` | string | - | Secret access key (required) |
+| `bucket` | string | - | Bucket name (required) |
+| `publicUrl` | string | - | Public URL base (required) |
+| `forcePathStyle` | boolean | `true` | Use path-style URLs |
+| `acl` | string | `"private"` | Default ACL for uploads |
+
+## Testing
+
+See [TESTING.md](./TESTING.md) for comprehensive testing guide.
+
+## Examples
+
+See [EXAMPLES.md](./EXAMPLES.md) for complete real-world examples.
+
+## License
+
+ISC