@visulima/storage 1.0.0-alpha.9 → 1.0.0

package/README.md

@@ -6,7 +6,7 @@
 
 </a>
 
-<h3 align="center">Visulima upload - Store files in a web-accessible location via a simplified API. Can automatically scale and rotate images. Includes S3, Azure, GCS and local filesystem-based backends with the most convenient features of each.</h3>
+<h3 align="center">Server-side file-storage abstraction - store files in a web-accessible location via a simplified API. Includes S3, Azure, GCS, local filesystem and 20+ other backends with TUS/multipart/REST upload handlers and the most convenient features of each.</h3>
 
 <!-- END_PACKAGE_OG_IMAGE_PLACEHOLDER -->
 
@@ -26,520 +26,136 @@
   <sub>Built with ❤︎ by <a href="https://twitter.com/_prisis_">Daniel Bannert</a></sub>
 </div>
 
-## Features
-
-- **Multiple Upload Handlers**: Multipart (form-based), REST (direct binary), and TUS (resumable) uploads
-- **Chunked Uploads**: REST handler supports client-side chunked uploads for large files
-- **Automatic Retry**: Built-in retry mechanism with exponential backoff for all storage backends (S3, Azure, GCS, Vercel Blob, Netlify Blob)
-- Parent directories are created automatically as needed (like S3 and Azure)
-- Content types are inferred from file extensions (like the filesystem)
-- Files are by default marked as readable via the web (like a filesystem + web server)
-- Images can be automatically scaled to multiple sizes
-- Images can be cropped
-- Images are automatically rotated if necessary for proper display on the web (i.e. iPhone photos with rotation hints are right side up)
-- Image width, image height and correct file extension are made available to the developer
-- Non-image files are also supported
-- Web access to files can be disabled and reenabled
-- GIF is supported, including animation, with full support for scaling and cropping
-- Batch operations: Delete multiple files in a single request
-- On fire about minimizing file sizes for your resized images? You can plug in `imagemin` and compatible tools using the `postprocessors` option.
-
-## Installation
+## Two surfaces, one library
 
-```sh
-npm install @visulima/upload
-```
-
-```sh
-yarn add @visulima/upload
-```
-
-```sh
-pnpm add @visulima/upload
-```
-
-## Install requirements peer storage
-
-### AWS S3
-
-```sh
-npm install @aws-sdk/client-s3 @aws-sdk/credential-providers @aws-sdk/s3-request-presigner @aws-sdk/signature-v4-crt aws-crt @aws-sdk/types
-```
-
-```sh
-yarn add @aws-sdk/client-s3 @aws-sdk/credential-providers @aws-sdk/s3-request-presigner @aws-sdk/signature-v4-crt aws-crt @aws-sdk/types
-```
-
-```sh
-pnpm add @aws-sdk/client-s3 @aws-sdk/credential-providers @aws-sdk/s3-request-presigner @aws-sdk/signature-v4-crt aws-crt @aws-sdk/types
-```
-
-### Azure Blob Storage
-
-```sh
-npm install @azure/storage-blob
-```
-
-```sh
-yarn add @azure/storage-blob
-```
-
-```sh
-pnpm add @azure/storage-blob
-```
-
-> Note: If you hit this error: "TypeError: Expected signal to be an instanceof AbortSignal" [#784](https://github.com/node-fetch/node-fetch/issues/784) you need to install node-fetch 4.0.0-beta or higher.
+- **`Files`** — one-liner API for ad-hoc storage operations. Web-standard body types (`Blob`, `Uint8Array`, `ReadableStream`), bulk-array overloads, per-call retry/timeout/abort, lifecycle hooks, and a `.raw` escape hatch. Use this when you just need to put files somewhere.
+- **`BaseStorage` adapters** (`S3Storage`, `DiskStorage`, `BunnyStorage`, …) — the full upload-server framework. TUS / multipart / REST handlers, lifecycle hooks, validators, image / video / audio transformers, OpenAPI export. Use this when you're hosting an upload service.
 
-### Google Cloud Storage
+Both surfaces wrap the same adapters — swap providers without touching call sites.
 
-```sh
-npm install @google-cloud/storage node-fetch gaxios
-```
-
-```sh
-yarn add @google-cloud/storage node-fetch gaxios
-```
+## Install
 
 ```sh
-pnpm add @google-cloud/storage node-fetch gaxios
-```
-
-## Caching
-
-The storage package supports caching to improve performance and reduce API calls. You provide your own cache implementation that follows the simple `Cache` interface.
-
-### LRU Cache
-
-Use the built-in LRU cache for simple in-memory caching:
-
-```typescript
-import { LRUCache } from "lru-cache";
-import { DiskStorage } from "@visulima/storage";
-
-const cache = new LRUCache({
-    max: 1000, // Maximum number of items
-    ttl: 3600000, // 1 hour in milliseconds
-});
-
-const storage = new DiskStorage({
-    directory: "/uploads",
-    cache,
-});
+npm install @visulima/storage
+# or yarn add / pnpm add
 ```
 
-### Custom Cache Implementation
-
-Implement the `Cache` interface for any cache provider:
-
-```typescript
-import { DiskStorage, type Cache } from "@visulima/storage";
-import { Redis } from "ioredis";
-
-// Custom Redis cache implementation
-class RedisCache implements Cache<string, any> {
-    constructor(private redis: Redis) {}
-
-    async get(key: string): Promise<any | undefined> {
-        const value = await this.redis.get(key);
-        return value ? JSON.parse(value) : undefined;
-    }
-
-    async set(key: string, value: any): Promise<boolean> {
-        await this.redis.set(key, JSON.stringify(value));
-        return true;
-    }
-
-    async delete(key: string): Promise<boolean> {
-        await this.redis.del(key);
-        return true;
-    }
-
-    async clear(): Promise<void> {
-        await this.redis.flushall();
-    }
+Each provider has its own peer dependency (e.g. `@aws-sdk/client-s3` for S3, `@azure/storage-blob` for Azure). See the relevant [service page](https://visulima.com/docs/packages/storage/services) for the exact install command.
 
-    async has(key: string): Promise<boolean> {
-        const exists = await this.redis.exists(key);
-        return exists === 1;
-    }
-}
+## Quick start with `Files`
 
-const redis = new Redis();
-const cache = new RedisCache(redis);
+```ts
+import { Files } from "@visulima/storage";
+import { S3Storage } from "@visulima/storage/provider/aws";
 
-const storage = new DiskStorage({
-    directory: "/uploads",
-    cache,
+const files = new Files({
+    adapter: new S3Storage({ bucket: "uploads", region: "us-east-1" }),
 });
-```
 
-### BentoCache Integration
-
-For advanced multi-tier caching, use BentoCache with the adapter:
-
-```typescript
-import { BentoCache, bentostore } from "bentocache";
-import { memoryDriver } from "bentocache/drivers/memory";
-import { redisDriver } from "bentocache/drivers/redis";
-import { BentoCacheAdapter } from "@visulima/storage/utils/cache";
-
-const bento = new BentoCache({
-    default: "storage",
-    stores: {
-        storage: bentostore()
-            .useL1Layer(memoryDriver({ maxSize: "10mb" }))
-            .useL2Layer(
-                redisDriver({
-                    connection: { host: "127.0.0.1", port: 6379 },
-                }),
-            ),
-    },
-});
+await files.upload("avatars/abc.png", buffer, { contentType: "image/png" });
 
-const cache = new BentoCacheAdapter({
-    bento,
-    namespace: "storage",
-    defaultTtl: 3600000, // 1 hour
-});
+const url = await files.url("avatars/abc.png", { expiresIn: 900 });
+const { body } = await files.download("avatars/abc.png");
 
-const storage = new DiskStorage({
-    directory: "/uploads",
-    cache,
-});
+// Stream large objects without buffering the whole payload in memory:
+const { body: stream } = await files.downloadStream("avatars/abc.png");
 ```
 
-### Transformer Caching
+Swap the adapter to change provider — everything else stays the same:
 
-```typescript
-import { MediaTransformer } from "@visulima/upload/transformer";
-import ImageTransformer from "@visulima/upload/transformer/image";
-import VideoTransformer from "@visulima/upload/transformer/video";
-import { LRUCache } from "lru-cache";
+```ts
+import { Files, DiskStorage } from "@visulima/storage";
+import { BunnyStorage } from "@visulima/storage/provider/bunny";
 
-const transformer = new MediaTransformer(storage, {
-    cache: new LRUCache({ max: 100, ttl: 3600000 }),
-    ImageTransformer: ImageTransformer,
-    VideoTransformer: VideoTransformer,
-});
+new Files({ adapter: new DiskStorage({ directory: "./uploads" }) });
+new Files({ adapter: new BunnyStorage({ zone, accessKey, region: "de" }) });
 ```
 
-### Cache Interface
-
-Any cache implementation must implement this interface:
+→ See the [Files facade reference](https://visulima.com/docs/packages/storage/files-facade) for the full method surface (`upload`, `download`, `downloadStream`, `head`, `exists`, `delete`, `copy`, `move`, `list`, `listAll`, `url`, `signedUploadUrl`, `.raw`) and body-type matrix. `list({ delimiter: "/" })` returns directory-style `{ files, prefixes }`; pass `{ readonly: true }` (or call `files.readonly()`) for a locked view, and read `files.capabilities` to branch on adapter support. Top-level `transfer(source, destination)` streams whole buckets between adapters; `sync(source, destination)` does an incremental, optionally-pruning mirror (with `dryRun`).
 
-```typescript
-interface Cache<K = string, V = any> {
-    get(key: K): V | undefined | Promise<V | undefined>;
-    set(key: K, value: V, options?: { ttl?: number }): boolean | Promise<boolean>;
-    delete(key: K): boolean | Promise<boolean>;
-    clear(): void | Promise<void>;
-    has(key: K): boolean | Promise<boolean>;
-}
-```
+## Building an upload server?
 
-For more information, see the [BentoCache documentation](https://bentocache.dev/docs/introduction).
+If you're handling client uploads from a browser — large files, resumable transfers, form posts — use the adapter directly with one of the HTTP handlers.
 
-## Retry Mechanism
-
-All storage backends (AWS S3, Azure, GCS, Vercel Blob, Netlify Blob) include automatic retry logic for transient failures. This ensures your uploads and file operations are resilient to network issues, rate limits, and temporary service unavailability.
-
-### Default Behavior
-
-By default, storage operations will retry up to 3 times with exponential backoff:
-
-- Initial delay: 1 second
-- Backoff multiplier: 2x (delays double each retry: 1s → 2s → 4s)
-- Maximum delay: 30 seconds
-- Retryable status codes: 408, 429, 500, 502, 503, 504
-
-### Basic Configuration
-
-Configure retry behavior when creating your storage instance:
-
-```typescript
-import { S3Storage } from "@visulima/storage";
-
-const storage = new S3Storage({
-    bucket: "my-bucket",
-    region: "us-east-1",
-    retryConfig: {
-        maxRetries: 5,
-        initialDelay: 2000,
-        backoffMultiplier: 1.5,
-        maxDelay: 60_000,
-    },
-});
-```
-
-### Advanced Configuration
-
-Customize retry logic with backend-specific error detection:
-
-```typescript
-import { AzureStorage } from "@visulima/storage";
-
-const storage = new AzureStorage({
-    containerName: "uploads",
-    accountName: "myaccount",
-    accountKey: "mykey",
-    retryConfig: {
-        maxRetries: 3,
-        initialDelay: 1000,
-        backoffMultiplier: 2,
-        maxDelay: 30_000,
-        retryableStatusCodes: [408, 429, 500, 502, 503, 504],
-        shouldRetry: (error: unknown) => {
-            // Custom retry logic
-            if (error instanceof Error) {
-                const errorCode = (error as any).code;
-
-                // Retry on network errors
-                if (errorCode === "ECONNRESET" || errorCode === "ETIMEDOUT") {
-                    return true;
-                }
-            }
-
-            // Retry on specific HTTP status codes
-            if ((error as any).statusCode && [429, 503].includes((error as any).statusCode)) {
-                return true;
-            }
-
-            return false;
-        },
-    },
-});
-```
-
-### Retry Configuration Options
-
-```typescript
-interface RetryConfig {
-    /** Maximum number of retry attempts (default: 3) */
-    maxRetries?: number;
-
-    /** Initial delay in milliseconds before first retry (default: 1000) */
-    initialDelay?: number;
-
-    /** Multiplier for exponential backoff (default: 2) */
-    backoffMultiplier?: number;
-
-    /** Maximum delay in milliseconds between retries (default: 30000) */
-    maxDelay?: number;
-
-    /** HTTP status codes that should trigger a retry (default: [408, 429, 500, 502, 503, 504]) */
-    retryableStatusCodes?: number[];
-
-    /** Custom function to determine if an error should be retried */
-    shouldRetry?: (error: unknown) => boolean;
-
-    /** Custom function to calculate delay for a specific retry attempt */
-    calculateDelay?: (attempt: number, error: unknown) => number | undefined;
-}
-```
-
-### Using Retry Utilities Directly
-
-You can also use the retry utilities for custom operations:
-
-```typescript
-import { retry, createRetryWrapper, isRetryableError } from "@visulima/storage";
-
-// One-off retry
-const result = await retry(
-    async () => {
-        // Your operation here
-        return await someOperation();
-    },
-    {
-        maxRetries: 3,
-        initialDelay: 1000,
-    },
-);
-
-// Create a reusable retry wrapper
-const retryWrapper = createRetryWrapper({
-    maxRetries: 5,
-    initialDelay: 2000,
-});
-
-const result = await retryWrapper(async () => {
-    return await someOperation();
-});
-
-// Check if an error is retryable
-if (isRetryableError(error)) {
-    // Handle retryable error
-}
-```
-
-### Supported Error Types
-
-The retry mechanism automatically handles:
-
-- **Network errors**: `ECONNRESET`, `ETIMEDOUT`, `ENOTFOUND`, `ECONNREFUSED`, `EAI_AGAIN`
-- **AWS SDK errors**: Server faults, retryable status codes, SDK v2/v3 error formats
-- **Azure Storage errors**: HTTP status codes, network connection issues
-- **HTTP errors**: 408 (Request Timeout), 429 (Too Many Requests), 5xx (Server Errors)
-
-### Examples by Storage Backend
-
-#### AWS S3
+```ts
+import { DiskStorage } from "@visulima/storage";
+import { Multipart, Rest } from "@visulima/storage/handler/http/node";
 
-```typescript
-import { S3Storage } from "@visulima/storage";
+const storage = new DiskStorage({ directory: "./uploads" });
 
-const storage = new S3Storage({
-    bucket: "my-bucket",
-    region: "us-east-1",
-    retryConfig: {
-        maxRetries: 3,
-        // AWS SDK errors are automatically detected
-    },
+const multipart = new Multipart({ storage });
+app.use("/upload", multipart.handle, (req, res) => {
+    res.json(req.body); // handler writes the stored file metadata to req.body
 });
-```
-
-#### Azure Blob Storage
 
-```typescript
-import { AzureStorage } from "@visulima/storage";
-
-const storage = new AzureStorage({
-    containerName: "uploads",
-    accountName: "myaccount",
-    accountKey: "mykey",
-    retryConfig: {
-        maxRetries: 5,
-        initialDelay: 2000,
-    },
-});
+const rest = new Rest({ storage });
+app.use("/files", rest.handle);
 ```
 
-#### Google Cloud Storage
+Three handlers cover the common upload patterns:
 
-GCS already has built-in retry support via `gaxios`. The retry mechanism works alongside GCS's native retry logic:
+- **Multipart** — `multipart/form-data` from HTML forms.
+- **REST** — direct binary `POST` / `PUT`. Optional client-side chunking for large files.
+- **TUS** — resumable uploads ([tus.io](https://tus.io)) for unreliable networks and very large files.
 
-```typescript
-import { GCStorage } from "@visulima/storage";
+→ Runtime adapters live under `@visulima/storage/handler/http/{node,fetch,hono,nextjs,solid-start,bun,deno,cloudflare,edge}`. See [framework integrations](https://visulima.com/docs/packages/storage/framework).
 
-const storage = new GCStorage({
-    bucket: "my-bucket",
-    projectId: "my-project",
-    retryConfig: {
-        maxRetries: 3,
-        // Works with GCS's existing retryOptions
-    },
-});
-```
+## Choosing your surface
 
-#### Vercel Blob
+| You want to…                                           | Use                                         |
+| ------------------------------------------------------ | ------------------------------------------- |
+| Save a `Blob` / `Buffer` to S3 and get a signed URL    | `Files`                                     |
+| Drop in a file picker / "import from cloud" flow       | `Files` with a consumer-provider adapter    |
+| Expose ad-hoc storage to an AI agent or LLM tool       | `Files` + `@visulima/storage/ai/*` subpaths |
+| Build a TUS / multipart / chunked upload endpoint      | adapter + handler                           |
+| Transform images / video / audio on the fly            | adapter + transformer                       |
+| Hook `onCreate` / `onComplete` / `onDelete` lifecycles | adapter                                     |
+| Export OpenAPI for an upload endpoint                  | adapter                                     |
 
-```typescript
-import { VercelBlobStorage } from "@visulima/storage";
+## Providers
 
-const storage = new VercelBlobStorage({
-    token: process.env.BLOB_READ_WRITE_TOKEN,
-    retryConfig: {
-        maxRetries: 3,
-        retryableStatusCodes: [408, 429, 500, 502, 503, 504],
-    },
-});
-```
+Object storage (service credentials, presigned URLs, S3-style):
 
-#### Netlify Blob
+- AWS S3 · Azure Blob · Google Cloud Storage · Vercel Blob · Netlify Blobs · Local disk
 
-```typescript
-import { NetlifyBlobStorage } from "@visulima/storage";
+S3-compatible (branded client configs under `provider/aws/s3/clients`):
 
-const storage = new NetlifyBlobStorage({
-    storeName: "uploads",
-    retryConfig: {
-        maxRetries: 3,
-    },
-});
-```
+- Cloudflare R2 · DigitalOcean Spaces · MinIO · Hetzner · Storj · Backblaze B2 · Tigris · Wasabi · Akamai · Scaleway · Vultr · Exoscale · Filebase · iDrive e2 · IBM Cloud Object Storage · Oracle Cloud (OCI) · OVHcloud · Alibaba Cloud OSS · Tencent Cloud COS · Yandex Object Storage
 
-## Chunked Uploads
+Consumer providers (user OAuth or service-credential, all peer-dep gated):
 
-The REST handler supports client-side chunked uploads for large files. This allows you to upload files in smaller pieces, reducing memory usage and enabling resumable uploads.
+- Dropbox · Google Drive · Microsoft OneDrive · SharePoint · Box · Supabase · UploadThing · Bunny Storage · Cloudinary · Firebase Storage · PocketBase
 
-### Initializing a Chunked Upload
+→ See the [capability matrix](https://visulima.com/docs/packages/storage/services/capabilities) for what each provider supports (presign, copy, list, signed uploads, …).
 
-```typescript
-import { Rest } from "@visulima/upload/handler/rest";
+## AI tool integrations
 
-const rest = new Rest({ storage });
-
-// Initialize chunked upload
-const initResponse = await fetch("/files", {
-    method: "POST",
-    headers: {
-        "X-Chunked-Upload": "true",
-        "X-Total-Size": "10485760", // Total file size in bytes
-        "Content-Length": "0",
-        "Content-Type": "application/octet-stream",
-    },
-});
-
-const { id } = await initResponse.json();
-// id is the upload session ID
-```
-
-### Uploading Chunks
-
-```typescript
-// Upload chunk 1 (bytes 0-524288)
-await fetch(`/files/${id}`, {
-    method: "PATCH",
-    headers: {
-        "X-Chunk-Offset": "0",
-        "Content-Length": "524288",
-        "Content-Type": "application/octet-stream",
-    },
-    body: chunk1,
-});
+The package ships four optional subpaths that expose `Files` as tools an LLM can call — same eight canonical operations wrapped in each SDK's native tool format, with per-tool approval gating.
 
-// Upload chunk 2 (bytes 524288-1048576) - can be out of order
-await fetch(`/files/${id}`, {
-    method: "PATCH",
-    headers: {
-        "X-Chunk-Offset": "524288",
-        "Content-Length": "524288",
-        "Content-Type": "application/octet-stream",
-    },
-    body: chunk2,
-});
-```
+```ts
+import { Files } from "@visulima/storage";
+import { S3Storage } from "@visulima/storage/provider/aws";
+import { createFileTools } from "@visulima/storage/ai/sdk";
 
-### Checking Upload Progress
-
-```typescript
-// Check upload status
-const statusResponse = await fetch(`/files/${id}`, {
-    method: "HEAD",
-});
-
-const offset = statusResponse.headers.get("X-Upload-Offset");
-const complete = statusResponse.headers.get("X-Upload-Complete");
-const chunks = JSON.parse(statusResponse.headers.get("X-Received-Chunks") || "[]");
-
-console.log(`Uploaded: ${offset} bytes, Complete: ${complete}`);
+const files = new Files({ adapter: new S3Storage({ bucket: "uploads", region: "us-east-1" }) });
+const tools = createFileTools({ files });
 ```
 
-### Features
-
-- **Out-of-Order Chunks**: Chunks can be uploaded in any order
-- **Idempotency**: Duplicate chunks are safely ignored
-- **Resumable**: Check progress and resume from last uploaded chunk
-- **Progress Tracking**: Real-time upload progress via HEAD requests
-- **Chunk Size Limits**: Maximum 100MB per chunk (configurable)
+Subpaths: `@visulima/storage/ai/{sdk,openai,claude,tanstack}` — Vercel AI SDK, OpenAI Responses + Agents (both under `/openai`), Claude Agent SDK, and TanStack AI.
 
-### Response Headers
+## Documentation
 
-- `X-Upload-ID`: Upload session ID (returned on initialization)
-- `X-Chunked-Upload`: Indicates chunked upload mode
-- `X-Upload-Offset`: Current upload offset in bytes
-- `X-Upload-Complete`: "true" when upload is complete, "false" otherwise
-- `X-Received-Chunks`: JSON array of received chunks `[{ offset, length }]`
+- [Introduction](https://visulima.com/docs/packages/storage/introduction) — the two surfaces, when to use each
+- [Files facade](https://visulima.com/docs/packages/storage/files-facade) — the eight-method reference
+- [Services](https://visulima.com/docs/packages/storage/services) — per-provider configuration and capability matrix
+- [Framework integrations](https://visulima.com/docs/packages/storage/framework) — Node, Fetch, Hono, Next.js, SolidStart, Bun, Deno, Cloudflare
+- [Custom storage](https://visulima.com/docs/packages/storage/custom-storage) — write your own adapter against `BaseStorage`
+- [Transformers](https://visulima.com/docs/packages/storage/transformers) — image / video / audio pipelines, caching
+- [TUS handler](https://visulima.com/docs/packages/storage/tus-handler) · [Chunked uploads](https://visulima.com/docs/packages/storage/chunked-uploads) · [Authenticated uploads](https://visulima.com/docs/packages/storage/authenticated-file-uploads)
+- [Batch operations](https://visulima.com/docs/packages/storage/batch-operations) · [Retry mechanism](https://visulima.com/docs/packages/storage/retry-mechanism) · [Caching](https://visulima.com/docs/packages/storage/caching)
+- [Error handling](https://visulima.com/docs/packages/storage/error-handling) — `UploadError`, `ERRORS` enum, `wrapStorageError`
+- [Observability](https://visulima.com/docs/packages/storage/observability) — metrics, OpenTelemetry, structured logs
+- [OpenAPI export](https://visulima.com/docs/packages/storage/openapi)
 
 ## Supported Node.js Versions