@visulima/storage 1.0.0-alpha.3 → 1.0.0-alpha.30

package/README.md

@@ -26,520 +26,133 @@
   <sub>Built with ❤︎ by <a href="https://twitter.com/_prisis_">Daniel Bannert</a></sub>
 </div>
 
-## Features
-
-- **Multiple Upload Handlers**: Multipart (form-based), REST (direct binary), and TUS (resumable) uploads
-- **Chunked Uploads**: REST handler supports client-side chunked uploads for large files
-- **Automatic Retry**: Built-in retry mechanism with exponential backoff for all storage backends (S3, Azure, GCS, Vercel Blob, Netlify Blob)
-- Parent directories are created automatically as needed (like S3 and Azure)
-- Content types are inferred from file extensions (like the filesystem)
-- Files are by default marked as readable via the web (like a filesystem + web server)
-- Images can be automatically scaled to multiple sizes
-- Images can be cropped
-- Images are automatically rotated if necessary for proper display on the web (i.e. iPhone photos with rotation hints are right side up)
-- Image width, image height and correct file extension are made available to the developer
-- Non-image files are also supported
-- Web access to files can be disabled and reenabled
-- GIF is supported, including animation, with full support for scaling and cropping
-- Batch operations: Delete multiple files in a single request
-- On fire about minimizing file sizes for your resized images? You can plug in `imagemin` and compatible tools using the `postprocessors` option.
-
-## Installation
+## Two surfaces, one library
 
-```sh
-npm install @visulima/upload
-```
-
-```sh
-yarn add @visulima/upload
-```
-
-```sh
-pnpm add @visulima/upload
-```
-
-## Install requirements peer storage
-
-### AWS S3
-
-```sh
-npm install @aws-sdk/client-s3 @aws-sdk/credential-providers @aws-sdk/s3-request-presigner @aws-sdk/signature-v4-crt aws-crt @aws-sdk/types
-```
-
-```sh
-yarn add @aws-sdk/client-s3 @aws-sdk/credential-providers @aws-sdk/s3-request-presigner @aws-sdk/signature-v4-crt aws-crt @aws-sdk/types
-```
-
-```sh
-pnpm add @aws-sdk/client-s3 @aws-sdk/credential-providers @aws-sdk/s3-request-presigner @aws-sdk/signature-v4-crt aws-crt @aws-sdk/types
-```
-
-### Azure Blob Storage
-
-```sh
-npm install @azure/storage-blob
-```
-
-```sh
-yarn add @azure/storage-blob
-```
-
-```sh
-pnpm add @azure/storage-blob
-```
-
-> Note: If you hit this error: "TypeError: Expected signal to be an instanceof AbortSignal" [#784](https://github.com/node-fetch/node-fetch/issues/784) you need to install node-fetch 4.0.0-beta or higher.
+- **`Files`** — one-liner API for ad-hoc storage operations. Web-standard body types (`Blob`, `Uint8Array`, `ReadableStream`), eight methods, and a `.raw` escape hatch. Use this when you just need to put files somewhere.
+- **`BaseStorage` adapters** (`S3Storage`, `DiskStorage`, `BunnyStorage`, …) — the full upload-server framework. TUS / multipart / REST handlers, lifecycle hooks, validators, image / video / audio transformers, OpenAPI export. Use this when you're hosting an upload service.
 
-### Google Cloud Storage
+Both surfaces wrap the same adapters — swap providers without touching call sites.
 
-```sh
-npm install @google-cloud/storage node-fetch gaxios
-```
-
-```sh
-yarn add @google-cloud/storage node-fetch gaxios
-```
+## Install
 
 ```sh
-pnpm add @google-cloud/storage node-fetch gaxios
-```
-
-## Caching
-
-The storage package supports caching to improve performance and reduce API calls. You provide your own cache implementation that follows the simple `Cache` interface.
-
-### LRU Cache
-
-Use the built-in LRU cache for simple in-memory caching:
-
-```typescript
-import { LRUCache } from "lru-cache";
-import { DiskStorage } from "@visulima/storage";
-
-const cache = new LRUCache({
-    max: 1000, // Maximum number of items
-    ttl: 3600000, // 1 hour in milliseconds
-});
-
-const storage = new DiskStorage({
-    directory: "/uploads",
-    cache,
-});
-```
-
-### Custom Cache Implementation
-
-Implement the `Cache` interface for any cache provider:
-
-```typescript
-import { DiskStorage, type Cache } from "@visulima/storage";
-import { Redis } from "ioredis";
-
-// Custom Redis cache implementation
-class RedisCache implements Cache<string, any> {
-    constructor(private redis: Redis) {}
-
-    async get(key: string): Promise<any | undefined> {
-        const value = await this.redis.get(key);
-        return value ? JSON.parse(value) : undefined;
-    }
-
-    async set(key: string, value: any): Promise<boolean> {
-        await this.redis.set(key, JSON.stringify(value));
-        return true;
-    }
-
-    async delete(key: string): Promise<boolean> {
-        await this.redis.del(key);
-        return true;
-    }
-
-    async clear(): Promise<void> {
-        await this.redis.flushall();
-    }
-
-    async has(key: string): Promise<boolean> {
-        const exists = await this.redis.exists(key);
-        return exists === 1;
-    }
-}
-
-const redis = new Redis();
-const cache = new RedisCache(redis);
-
-const storage = new DiskStorage({
-    directory: "/uploads",
-    cache,
-});
-```
-
-### BentoCache Integration
-
-For advanced multi-tier caching, use BentoCache with the adapter:
-
-```typescript
-import { BentoCache, bentostore } from "bentocache";
-import { memoryDriver } from "bentocache/drivers/memory";
-import { redisDriver } from "bentocache/drivers/redis";
-import { BentoCacheAdapter } from "@visulima/storage/utils/cache";
-
-const bento = new BentoCache({
-    default: "storage",
-    stores: {
-        storage: bentostore()
-            .useL1Layer(memoryDriver({ maxSize: "10mb" }))
-            .useL2Layer(
-                redisDriver({
-                    connection: { host: "127.0.0.1", port: 6379 },
-                }),
-            ),
-    },
-});
-
-const cache = new BentoCacheAdapter({
-    bento,
-    namespace: "storage",
-    defaultTtl: 3600000, // 1 hour
-});
-
-const storage = new DiskStorage({
-    directory: "/uploads",
-    cache,
-});
-```
-
-### Transformer Caching
-
-```typescript
-import { MediaTransformer } from "@visulima/upload/transformer";
-import ImageTransformer from "@visulima/upload/transformer/image";
-import VideoTransformer from "@visulima/upload/transformer/video";
-import { LRUCache } from "lru-cache";
-
-const transformer = new MediaTransformer(storage, {
-    cache: new LRUCache({ max: 100, ttl: 3600000 }),
-    ImageTransformer: ImageTransformer,
-    VideoTransformer: VideoTransformer,
-});
-```
-
-### Cache Interface
-
-Any cache implementation must implement this interface:
-
-```typescript
-interface Cache<K = string, V = any> {
-    get(key: K): V | undefined | Promise<V | undefined>;
-    set(key: K, value: V, options?: { ttl?: number }): boolean | Promise<boolean>;
-    delete(key: K): boolean | Promise<boolean>;
-    clear(): void | Promise<void>;
-    has(key: K): boolean | Promise<boolean>;
-}
+npm install @visulima/storage
+# or yarn add / pnpm add
 ```
 
-For more information, see the [BentoCache documentation](https://bentocache.dev/docs/introduction).
+Each provider has its own peer dependency (e.g. `@aws-sdk/client-s3` for S3, `@azure/storage-blob` for Azure). See the relevant [service page](https://visulima.com/docs/packages/storage/services) for the exact install command.
 
-## Retry Mechanism
+## Quick start with `Files`
 
-All storage backends (AWS S3, Azure, GCS, Vercel Blob, Netlify Blob) include automatic retry logic for transient failures. This ensures your uploads and file operations are resilient to network issues, rate limits, and temporary service unavailability.
+```ts
+import { Files } from "@visulima/storage";
+import { S3Storage } from "@visulima/storage/provider/aws";
 
-### Default Behavior
-
-By default, storage operations will retry up to 3 times with exponential backoff:
-
-- Initial delay: 1 second
-- Backoff multiplier: 2x (delays double each retry: 1s → 2s → 4s)
-- Maximum delay: 30 seconds
-- Retryable status codes: 408, 429, 500, 502, 503, 504
-
-### Basic Configuration
-
-Configure retry behavior when creating your storage instance:
-
-```typescript
-import { S3Storage } from "@visulima/storage";
-
-const storage = new S3Storage({
-    bucket: "my-bucket",
-    region: "us-east-1",
-    retryConfig: {
-        maxRetries: 5,
-        initialDelay: 2000,
-        backoffMultiplier: 1.5,
-        maxDelay: 60_000,
-    },
+const files = new Files({
+    adapter: new S3Storage({ bucket: "uploads", region: "us-east-1" }),
 });
-```
-
-### Advanced Configuration
-
-Customize retry logic with backend-specific error detection:
-
-```typescript
-import { AzureStorage } from "@visulima/storage";
-
-const storage = new AzureStorage({
-    containerName: "uploads",
-    accountName: "myaccount",
-    accountKey: "mykey",
-    retryConfig: {
-        maxRetries: 3,
-        initialDelay: 1000,
-        backoffMultiplier: 2,
-        maxDelay: 30_000,
-        retryableStatusCodes: [408, 429, 500, 502, 503, 504],
-        shouldRetry: (error: unknown) => {
-            // Custom retry logic
-            if (error instanceof Error) {
-                const errorCode = (error as any).code;
-
-                // Retry on network errors
-                if (errorCode === "ECONNRESET" || errorCode === "ETIMEDOUT") {
-                    return true;
-                }
-            }
-
-            // Retry on specific HTTP status codes
-            if ((error as any).statusCode && [429, 503].includes((error as any).statusCode)) {
-                return true;
-            }
-
-            return false;
-        },
-    },
-});
-```
-
-### Retry Configuration Options
-
-```typescript
-interface RetryConfig {
-    /** Maximum number of retry attempts (default: 3) */
-    maxRetries?: number;
-
-    /** Initial delay in milliseconds before first retry (default: 1000) */
-    initialDelay?: number;
-
-    /** Multiplier for exponential backoff (default: 2) */
-    backoffMultiplier?: number;
-
-    /** Maximum delay in milliseconds between retries (default: 30000) */
-    maxDelay?: number;
 
-    /** HTTP status codes that should trigger a retry (default: [408, 429, 500, 502, 503, 504]) */
-    retryableStatusCodes?: number[];
+await files.upload("avatars/abc.png", buffer, { contentType: "image/png" });
 
-    /** Custom function to determine if an error should be retried */
-    shouldRetry?: (error: unknown) => boolean;
-
-    /** Custom function to calculate delay for a specific retry attempt */
-    calculateDelay?: (attempt: number, error: unknown) => number | undefined;
-}
+const url = await files.url("avatars/abc.png", { expiresIn: 900 });
+const { body } = await files.download("avatars/abc.png");
 ```
 
-### Using Retry Utilities Directly
-
-You can also use the retry utilities for custom operations:
-
-```typescript
-import { retry, createRetryWrapper, isRetryableError } from "@visulima/storage";
-
-// One-off retry
-const result = await retry(
-    async () => {
-        // Your operation here
-        return await someOperation();
-    },
-    {
-        maxRetries: 3,
-        initialDelay: 1000,
-    },
-);
-
-// Create a reusable retry wrapper
-const retryWrapper = createRetryWrapper({
-    maxRetries: 5,
-    initialDelay: 2000,
-});
+Swap the adapter to change provider — everything else stays the same:
 
-const result = await retryWrapper(async () => {
-    return await someOperation();
-});
+```ts
+import { Files, DiskStorage } from "@visulima/storage";
+import { BunnyStorage } from "@visulima/storage/provider/bunny";
 
-// Check if an error is retryable
-if (isRetryableError(error)) {
-    // Handle retryable error
-}
+new Files({ adapter: new DiskStorage({ directory: "./uploads" }) });
+new Files({ adapter: new BunnyStorage({ zone, accessKey, region: "de" }) });
 ```
 
-### Supported Error Types
-
-The retry mechanism automatically handles:
+→ See the [Files facade reference](https://visulima.com/docs/packages/storage/files-facade) for the full method surface (`upload`, `download`, `head`, `delete`, `copy`, `list`, `url`, `signedUploadUrl`, `.raw`) and body-type matrix.
 
-- **Network errors**: `ECONNRESET`, `ETIMEDOUT`, `ENOTFOUND`, `ECONNREFUSED`, `EAI_AGAIN`
-- **AWS SDK errors**: Server faults, retryable status codes, SDK v2/v3 error formats
-- **Azure Storage errors**: HTTP status codes, network connection issues
-- **HTTP errors**: 408 (Request Timeout), 429 (Too Many Requests), 5xx (Server Errors)
+## Building an upload server?
 
-### Examples by Storage Backend
+If you're handling client uploads from a browser — large files, resumable transfers, form posts — use the adapter directly with one of the HTTP handlers.
 
-#### AWS S3
-
-```typescript
-import { S3Storage } from "@visulima/storage";
-
-const storage = new S3Storage({
-    bucket: "my-bucket",
-    region: "us-east-1",
-    retryConfig: {
-        maxRetries: 3,
-        // AWS SDK errors are automatically detected
-    },
-});
-```
-
-#### Azure Blob Storage
+```ts
+import { DiskStorage } from "@visulima/storage";
+import { Multipart, Rest } from "@visulima/storage/handler/http/node";
 
-```typescript
-import { AzureStorage } from "@visulima/storage";
+const storage = new DiskStorage({ directory: "./uploads" });
 
-const storage = new AzureStorage({
-    containerName: "uploads",
-    accountName: "myaccount",
-    accountKey: "mykey",
-    retryConfig: {
-        maxRetries: 5,
-        initialDelay: 2000,
-    },
+const multipart = new Multipart({ storage });
+app.use("/upload", multipart.handle, (req, res) => {
+    res.json(req.body); // handler writes the stored file metadata to req.body
 });
-```
-
-#### Google Cloud Storage
 
-GCS already has built-in retry support via `gaxios`. The retry mechanism works alongside GCS's native retry logic:
-
-```typescript
-import { GCStorage } from "@visulima/storage";
-
-const storage = new GCStorage({
-    bucket: "my-bucket",
-    projectId: "my-project",
-    retryConfig: {
-        maxRetries: 3,
-        // Works with GCS's existing retryOptions
-    },
-});
+const rest = new Rest({ storage });
+app.use("/files", rest.handle);
 ```
 
-#### Vercel Blob
+Three handlers cover the common upload patterns:
 
-```typescript
-import { VercelBlobStorage } from "@visulima/storage";
+- **Multipart** — `multipart/form-data` from HTML forms.
+- **REST** — direct binary `POST` / `PUT`. Optional client-side chunking for large files.
+- **TUS** — resumable uploads ([tus.io](https://tus.io)) for unreliable networks and very large files.
 
-const storage = new VercelBlobStorage({
-    token: process.env.BLOB_READ_WRITE_TOKEN,
-    retryConfig: {
-        maxRetries: 3,
-        retryableStatusCodes: [408, 429, 500, 502, 503, 504],
-    },
-});
-```
+→ Runtime adapters live under `@visulima/storage/handler/http/{node,fetch,hono,nextjs,solid-start,bun,deno,cloudflare,edge}`. See [framework integrations](https://visulima.com/docs/packages/storage/framework).
 
-#### Netlify Blob
+## Choosing your surface
 
-```typescript
-import { NetlifyBlobStorage } from "@visulima/storage";
+| You want to…                                           | Use                                         |
+| ------------------------------------------------------ | ------------------------------------------- |
+| Save a `Blob` / `Buffer` to S3 and get a signed URL    | `Files`                                     |
+| Drop in a file picker / "import from cloud" flow       | `Files` with a consumer-provider adapter    |
+| Expose ad-hoc storage to an AI agent or LLM tool       | `Files` + `@visulima/storage/ai/*` subpaths |
+| Build a TUS / multipart / chunked upload endpoint      | adapter + handler                           |
+| Transform images / video / audio on the fly            | adapter + transformer                       |
+| Hook `onCreate` / `onComplete` / `onDelete` lifecycles | adapter                                     |
+| Export OpenAPI for an upload endpoint                  | adapter                                     |
 
-const storage = new NetlifyBlobStorage({
-    storeName: "uploads",
-    retryConfig: {
-        maxRetries: 3,
-    },
-});
-```
+## Providers
 
-## Chunked Uploads
+Object storage (service credentials, presigned URLs, S3-style):
 
-The REST handler supports client-side chunked uploads for large files. This allows you to upload files in smaller pieces, reducing memory usage and enabling resumable uploads.
+- AWS S3 · Azure Blob · Google Cloud Storage · Vercel Blob · Netlify Blobs · Local disk
 
-### Initializing a Chunked Upload
+S3-compatible (branded client configs under `provider/aws/s3/clients`):
 
-```typescript
-import { Rest } from "@visulima/upload/handler/rest";
+- Cloudflare R2 · DigitalOcean Spaces · MinIO · Hetzner · Storj · Backblaze B2 · Tigris · Wasabi · Akamai · Scaleway · Vultr · Exoscale · Filebase · iDrive e2 · IBM Cloud Object Storage · Oracle Cloud (OCI) · OVHcloud · Alibaba Cloud OSS · Tencent Cloud COS · Yandex Object Storage
 
-const rest = new Rest({ storage });
+Consumer providers (user OAuth or service-credential, all peer-dep gated):
 
-// Initialize chunked upload
-const initResponse = await fetch("/files", {
-    method: "POST",
-    headers: {
-        "X-Chunked-Upload": "true",
-        "X-Total-Size": "10485760", // Total file size in bytes
-        "Content-Length": "0",
-        "Content-Type": "application/octet-stream",
-    },
-});
+- Dropbox · Google Drive · Microsoft OneDrive · SharePoint · Box · Supabase · UploadThing · Bunny Storage · Cloudinary · Firebase Storage · PocketBase
 
-const { id } = await initResponse.json();
-// id is the upload session ID
-```
+→ See the [capability matrix](https://visulima.com/docs/packages/storage/services/capabilities) for what each provider supports (presign, copy, list, signed uploads, …).
 
-### Uploading Chunks
-
-```typescript
-// Upload chunk 1 (bytes 0-524288)
-await fetch(`/files/${id}`, {
-    method: "PATCH",
-    headers: {
-        "X-Chunk-Offset": "0",
-        "Content-Length": "524288",
-        "Content-Type": "application/octet-stream",
-    },
-    body: chunk1,
-});
+## AI tool integrations
 
-// Upload chunk 2 (bytes 524288-1048576) - can be out of order
-await fetch(`/files/${id}`, {
-    method: "PATCH",
-    headers: {
-        "X-Chunk-Offset": "524288",
-        "Content-Length": "524288",
-        "Content-Type": "application/octet-stream",
-    },
-    body: chunk2,
-});
-```
+The package ships four optional subpaths that expose `Files` as tools an LLM can call — same eight canonical operations wrapped in each SDK's native tool format, with per-tool approval gating.
 
-### Checking Upload Progress
+```ts
+import { Files } from "@visulima/storage";
+import { S3Storage } from "@visulima/storage/provider/aws";
+import { createFileTools } from "@visulima/storage/ai/sdk";
 
-```typescript
-// Check upload status
-const statusResponse = await fetch(`/files/${id}`, {
-    method: "HEAD",
-});
-
-const offset = statusResponse.headers.get("X-Upload-Offset");
-const complete = statusResponse.headers.get("X-Upload-Complete");
-const chunks = JSON.parse(statusResponse.headers.get("X-Received-Chunks") || "[]");
-
-console.log(`Uploaded: ${offset} bytes, Complete: ${complete}`);
+const files = new Files({ adapter: new S3Storage({ bucket: "uploads", region: "us-east-1" }) });
+const tools = createFileTools({ files });
 ```
 
-### Features
-
-- **Out-of-Order Chunks**: Chunks can be uploaded in any order
-- **Idempotency**: Duplicate chunks are safely ignored
-- **Resumable**: Check progress and resume from last uploaded chunk
-- **Progress Tracking**: Real-time upload progress via HEAD requests
-- **Chunk Size Limits**: Maximum 100MB per chunk (configurable)
+Subpaths: `@visulima/storage/ai/{sdk,openai,claude,tanstack}` — Vercel AI SDK, OpenAI Responses + Agents (both under `/openai`), Claude Agent SDK, and TanStack AI.
 
-### Response Headers
+## Documentation
 
-- `X-Upload-ID`: Upload session ID (returned on initialization)
-- `X-Chunked-Upload`: Indicates chunked upload mode
-- `X-Upload-Offset`: Current upload offset in bytes
-- `X-Upload-Complete`: "true" when upload is complete, "false" otherwise
-- `X-Received-Chunks`: JSON array of received chunks `[{ offset, length }]`
+- [Introduction](https://visulima.com/docs/packages/storage/introduction) — the two surfaces, when to use each
+- [Files facade](https://visulima.com/docs/packages/storage/files-facade) — the eight-method reference
+- [Services](https://visulima.com/docs/packages/storage/services) — per-provider configuration and capability matrix
+- [Framework integrations](https://visulima.com/docs/packages/storage/framework) — Node, Fetch, Hono, Next.js, SolidStart, Bun, Deno, Cloudflare
+- [Custom storage](https://visulima.com/docs/packages/storage/custom-storage) — write your own adapter against `BaseStorage`
+- [Transformers](https://visulima.com/docs/packages/storage/transformers) — image / video / audio pipelines, caching
+- [TUS handler](https://visulima.com/docs/packages/storage/tus-handler) · [Chunked uploads](https://visulima.com/docs/packages/storage/chunked-uploads) · [Authenticated uploads](https://visulima.com/docs/packages/storage/authenticated-file-uploads)
+- [Batch operations](https://visulima.com/docs/packages/storage/batch-operations) · [Retry mechanism](https://visulima.com/docs/packages/storage/retry-mechanism) · [Caching](https://visulima.com/docs/packages/storage/caching)
+- [Error handling](https://visulima.com/docs/packages/storage/error-handling) — `UploadError`, `ERRORS` enum, `wrapStorageError`
+- [Observability](https://visulima.com/docs/packages/storage/observability) — metrics, OpenTelemetry, structured logs
+- [OpenAPI export](https://visulima.com/docs/packages/storage/openapi)
 
 ## Supported Node.js Versions
 
@@ -564,14 +177,14 @@ This is an open source project and will always remain free to use. If you think
 
 ## License
 
-The visulima uploads is open-sourced software licensed under the [MIT][license]
+The visulima storage is open-sourced software licensed under the [MIT][license]
 
 <!-- badges -->
 
-[license-badge]: https://img.shields.io/npm/l/@visulima/upload?style=for-the-badge
+[license-badge]: https://img.shields.io/npm/l/@visulima/storage?style=for-the-badge
 [license]: https://github.com/visulima/visulima/blob/main/LICENSE
-[npm-downloads-badge]: https://img.shields.io/npm/dm/@visulima/upload?style=for-the-badge
-[npm-downloads]: https://www.npmjs.com/package/@visulima/upload
+[npm-downloads-badge]: https://img.shields.io/npm/dm/@visulima/storage?style=for-the-badge
+[npm-downloads]: https://www.npmjs.com/package/@visulima/storage
 [prs-welcome-badge]: https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=for-the-badge
 [prs-welcome]: https://github.com/visulima/visulima/blob/main/.github/CONTRIBUTING.md
 [chat-badge]: https://img.shields.io/discord/932323359193186354.svg?style=for-the-badge