@lumeweb/pinner 0.0.1 → 0.1.1

package/README.md

@@ -1,45 +1,724 @@
 # @lumeweb/pinner
 
-## ⚠️ IMPORTANT NOTICE ⚠️
+A TypeScript library for uploading files to IPFS and managing pinning operations with support for multiple upload protocols, custom storage backends, and flexible configuration.
 
-**This package is created solely for the purpose of setting up OIDC (OpenID Connect) trusted publishing with npm.**
+## Features
 
-This is **NOT** a functional package and contains **NO** code or functionality beyond the OIDC setup configuration.
+- **Multiple Upload Methods**: TUS resumable uploads, XHR uploads, and direct CAR file uploads
+- **Directory Support**: Upload multiple files as a directory to IPFS
+- **Pin Management**: Add, list, remove, and check status of pinned content
+- **Custom Blockstore**: Flexible storage backend using unstorage (IndexedDB, filesystem, Redis, etc.)
+- **Adapters**: Built-in Pinata adapter with extensible adapter pattern
+- **Encoders**: Support for CSV, JSON, text, base64, and URL encoding
+- **Progress Tracking**: Real-time upload progress and operation polling
+- **Cross-Platform**: Works in both browser and Node.js environments
+- **TypeScript**: Fully typed with comprehensive type definitions
 
-## Purpose
+## Installation
 
-This package exists to:
-1. Configure OIDC trusted publishing for the package name `@lumeweb/pinner`
-2. Enable secure, token-less publishing from CI/CD workflows
-3. Establish provenance for packages published under this name
+```bash
+pnpm add @lumeweb/pinner
+```
 
-## What is OIDC Trusted Publishing?
+## Quick Start
 
-OIDC trusted publishing allows package maintainers to publish packages directly from their CI/CD workflows without needing to manage npm access tokens. Instead, it uses OpenID Connect to establish trust between the CI/CD provider (like GitHub Actions) and npm.
+```typescript
+import { Pinner } from "@lumeweb/pinner";
 
-## Setup Instructions
+// Initialize with JWT token
+const pinner = new Pinner({
+  jwt: "your-jwt-token",
+  endpoint: "https://ipfs.pinner.xyz",
+  gateway: "https://dweb.link"
+});
 
-To properly configure OIDC trusted publishing for this package:
+// Upload a file
+const file = new File(["Hello, IPFS!"], "hello.txt", { type: "text/plain" });
+const operation = await pinner.upload(file);
 
-1. Go to [npmjs.com](https://www.npmjs.com/) and navigate to your package settings
-2. Configure the trusted publisher (e.g., GitHub Actions)
-3. Specify the repository and workflow that should be allowed to publish
-4. Use the configured workflow to publish your actual package
+// Wait for completion
+const result = await operation.result;
+console.log("CID:", result.cid);
+console.log("URL:", result.url);
 
-## DO NOT USE THIS PACKAGE
+// List pins
+const pins = await pinner.listPins();
+console.log("Pinned content:", pins);
+```
 
-This package is a placeholder for OIDC configuration only. It:
-- Contains no executable code
-- Provides no functionality
-- Should not be installed as a dependency
-- Exists only for administrative purposes
+## Configuration
 
-## More Information
+The `Pinner` class accepts a `PinnerConfig` object:
 
-For more details about npm's trusted publishing feature, see:
-- [npm Trusted Publishing Documentation](https://docs.npmjs.com/generating-provenance-statements)
-- [GitHub Actions OIDC Documentation](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect)
+```typescript
+interface PinnerConfig {
+  // Required
+  jwt: string;
+
+  // Optional
+  endpoint?: string;               // Default: "https://ipfs.pinner.xyz"
+  gateway?: string;                // Default: "https://dweb.link"
+  allowedFileTypes?: string[];     // MIME types allowed for upload
+  fetch?: typeof fetch;            // Custom fetch implementation
+  datastore?: Datastore;           // Custom datastore for Helia
+  storage?: Storage;               // Custom unstorage instance
+  datastoreName?: string;          // Base name for storage (default: "pinner-helia-data")
+}
+```
+
+## Upload Methods
+
+### Basic Upload
+
+```typescript
+// Upload with default options
+const operation = await pinner.upload(file);
+
+// Upload with custom options
+const operation = await pinner.upload(file, {
+  metadata: { name: "My File" },
+  timeout: 30000
+});
+
+// Wait for result
+const result = await operation.result;
+```
+
+### Upload and Wait
+
+Convenience method for simple use cases:
+
+```typescript
+const result = await pinner.uploadAndWait(file);
+console.log("CID:", result.cid);
+```
+
+### Directory Upload
+
+```typescript
+const files = [
+  new File(["content1"], "file1.txt"),
+  new File(["content2"], "file2.txt")
+];
+
+const operation = await pinner.uploadDirectory(files);
+const result = await operation.result;
+```
+
+### CAR File Upload
+
+Upload pre-generated CAR files without preprocessing:
+
+```typescript
+const carFile = new File([carData], "content.car", { type: "application/vnd.ipld.car" });
+const operation = await pinner.uploadCar(carFile);
+const result = await operation.result;
+```
+
+### Upload Builder Pattern
+
+Use the builder API for more control:
+
+```typescript
+// Upload JSON
+const operation = await pinner.upload.json({ foo: "bar" });
+
+// Upload text
+const operation = await pinner.upload.text("Hello, world!");
+
+// Upload CSV
+const operation = await pinner.upload.csv([
+  { name: "Alice", age: 30 },
+  { name: "Bob", age: 25 }
+]);
+```
+
+## Progress Tracking
+
+Upload operations return an `UploadOperation` with progress tracking:
+
+```typescript
+const operation = await pinner.upload(file);
+
+// Listen to progress events
+operation.on("progress", (progress) => {
+  console.log(`Progress: ${progress.progress}%`);
+  console.log(`Speed: ${progress.speed} bytes/sec`);
+});
+
+// Listen to completion
+operation.on("complete", (result) => {
+  console.log("Upload complete:", result.cid);
+});
+
+// Listen to errors
+operation.on("error", (error) => {
+  console.error("Upload failed:", error);
+});
+
+// Or await the result directly
+const result = await operation.result;
+```
+
+## Pin Management
+
+### Pin by CID
+
+```typescript
+import { CID } from "multiformats/cid";
+
+// Pin existing content
+const cid = CID.parse("Qm...");
+await pinner.pinByHash(cid);
+
+// With options
+await pinner.pinByHash(cid, {
+  name: "My Pin",
+  metadata: { key: "value" }
+});
+```
+
+### List Pins
+
+```typescript
+// List all pins
+const pins = await pinner.listPins();
+
+// List with filters
+const pins = await pinner.listPins({
+  status: "pinned",
+  limit: 10,
+  offset: 0
+});
+```
+
+### Get Pin Status
+
+```typescript
+const pin = await pinner.getPinStatus(cid);
+console.log("Pin status:", pin.status);
+console.log("Created:", pin.created);
+```
+
+### Check if Pinned
+
+```typescript
+const isPinned = await pinner.isPinned(cid);
+if (isPinned) {
+  console.log("Content is pinned");
+}
+```
+
+### Update Metadata
+
+```typescript
+await pinner.setPinMetadata(cid, {
+  name: "Updated Name",
+  description: "Updated description"
+});
+```
+
+### Remove Pin
+
+```typescript
+await pinner.unpin(cid);
+```
+
+## Operation Polling
+
+Wait for operations to complete with custom polling options:
+
+```typescript
+// Poll with default settings
+const result = await pinner.waitForOperation(operationId);
+
+// Poll with custom options
+const result = await pinner.waitForOperation(operationId, {
+  interval: 1000,      // Check every 1 second
+  timeout: 60000,      // Timeout after 60 seconds
+  settledStates: ["completed", "failed"]
+});
+```
+
+## Custom Blockstore
+
+Configure a custom storage backend using unstorage:
+
+```typescript
+import { Pinner, createBlockstore, setDriverFactory } from "@lumeweb/pinner";
+import { createStorage } from "unstorage";
+import redisDriver from "unstorage/drivers/redis";
+
+// Create Redis storage
+const storage = createStorage({
+  driver: redisDriver({
+    host: "localhost",
+    port: 6379,
+    base: "pinner:"
+  })
+});
+
+// Initialize with custom storage
+const pinner = new Pinner({
+  jwt: "your-token",
+  storage
+});
+```
+
+### Blockstore Options
+
+```typescript
+import { createBlockstore } from "@lumeweb/pinner";
+
+// Auto-configure (browser: IndexedDB, Node.js: filesystem)
+const blockstore = createBlockstore();
+
+// Custom driver
+const blockstore = createBlockstore({
+  driver: localStorageDriver({ base: "my-app:" })
+});
+
+// Pre-configured storage
+const blockstore = createBlockstore({
+  storage: myStorageInstance
+});
+
+// Disable auto-configuration (uses memory)
+const blockstore = createBlockstore({
+  autoConfigure: false
+});
+```
+
+See [blockstore/README.md](./src/blockstore/README.md) for detailed blockstore documentation.
+
+## Adapters
+
+### Pinata Adapters
+
+The Pinata adapters provide Pinata SDK API compatibility for the Pinner client, allowing applications written for the Pinata SDK to work with Lume's IPFS pinning infrastructure with minimal code changes.
+
+**Attribution**: These adapters include TypeScript type definitions and API interfaces adapted from the Pinata SDK for compatibility purposes. The original Pinata SDK is available at:
+- **Pinata SDK 2.x**: https://github.com/PinataCloud/pinata/commit/cdc0c06116aaadaf7c4b287a2673cd23b6ba1125
+- **Pinata SDK 1.x**: https://github.com/PinataCloud/pinata/commit/c141177ff3036e46fa7b95fcc68c159b58817836
+
+The adapters provide Pinata SDK API compatibility but route all operations through Lume's IPFS pinning infrastructure. They do NOT use Pinata's servers or services.
+
+#### Available Adapters
+
+- **V2 Adapter** (`pinataAdapter`): Compatible with Pinata SDK 2.x API (recommended, latest)
+- **Legacy Adapter** (`pinataLegacyAdapter`): Compatible with Pinata SDK 1.x API
+
+See [adapters/README.md](./src/adapters/README.md) for comprehensive documentation including migration guides, feature support tables, and detailed examples.
+
+#### Setup
+
+```typescript
+import { Pinner, pinataAdapter, pinataLegacyAdapter } from "@lumeweb/pinner";
+
+// Initialize Pinner
+const pinner = new Pinner({
+  jwt: "your-jwt-token",
+  endpoint: "https://your-pinning-service-endpoint.com"
+});
+
+// Create Pinata V2 adapter (recommended)
+const pinata = pinataAdapter(pinner);
+
+// Or use the legacy adapter
+const pinataLegacy = pinataLegacyAdapter(pinner);
+```
+
+#### Upload Methods
+
+The Pinata adapter provides multiple upload methods with a fluent builder pattern:
+
+##### Upload a File
+
+```typescript
+// Simple file upload
+const result = await pinata.upload.public.file(file).execute();
+console.log("CID:", result.IpfsHash);
+console.log("Size:", result.PinSize);
+
+// Upload with metadata
+const result = await pinata.upload.public.file(file)
+  .name("My File")
+  .keyvalues({ key: "value" })
+  .execute();
+```
+
+##### Upload Multiple Files (Directory)
+
+```typescript
+const files = [
+  new File(["content1"], "file1.txt"),
+  new File(["content2"], "file2.txt")
+];
+
+const result = await pinata.upload.public.fileArray(files)
+  .name("My Directory")
+  .keyvalues({ type: "directory" })
+  .execute();
+```
+
+##### Upload JSON Data
+
+```typescript
+const data = { foo: "bar", number: 42 };
+
+const result = await pinata.upload.public.json(data)
+  .name("data.json")
+  .keyvalues({ format: "json" })
+  .execute();
+```
+
+##### Upload Base64 String
+
+```typescript
+const base64String = "SGVsbG8sIHdvcmxkIQ==";
+
+const result = await pinata.upload.public.base64(base64String)
+  .name("base64-file.txt")
+  .execute();
+```
+
+##### Upload from URL
+
+```typescript
+const result = await pinata.upload.public.url("https://example.com/data.json")
+  .name("downloaded-file.json")
+  .execute();
+```
+
+##### Pin by CID
+
+```typescript
+// Pin existing content
+await pinata.upload.public.cid("Qm...").execute();
+
+// Pin with metadata
+await pinata.upload.public.cid("Qm...")
+  .name("Existing Content")
+  .keyvalues({ source: "external" })
+  .execute();
+```
+
+#### Pin Management
+
+##### Pin by Hash
+
+```typescript
+await pinata.pinByHash("Qm...", {
+  name: "My Pin",
+  keyvalues: { key: "value" }
+});
+```
+
+##### Unpin Content
+
+```typescript
+await pinata.unpin("Qm...");
+```
+
+##### Get Pin Status
+
+```typescript
+const pin = await pinata.getPinStatus("Qm...");
+console.log("Pin ID:", pin.id);
+console.log("IPFS Hash:", pin.ipfsPinHash);
+console.log("Size:", pin.size);
+console.log("Date Pinned:", pin.datePinned);
+console.log("Metadata:", pin.metadata);
+```
+
+##### Check if Pinned
+
+```typescript
+const isPinned = await pinata.isPinned("Qm...");
+if (isPinned) {
+  console.log("Content is pinned");
+}
+```
+
+##### Update Pin Metadata
+
+```typescript
+await pinata.setPinMetadata("Qm...", {
+  name: "Updated Name",
+  key: "value"
+});
+```
+
+#### Files Management
+
+##### List Files
+
+```typescript
+// List all files
+const files = await pinata.files.public.list().execute();
+
+// List with pagination
+const files = await pinata.files.public.list()
+  .limit(10)
+  .pageToken("next-page-token")
+  .execute();
+
+files.forEach(file => {
+  console.log("ID:", file.id);
+  console.log("CID:", file.cid);
+  console.log("Size:", file.size);
+  console.log("Name:", file.name);
+  console.log("Created:", file.createdAt);
+});
+```
+
+##### Get File by ID
+
+```typescript
+const file = await pinata.files.public.get("Qm...");
+console.log("File details:", file);
+```
+
+#### Pinata SDK Compatibility
+
+The adapter follows the Pinata SDK's API conventions, making migration straightforward:
+
+**Pinata SDK:**
+```typescript
+import pinataSDK from '@pinata/sdk';
+const pinata = pinataSDK('apiKey', 'apiSecret');
+
+const result = await pinata.pinFileToIPFS(file, {
+  pinataMetadata: { name: 'My File' },
+  pinataOptions: { cidVersion: 1 }
+});
+```
+
+**Pinner with Pinata Adapter:**
+```typescript
+import { Pinner, pinataAdapter } from '@lumeweb/pinner';
+
+const pinner = new Pinner({ jwt: 'your-jwt-token' });
+const pinata = pinataAdapter(pinner);
+
+const result = await pinata.upload.public.file(file)
+  .name('My File')
+  .execute();
+```
+
+#### Result Format
+
+Upload operations return a `PinataUploadResult`:
+
+```typescript
+interface PinataUploadResult {
+  IpfsHash: string;        // IPFS CID
+  PinSize: number;         // Size in bytes
+  Timestamp: string;       // ISO timestamp
+  isDuplicate: boolean;    // Whether content was already pinned
+}
+```
+
+#### Error Handling
+
+The adapter provides specific errors:
+
+```typescript
+import { PinataAdapterError } from "@lumeweb/pinner/adapters/pinata";
+
+try {
+  const result = await pinata.upload.file(file).execute();
+} catch (error) {
+  if (error instanceof PinataAdapterError) {
+    switch (error.code) {
+      case "UPLOAD_FAILED":
+        console.error("Upload failed:", error.message);
+        break;
+      case "EMPTY_FILE_ARRAY":
+        console.error("Cannot upload empty file array");
+        break;
+      case "INVALID_CID":
+        console.error("Invalid CID:", error.message);
+        break;
+    }
+  }
+}
+```
+
+## Encoders
+
+The library provides several encoders for different data formats:
+
+```typescript
+// JSON encoder
+const operation = await pinner.upload.json({ data: "value" });
+
+// CSV encoder
+const operation = await pinner.upload.csv([
+  { column1: "value1", column2: "value2" }
+]);
+
+// Text encoder
+const operation = await pinner.upload.text("Plain text content");
+
+// Base64 encoder
+const operation = await pinner.upload.base64("SGVsbG8sIHdvcmxkIQ==");
+
+// URL encoder
+const operation = await pinner.upload.url("https://example.com/data");
+```
+
+## Error Handling
+
+The library provides specific error types:
+
+```typescript
+import {
+  PinnerError,
+  ConfigurationError,
+  AuthenticationError,
+  UploadError,
+  NetworkError,
+  ValidationError,
+  EmptyFileError,
+  TimeoutError,
+  PinError,
+  NotFoundError,
+  RateLimitError
+} from "@lumeweb/pinner";
+
+try {
+  const result = await pinner.uploadAndWait(file);
+} catch (error) {
+  if (error instanceof AuthenticationError) {
+    console.error("Authentication failed:", error.message);
+  } else if (error instanceof NetworkError) {
+    console.error("Network error:", error.message);
+  } else if (error instanceof ValidationError) {
+    console.error("Validation error:", error.message);
+  } else {
+    console.error("Unknown error:", error);
+  }
+}
+```
+
+### Type Guards
+
+```typescript
+import { isRetryable, isAuthenticationError } from "@lumeweb/pinner";
+
+if (isRetryable(error)) {
+  // Retry the operation
+}
+
+if (isAuthenticationError(error)) {
+  // Re-authenticate
+}
+```
+
+## Stream Utilities
+
+```typescript
+import {
+  streamToBlob,
+  calculateStreamSize,
+  asyncGeneratorToReadableStream,
+  readableStreamToAsyncIterable
+} from "@lumeweb/pinner";
+
+// Convert stream to blob
+const blob = await streamToBlob(stream);
+
+// Calculate stream size
+const size = await calculateStreamSize(stream);
+
+// Convert async generator to readable stream
+const readableStream = asyncGeneratorToReadableStream(asyncGenerator);
+
+// Convert readable stream to async iterable
+const asyncIterable = readableStreamToAsyncIterable(readableStream);
+```
+
+## Testing
+
+The library includes comprehensive tests:
+
+```bash
+# Run all tests
+pnpm test
+
+# Run tests with coverage
+pnpm coverage
+
+# Run specific test project
+vitest run --project node-upload-unit
+vitest run --project browser-upload-integration
+```
+
+### Test Structure
+
+- **Unit Tests**: Test individual components with mocks
+- **Integration Tests**: Test real-world scenarios without mocks
+- **Browser Tests**: Run tests in Chromium using Playwright
+- **Node Tests**: Run tests in Node.js environment
+
+## Build
+
+```bash
+# Build the library
+pnpm build
+
+# Type checking
+pnpm lint
+```
+
+The library outputs:
+- ESM: `dist/esm/`
+- CJS: `dist/cjs/`
+- Type definitions: `dist/esm/*.d.ts`
+
+## Package Exports
+
+```typescript
+// Main exports
+import { Pinner } from "@lumeweb/pinner";
+
+// Upload types and utilities
+import {
+  UploadManager,
+  UploadResult,
+  UploadOptions,
+  UploadProgress
+} from "@lumeweb/pinner";
+
+// Pin types
+import type {
+  RemotePins,
+  RemotePin,
+  RemoteAddOptions,
+  RemoteLsOptions
+} from "@lumeweb/pinner";
+
+// Blockstore
+import {
+  createBlockstore,
+  createDatastore,
+  setDriverFactory
+} from "@lumeweb/pinner";
+
+// Adapters
+import { pinataAdapter, pinataLegacyAdapter } from "@lumeweb/pinner";
+
+// Blockstore module
+import { UnstorageBlockstore } from "@lumeweb/pinner/blockstore";
+```
+
+## License
+
+MIT
+
+## Contributing
+
+Contributions are welcome! Please ensure all tests pass and follow the existing code style.
 
----
 
-**Maintained for OIDC setup purposes only**