@uploadista/core 0.0.17 → 0.0.18-beta.10

package/docs/DEAD-LETTER-QUEUE.md

@@ -0,0 +1,374 @@
+# Dead Letter Queue (DLQ) Documentation
+
+## Overview
+
+The Dead Letter Queue (DLQ) provides automatic capture and retry capabilities for failed flow jobs. When a flow execution fails, the DLQ preserves the complete failure context including inputs, partial results, and error details for debugging, automatic retry, or manual intervention.
+
+## Key Features
+
+- **Automatic Failure Capture**: Failed flow jobs are automatically captured with full execution context
+- **Configurable Retry Policies**: Support for immediate, fixed delay, and exponential backoff strategies
+- **Error Filtering**: Configure which errors should be retried vs. non-retryable
+- **Admin API**: RESTful endpoints for DLQ management (list, retry, resolve, cleanup)
+- **Observability**: Event-based metrics and tracing for monitoring
+- **TTL-based Cleanup**: Automatic expiration of old DLQ items
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                             Flow Execution                                   │
+└──────────────────────────────────┬──────────────────────────────────────────┘
+                                   │
+                              ┌────▼────┐
+                              │  Fail?  │
+                              └────┬────┘
+                               Yes │
+                    ┌──────────────▼──────────────────────┐
+                    │      DeadLetterQueueService         │
+                    │  ┌────────────────────────────────┐ │
+                    │  │     DeadLetterItem             │ │
+                    │  │  - jobId, flowId, storageId    │ │
+                    │  │  - error details, inputs       │ │
+                    │  │  - nodeResults, retryHistory   │ │
+                    │  └────────────────────────────────┘ │
+                    └──────────────────┬──────────────────┘
+                                       │
+                    ┌──────────────────▼──────────────────┐
+                    │       DeadLetterQueueKVStore        │
+                    │         (Persistent Storage)        │
+                    └─────────────────────────────────────┘
+```
+
+## Quick Start
+
+### 1. Enable DLQ in Flow Configuration
+
+```typescript
+import { createFlowWithSchema, type FlowDeadLetterQueueConfig } from "@uploadista/core";
+
+const flowConfig = {
+  flowId: "image-pipeline",
+  name: "Image Processing Pipeline",
+  nodes: [...],
+  edges: [...],
+  inputSchema: imageInputSchema,
+  outputSchema: imageOutputSchema,
+  // Enable DLQ with custom retry policy
+  deadLetterQueue: {
+    enabled: true,
+    retryPolicy: {
+      enabled: true,
+      maxRetries: 5,
+      backoff: {
+        type: "exponential",
+        initialDelayMs: 1000,
+        maxDelayMs: 300000, // 5 minutes
+        multiplier: 2,
+        jitter: true
+      },
+      nonRetryableErrors: ["VALIDATION_ERROR", "AUTH_ERROR"],
+      ttlMs: 604800000 // 7 days
+    }
+  }
+};
+```
+
+### 2. Provide DLQ Service Layer
+
+```typescript
+import {
+  DeadLetterQueueService,
+  deadLetterQueueService,
+  deadLetterQueueKvStore,
+  BaseKvStoreService
+} from "@uploadista/core";
+
+// Provide the DLQ service in your Effect layer stack
+const program = myFlowProgram.pipe(
+  Effect.provide(deadLetterQueueService),
+  Effect.provide(deadLetterQueueKvStore),
+  Effect.provide(baseKvStoreLayer)
+);
+```
+
+### 3. Access DLQ in Admin Handlers
+
+```typescript
+import { DeadLetterQueueService } from "@uploadista/core";
+
+const adminHandler = Effect.gen(function* () {
+  const dlq = yield* DeadLetterQueueService;
+
+  // Get DLQ statistics
+  const stats = yield* dlq.getStats();
+  console.log(`Total DLQ items: ${stats.totalItems}`);
+
+  // List pending items
+  const { items, total } = yield* dlq.list({ status: "pending" });
+
+  // Manual retry
+  const item = yield* dlq.get(itemId);
+  yield* dlq.markRetrying(item.id);
+  // ... re-execute flow with item.inputs ...
+  yield* dlq.markResolved(item.id);
+});
+```
+
+## Retry Policies
+
+### Backoff Strategies
+
+#### Immediate
+Retry immediately without delay. Use for transient errors that may succeed on immediate retry.
+
+```typescript
+const immediatePolicy = {
+  enabled: true,
+  maxRetries: 3,
+  backoff: { type: "immediate" }
+};
+```
+
+#### Fixed Delay
+Wait a fixed duration between retries.
+
+```typescript
+const fixedPolicy = {
+  enabled: true,
+  maxRetries: 5,
+  backoff: {
+    type: "fixed",
+    delayMs: 5000 // 5 seconds between retries
+  }
+};
+```
+
+#### Exponential Backoff
+Progressively longer delays with optional jitter.
+
+```typescript
+const exponentialPolicy = {
+  enabled: true,
+  maxRetries: 5,
+  backoff: {
+    type: "exponential",
+    initialDelayMs: 1000,   // Start with 1 second
+    maxDelayMs: 300000,     // Cap at 5 minutes
+    multiplier: 2,          // Double each time
+    jitter: true            // Add randomness to prevent thundering herd
+  }
+};
+// Delays: ~1s, ~2s, ~4s, ~8s, ~16s, ... capped at 5min
+```
+
+### Error Filtering
+
+Control which errors trigger retries:
+
+```typescript
+const filteredPolicy = {
+  enabled: true,
+  maxRetries: 3,
+  backoff: { type: "exponential", ... },
+  // Only retry these errors
+  retryableErrors: ["NETWORK_ERROR", "TIMEOUT_ERROR"],
+  // Never retry these (takes precedence)
+  nonRetryableErrors: ["VALIDATION_ERROR", "AUTH_ERROR", "PERMISSION_DENIED"]
+};
+```
+
+## DLQ Item Lifecycle
+
+```
+┌─────────┐    Add      ┌─────────┐   Retry   ┌──────────┐
+│  Flow   │ ─────────▶  │ pending │ ────────▶ │ retrying │
+│ Failure │             └────┬────┘           └────┬─────┘
+└─────────┘                  │                     │
+                             │               ┌─────┴─────┐
+                             │           Success      Failure
+                             │               │           │
+                     Max Retries         ┌───▼────┐  ┌───▼────┐
+                         Reached         │resolved│  │pending │
+                             │           └────────┘  └────────┘
+                         ┌───▼─────┐                     │
+                         │exhausted│◀────────────────────┘
+                         └─────────┘                Max retries
+```
+
+### Status Meanings
+
+- **pending**: Awaiting retry (scheduled or manual)
+- **retrying**: Currently being retried
+- **exhausted**: Max retries reached, requires manual intervention
+- **resolved**: Successfully retried or manually resolved
+
+## Admin API Endpoints
+
+The DLQ provides RESTful admin endpoints for management:
+
+### List DLQ Items
+```
+GET /api/admin/dlq
+Query params: status, flowId, clientId, limit, offset
+```
+
+### Get Single Item
+```
+GET /api/admin/dlq/:itemId
+```
+
+### Retry Single Item
+```
+POST /api/admin/dlq/:itemId/retry
+```
+
+### Retry All Items
+```
+POST /api/admin/dlq/retry-all
+Body: { status?: "pending", flowId?: string }
+```
+
+### Delete Item
+```
+DELETE /api/admin/dlq/:itemId
+```
+
+### Mark as Resolved
+```
+POST /api/admin/dlq/:itemId/resolve
+```
+
+### Cleanup Old Items
+```
+POST /api/admin/dlq/cleanup
+Body: { olderThan?: Date, status?: "exhausted" | "resolved" }
+```
+
+### Get Statistics
+```
+GET /api/admin/dlq/stats
+```
+
+## Observability Events
+
+The DLQ emits events for monitoring and alerting:
+
+| Event | Description |
+|-------|-------------|
+| `dlq-item-added` | Job added to DLQ |
+| `dlq-retry-start` | Retry attempt started |
+| `dlq-retry-success` | Retry succeeded |
+| `dlq-retry-failed` | Retry failed |
+| `dlq-item-exhausted` | Max retries reached |
+| `dlq-item-resolved` | Item marked resolved |
+
+### Example Event Handler
+
+```typescript
+const eventHandler = (event: FlowEvent) => {
+  switch (event.eventType) {
+    case EventType.DlqItemAdded:
+      metrics.increment("dlq.items.added");
+      alerting.notify(`Job ${event.jobId} added to DLQ: ${event.errorMessage}`);
+      break;
+    case EventType.DlqItemExhausted:
+      metrics.increment("dlq.items.exhausted");
+      alerting.critical(`Job ${event.jobId} exhausted all retries`);
+      break;
+  }
+};
+```
+
+## Best Practices
+
+### 1. Configure Appropriate Retry Limits
+- Use fewer retries (2-3) for validation errors
+- Use more retries (5-10) for external service calls
+- Consider the total retry duration vs. business requirements
+
+### 2. Filter Non-Retryable Errors
+Always configure `nonRetryableErrors` to skip permanent failures:
+- `VALIDATION_ERROR` - Invalid input data
+- `AUTH_ERROR` - Authentication failures
+- `PERMISSION_DENIED` - Authorization failures
+- `NOT_FOUND` - Missing resources
+
+### 3. Set Appropriate TTL
+- Short TTL (1-2 days) for time-sensitive flows
+- Longer TTL (7-30 days) for debugging needs
+- Consider storage costs for high-volume systems
+
+### 4. Monitor DLQ Growth
+Set up alerts for:
+- DLQ size exceeding threshold
+- High rate of exhausted items
+- Specific error codes appearing frequently
+
+### 5. Regular Cleanup
+Schedule periodic cleanup of resolved and exhausted items:
+
+```typescript
+// Daily cleanup of items older than 7 days
+const dailyCleanup = Effect.gen(function* () {
+  const dlq = yield* DeadLetterQueueService;
+  const weekAgo = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);
+  const result = yield* dlq.cleanup({ olderThan: weekAgo });
+  console.log(`Cleaned up ${result.deleted} DLQ items`);
+});
+```
+
+## TypeScript Types
+
+### DeadLetterItem
+
+```typescript
+interface DeadLetterItem {
+  id: string;
+  jobId: string;
+  flowId: string;
+  storageId: string;
+  clientId: string | null;
+  error: DeadLetterError;
+  inputs: Record<string, unknown>;
+  nodeResults: Record<string, unknown>;
+  failedAtNodeId?: string;
+  retryCount: number;
+  maxRetries: number;
+  nextRetryAt?: Date;
+  retryHistory: DeadLetterRetryAttempt[];
+  createdAt: Date;
+  updatedAt: Date;
+  expiresAt?: Date;
+  status: DeadLetterItemStatus;
+}
+```
+
+### RetryPolicy
+
+```typescript
+interface RetryPolicy {
+  enabled: boolean;
+  maxRetries: number;
+  backoff: BackoffStrategy;
+  retryableErrors?: string[];
+  nonRetryableErrors?: string[];
+  ttlMs?: number;
+}
+
+type BackoffStrategy =
+  | { type: "immediate" }
+  | { type: "fixed"; delayMs: number }
+  | { type: "exponential"; initialDelayMs: number; maxDelayMs: number; multiplier: number; jitter: boolean };
+```
+
+## Migration Guide
+
+### From v0.x (No DLQ) to v1.x (With DLQ)
+
+1. Add DLQ KV store to your base store configuration
+2. Provide the `deadLetterQueueService` layer
+3. Optionally configure flow-level retry policies
+4. Implement admin UI or CLI for DLQ management
+
+The DLQ integration is fully optional and backward compatible. Flows without explicit DLQ configuration will work as before.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@uploadista/core",
-  "version": "0.0.17",
+  "version": "0.0.18-beta.10",
   "description": "Core package of Uploadista",
   "license": "MIT",
   "author": "Uploadista",
@@ -62,16 +62,21 @@
     }
   },
   "dependencies": {
-    "effect": "3.19.6",
-    "zod": "4.1.13"
+    "micromustache": "^8.0.3"
+  },
+  "peerDependencies": {
+    "effect": "^3.0.0",
+    "zod": "^4.0.0"
   },
   "devDependencies": {
     "@effect/vitest": "0.27.0",
     "@types/node": "24.10.1",
+    "effect": "3.19.8",
     "tsd": "0.33.0",
-    "tsdown": "0.16.6",
-    "vitest": "4.0.13",
-    "@uploadista/typescript-config": "0.0.17"
+    "tsdown": "0.16.8",
+    "vitest": "4.0.14",
+    "zod": "4.1.13",
+    "@uploadista/typescript-config": "0.0.18-beta.10"
   },
   "publishConfig": {
     "access": "public",

package/src/errors/uploadista-error.ts

@@ -48,6 +48,8 @@ export type UploadistaErrorCode =
   | "FFMPEG_NOT_INSTALLED"
   | "INVALID_NODE_TYPE"
   | "TYPE_CATEGORY_MISMATCH"
+  | "INVALID_INPUT_TYPE"
+  | "INVALID_OUTPUT_TYPE"
   | "OUTPUT_NOT_FOUND"
   | "MULTIPLE_OUTPUTS_FOUND"
   | "VIRUS_SCAN_FAILED"
@@ -60,7 +62,8 @@ export type UploadistaErrorCode =
   | "OCR_FAILED"
   | "PDF_ENCRYPTED"
   | "PDF_CORRUPTED"
-  | "PAGE_RANGE_INVALID";
+  | "PAGE_RANGE_INVALID"
+  | "CIRCUIT_BREAKER_OPEN";
 
 /**
  * Catalog of all predefined errors in the Uploadista system.
@@ -236,6 +239,14 @@ export const ERROR_CATALOG: Readonly<
     status: 500,
     body: "Node type category does not match the node configuration\n",
   },
+  INVALID_INPUT_TYPE: {
+    status: 500,
+    body: "The input type is not registered\n",
+  },
+  INVALID_OUTPUT_TYPE: {
+    status: 500,
+    body: "The output type is not registered\n",
+  },
   OUTPUT_NOT_FOUND: {
     status: 404,
     body: "No output of the specified type was found\n",
@@ -288,6 +299,10 @@ export const ERROR_CATALOG: Readonly<
     status: 400,
     body: "The specified page range is invalid\n",
   },
+  CIRCUIT_BREAKER_OPEN: {
+    status: 503,
+    body: "Circuit breaker is open - service temporarily unavailable\n",
+  },
 } as const;
 
 /**

package/src/flow/README.md

@@ -437,6 +437,108 @@ const conditionalNode: FlowNode<MyData, MyData> = {
 };
 ```
 
+### 6. File Naming for Transform Nodes
+
+Transform nodes (nodes that produce new files) support automatic file naming to avoid confusion in multi-output flows. When processing multiple files through a pipeline, automatic naming helps distinguish between original and processed versions.
+
+#### Naming Modes
+
+The flow engine supports three naming modes:
+
+- **None**: Keep the original filename unchanged
+- **Auto** (default): Automatically add a suffix based on the operation (e.g., `photo.jpg` → `photo-800x600.jpg`)
+- **Custom**: Use a template pattern or custom function
+
+#### Auto Naming
+
+When enabled, each transform node type adds a relevant suffix:
+
+| Node | Auto Suffix | Example |
+|------|-------------|---------|
+| resize | `${width}x${height}` | `photo-800x600.jpg` |
+| optimize | `${format}` | `photo-webp.webp` |
+| transform-image | `transformed` | `photo-transformed.jpg` |
+| remove-background | `nobg` | `photo-nobg.png` |
+| resize-video | `${width}x${height}` | `video-720p.mp4` |
+| transcode | `${format}` | `video-mp4.mp4` |
+| trim | `trimmed` | `video-trimmed.mp4` |
+| thumbnail | `thumb` | `video-thumb.jpg` |
+| split-pdf | `page-${pageNumber}` | `doc-page-1.pdf` |
+| merge-pdf | `merged` | `docs-merged.pdf` |
+
+#### Usage Example
+
+```typescript
+import { createResizeNode } from "@uploadista/flow-image-nodes";
+
+// Default: Auto naming enabled
+const resizeNode = yield* createResizeNode("resize", {
+  width: 800,
+  height: 600,
+}, {
+  naming: { mode: "auto" }, // Output: "photo-800x600.jpg"
+});
+
+// Custom naming with template
+const resizeNodeCustom = yield* createResizeNode("resize", {
+  width: 800,
+  height: 600,
+}, {
+  naming: {
+    mode: "custom",
+    pattern: "{{baseName}}-{{nodeType}}-{{width}}w.{{extension}}",
+  }, // Output: "photo-resize-800w.jpg"
+});
+
+// Custom naming with function
+const resizeNodeFn = yield* createResizeNode("resize", {
+  width: 800,
+  height: 600,
+}, {
+  naming: {
+    mode: "custom",
+    rename: (file, ctx) => `processed-${ctx.baseName}.${ctx.extension}`,
+  }, // Output: "processed-photo.jpg"
+});
+
+// Disable naming
+const resizeNodeNone = yield* createResizeNode("resize", {
+  width: 800,
+  height: 600,
+}, {
+  naming: { mode: "none" }, // Output: "photo.jpg" (unchanged)
+});
+```
+
+#### Available Template Variables
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `{{baseName}}` | Filename without extension | `photo` |
+| `{{extension}}` | File extension | `jpg` |
+| `{{fileName}}` | Full filename | `photo.jpg` |
+| `{{nodeType}}` | Type of processing node | `resize` |
+| `{{nodeId}}` | Node identifier | `resize-1` |
+| `{{flowId}}` | Flow identifier | `flow-abc` |
+| `{{jobId}}` | Job identifier | `job-123` |
+| `{{timestamp}}` | Processing timestamp | `2024-01-15T10:30:00Z` |
+| `{{width}}` | Output width (when applicable) | `800` |
+| `{{height}}` | Output height (when applicable) | `600` |
+| `{{format}}` | Output format (when applicable) | `webp` |
+| `{{quality}}` | Quality setting (when applicable) | `80` |
+| `{{pageNumber}}` | Page number (for PDF split) | `1` |
+
+#### Metadata-Only Nodes
+
+Nodes that only extract metadata (don't transform file bytes) don't support file naming:
+- `describe-image-node` - AI image description
+- `describe-document-node` - PDF metadata extraction
+- `describe-video-node` - Video metadata extraction
+- `extract-text-node` - PDF text extraction
+- `ocr-node` - OCR text extraction
+- `convert-to-markdown-node` - Markdown extraction to metadata
+- `scan-virus-node` - Virus scanning
+
 ## Best Practices
 
 1. **Use descriptive node names**: Make your nodes easy to understand and debug