@uploadista/core 0.0.2

package/docs/PARALLEL_EXECUTION_QUICKSTART.md

@@ -0,0 +1,142 @@
+# Parallel Execution - Quick Start Guide
+
+## TL;DR
+
+The `ParallelScheduler` now uses Effect's native concurrency primitives instead of broken Semaphore dependencies. Enable parallel execution with one config option.
+
+## Enable Parallel Execution
+
+Add `parallelExecution` to your flow config:
+
+```typescript
+const flow = yield* createFlowWithSchema({
+  flowId: 'my-flow',
+  name: 'My Flow',
+  nodes: [inputNode, processA, processB, mergeNode, outputNode],
+  edges: [
+    { source: 'input', target: 'processA' },
+    { source: 'input', target: 'processB' },
+    { source: 'processA', target: 'merge' },
+    { source: 'processB', target: 'merge' },
+    { source: 'merge', target: 'output' }
+  ],
+  inputSchema: myInputSchema,
+  outputSchema: myOutputSchema,
+  parallelExecution: {
+    enabled: true,
+    maxConcurrency: 4,
+  },
+});
+```
+
+## How It Works
+
+1. **Dependency Analysis**: Nodes are automatically grouped by their dependencies
+2. **Level-Based Execution**:
+   - Nodes with no dependencies (Level 0)
+   - Nodes depending on Level 0 (Level 1)
+   - And so on...
+3. **Parallel Within Levels**: Nodes at the same level run in parallel
+4. **Sequential Between Levels**: Levels run one after another
+
+### Example Flow
+
+```
+Input → [Resize ║ Optimize] → Merge → Output
+         └─ Level 1 (parallel) ─┘
+```
+
+- Input runs first
+- Resize and Optimize run in parallel
+- Merge waits for both to complete
+- Output runs last
+
+## Configuration
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `enabled` | `false` | Enable/disable parallel execution |
+| `maxConcurrency` | `4` | Max concurrent nodes per level |
+
+## What Changed
+
+### ParallelScheduler
+
+**Before** (broken):
+```typescript
+// This didn't work - Semaphore doesn't exist
+const resourceSemaphore = semaphore(maxConcurrency); // ERROR
+async executeNodesInParallel(executors) { ... }
+```
+
+**After** (working):
+```typescript
+// Uses Effect's Effect.all() with concurrency option
+executeNodesInParallel<T, E, R>(
+  nodeExecutors: Array<() => Effect.Effect<T, E, R>>,
+): Effect.Effect<T[], E, R> {
+  return Effect.all(nodeExecutors.map((executor) => executor()), {
+    concurrency: this.maxConcurrency,
+  });
+}
+```
+
+### Flow Engine
+
+Automatic detection of `parallelExecution.enabled`:
+- If `true`: Uses `ParallelScheduler` with level-based execution
+- If `false` (default): Uses original sequential execution
+
+## Performance Example
+
+```
+Sequential:  Input(1s) → Resize(2s) → Optimize(2s) → Merge(1s) → Output(1s) = 7s
+Parallel:    Input(1s) → [Resize(2s) ║ Optimize(2s)] → Merge(1s) → Output(1s) = 5s
+             Speedup: 28% faster
+```
+
+## Debugging
+
+Enable debug logs to see execution levels:
+
+```
+Flow my-flow: Executing in parallel mode (maxConcurrency: 4)
+Flow my-flow: Grouped nodes into 4 execution levels
+Flow my-flow: Executing level 0 with nodes: input_1
+Flow my-flow: Executing level 1 with nodes: resize_1, optimize_1
+Flow my-flow: Executing level 2 with nodes: merge_1
+Flow my-flow: Executing level 3 with nodes: output_1
+```
+
+## Introspection
+
+```typescript
+const scheduler = new ParallelScheduler({ maxConcurrency: 4 });
+
+// See how nodes are grouped
+const levels = scheduler.groupNodesByExecutionLevel(nodes, edges);
+levels.forEach(level => {
+  console.log(`Level ${level.level}: ${level.nodes.join(', ')}`);
+});
+
+// Get stats
+const stats = scheduler.getStats();
+console.log(`Max concurrency: ${stats.maxConcurrency}`);
+```
+
+## Backward Compatibility
+
+✅ Existing flows work unchanged (parallel execution disabled by default)
+✅ No breaking changes
+✅ Add config option to opt-in to parallel execution
+
+## Constraints
+
+- **Dependencies enforced**: Nodes can't run before their inputs are ready
+- **Level structure**: All nodes in a level must complete before next level starts
+- **Concurrency limit**: Won't exceed `maxConcurrency` simultaneous executions
+- **Error handling**: First error in a level stops the flow
+
+## For More Details
+
+See `/src/flow/PARALLEL_EXECUTION.md` for comprehensive documentation.

package/docs/PARALLEL_EXECUTION_REFACTOR.md

@@ -0,0 +1,184 @@
+# Parallel Execution Refactoring - Summary
+
+## Overview
+
+Successfully refactored the `ParallelScheduler` from a Promise-based implementation with undefined `Semaphore` dependencies to a modern Effect-based pattern aligned with the flow engine's architecture.
+
+## Changes Made
+
+### 1. **ParallelScheduler Refactored** (`src/flow/parallel-scheduler.ts`)
+
+#### Issues Fixed
+- ❌ Removed undefined `Semaphore` import and usage
+- ❌ Removed `semaphore()` function call (didn't exist)
+- ✅ Replaced with Effect's native concurrency primitives
+
+#### Key Changes
+- Converted `executeNodesInParallel()` from Promise-based to Effect-based
+- Now uses `Effect.all(nodeExecutors, { concurrency: maxConcurrency })`
+- Improved documentation with comprehensive JSDoc comments
+- All methods now properly typed with Effect generics
+
+```typescript
+// Before (broken):
+async executeNodesInParallel<T>(
+  nodeExecutors: Array<() => Promise<T>>,
+): Promise<T[]> {
+  const permit = await this.resourceSemaphore.acquire(); // ERROR: doesn't exist
+  // ...
+}
+
+// After (working):
+executeNodesInParallel<T, E, R>(
+  nodeExecutors: Array<() => Effect.Effect<T, E, R>>,
+): Effect.Effect<T[], E, R> {
+  return Effect.all(nodeExecutors.map((executor) => executor()), {
+    concurrency: this.maxConcurrency,
+  });
+}
+```
+
+### 2. **Flow Engine Integration** (`src/flow/flow.ts`)
+
+#### Added Parallel Execution Support
+- Imported `ParallelScheduler`
+- Added parallel execution path in `executeFlow` function
+- Maintains backward compatibility with sequential execution (default)
+
+#### Execution Strategy
+```typescript
+if (useParallelExecution) {
+  // Parallel execution using execution levels
+  const scheduler = new ParallelScheduler({ maxConcurrency: 4 });
+  const executionLevels = scheduler.groupNodesByExecutionLevel(nodes, edges);
+
+  // Execute each level sequentially, but nodes within level in parallel
+  for (const level of executionLevels) {
+    const levelResults = yield* scheduler.executeNodesInParallel(nodeExecutors);
+    // Process results...
+  }
+} else {
+  // Sequential execution (original behavior)
+  for (let i = startIndex; i < executionOrder.length; i++) {
+    // ...
+  }
+}
+```
+
+#### Key Features
+- ✅ Level-based execution (respects dependencies)
+- ✅ Configurable concurrency via `maxConcurrency`
+- ✅ Proper pause/resume handling in parallel mode
+- ✅ Debug logging at each level
+- ✅ Backward compatible (sequential is default)
+
+### 3. **Documentation Updated** (`src/flow/PARALLEL_EXECUTION.md`)
+
+Comprehensive guide covering:
+- ✅ Effect-based implementation details
+- ✅ Configuration options
+- ✅ Usage examples
+- ✅ Performance characteristics
+- ✅ Error handling
+- ✅ Debugging and monitoring
+- ✅ Future enhancements
+
+## Configuration
+
+Parallel execution is **disabled by default** and can be enabled per flow:
+
+```typescript
+const flow = yield* createFlowWithSchema({
+  flowId: 'image-pipeline',
+  name: 'Image Processing Pipeline',
+  nodes: [inputNode, resizeNode, optimizeNode, mergeNode, storageNode],
+  edges: [
+    { source: 'input', target: 'resize' },
+    { source: 'input', target: 'optimize' },
+    { source: 'resize', target: 'merge' },
+    { source: 'optimize', target: 'merge' },
+    { source: 'merge', target: 'storage' }
+  ],
+  inputSchema: myInputSchema,
+  outputSchema: myOutputSchema,
+  parallelExecution: {
+    enabled: true,
+    maxConcurrency: 4, // Limit concurrent nodes per level
+  },
+});
+```
+
+## Architecture
+
+### Execution Levels
+Nodes are automatically grouped into levels based on dependencies:
+
+```
+Level 0: [input]                    ← No dependencies
+Level 1: [resize, optimize]         ← Both depend on input
+Level 2: [merge]                    ← Depends on resize and optimize
+Level 3: [storage]                  ← Depends on merge
+```
+
+Within each level:
+- All nodes execute **in parallel**
+- Concurrency is limited by `maxConcurrency`
+- Uses Effect's `Effect.all()` for safe concurrent execution
+
+Between levels:
+- Execution is **strictly sequential**
+- Dependencies are guaranteed to complete before dependents run
+
+### Dependency Analysis
+Uses **Kahn's algorithm** for topological sorting:
+1. Build dependency graph from edges
+2. Calculate in-degree for each node
+3. Find nodes with in-degree 0
+4. Group into levels and update in-degrees
+5. Repeat until all nodes are processed
+
+## Testing
+
+✅ **Build Status**: All 27 packages compile successfully
+- `pnpm build` completes with 0 errors
+- No TypeScript diagnostics
+- ParallelScheduler and flow.ts both type-check cleanly
+
+## Backward Compatibility
+
+✅ **100% Backward Compatible**
+- Existing flows work without any changes
+- Sequential execution remains the default
+- No breaking changes to public APIs
+- `parallelExecution` is an optional config property
+
+## Migration Path
+
+1. **Current flows**: Continue working as-is (sequential by default)
+2. **Enable parallel execution**: Add `parallelExecution: { enabled: true }` to existing flows
+3. **Tune performance**: Adjust `maxConcurrency` based on your hardware and workload
+
+## Files Modified
+
+1. `src/flow/parallel-scheduler.ts` - Complete refactor to Effect-based API
+2. `src/flow/flow.ts` - Integrated parallel scheduler into executeFlow
+3. `src/flow/PARALLEL_EXECUTION.md` - Updated documentation
+4. ✅ All type definitions already in place (`src/flow/types/flow-types.ts`)
+
+## Benefits
+
+- ✅ **No External Dependencies**: Uses Effect's built-in concurrency
+- ✅ **Type Safe**: Full TypeScript support with proper Effect typing
+- ✅ **Resource Safe**: Concurrency limits prevent resource exhaustion
+- ✅ **Performance**: Independent nodes can run in parallel
+- ✅ **Maintainable**: Clean, well-documented code following Effect patterns
+- ✅ **Debuggable**: Built-in logging and scheduler introspection
+
+## Future Enhancements
+
+Potential improvements for future releases:
+- Dynamic concurrency adjustment based on resource availability
+- Per-node concurrency limits for resource-intensive operations
+- Parallel level visualization in flow builder UI
+- Performance metrics collection and reporting
+- Adaptive scheduling based on node execution times

package/package.json

@@ -0,0 +1,80 @@
+{
+  "name": "@uploadista/core",
+  "version": "0.0.2",
+  "description": "Core package of Uploadista",
+  "license": "MIT",
+  "author": "Uploadista",
+  "type": "module",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "default": "./dist/index.js"
+    },
+    "./errors": {
+      "types": "./dist/errors/index.d.ts",
+      "import": "./dist/errors/index.js",
+      "default": "./dist/errors/index.js"
+    },
+    "./types": {
+      "types": "./dist/types/index.d.ts",
+      "import": "./dist/types/index.js",
+      "default": "./dist/types/index.js"
+    },
+    "./flow": {
+      "types": "./dist/flow/index.d.ts",
+      "import": "./dist/flow/index.js",
+      "default": "./dist/flow/index.js"
+    },
+    "./upload": {
+      "types": "./dist/upload/index.d.ts",
+      "import": "./dist/upload/index.js",
+      "default": "./dist/upload/index.js"
+    },
+    "./logger/*": {
+      "types": "./dist/logger/*.d.ts",
+      "import": "./dist/logger/*.js",
+      "default": "./dist/logger/*.js"
+    },
+    "./streams/*": {
+      "types": "./dist/streams/*.d.ts",
+      "import": "./dist/streams/*.js",
+      "default": "./dist/streams/*.js"
+    },
+    "./utils/*": {
+      "types": "./dist/utils/*.d.ts",
+      "import": "./dist/utils/*.js",
+      "default": "./dist/utils/*.js"
+    },
+    "./websocket": {
+      "types": "./dist/websocket/index.d.ts",
+      "import": "./dist/websocket/index.js",
+      "default": "./dist/websocket/index.js"
+    }
+  },
+  "dependencies": {
+    "effect": "3.18.4",
+    "zod": "4.1.12"
+  },
+  "devDependencies": {
+    "@types/node": "24.8.1",
+    "tsdown": "0.15.7",
+    "vitest": "3.2.4",
+    "@uploadista/typescript-config": "0.0.2"
+  },
+  "publishConfig": {
+    "access": "public",
+    "registry": "https://registry.npmjs.org/"
+  },
+  "scripts": {
+    "build": "tsc -b",
+    "release": "tsdown && bumpp && pnpm publish --access restricted --no-git-checks",
+    "format": "biome format --write ./src",
+    "lint": "biome lint --write ./src",
+    "check": "biome check --write ./src",
+    "test": "vitest",
+    "test:run": "vitest run",
+    "test:watch": "vitest --watch",
+    "typecheck": "tsc --noEmit"
+  }
+}

package/src/errors/__tests__/uploadista-error.test.ts

@@ -0,0 +1,251 @@
+import { describe, expect, it } from "vitest";
+
+import {
+  ERROR_CATALOG,
+  httpFailure,
+  isUploadistaError,
+  UploadistaError,
+  type UploadistaErrorCode,
+} from "../uploadista-error";
+import { Effect } from "effect";
+
+describe("ERROR_CATALOG", () => {
+  it("should contain all error codes with status and body", () => {
+    const errorCodes: UploadistaErrorCode[] = [
+      "MISSING_OFFSET",
+      "ABORTED",
+      "INVALID_TERMINATION",
+      "ERR_LOCK_TIMEOUT",
+      "INVALID_CONTENT_TYPE",
+      "FLOW_STRUCTURE_ERROR",
+      "FLOW_CYCLE_ERROR",
+      "FLOW_NODE_NOT_FOUND",
+      "FLOW_NODE_ERROR",
+      "DATASTORE_NOT_FOUND",
+      "FILE_NOT_FOUND",
+      "INVALID_OFFSET",
+      "FILE_NO_LONGER_EXISTS",
+      "ERR_SIZE_EXCEEDED",
+      "ERR_MAX_SIZE_EXCEEDED",
+      "INVALID_LENGTH",
+      "INVALID_METADATA",
+      "UNKNOWN_ERROR",
+      "FILE_WRITE_ERROR",
+      "UPLOAD_ID_NOT_FOUND",
+      "FLOW_OUTPUT_VALIDATION_ERROR",
+      "FLOW_INPUT_VALIDATION_ERROR",
+    ];
+
+    for (const code of errorCodes) {
+      expect(ERROR_CATALOG[code]).toBeDefined();
+      expect(ERROR_CATALOG[code]).toHaveProperty("status");
+      expect(ERROR_CATALOG[code]).toHaveProperty("body");
+      expect(typeof ERROR_CATALOG[code].status).toBe("number");
+      expect(typeof ERROR_CATALOG[code].body).toBe("string");
+    }
+  });
+
+  it("should have appropriate HTTP status codes", () => {
+    expect(ERROR_CATALOG.MISSING_OFFSET.status).toBe(403);
+    expect(ERROR_CATALOG.FILE_NOT_FOUND.status).toBe(404);
+    expect(ERROR_CATALOG.INVALID_OFFSET.status).toBe(409);
+    expect(ERROR_CATALOG.FILE_NO_LONGER_EXISTS.status).toBe(410);
+    expect(ERROR_CATALOG.ERR_SIZE_EXCEEDED.status).toBe(413);
+    expect(ERROR_CATALOG.UNKNOWN_ERROR.status).toBe(500);
+  });
+
+  it("should have meaningful error messages", () => {
+    expect(ERROR_CATALOG.MISSING_OFFSET.body).toContain("Upload-Offset");
+    expect(ERROR_CATALOG.FILE_NOT_FOUND.body).toContain("file");
+    expect(ERROR_CATALOG.FLOW_CYCLE_ERROR.body).toContain("cycle");
+    expect(ERROR_CATALOG.ERR_MAX_SIZE_EXCEEDED.body).toContain("Maximum size");
+  });
+});
+
+describe("UploadistaError", () => {
+  it("should create error with all properties", () => {
+    const error = new UploadistaError({
+      code: "FILE_NOT_FOUND",
+      status: 404,
+      body: "File not found",
+      cause: new Error("underlying error"),
+      details: { fileId: "123" },
+    });
+
+    expect(error.name).toBe("UploadistaError");
+    expect(error.code).toBe("FILE_NOT_FOUND");
+    expect(error.status).toBe(404);
+    expect(error.status_code).toBe(404); // legacy alias
+    expect(error.body).toBe("File not found");
+    expect(error.message).toBe("File not found");
+    expect(error.details).toEqual({ fileId: "123" });
+    expect((error as { cause?: unknown }).cause).toBeInstanceOf(Error);
+  });
+
+  it("should create error without optional properties", () => {
+    const error = new UploadistaError({
+      code: "UNKNOWN_ERROR",
+      status: 500,
+      body: "Something went wrong",
+    });
+
+    expect(error.code).toBe("UNKNOWN_ERROR");
+    expect(error.status).toBe(500);
+    expect(error.body).toBe("Something went wrong");
+    expect(error.details).toBeUndefined();
+    expect((error as { cause?: unknown }).cause).toBeUndefined();
+  });
+
+  it("should inherit from Error", () => {
+    const error = new UploadistaError({
+      code: "FILE_NOT_FOUND",
+      status: 404,
+      body: "File not found",
+    });
+
+    expect(error).toBeInstanceOf(Error);
+    expect(error).toBeInstanceOf(UploadistaError);
+  });
+
+  describe("fromCode", () => {
+    it("should create error from error code", () => {
+      const error = UploadistaError.fromCode("FILE_NOT_FOUND");
+
+      expect(error.code).toBe("FILE_NOT_FOUND");
+      expect(error.status).toBe(ERROR_CATALOG.FILE_NOT_FOUND.status);
+      expect(error.body).toBe(ERROR_CATALOG.FILE_NOT_FOUND.body);
+    });
+
+    it("should allow overriding status and body", () => {
+      const error = UploadistaError.fromCode("FILE_NOT_FOUND", {
+        status: 400,
+        body: "Custom message",
+      });
+
+      expect(error.code).toBe("FILE_NOT_FOUND");
+      expect(error.status).toBe(400);
+      expect(error.body).toBe("Custom message");
+    });
+
+    it("should include details and cause when provided", () => {
+      const cause = new Error("root cause");
+      const error = UploadistaError.fromCode("UNKNOWN_ERROR", {
+        details: { extra: "info" },
+        cause: cause,
+      });
+
+      expect(error.details).toEqual({ extra: "info" });
+      expect((error as { cause?: unknown }).cause).toBe(cause);
+    });
+
+    it("should work with all error codes", () => {
+      const errorCodes: UploadistaErrorCode[] = [
+        "MISSING_OFFSET",
+        "ABORTED",
+        "INVALID_TERMINATION",
+        "ERR_LOCK_TIMEOUT",
+        "INVALID_CONTENT_TYPE",
+        "FLOW_STRUCTURE_ERROR",
+        "FLOW_CYCLE_ERROR",
+        "FLOW_NODE_NOT_FOUND",
+        "FLOW_NODE_ERROR",
+        "DATASTORE_NOT_FOUND",
+        "FILE_NOT_FOUND",
+        "INVALID_OFFSET",
+        "FILE_NO_LONGER_EXISTS",
+        "ERR_SIZE_EXCEEDED",
+        "ERR_MAX_SIZE_EXCEEDED",
+        "INVALID_LENGTH",
+        "INVALID_METADATA",
+        "UNKNOWN_ERROR",
+        "FILE_WRITE_ERROR",
+        "UPLOAD_ID_NOT_FOUND",
+        "FLOW_OUTPUT_VALIDATION_ERROR",
+        "FLOW_INPUT_VALIDATION_ERROR",
+      ];
+
+      for (const code of errorCodes) {
+        const error = UploadistaError.fromCode(code);
+        expect(error.code).toBe(code);
+        expect(error.status).toBe(ERROR_CATALOG[code].status);
+        expect(error.body).toBe(ERROR_CATALOG[code].body);
+      }
+    });
+  });
+
+  describe("toFailure", () => {
+    it("should convert error to failure result", () => {
+      const error = UploadistaError.fromCode("FILE_NOT_FOUND");
+      const result = error.toEffect();
+
+      expect(Effect.isFailure(result)).toBe(true);
+      expect(Effect.isSuccess(result)).toBe(false);
+    });
+  });
+});
+
+describe("isUploadistaError", () => {
+  it("should return true for UploadistaError instances", () => {
+    const error = UploadistaError.fromCode("FILE_NOT_FOUND");
+    expect(isUploadistaError(error)).toBe(true);
+  });
+
+  it("should return false for regular Error instances", () => {
+    const error = new Error("regular error");
+    expect(isUploadistaError(error)).toBe(false);
+  });
+
+  it("should return false for non-error values", () => {
+    expect(isUploadistaError("string")).toBe(false);
+    expect(isUploadistaError(null)).toBe(false);
+    expect(isUploadistaError(undefined)).toBe(false);
+    expect(isUploadistaError({})).toBe(false);
+    expect(isUploadistaError(123)).toBe(false);
+  });
+
+  it("should work as type guard", () => {
+    const error: unknown = UploadistaError.fromCode("FILE_NOT_FOUND");
+
+    if (isUploadistaError(error)) {
+      // TypeScript should know error is UploadistaError
+      expect(error.code).toBe("FILE_NOT_FOUND");
+      expect(error.status).toBe(404);
+    } else {
+      expect.fail("Should be UploadistaError");
+    }
+  });
+});
+
+describe("httpFailure", () => {
+  it("should create failure result from error code", () => {
+    const result = httpFailure("FILE_NOT_FOUND");
+
+    expect(Effect.isFailure(result)).toBe(true);
+    expect(Effect.isSuccess(result)).toBe(false);
+  });
+
+  it("should allow overriding error properties", () => {
+    const result = httpFailure("UNKNOWN_ERROR", {
+      status: 503,
+      body: "Service unavailable",
+      details: { retryAfter: 60 },
+    });
+
+    expect(Effect.isFailure(result)).toBe(true);
+    expect(Effect.isSuccess(result)).toBe(false);
+  });
+
+  it("should work with all error codes", () => {
+    const errorCodes: UploadistaErrorCode[] = [
+      "MISSING_OFFSET",
+      "FLOW_NODE_ERROR",
+      "INVALID_METADATA",
+    ];
+
+    for (const code of errorCodes) {
+      const result = httpFailure(code);
+      expect(Effect.isFailure(result)).toBe(true);
+      expect(Effect.isSuccess(result)).toBe(false);
+    }
+  });
+});

package/src/errors/index.ts

@@ -0,0 +1,2 @@
+export * from "./uploadista-error.js";
+export { ERROR_CATALOG as ERRORS } from "./uploadista-error.js";