@decaf-ts/core 0.28.1 → 0.30.0

package/README.md

@@ -11,7 +11,7 @@ Decaf Core provides the foundational building blocks for the Decaf TypeScript ec
 *   **`Repository`**: A class that implements the repository pattern, providing a consistent API for CRUD operations and querying.
 *   **`Adapter`**: An abstract class that defines the interface for connecting to different database backends.
 *   **`Statement`**: A query builder for creating complex database queries in a fluent, type-safe manner.
-*   **`TaskEngine`**: A system for managing background jobs and asynchronous operations.
+*   **`TaskEngine`**: A system for managing background jobs and asynchronous operations, including composite-step concurrency controls.
 *   **`ModelService` and `PersistenceService`**: Base classes for creating services that encapsulate business logic and data access.
 *   **Migrations**: A system for managing database schema changes over time.
 *   **RAM Adapter**: An in-memory adapter for testing and development.
@@ -39,7 +39,7 @@ Decaf Core provides the foundational building blocks for the Decaf TypeScript ec
 
 Documentation [here](https://decaf-ts.github.io/injectable-decorators/), Test results [here](https://decaf-ts.github.io/injectable-decorators/workdocs/reports/html/test-report.html) and Coverage [here](https://decaf-ts.github.io/injectable-decorators/workdocs/reports/coverage/lcov-report/index.html)
 
-Minimal size: 47.5 KB kb gzipped
+Minimal size: 48 KB kb gzipped
 
 
 # Core Package — Detailed Description
@@ -48,7 +48,7 @@ The Decaf Core package provides a cohesive set of primitives for building strong
 
 - Models (from @decaf-ts/decorator-validation) enhanced with identity and persistence metadata.
 - A Repository abstraction that encapsulates CRUD, querying, and observation.
-- A powerful Task Engine for defining, scheduling, and executing background jobs with support for worker threads.
+- A powerful Task Engine for defining, scheduling, and executing background jobs with support for worker threads and composite-step concurrency controls.
 - Adapters that bridge repositories to underlying storage (in-memory, HTTP, TypeORM, etc.).
 - A fluent Query DSL (Statement/Condition) with pagination.
 - Lightweight dependency injection utilities to auto-resolve repositories.
@@ -88,7 +88,7 @@ Below is an overview of the main modules and their public APIs exposed by core.
 
 ## 2. Task Engine Module
 A robust system for managing background jobs.
-- **`TaskEngine<A>`**: The core engine that polls for and executes tasks. Manages the task lifecycle, concurrency, and worker threads.
+- **`TaskEngine<A>`**: The core engine that polls for and executes tasks. Manages the task lifecycle, global runnable-task concurrency, composite-step concurrency, and worker threads.
 - **`TaskService<A>`**: A high-level service providing a clean API for interacting with the `TaskEngine`. It's the recommended entry point for managing tasks.
   - `push(task, track?)`: Submits a new task for execution.
   - `schedule(task, track?).for(date)`: Schedules a task to run at a specific time.
@@ -96,14 +96,18 @@ A robust system for managing background jobs.
 - **Models**:
   - `TaskModel`: Represents a task, its status (`PENDING`, `RUNNING`, `SUCCEEDED`, `FAILED`), input, and configuration (e.g., `maxAttempts`, `backoff`). Can be `ATOMIC` or `COMPOSITE`.
   - `TaskEventModel`: Logs status changes and progress for a task.
+  - `TaskStepSpecModel`: Describes composite steps, including `classification`, `name`, `lock`, `dependsOn`, and `allowConcurrent` for opt-in parallel execution of compatible steps.
 - **Builders**:
-  - `TaskBuilder`: A fluent API for constructing `TaskModel` instances.
+  - `TaskBuilder`: A fluent API for constructing atomic `TaskModel` instances.
   - `CompositeTaskBuilder`: A builder for creating multi-step (`COMPOSITE`) tasks.
+  - `TaskStepSpecBuilder`: A fluent API for constructing composite step specs, including `setAllowConcurrent(false)` by default.
 - **Handlers & Tracking**:
   - `ITaskHandler`: The interface to implement for defining the logic of a task. Handlers are registered with the `TaskHandlerRegistry`.
   - `TaskTracker`: An object returned when tracking a task, allowing you to await its completion and receive progress updates.
 - **Worker Threads**: The engine can be configured to run tasks in Node.js `worker_threads`, providing true parallelism and non-blocking execution for CPU-intensive jobs. Configuration is done via the `workerPool` and `workerAdapter` properties in the `TaskEngineConfig`.
 
+Composite-step concurrency is opt-in at the step level. `allowConcurrent` exists on `TaskStepSpecBuilder`, defaults to `false`, and only affects composite steps that share the same `lock`. The per-task `maxConcurrentCompositeSteps` limit defaults to `-1` and is separate from the engine-wide `concurrency` limit for runnable tasks. The engine also uses a shared task-context write lock so logs and results remain serialized while compatible steps run in parallel.
+
 ## 3. Persistence Module
 - **`Adapter<N, Q, R, Ctx>`**: The bridge between a repository and the back-end storage.
   - Handles CRUD operations, raw queries, and model/record transformation (`prepare`/`revert`).
@@ -430,7 +434,7 @@ const activeUsers = await userService.findActiveUsers();
 
 ## Task Engine
 
-The `TaskEngine` is a powerful tool for managing background jobs.
+The `TaskEngine` is a powerful tool for managing background jobs, including composite-step concurrency controls.
 
 ### Creating a Task Handler
 
@@ -518,6 +522,7 @@ await taskEngine.start();
 | `bus` | Optional `TaskEventBus` that receives progress/log/status events. |
 | `workerId` | Uniquely identifies the worker claiming leases. Each engine (including CLI migrations) must use a different `workerId` so leases do not clash. |
 | `concurrency` | Number of work units to execute in parallel (set to `1` when migration steps must stay sequential). |
+| `maxConcurrentCompositeSteps` | Maximum number of composite steps that may run at once for a single composite task. Defaults to `-1`, which means no limit. |
 | `leaseMs` | How long a running task can go without a heartbeat before it is re-queued. |
 | `pollMsIdle` | Poll interval when the queue is empty. |
 | `pollMsBusy` | Poll interval while tasks are running (shorter than `pollMsIdle`). |
@@ -534,6 +539,9 @@ await taskEngine.start();
 - `pipe(...log)` and `flush()`: buffer logs that eventually feed into `TaskEventType.LOG`.
 - `heartbeat()`: extend the lease before it expires (used in long-running handlers).
 - `scheduleCompositeSteps(...)`: dynamically insert extra steps when building migration tasks.
+- `stepWriteLock`: a shared lock used by composite steps to serialize writes to the task context, so logs and results stay consistent even when compatible steps run concurrently.
+
+Composite-step concurrency is opt-in and step-scoped. `allowConcurrent` lives on `TaskStepSpecBuilder`, defaults to `false`, and only affects composite steps that share the same `lock`. The per-task `maxConcurrentCompositeSteps` limit defaults to `-1` and is independent from the engine-wide `concurrency` limit for runnable tasks.
 
 ### Task Engine migration guardrails
 
@@ -560,7 +568,7 @@ The CLI already follows this pattern and explicitly prevents the task engine ada
 
 `TaskService.track(id)` wires the CLI logger to the matching `TaskTracker` so status/progress logs stream through your console before `TaskTracker.wait()` resolves. If a migration task fails, call `MigrationService.retry(taskId)`—it uses repository overrides to reset `status` to `PENDING`, clear `error`/lease metadata, and re-queue the work—then `taskService.track(id)` again so the TaskEngine reclaims it.
 
-Composite tasks are ordered by the sequence you pass to `CompositeTaskBuilder` or by using the `dependsOn`/`dependencies` array. Each step has a `classification` (matching a handler), an optional `name`, and `lock`/`dependsOn` metadata (`TaskStepSpecModel`). Locks avoid concurrent execution, and dependencies support either `<taskId>` or `<taskId>:<stepRef>` shorthand so you can mix tasks and steps as prerequisites.
+Composite tasks are ordered by the sequence you pass to `CompositeTaskBuilder` or by using the `dependsOn`/`dependencies` array. Each step has a `classification` (matching a handler), an optional `name`, and `lock`/`dependsOn` metadata (`TaskStepSpecModel`). Use `TaskStepSpecBuilder.setAllowConcurrent(true)` to let compatible steps that share the same lock value run together; the default remains `false`. The engine still serializes context writes through the shared task-context lock so logs and results are persisted safely. Dependencies support either `<taskId>` or `<taskId>:<stepRef>` shorthand so you can mix tasks and steps as prerequisites.
 
 Task attempts are bounded by `maxAttempts` and `backoff` (configured via builders). The engine records each attempt and automatically escalates to `WAITING_RETRY`/`RUNNING` states; if a task exhausts retries, the service surfaces the final error via `TaskTracker.wait()` so your migration command can decide between retrying or aborting.