mongodash 2.1.0 → 2.1.1

package/docs/reactive-tasks.md

@@ -4,45 +4,29 @@ A powerful, distributed task execution system built on top of [MongoDB Change St
 
 Reactive Tasks allow you to define background jobs that trigger automatically when your data changes. This enables **Model Data-Driven Flows**, where business logic is triggered by state changes (e.g., `status: 'paid'`) rather than explicit calls. The system handles **concurrency**, **retries**, **deduplication**, and **monitoring** out of the box.
 
-## Features
+## Overview
+
+### Features
 
 -   **Reactive**: Tasks triggered instantly (near-real-time) by database changes (insert/update).
 -   **Distributed**: Safe to run on multiple instances (Kubernetes/Serverless). Only one instance processes a specific task for a specific document at a time.
 -   **Efficient Listener**: Regardless of the number of application instances, **only one instance (the leader)** listens to the MongoDB Change Stream. This minimizes database load significantly (O(1) connections), though it implies that the total ingestion throughput is limited by the single leader instance.
 -   **Reliable**: Built-in retry mechanisms (exponential backoff) and "Dead Letter Queue" logic.
 -   **Efficient**: Uses MongoDB Driver for low-latency updates and avoids polling where possible.
--   **Memory Efficiency**: The system is designed to handle large datasets. During live scheduling (Change Streams), reconciliation, and periodic cleanup, the library only loads the `_id`'s of the source documents into memory, keeping the footprint low regardless of the collection size. Note that task *storage* size depends on your `watchProjection` configuration—see [Storage Optimization](#change-detection--storage-optimization).
+-   **Memory Efficiency**: The system is designed to handle large datasets. During live scheduling (Change Streams), reconciliation, and periodic cleanup, the library only loads the `_id`'s of the source documents into memory, keeping the footprint low regardless of the collection size. Note that task *storage* size depends on your `watchProjection` configuration—see [Storage Optimization](#change-detection-and-storage-optimization).
 -   **Observability**: First-class Prometheus metrics support.
 -   **Dashboard**: A visual [Dashboard](./dashboard.md) to monitor, retry, and debug tasks.
 
-## Architecture & Scalability
-
-The system uses a **Leader-Worker** architecture to balance efficiency and scalability.
-
-### 1. The Leader (Planner)
--   **Role**: A single instance is elected as the **Leader**.
--   **Responsibility**: It listens to the MongoDB Change Stream, calculates the necessary tasks (based on `watchProjection`), and persists them into the `_tasks` collection. To minimize memory usage, it only fetches the document `_id` from the Change Stream event.
-    > [!NOTE]
-    > **Database Resolution**: The Change Stream is established on the database of the **first registered reactive task**.
--   **Resilience**: Leadership is maintained via a distributed lock with a heartbeat. If the leader crashes, another instance automatically takes over (Failover).
-
-### 2. The Workers (Executors)
--   **Role**: *Every* instance (including the leader) runs a set of **Worker** threads (managed by the event loop).
--   **Responsibility**: Workers poll the `_tasks` collection for `pending` jobs, lock them, and execute the `handler`.
--   **Adaptive Polling**: Workers use an **adaptive polling** mechanism.
-    -   **Idle**: If no tasks are found, the polling frequency automatically lowers (saves CPU/IO).
-    -   **Busy**: If tasks are found (or the **local** Leader signals new work), the frequency speeds up immediately to process the queue as fast as possible. Workers on other instances will speed up once they independently find a task during their regular polling.
-
-## Reactive vs Scheduled Tasks
+### Reactive vs Scheduled Tasks
 
 It is important to distinguish between Reactive Tasks and standard schedulers (like Agenda or BullMQ).
 
 -   **Reactive Tasks (Reactors)**: Triggered by **state changes** (data). "When Order is Paid, send email". This guarantees consistency with data.
 -   **Schedulers**: Triggered by **time**. "Send email at 2:00 PM".
 
-Reactive Tasks support time-based operations via `debounceMs` (e.g., "Wait 1m after data change to settle") and `deferCurrent` (e.g., "Retry in 5m"), but they are fundamentally event-driven. If you need purely time-based jobs (e.g., "Daily Report" without any data change trigger), you can trigger them via a [Cron job](./cron-tasks.md), although you can model them as "Run on insert to 'daily_reports' collection".
+Reactive Tasks support time-based operations via `debounce` (e.g., "Wait 1m after data change to settle") and `deferCurrent` (e.g., "Retry in 5m"), but they are fundamentally event-driven. If you need purely time-based jobs (e.g., "Daily Report" without any data change trigger), you can trigger them via a [Cron job](./cron-tasks.md), although you can model them as "Run on insert to 'daily_reports' collection".
 
-## Advantages over Standard Messaging
+### Advantages over Standard Messaging
 
 Using Reactive Tasks instead of a traditional message broker (RabbitMQ, Kafka) provides distinct architectural benefits:
 
@@ -58,7 +42,6 @@ Using Reactive Tasks instead of a traditional message broker (RabbitMQ, Kafka) p
     -   The task queue is stored in a standard MongoDB collection (`[collection]_tasks`), not in a hidden broker queue.
     -   You can use standard tools (MongoDB Compass, Atlas Data Explorer, simple queries) to inspect pending jobs, debug failures, and analyze queue distribution without needing specialized queue management interfaces.
 
-
 ## Getting Started
 
 ### 1. Initialization
@@ -113,74 +96,7 @@ import { startReactiveTasks } from 'mongodash';
 await startReactiveTasks();
 ```
 
-### 4. Common Use Cases
-
-Reactive Tasks are versatile. Here are a few patterns you can implement:
-
-#### A. Webhook Delivery & Data Sync
-Perfect for reliable delivery of data to external systems. If the external API is down, Mongodash will automatically retry with exponential backoff.
-
-```typescript
-await reactiveTask({
-    task: 'sync-order-to-erp',
-    collection: 'orders',
-    filter: { status: 'paid' }, // Only sync when paid
-    watchProjection: { status: 1 },  // Only check when status changes
-    
-    handler: async (context) => {
-        const order = await context.getDocument();
-        await axios.post('https://erp-system.com/api/orders', order);
-    }
-});
-```
-
-#### B. Async Statistics Recalculation
-Offload heavy calculations from the main request path. When a raw document changes, update the aggregated view in the background.
-
-```typescript
-await reactiveTask({
-    task: 'recalc-product-rating',
-    collection: 'reviews',
-    debounce: '5s', // Re-calc at most once every 5 seconds per product
-    
-    handler: async (context) => {
-        // We only watched 'status', so we might need the full doc? 
-        // Or if we have the ID, that's enough for aggregation:
-        const { docId } = context; 
-
-        // Calculate new average
-        const stats = await calculateAverageRating(docId);
-        
-        // Update product document
-        await db.collection('products').updateOne(
-            { _id: docId },
-            { $set: { rating: stats.rating, reviewCount: stats.count } }
-        );
-    }
-});
-```
-
-#### C. Pub-Sub (Event Bus)
-Use Reactive Tasks as a distributed Event Bus. By creating an events collection and watching only the `_id`, you effectively create a listener that triggers **only on new insertions**.
-
-```typescript
-await reactiveTask({
-    task: 'send-welcome-sequence',
-    collection: 'app_events',
-    
-    // TRICK: _id never changes. 
-    // This config ensures the handler ONLY runs when a new document is inserted.
-    watchProjection: { _id: 1 }, 
-    filter: { type: 'user-registered' },
-    
-    handler: async (context) => {
-        const event = await context.getDocument();
-        await emailService.sendWelcome(event.payload.email);
-    }
-});
-```
-
-### 5. Advanced Initialization
+### 4. Advanced Configuration
 
 You can customize the scheduler behavior via `mongodash.init`:
 
@@ -239,6 +155,8 @@ await mongodash.init({
 });
 ```
 
+## Writing Tasks
+
 ### Task Options
 
 | Option | Type | Description |
@@ -247,35 +165,81 @@ await mongodash.init({
 | `collection` | `string` | **Required**. Name of the MongoDB collection to watch. |
 | `handler` | `(context) => Promise<void>` | **Required**. Async function to process the task. Use `context.getDocument()` to get the document. |
 | `filter` | `Document` | Standard Query (e.g., `{ status: 'pending' }`) OR Aggregation Expression (e.g., `{ $eq: ['$status', 'pending'] }`). Aggregation syntax unlocks powerful features like using `$$NOW` for time-based filtering. |
-| `watchProjection` | `Document` | MongoDB Projection. Task only triggers if the projected result changes. Supports inclusion `{ a: 1 }` and computed fields. |
+| `watchProjection` | `Document` | MongoDB Projection. Task only re-triggers if the projected result changes. Supports inclusion `{ a: 1 }` and computed fields. |
 | `debounce` | `number \| string` | Debounce window (ms or duration string). Default: `1000`. Useful to group rapid updates. |
 | `retryPolicy` | `RetryPolicy` | Configuration for retries on failure. |
 | `cleanupPolicy` | `CleanupPolicy` | Configuration for automatic cleanup of orphaned task records. See [Cleanup Policy](#cleanup-policy). |
 | `executionHistoryLimit` | `number` | Number of past execution entries to keep in `_tasks` doc. Default: `5`. |
-| `evolution` | `EvolutionConfig` | Configuration for handling task logic updates (versioning, reconciliation policies). |
+| `evolution` | `EvolutionConfig` | Configuration for handling task logic updates (versioning, reconciliation policies). See [Filter Evolution & Reconciliation](#filter-evolution-and-reconciliation). |
 
-### Change Detection & Storage Optimization
+### Common Use Cases
 
-To ensure reliability and efficiency, the system needs to determine *when* to trigger a task.
+Reactive Tasks are versatile. Here are a few patterns you can implement:
 
-**How it works:**
-1.  **State Persistence**: For every source document, a corresponding "task document" is stored in the `[collection]_tasks` collection.
-2.  **Snapshotting**: This task document holds a snapshot of the source document's fields (specifically, the result of `watchProjection`).
-3.  **Diffing**: When an event occurs (or during reconciliation), the system compares the current state of the document against the stored snapshot (`lastObservedValues`).
-4.  **No-Op**: If the watched fields haven't changed, **no task is triggered**. This guarantees reliability and prevents redundant processing.
+#### A. Webhook Delivery & Data Sync
+Perfect for reliable delivery of data to external systems. If the external API is down, Mongodash will automatically retry with exponential backoff.
 
-**Storage Implications:**
--   **Task Persistence**: The task document remains in the `_tasks` collection as long as the source document exists. It is only removed when the source document is deleted.
--   **Optimization**: If `watchProjection` is **not defined**, the system copies the **entire source document** into the task document.
--   **Recommendation**: For collections with **large documents** or **large datasets**, always define `watchProjection`. This significantly reduces storage usage and improves performance by copying only the necessary data subset.
--   **Tip**: If you want to trigger the task on *any* change but avoid storing the full document, watch a versioning field like `updatedAt`, `lastModifiedAt`, or `_version`.
-    ```typescript
-    // Triggers on any update (assuming your app updates 'updatedAt'), 
-    // but stores ONLY the 'updatedAt' date in the tasks collection.
-    watchProjection: { updatedAt: 1 } 
-    ```
+```typescript
+await reactiveTask({
+    task: 'sync-order-to-erp',
+    collection: 'orders',
+    filter: { status: 'paid' }, // Only sync when paid
+    watchProjection: { status: 1 },  // Only check when status changes
+    
+    handler: async (context) => {
+        const order = await context.getDocument();
+        await axios.post('https://erp-system.com/api/orders', order);
+    }
+});
+```
+
+#### B. Async Statistics Recalculation
+Offload heavy calculations from the main request path. When a raw document changes, update the aggregated view in the background.
+
+```typescript
+await reactiveTask({
+    task: 'recalc-product-rating',
+    collection: 'reviews',
+    debounce: '5s', // Re-calc at most once every 5 seconds per product
+    
+    handler: async (context) => {
+        // We only watched 'status', so we might need the full doc? 
+        // Or if we have the ID, that's enough for aggregation:
+        const { docId } = context; 
+
+        // Calculate new average
+        const stats = await calculateAverageRating(docId);
+        
+        // Update product document
+        await db.collection('products').updateOne(
+            { _id: docId },
+            { $set: { rating: stats.rating, reviewCount: stats.count } }
+        );
+    }
+});
+```
+
+#### C. Pub-Sub (Event Bus)
+Use Reactive Tasks as a distributed Event Bus. By creating an events collection and watching only the `_id`, you effectively create a listener that triggers **only on new insertions**.
+
+```typescript
+await reactiveTask({
+    task: 'send-welcome-sequence',
+    collection: 'app_events',
+    
+    // TRICK: _id never changes. 
+    // This config ensures the handler ONLY runs when a new document is inserted.
+    watchProjection: { _id: 1 }, 
+    filter: { type: 'user-registered' },
+    
+    handler: async (context) => {
+        const event = await context.getDocument();
+        await emailService.sendWelcome(event.payload.email);
+    }
+});
+```
 
-### Late-Binding Filter Check & `getDocument`
+### The Handler Context: `getDocument` & Safety Checks
 
 Critically, the library performs a **runtime check** when you call `await context.getDocument()` inside your handler.
 
@@ -342,142 +306,132 @@ handler: async (context) => {
 }
 ```
 
+### Flow Control (Defer / Throttle)
 
+Sometimes you need dynamic control over task execution speed based on external factors (e.g., rate limits of a 3rd party API) or business logic.
 
-### Cleanup Policy
-
-The Cleanup Policy controls automatic deletion of orphaned task records — tasks whose source documents have been deleted or no longer match the configured filter.
-
-#### Configuration
-
-```typescript
-cleanupPolicy?: {
-    deleteWhen?: 'sourceDocumentDeleted' | 'sourceDocumentDeletedOrNoLongerMatching' | 'never';
-    keepFor?: string | number;
-}
-```
-
-| Property | Type | Default | Description |
-|----------|------|---------|-------------|
-| `deleteWhen` | `string` | `'sourceDocumentDeleted'` | When to trigger task deletion |
-| `keepFor` | `string \| number` | `'24h'` | Grace period before deletion (e.g., `'1h'`, `'7d'`, or `86400000` ms) |
-
-#### Deletion Strategies (`deleteWhen`)
-
-| Strategy | Behavior |
-|----------|----------|
-| `sourceDocumentDeleted` | **Default.** Task deleted only when its source document is deleted from the database. Filter mismatches are ignored. |
-| `sourceDocumentDeletedOrNoLongerMatching` | Task deleted when source document is deleted **OR** when it no longer matches the task's `filter`. Useful for cases the change of document is permament and it is not expected the document could match in the future again and retrigger because of that. Also useful for  `$$NOW`-based or dynamic filters. |
-| `never` | Tasks are never automatically deleted. Use for audit trails or manual cleanup scenarios. |
-
-#### Grace Period Calculation
-
-The `keepFor` grace period is measured from `MAX(updatedAt, lastFinalizedAt)`:
+The `handler` receives a `context` object that exposes flow control methods.
 
-- **`updatedAt`**: When the source document's watched fields (`watchProjection`) last changed
-- **`lastFinalizedAt`**: When a worker last completed or failed the task
+#### 1. Deferral (`deferCurrent`)
 
-This ensures tasks are protected if either:
-1. The source data changed recently, OR
-2. A worker processed the task recently
+Delays the **current** task execution. The task is put back into the queue specifically for this document and will not be picked up again until the specified time.
 
-#### Example: Dynamic Filter Cleanup
+This is useful for:
+*   **Rate Limits**: "API returned 429, try again in 30 seconds."
+*   **Business Waits**: "Customer created, but wait 1 hour before sending first email."
 
 ```typescript
 await reactiveTask({
-    task: 'remind-pending-order',
-    collection: 'orders',
-    // Match orders pending for more than 24 hours
-    filter: { $expr: { $gt: ['$$NOW', { $add: ['$createdAt', 24 * 60 * 60 * 1000] }] } },
-    
-    cleanupPolicy: {
-        deleteWhen: 'sourceDocumentDeletedOrNoLongerMatching',
-        keepFor: '1h', // Keep it at least 1 hour after last scheduled matching or finalization
-    },
-    
-    handler: async (order) => { /* Send reminder email */ }
+    task: 'send-webhook',
+    collection: 'events',
+    handler: async (context) => {
+        const doc = await context.getDocument();
+        try {
+            await sendWebhook(doc);
+        } catch (err) {
+            if (err.status === 429) {
+                const retryAfter = err.headers['retry-after'] || 30; // seconds
+                
+                // Defer THIS task only. 
+                // It resets status to 'pending' and schedules it for future.
+                // It does NOT increment attempt count (it's not a failure).
+                context.deferCurrent(retryAfter * 1000); 
+                return;
+            }
+            throw err; // Use standard retry policy for other errors
+        }
+    }
 });
 ```
 
-#### Scheduler-Level Configuration
+#### 2. Throttling (`throttleAll`)
 
-Control how often the cleanup runs using `reactiveTaskCleanupInterval` in scheduler options. Cleanup is performed in **batches** (default 1000 items) to ensure stability on large datasets.
+Pauses all FUTURE tasks of this type for a specified duration. This serves as a "Circuit Breaker" when an external system (e.g., CRM, Payment Gateway) is unresponsive or returns overload errors (503, 429).
 
 ```typescript
-scheduler.configure({
-    reactiveTaskCleanupInterval: '12h', // Run cleanup every 12 hours (default: '24h')
-});
+context.throttleAll(60 * 1000); // Pause this task type for 1 minute
 ```
 
-Supported formats:
-- Duration string: `'1h'`, `'24h'`, `'7d'`
-- Milliseconds: `86400000`
-- Cron expression: `'CRON 0 3 * * *'` (e.g., daily at 3 AM)
+> [!IMPORTANT]
+> **Cluster Behavior (Instance-Local)**
+> `throttleAll` operates only in the memory of the current instance (worker).
+> In a distributed environment (e.g., Kubernetes with multiple pods), other instances will not know about the issue immediately. They will continue processing until they independently encounter the error and trigger their own `throttleAll`.
+>
+> **Result**: The load on the external service will not drop to zero immediately but will decrease gradually as individual instances hit the "circuit breaker".
 
+> [!NOTE]
+> **Current Task**
+> `throttleAll` does not affect the currently running task. If you want to postpone the current task (so it counts as pending and retries after the pause), you must explicitly call `deferCurrent()`.
 
-### Filter Evolution & Reconciliation
-
-Reactive Tasks are designed to evolve with your application. As you deploy new versions of your code, you might change the `filter`, `watchProjection`, or the `handler` logic itself. The system automatically detects these changes and adapts the task state accordingly.
-
-You can control this behavior using the optional `evolution` configuration:
+**Example (Service Down):**
 
 ```typescript
 await reactiveTask({
-    task: 'process-order',
-    collection: 'orders',
-    filter: { status: 'paid', amount: { $gt: 100 } },
-    
-    // Logic Versioning
-    evolution: {
-        // Increment this when you change the handler code and want to re-process tasks
-        handlerVersion: 2, 
-        
-        // What to do when version increments?
-        // - 'none': Do nothing (default).
-        // - 'reprocess_failed': Reset all 'failed' tasks to 'pending' to retry with new code.
-        // - 'reprocess_all': Reset ALL tasks (even completed ones) to 'pending'.
-        onHandlerVersionChange: 'reprocess_failed',
-        
-        // If 'filter' or 'watchProjection' changes, should we run reconciliation?
-        // Default: true
-        reconcileOnTriggerChange: true
-    },
+    task: 'sync-to-crm',
+    collection: 'users',
+    handler: async (context) => {
+        // Note: You can throttle even before fetching the doc if you know the service is down!
+        try {
+            const doc = await context.getDocument();
+            await crmApi.update(doc);
+        } catch (err) {
+            // If service is unavailable (503) or circuit breaker is open
+            if (err.status === 503 || err.isCircuitBreakerOpen) {
+                console.warn('CRM is down, pausing tasks for 1 minute.');
 
-    handler: async (order) => { /* ... */ }
+                // 1. Stop processing future tasks of this type on this instance
+                context.throttleAll(60 * 1000);
+
+                // 2. Defer the CURRENT task so it retries after the pause
+                context.deferCurrent(60 * 1000);
+                return;
+            }
+            throw err; // Standard retry policy for other errors
+        }
+    }
 });
 ```
 
-#### 1. Trigger Evolution (Filter / Projection)
+### Idempotency & Re-execution
 
-When the scheduler starts, it compares the current `filter` and `watchProjection` with the stored configuration from the previous deployment.
+The system is designed with an **At-Least-Once** execution guarantee. This is a fundamental property of distributed systems that value reliability over "exactly-once".
 
-*   **Narrowing the Filter** (e.g., `amount > 50` → `amount > 100`):
-    *   **Pending Tasks**: Workers will pick up pending tasks. Before execution, they perform a "Late-Binding Check". If the document no longer matches the new filter (e.g. amount is 75), the task is **skipped** (completed) without running the handler.
-    *   **Existing Tasks**: Tasks for documents that no longer match are **not deleted** immediately; they remain as history but won't satisfy the filter for future updates. See the cleanup policies for more details.
+While the system strives to execute your handler exactly once per event, there are specific scenarios where it might execute multiple times for the same document state. Therefore, **your `handler` must be idempotent**.
 
-*   **Widening the Filter** (e.g., `amount > 100` → `amount > 50`):
-    *   **Reconciliation**: The system detects the filter change and automatically triggers a **Reconciliation** scan for this specific task.
-    *   **Backfilling**: It scans the collection for documents that *now* match the new filter (e.g. amount 75) but don't have a task yet. It schedules new tasks for them immediately.
-    *   *Note*: This ensures specific newly-matched documents get processed without needing a manual migration script.
+#### Common Re-execution Scenarios
 
-    > [!WARNING]
-    > **Dynamic Filters (e.g., `$$NOW`)**: If your filter uses time-based expressions to "widen" the range automatically over time (e.g. `{ $expr: { $lte: ['$releaseDate', '$$NOW'] } }`), this does **NOT** trigger reconciliation. The scheduler only detects changes to the *filter definition object*. Documents that match purely because time has passed (without a data change) will **not** be picked up. For time-based triggers, use a [Cron Task](./cron-tasks.md).
+1.  **Transient Failures (Retries)**: If a worker crashes or loses network connectivity during execution (before marking the task `completed`), the lock will expire. Another worker will pick up the task and retry it.
+2.  **Reconciliation Recovery**: If task records are deleted (e.g. manual cleanup) but source documents remain, once a reconciliation runs, it recreates them as `pending`.
+3.  **Filter Re-matching** If a document is no longer matching the task filter, the task is deleted because the **sourceDocumentDeletedOrNoLongerMatching** cleanup policy is used and then the document is changed back again to match the task filter, the task will be recreated as `pending`.
+4.  **Explicit Reprocessing**: You might trigger re-execution manually (via `retryReactiveTasks`) or through schema evolution policies (`reprocess_all`).
 
-#### 2. Logic Evolution (Handler Versioning)
+#### Designing Idempotent Handlers
 
-Sometimes you fix a bug in your handler and want to retry failed tasks, or you implement a new feature (e.g. generic data migration) and want to re-run the task for *every* document.
+Ensure your handler allows multiple executions without adverse side effects.
 
-*   **Versioning**: Increment `evolution.handlerVersion` (integer, default 1).
-*   **Policies (`onHandlerVersionChange`)**:
-    *   `'none'`: The system acknowledges the new version but doesn't touch existing task states. New executions will use the new code.
-    *   `'reprocess_failed'`: Finds all tasks currently in `failed` status and resets them to `pending` (resetting attempts count). Useful for bug fixes.
-    *   `'reprocess_all'`: Resets **ALL** tasks (failed, completed) to `pending`. Useful for migrations or re-calculating data for the entire dataset.
+**Example**:
+```typescript
+handler: async (context) => {
+    // 1. Fetch document (with verification)
+    const order = await context.getDocument();
 
-> [!TIP]
-> Use `reprocess_failed` for bug fixes and `reprocess_all` sparingly for data migrations. The system automatically handles the "reset" operation efficiently using database-side updates.
+    // 2. Check if the work is already done
+    if (order.emailSent) return;
+    
+    // 3. Perform the side-effect
+    await sendEmail(order.userId, "Order Received");
+    
+    // 4. Mark as done (using atomic update)
+    await db.collection('orders').updateOne(
+        { _id: order._id }, 
+        { $set: { emailSent: true } }
+    );
+}
+```
 
-### Retry Policy
+## Policies & Lifecycle
 
+### Retry Policy
 
 You can configure the retry behavior using the `retryPolicy` option.
 
@@ -487,7 +441,7 @@ You can configure the retry behavior using the `retryPolicy` option.
 | :--- | :--- | :--- | :--- |
 | `type` | `string` | **Required** | `'fixed'`, `'linear'`, `'exponential'`, `'series'`, or `'cron'` |
 | `maxAttempts` | `number` | `5`* | Maximum total attempts (use `-1` for unlimited). |
-| `maxDuration` | `string \| number` | `undefined` | Stop retrying if elapsed time exceeds this value. |
+| `maxDuration` | `string \| number` | `undefined` | Stop retrying if elapsed time since the **first failure** in the current sequence exceeds this value. |
 | `resetRetriesOnDataChange` | `boolean` | `true` | Reset attempt count if the source document changes. |
 
 *\* If `maxDuration` is specified, `maxAttempts` defaults to unlimited.*
@@ -504,7 +458,7 @@ You can configure the retry behavior using the `retryPolicy` option.
 | **`series`** | `intervals` | - | Array of fixed delays (e.g., `['1m', '5m', '15m']`). |
 | **`cron`** | `expression` | - | Standard cron string for scheduling retries. |
 
-### Examples
+#### Examples
 
 ```typescript
 // 1. Give up after 24 hours (infinite attempts within that window)
@@ -538,95 +492,139 @@ retryPolicy: {
 }
 ```
 
-### Flow Control (Defer / Throttle)
+### Cleanup Policy
 
-Sometimes you need dynamic control over task execution speed based on external factors (e.g., rate limits of a 3rd party API) or business logic.
+The Cleanup Policy controls automatic deletion of orphaned task records — tasks whose source documents have been deleted or no longer match the configured filter.
 
-The `handler` receives a `context` object that exposes flow control methods.
+#### Configuration
 
-#### 1. Deferral (`deferCurrent`)
+```typescript
+cleanupPolicy?: {
+    deleteWhen?: 'sourceDocumentDeleted' | 'sourceDocumentDeletedOrNoLongerMatching' | 'never';
+    keepFor?: string | number;
+}
+```
 
-Delays the **current** task execution. The task is put back into the queue specifically for this document and will not be picked up again until the specified time.
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `deleteWhen` | `string` | `'sourceDocumentDeleted'` | When to trigger task deletion |
+| `keepFor` | `string \| number` | `'24h'` | Grace period before deletion (e.g., `'1h'`, `'7d'`, or `86400000` ms) |
 
-This is useful for:
-*   **Rate Limits**: "API returned 429, try again in 30 seconds."
-*   **Business Waits**: "Customer created, but wait 1 hour before sending first email."
+#### Deletion Strategies (`deleteWhen`)
+
+| Strategy | Behavior |
+|----------|----------|
+| `sourceDocumentDeleted` | **Default.** Task deleted only when its source document is deleted from the database. Filter mismatches are ignored. |
+| `sourceDocumentDeletedOrNoLongerMatching` | Task deleted when source document is deleted **OR** when it no longer matches the task's `filter`. Useful for cases the change of document is permament and it is not expected the document could match in the future again and retrigger because of that. Also useful for  `$$NOW`-based or dynamic filters. |
+| `never` | Tasks are never automatically deleted. Use for audit trails or manual cleanup scenarios. |
+
+#### Grace Period Calculation
+
+The `keepFor` grace period is measured from `MAX(updatedAt, lastFinalizedAt)`:
+
+- **`updatedAt`**: When the source document's watched fields (`watchProjection`) last changed
+- **`lastFinalizedAt`**: When a worker last completed or failed the task
+
+This ensures tasks are protected if either:
+1. The source data changed recently, OR
+2. A worker processed the task recently
+
+#### Example: Dynamic Filter Cleanup
 
 ```typescript
 await reactiveTask({
-    task: 'send-webhook',
-    collection: 'events',
-    handler: async (context) => {
-        const doc = await context.getDocument();
-        try {
-            await sendWebhook(doc);
-        } catch (err) {
-            if (err.status === 429) {
-                const retryAfter = err.headers['retry-after'] || 30; // seconds
-                
-                // Defer THIS task only. 
-                // It resets status to 'pending' and schedules it for future.
-                // It does NOT increment attempt count (it's not a failure).
-                context.deferCurrent(retryAfter * 1000); 
-                return;
-            }
-            throw err; // Use standard retry policy for other errors
-        }
-    }
+    task: 'remind-pending-order',
+    collection: 'orders',
+    // Match orders pending for more than 24 hours
+    filter: { $expr: { $gt: ['$$NOW', { $add: ['$createdAt', 24 * 60 * 60 * 1000] }] } },
+    
+    cleanupPolicy: {
+        deleteWhen: 'sourceDocumentDeletedOrNoLongerMatching',
+        keepFor: '1h', // Keep it at least 1 hour after last scheduled matching or finalization
+    },
+    
+    handler: async (order) => { /* Send reminder email */ }
 });
 ```
 
-#### 2. Throttling (`throttleAll`)
+#### Scheduler-Level Configuration
 
-Pauses all FUTURE tasks of this type for a specified duration. This serves as a "Circuit Breaker" when an external system (e.g., CRM, Payment Gateway) is unresponsive or returns overload errors (503, 429).
+Control how often the cleanup runs using `reactiveTaskCleanupInterval` in scheduler options. Cleanup is performed in **batches** (default 1000 items) to ensure stability on large datasets.
 
 ```typescript
-context.throttleAll(60 * 1000); // Pause this task type for 1 minute
+await mongodash.init({
+    // ...
+    reactiveTaskCleanupInterval: '12h', // Run cleanup every 12 hours (default: '24h')
+});
 ```
 
-> [!IMPORTANT]
-> **Cluster Behavior (Instance-Local)**
-> `throttleAll` operates only in the memory of the current instance (worker).
-> In a distributed environment (e.g., Kubernetes with multiple pods), other instances will not know about the issue immediately. They will continue processing until they independently encounter the error and trigger their own `throttleAll`.
->
-> **Result**: The load on the external service will not drop to zero immediately but will decrease gradually as individual instances hit the "circuit breaker".
+Supported formats:
+- Duration string: `'1h'`, `'24h'`, `'7d'`
+- Milliseconds: `86400000`
+- Cron expression: `'CRON 0 3 * * *'` (e.g., daily at 3 AM)
 
-> [!NOTE]
-> **Current Task**
-> `throttleAll` does not affect the currently running task. If you want to postpone the current task (so it counts as pending and retries after the pause), you must explicitly call `deferCurrent()`.
+### Filter Evolution and Reconciliation
 
-**Example (Service Down):**
+Reactive Tasks are designed to evolve with your application. As you deploy new versions of your code, you might change the `filter`, `watchProjection`, or the `handler` logic itself. The system automatically detects these changes and adapts the task state accordingly.
+
+You can control this behavior using the optional `evolution` configuration:
 
 ```typescript
 await reactiveTask({
-    task: 'sync-to-crm',
-    collection: 'users',
-    handler: async (context) => {
-        // Note: You can throttle even before fetching the doc if you know the service is down!
-        try {
-            const doc = await context.getDocument();
-            await crmApi.update(doc);
-        } catch (err) {
-            // If service is unavailable (503) or circuit breaker is open
-            if (err.status === 503 || err.isCircuitBreakerOpen) {
-                console.warn('CRM is down, pausing tasks for 1 minute.');
-
-                // 1. Stop processing future tasks of this type on this instance
-                context.throttleAll(60 * 1000);
+    task: 'process-order',
+    collection: 'orders',
+    filter: { status: 'paid', amount: { $gt: 100 } },
+    
+    // Logic Versioning
+    evolution: {
+        // Increment this when you change the handler code and want to re-process tasks
+        handlerVersion: 2, 
+        
+        // What to do when version increments?
+        // - 'none': Do nothing (default).
+        // - 'reprocess_failed': Reset all 'failed' tasks to 'pending' to retry with new code.
+        // - 'reprocess_all': Reset ALL tasks (even completed ones) to 'pending'.
+        onHandlerVersionChange: 'reprocess_failed',
+        
+        // If 'filter' or 'watchProjection' changes, should we run reconciliation?
+        // Default: true
+        reconcileOnTriggerChange: true
+    },
 
-                // 2. Defer the CURRENT task so it retries after the pause
-                context.deferCurrent(60 * 1000);
-                return;
-            }
-            throw err; // Standard retry policy for other errors
-        }
-    }
+    handler: async (order) => { /* ... */ }
 });
 ```
 
+#### 1. Trigger Evolution (Filter / Projection)
 
+When the scheduler starts, it compares the current `filter` and `watchProjection` with the stored configuration from the previous deployment.
 
-### Reconciliation & Reliability
+*   **Narrowing the Filter** (e.g., `amount > 50` → `amount > 100`):
+    *   **Pending Tasks**: Workers will pick up pending tasks. Before execution, they perform a "Late-Binding Check". If the document no longer matches the new filter (e.g. amount is 75), the task is **skipped** (completed) without running the handler.
+    *   **Existing Tasks**: Tasks for documents that no longer match are **not deleted** immediately; they remain as history but won't satisfy the filter for future updates. See the cleanup policies for more details.
+
+*   **Widening the Filter** (e.g., `amount > 100` → `amount > 50`):
+    *   **Reconciliation**: The system detects the filter change and automatically triggers a **Reconciliation** scan for this specific task.
+    *   **Backfilling**: It scans the collection for documents that *now* match the new filter (e.g. amount 75) but don't have a task yet. It schedules new tasks for them immediately.
+    *   *Note*: This ensures specific newly-matched documents get processed without needing a manual migration script.
+
+    > [!WARNING]
+    > **Dynamic Filters (e.g., `$$NOW`)**: If your filter uses time-based expressions to "widen" the range automatically over time (e.g. `{ $expr: { $lte: ['$releaseDate', '$$NOW'] } }`), this does **NOT** trigger reconciliation. The scheduler only detects changes to the *filter definition object*. Documents that match purely because time has passed (without a data change) will **not** be picked up. For time-based triggers, use a [Cron Task](./cron-tasks.md).
+
+#### 2. Logic Evolution (Handler Versioning)
+
+Sometimes you fix a bug in your handler and want to retry failed tasks, or you implement a new feature (e.g. generic data migration) and want to re-run the task for *every* document.
+
+*   **Versioning**: Increment `evolution.handlerVersion` (integer, default 1).
+*   **Policies (`onHandlerVersionChange`)**:
+    *   `'none'`: The system acknowledges the new version but doesn't touch existing task states. New executions will use the new code.
+    *   `'reprocess_failed'`: Finds all tasks currently in `failed` status and resets them to `pending` (resetting attempts count). Useful for bug fixes.
+    *   `'reprocess_all'`: Resets **ALL** tasks (failed, completed) to `pending`. Useful for migrations or re-calculating data for the entire dataset.
+
+> [!TIP]
+> Use `reprocess_failed` for bug fixes and `reprocess_all` sparingly for data migrations. The system automatically handles the "reset" operation efficiently using database-side updates.
+
+#### Reconciliation & Reliability
 
 The system includes a self-healing mechanism called **Reconciliation**.
 
@@ -658,60 +656,113 @@ Reconciliation is **persistent and resilient**.
     -   If the `watchProjection` hasn't changed since the last run (comparing `lastObservedValues`), the task is **not** re-triggered.
     -   **Recommendation**: Carefully configure `filter` and `watchProjection` to minimize unnecessary processing during reconciliation.
 
-## Idempotency & Re-execution
+### Change Detection and Storage Optimization
 
-The system is designed with an **At-Least-Once** execution guarantee. This is a fundamental property of distributed systems that value reliability over "exactly-once".
+To ensure reliability and efficiency, the system needs to determine *when* to trigger a task.
 
-While the system strives to execute your handler exactly once per event, there are specific scenarios where it might execute multiple times for the same document state. Therefore, **your `handler` must be idempotent**.
+**How it works:**
+1.  **State Persistence**: For every source document, a corresponding "task document" is stored in the `[collection]_tasks` collection.
+2.  **Snapshotting**: This task document holds a snapshot of the source document's fields (specifically, the result of `watchProjection`).
+3.  **Diffing**: When an event occurs (or during reconciliation), the system compares the current state of the document against the stored snapshot (`lastObservedValues`).
+4.  **No-Op**: If the watched fields haven't changed, **no task is triggered**. This guarantees reliability and prevents redundant processing.
+
+**Storage Implications:**
+-   **Task Persistence**: The task document remains in the `_tasks` collection as long as the source document exists. It is only removed when the source document is deleted.
+-   **Optimization**: If `watchProjection` is **not defined**, the system copies the **entire source document** into the task document.
+-   **Recommendation**: For collections with **large documents** or **large datasets**, always define `watchProjection`. This significantly reduces storage usage and improves performance by copying only the necessary data subset.
+-   **Tip**: If you want to trigger the task on *any* change but avoid storing the full document, watch a versioning field like `updatedAt`, `lastModifiedAt`, or `_version`.
+    ```typescript
+    // Triggers on any update (assuming your app updates 'updatedAt'), 
+    // but stores ONLY the 'updatedAt' date in the tasks collection.
+    watchProjection: { updatedAt: 1 } 
+    ```
 
-### Common Re-execution Scenarios
+## Operations & Monitoring
 
-1.  **Transient Failures (Retries)**: If a worker crashes or loses network connectivity during execution (before marking the task `completed`), the lock will expire. Another worker will pick up the task and retry it.
-2.  **Reconciliation Recovery**: If task records are deleted (e.g. manual cleanup) but source documents remain, once a reconciliation runs, it recreates them as `pending`.
-3.  **Filter Re-matching** If a document is no longer matching the task filter, the task is deleted because the **sourceDocumentDeletedOrNoLongerMatching** cleanup policy is used and then the document is changed back again to match the task filter, the task will be recreated as `pending`.
-4.  **Explicit Reprocessing**: You might trigger re-execution manually (via `retryReactiveTasks`) or through schema evolution policies (`reprocess_all`).
+### Task Management & DLQ
 
-### Designing Idempotent Handlers
+You can programmatically manage tasks, investigate failures, and handle Dead Letter Queues (DLQ) using the exported management API.
 
-Ensure your handler allows multiple executions without adverse side effects.
+These functions allow you to build custom admin UIs or automated recovery workflows.
+
+#### Listing Tasks
+
+Use `getReactiveTasks` to inspect the queue. You can filter by task name, status, error message, or properties of the **source document**.
 
-**Example**:
 ```typescript
-handler: async (context) => {
-    // 1. Fetch document (with verification)
-    const order = await context.getDocument();
+import { getReactiveTasks } from 'mongodash';
 
-    // 2. Check if the work is already done
-    if (order.emailSent) return;
-    
-    // 3. Perform the side-effect
-    await sendEmail(order.userId, "Order Received");
-    
-    // 4. Mark as done (using atomic update)
-    await db.collection('orders').updateOne(
-        { _id: order._id }, 
-        { $set: { emailSent: true } }
-    );
-}
+// list currently failed tasks
+const failedTasks = await getReactiveTasks({ 
+    task: 'send-welcome-email', 
+    status: 'failed' 
+});
+
+// list with pagination
+const page1 = await getReactiveTasks(
+    { task: 'send-welcome-email' }, 
+    { limit: 50, skip: 0, sort: { scheduledAt: -1 } }
+);
+
+// Advanced: Helper to find task by properties of the SOURCE document
+// This is powerful: "Find the task associated with Order #123"
+const orderTasks = await getReactiveTasks({
+    task: 'sync-order',
+    sourceDocFilter: { _id: 'order-123' }
+});
+
+// Advanced: Find tasks where source document matches complex filter
+// "Find sync tasks for all VIP users"
+const vipTasks = await getReactiveTasks({
+    task: 'sync-order',
+    sourceDocFilter: { isVip: true } 
+});
 ```
 
+#### Counting Tasks
 
+Use `countReactiveTasks` for metrics or UI badges.
 
-## Task States & Lifecycle
+```typescript
+import { countReactiveTasks } from 'mongodash';
 
-Every task record in the `_tasks` collection follows a specific lifecycle:
+const dlqSize = await countReactiveTasks({ 
+    task: 'send-welcome-email', 
+    status: 'failed' 
+});
+```
 
-| Status | Description |
-| :--- | :--- |
-| `pending` | Task is waiting to be processed by a worker. This is the initial state after scheduling or a re-trigger. |
-| `processing` | Task is currently locked and being worked on by an active worker instance. |
-| `processing_dirty` | **Concurrency Guard.** New data was detected while the worker was already processing the previous state. The task will be reset to `pending` immediately after the current run finishes to ensure no updates are missed. |
-| `completed` | Task was processed successfully or it was not matching the filter during the last attempt. |
-| `failed` | Task permanently failed after exceeding all retries or the `maxDuration` window. |
+#### Retrying Tasks
+
+Use `retryReactiveTasks` to manually re-trigger tasks. This is useful for DLQ recovery after fixing a bug.
+
+This operation is **concurrency-safe**. If a task is currently `processing`, it will be marked to re-run immediately after the current execution finishes (`processing_dirty`), ensuring no race conditions.
+
+```typescript
+import { retryReactiveTasks } from 'mongodash';
+
+// Retry ALL failed tasks for a specific job
+const result = await retryReactiveTasks({ 
+    task: 'send-welcome-email', 
+    status: 'failed' 
+});
+console.log(`Retried ${result.modifiedCount} tasks.`);
 
----
+// Retry specific task by Source Document ID
+await retryReactiveTasks({
+    task: 'sync-order',
+    sourceDocFilter: { _id: 'order-123' }
+});
 
-## Monitoring
+// Bulk Retry: Retry all tasks for "VIP" orders
+// This efficiently finds matching tasks and schedules them for execution.
+await retryReactiveTasks({
+    task: 'sync-order',
+    sourceDocFilter: { isVip: true }
+});
+```
+
+### Monitoring
 
 Mongodash provides built-in Prometheus metrics to monitor your reactive tasks.
 
@@ -721,7 +772,7 @@ Mongodash provides built-in Prometheus metrics to monitor your reactive tasks.
 > npm install prom-client
 > ```
 
-### Configuration
+#### Configuration
 
 Monitoring is configured in the initialization options under the `monitoring` key:
 
@@ -730,8 +781,8 @@ await mongodash.init({
     // ...
     monitoring: {
         enabled: true,           // Default: true
-        pushIntervalMs: 60000,   // How often instances synchronize metrics (default: 1m)
         scrapeMode: 'cluster',   // 'cluster' (default) or 'local'
+        pushIntervalMs: 60000,   // How often instances synchronize metrics (default: 1m). Relevant only if scrapeMode is 'cluster'.
         readPreference: 'secondaryPreferred' // 'primary', 'secondaryPreferred' etc.
     }
 });
@@ -741,7 +792,7 @@ await mongodash.init({
     - `'cluster'` (Default): Returns aggregated system-wide metrics. Any instance can respond to this request (by fetching state from the DB). It aggregates metrics from all other active instances. (Recommended for Load Balancers / Heroku)
     - `'local'`: Returns local metrics for THIS instance. If this instance is the Leader, it ALSO includes Global System Metrics (Queue Depth, Lag) so they are reported exactly once in the cluster. (Recommended for K8s Pod Monitors)
 
-### Retrieving Metrics
+#### Retrieving Metrics
 
 Expose the metrics endpoint (e.g., in Express):
 
@@ -760,7 +811,7 @@ app.get('/metrics', async (req, res) => {
 });
 ```
 
-### Available Metrics
+#### Available Metrics
 
 The system exposes the following metrics with standardized labels:
 
@@ -773,7 +824,7 @@ The system exposes the following metrics with standardized labels:
 | `reactive_tasks_change_stream_lag_seconds` | Gauge | *none* | Time difference between now and the last processed Change Stream event. |
 | `reactive_tasks_last_reconciliation_timestamp_seconds` | Gauge | *none* | Timestamp when the last full reconciliation (recovery) finished. |
 
-### Grafana Dashboard
+#### Grafana Dashboard
 
 A comprehensive **Grafana Dashboard** ("Reactive Tasks - System Overview") is included with the package.
 
@@ -788,90 +839,7 @@ You can find the dashboard JSON file at:
 
 Import this file directly into Grafana to get started.
 
-## Task Management & DLQ
-
-You can programmatically manage tasks, investigate failures, and handle Dead Letter Queues (DLQ) using the exported management API.
-
-These functions allow you to build custom admin UIs or automated recovery workflows.
-
-### Listing Tasks
-
-Use `getReactiveTasks` to inspect the queue. You can filter by task name, status, error message, or properties of the **source document**.
-
-```typescript
-import { getReactiveTasks } from 'mongodash';
-
-// list currently failed tasks
-const failedTasks = await getReactiveTasks({ 
-    task: 'send-welcome-email', 
-    status: 'failed' 
-});
-
-// list with pagination
-const page1 = await getReactiveTasks(
-    { task: 'send-welcome-email' }, 
-    { limit: 50, skip: 0, sort: { scheduledAt: -1 } }
-);
-
-// Advanced: Helper to find task by properties of the SOURCE document
-// This is powerful: "Find the task associated with Order #123"
-const orderTasks = await getReactiveTasks({
-    task: 'sync-order',
-    sourceDocFilter: { _id: 'order-123' }
-});
-
-// Advanced: Find tasks where source document matches complex filter
-// "Find sync tasks for all VIP users"
-const vipTasks = await getReactiveTasks({
-    task: 'sync-order',
-    sourceDocFilter: { isVip: true } 
-});
-```
-
-### Counting Tasks
-
-Use `countReactiveTasks` for metrics or UI badges.
-
-```typescript
-import { countReactiveTasks } from 'mongodash';
-
-const dlqSize = await countReactiveTasks({ 
-    task: 'send-welcome-email', 
-    status: 'failed' 
-});
-```
-
-### Retrying Tasks
-
-Use `retryReactiveTasks` to manually re-trigger tasks. This is useful for DLQ recovery after fixing a bug.
-
-This operation is **concurrency-safe**. If a task is currently `processing`, it will be marked to re-run immediately after the current execution finishes (`processing_dirty`), ensuring no race conditions.
-
-```typescript
-import { retryReactiveTasks } from 'mongodash';
-
-// Retry ALL failed tasks for a specific job
-const result = await retryReactiveTasks({ 
-    task: 'send-welcome-email', 
-    status: 'failed' 
-});
-console.log(`Retried ${result.modifiedCount} tasks.`);
-
-// Retry specific task by Source Document ID
-await retryReactiveTasks({
-    task: 'sync-order',
-    sourceDocFilter: { _id: 'order-123' }
-});
-
-// Bulk Retry: Retry all tasks for "VIP" orders
-// This efficiently finds matching tasks and schedules them for execution.
-await retryReactiveTasks({
-    task: 'sync-order',
-    sourceDocFilter: { isVip: true }
-});
-```
-
-## Graceful Shutdown
+### Graceful Shutdown
 
 When shutting down your application, call `stopReactiveTasks()` in your termination signal handlers to ensure in-progress tasks complete and resources are released cleanly.
 
@@ -912,3 +880,34 @@ process.on('SIGINT', () => gracefulShutdown('SIGINT'));   // Ctrl+C
 > [!NOTE]
 > **Self-Healing Design**: While graceful shutdown is recommended best practice, the system is designed to be resilient. If your application crashes or is forcefully terminated, task locks will automatically expire after a timeout (default: 1 minute), allowing other instances to pick up and process the unfinished tasks. Similarly, leadership locks expire, ensuring another instance takes over. This guarantees eventual task processing even in failure scenarios.
 
+## Architecture & Internals
+
+### Architecture & Scalability
+
+The system uses a **Leader-Worker** architecture to balance efficiency and scalability.
+
+#### 1. The Leader (Planner)
+-   **Role**: A single instance is elected as the **Leader**.
+-   **Responsibility**: It listens to the MongoDB Change Stream, calculates the necessary tasks (based on `watchProjection`), and persists them into the `_tasks` collection. To minimize memory usage, it only fetches the document `_id` from the Change Stream event.
+    > [!NOTE]
+    > **Database Resolution**: The Change Stream is established on the database of the **first registered reactive task**.
+-   **Resilience**: Leadership is maintained via a distributed lock with a heartbeat. If the leader crashes, another instance automatically takes over (Failover).
+
+#### 2. The Workers (Executors)
+-   **Role**: *Every* instance (including the leader) runs a set of **Worker** threads (managed by the event loop).
+-   **Responsibility**: Workers poll the `_tasks` collection for `pending` jobs, lock them, and execute the `handler`.
+-   **Adaptive Polling**: Workers use an **adaptive polling** mechanism.
+    -   **Idle**: If no tasks are found, the polling frequency automatically lowers (saves CPU/IO).
+    -   **Busy**: If tasks are found (or the **local** Leader signals new work), the frequency speeds up immediately to process the queue as fast as possible. Workers on other instances will speed up once they independently find a task during their regular polling.
+
+### Task States & Lifecycle
+
+Every task record in the `_tasks` collection follows a specific lifecycle:
+
+| Status | Description |
+| :--- | :--- |
+| `pending` | Task is waiting to be processed by a worker. This is the initial state after scheduling or a re-trigger. |
+| `processing` | Task is currently locked and being worked on by an active worker instance. |
+| `processing_dirty` | **Concurrency Guard.** New data was detected while the worker was already processing the previous state. The task will be reset to `pending` immediately after the current run finishes to ensure no updates are missed. |
+| `completed` | Task was processed successfully or it was not matching the filter during the last attempt. |
+| `failed` | Task permanently failed after exceeding all retries or the `maxDuration` window. |