mongodash 2.1.0 → 2.1.2

package/docs/reactive-tasks/guides.md

@@ -0,0 +1,237 @@
+# Guides & Patterns
+
+
+
+## Idempotency & Re-execution
+
+The system is designed with an **At-Least-Once** execution guarantee. This is a fundamental property of distributed systems that value reliability over "exactly-once".
+
+While the system strives to execute your handler exactly once per event, there are specific scenarios where it might execute multiple times for the same document state. Therefore, **your `handler` must be idempotent**.
+
+### Common Re-execution Scenarios
+
+1.  **Transient Failures ([Retries](./policy-retry.md))**: If a worker crashes or loses network connectivity during execution (before marking the task `completed`), the lock will expire. Another worker will pick up the task and retry it.
+2.  **[Reconciliation](./reconciliation.md) Recovery**: If task records are deleted (e.g. manual cleanup) but source documents remain, once a reconciliation runs, it recreates them as `pending`.
+3.  **Filter Re-matching** If a document is no longer matching the task filter, the task is deleted because the **[sourceDocumentDeletedOrNoLongerMatching](./policy-cleanup.md)** cleanup policy is used and then the document is changed back again to match the task filter, the task will be recreated as `pending`.
+4.  **Explicit Reprocessing**: You might trigger re-execution manually (via `retryReactiveTasks`) or through [schema evolution policies](./evolution.md) (`reprocess_all`).
+
+### Designing Idempotent Handlers
+
+Ensure your handler allows multiple executions without adverse side effects.
+
+**Example**:
+```typescript
+handler: async (context) => {
+    // 1. Fetch document (with verification)
+    const order = await context.getDocument();
+
+    // 2. Check if the work is already done
+    if (order.emailSent) return;
+    
+    // 3. Perform the side-effect
+    await sendEmail(order.userId, "Order Received");
+    
+    // 4. Mark as done (using atomic update)
+    await db.collection('orders').updateOne(
+        { _id: order._id }, 
+        { $set: { emailSent: true } }
+    );
+}
+```
+
+
+## The Handler Context
+
+### `getDocument` & Safety Checks
+
+Critically, the library performs a **runtime check** when you call `await context.getDocument()` inside your handler.
+
+1.  **Lock Task**: The worker locks the task.
+2.  **Fetch & Verify**: When you call `await context.getDocument()`, it performs an atomic fetch that ensures:
+    *   **Filter Match**: The document still matches your `filter` configuration.
+    *   **Data Consistency**: The watched fields (`watchProjection`) have NOT changed since the task was triggered (Optimistic Locking).
+    *   **Existence**: The document still exists.
+
+    If any of these conditions fail, `getDocument` throws a `TaskConditionFailedError`. The worker catches this error effectively **skipping** the task and marking it as 'completed'.
+
+**Why is this important?**
+*   **Race Conditions**: Imagine a "Back-In-Stock" task triggered when `inventory > 0`. If the item sells out immediately (`inventory` returns to `0`) *while* the task is waiting in the queue, this check prevents sending a false notification.
+*   **Optimistic Concurrency**: If the data changed significantly (e.g. `status` changed from `paid` to `refunded`) between trigger and execution, the task is skipped to effectively "cancel" the stale operation. A new task for the new state (`refunded`) will likely be in the queue anyway.
+
+#### Advanced Usage: Options & Transactions
+
+The `getDocument(options)` method accepts standard MongoDB `FindOptions`, allowing you to optimize performance or ensure consistency.
+
+**1. Projections (Partial Fetch)**
+If your source document is large but you only need a few fields, use `projection`.
+
+```typescript
+const user = await context.getDocument({ 
+    projection: { email: 1, firstName: 1 } 
+});
+```
+
+**2. Transactions (`session`)**
+To ensure atomic updates across multiple collections, pass a `session` to `getDocument`. This ensures that the document fetch and your subsequent writes happen within the same transaction snapshot.
+
+```typescript
+import { withTransaction } from 'mongodash';
+
+handler: async (context) => {
+    await withTransaction(async (session) => {
+        // Pass session to getDocument to participate in the transaction
+        const doc = await context.getDocument({ session });
+
+        // Perform other operations in the same transaction
+        await otherCollection.updateOne({ _id: doc.refId }, { $set: { ... } }, { session });
+    });
+}
+```
+
+**3. Locking Resources (`withLock`)**
+While the *task itself* is locked (ensuring only one worker processes this specific task instance), you might need to lock shared resources if your handler accesses data outside the source document.
+
+You can use `context.watchedValues` to get IDs needed for locking *before* you fetch the document.
+
+```typescript
+import { withLock } from 'mongodash';
+
+handler: async (context) => {
+    // Use watchedValues to get the ID for locking
+    const accountId = context.watchedValues.accountId;
+
+    // Lock a shared resource
+    await withLock(`account-update-${accountId}`, async () => {
+        // Now it is safe to fetch and process
+        const doc = await context.getDocument();
+        // ... safe exclusive access to the account ...
+    });
+}
+```
+
+### Flow Control (Defer / Throttle)
+
+Sometimes you need dynamic control over task execution speed based on external factors (e.g., rate limits of a 3rd party API) or business logic.
+
+The `handler` receives a `context` object that exposes flow control methods.
+
+#### 1. Deferral (`deferCurrent`)
+
+Delays the **current** task execution. The task is put back into the queue specifically for this document and will not be picked up again until the specified time.
+
+This is useful for:
+*   **Rate Limits**: "API returned 429, try again in 30 seconds."
+*   **Business Waits**: "Customer created, but wait 1 hour before sending first email."
+
+```typescript
+await reactiveTask({
+    task: 'send-webhook',
+    collection: 'events',
+    handler: async (context) => {
+        const doc = await context.getDocument();
+        try {
+            await sendWebhook(doc);
+        } catch (err) {
+            if (err.status === 429) {
+                const retryAfter = err.headers['retry-after'] || 30; // seconds
+                
+                // Defer THIS task only. 
+                // It resets status to 'pending' and schedules it for future.
+                // It does NOT increment attempt count (it's not a failure).
+                context.deferCurrent(retryAfter * 1000); 
+                return;
+            }
+            throw err; // Use standard retry policy for other errors
+        }
+    }
+});
+```
+
+#### 2. Throttling (`throttleAll`)
+
+Pauses all FUTURE tasks of this type for a specified duration. This serves as a "Circuit Breaker" when an external system (e.g., CRM, Payment Gateway) is unresponsive or returns overload errors (503, 429).
+
+```typescript
+context.throttleAll(60 * 1000); // Pause this task type for 1 minute
+```
+
+> [!IMPORTANT]
+> **Cluster Behavior (Instance-Local)**
+> `throttleAll` operates only in the memory of the current instance (worker).
+> In a distributed environment (e.g., Kubernetes with multiple pods), other instances will not know about the issue immediately. They will continue processing until they independently encounter the error and trigger their own `throttleAll`.
+>
+> **Result**: The load on the external service will not drop to zero immediately but will decrease gradually as individual instances hit the "circuit breaker".
+
+> [!NOTE]
+> **Current Task**
+> `throttleAll` does not affect the currently running task. If you want to postpone the current task (so it counts as pending and retries after the pause), you must explicitly call `deferCurrent()`.
+
+**Example (Service Down):**
+
+```typescript
+await reactiveTask({
+    task: 'sync-to-crm',
+    collection: 'users',
+    handler: async (context) => {
+        // Note: You can throttle even before fetching the doc if you know the service is down!
+        try {
+            const doc = await context.getDocument();
+            await crmApi.update(doc);
+        } catch (err) {
+            // If service is unavailable (503) or circuit breaker is open
+            if (err.status === 503 || err.isCircuitBreakerOpen) {
+                console.warn('CRM is down, pausing tasks for 1 minute.');
+
+                // 1. Stop processing future tasks of this type on this instance
+                context.throttleAll(60 * 1000);
+
+                // 2. Defer the CURRENT task so it retries after the pause
+                context.deferCurrent(60 * 1000);
+                return;
+            }
+            throw err; // Standard retry policy for other errors
+        }
+    }
+});
+```
+
+## Graceful Shutdown
+
+When shutting down your application, call `stopReactiveTasks()` in your termination signal handlers to ensure in-progress tasks complete and resources are released cleanly.
+
+**Recommended Pattern:**
+
+```typescript
+import { stopReactiveTasks } from 'mongodash';
+
+const gracefulShutdown = async (signal: string) => {
+  console.log(`${signal} received, shutting down...`);
+  
+  // Set timeout to force exit if shutdown hangs
+  const timeout = setTimeout(() => {
+    console.error('Shutdown timeout, forcing exit');
+    process.exit(1);
+  }, 30000);
+  
+  try {
+    await stopReactiveTasks();  // Stop tasks BEFORE closing DB
+    await server.close();        // Close your HTTP server
+    await db.disconnect();       // Close database connection
+    
+    clearTimeout(timeout);
+    process.exit(0);
+  } catch (err) {
+    console.error('Shutdown error:', err);
+    process.exit(1);
+  }
+};
+
+process.on('SIGTERM', () => gracefulShutdown('SIGTERM')); // Docker, K8s
+process.on('SIGINT', () => gracefulShutdown('SIGINT'));   // Ctrl+C
+```
+
+> [!IMPORTANT]
+> Always call `stopReactiveTasks()` **before** closing database connections, as the stop process needs to communicate with MongoDB.
+
+> [!NOTE]
+> **Self-Healing Design**: While graceful shutdown is recommended best practice, the system is designed to be resilient. If your application crashes or is forcefully terminated, task locks will automatically expire after a timeout (default: 1 minute), allowing other instances to pick up and process the unfinished tasks. Similarly, leadership locks expire, ensuring another instance takes over. This guarantees eventual task processing even in failure scenarios.

package/docs/reactive-tasks/index.md

@@ -0,0 +1,44 @@
+# Reactive Tasks
+
+A powerful, distributed task execution system built on top of [MongoDB Change Streams](https://www.mongodb.com/docs/manual/changeStreams/).
+
+Reactive Tasks allow you to define background jobs that trigger automatically when your data changes. This enables **Model Data-Driven Flows**, where business logic is triggered by state changes (e.g., `status: 'paid'`) rather than explicit calls. The system handles **concurrency**, **retries**, **deduplication**, and **monitoring** out of the box.
+
+## Features
+
+-   **Reactive**: Tasks triggered instantly (near-real-time) by database changes (insert/update).
+-   **Distributed**: Safe to run on multiple instances (Kubernetes/Serverless). Only one instance processes a specific task for a specific document at a time.
+-   **Efficient Listener**: Regardless of the number of application instances, **only one instance (the leader)** listens to the MongoDB Change Stream. This minimizes database load significantly (**O(1) connections**), though it implies that the total ingestion throughput is limited by the single leader instance.
+-   **[Reliable Retries](./policy-retry.md)**: Built-in retry mechanisms (exponential backoff) and "Dead Letter Queue" logic.
+-   **Efficient**: Uses MongoDB Driver for low-latency updates and avoids polling where possible.
+-   **Memory Efficiency**: The system is designed to handle large datasets. During live scheduling (Change Streams), reconciliation, and periodic cleanup, the library **only loads the `_id`'s** of the source documents into memory, keeping the footprint low regardless of the collection size. Note that task *storage* size depends on your `watchProjection` configuration—see [Storage Optimization](./core-concepts.md#change-detection-and-storage-optimization).
+-   **[Concurrency Control](./configuration.md)**: Limit parallel execution to protect downstream resources.
+-   **[Deduplication](./guides.md#idempotency--re-execution)**: Automatic debouncing ("wait for data to settle") and task merging.
+-   **[Observability](./monitoring.md)**: First-class Prometheus metrics support.
+-   **[Dashboard](../dashboard.md)**: A visual Dashboard to monitor, retry, and debug tasks.
+-   **Developer Friendly**: Zero-config local development, fully typed with TypeScript.
+
+## Reactive vs Scheduled Tasks
+
+It is important to distinguish between Reactive Tasks and standard schedulers (like Agenda or BullMQ).
+
+-   **Reactive Tasks (Reactors)**: Triggered by **state changes** (data). "When Order is Paid, send email". This guarantees consistency with data.
+-   **Schedulers**: Triggered by **time**. "Send email at 2:00 PM".
+
+Reactive Tasks support time-based operations via `debounce` (e.g., "Wait 1m after data change to settle") and `deferCurrent` (e.g., "Retry in 5m"), but they are fundamentally event-driven. If you need purely time-based jobs (e.g., "Daily Report" without any data change trigger), you can trigger them via a [Cron job](../cron-tasks.md), although you can model them as "Run on insert to 'daily_reports' collection".
+
+## Advantages over Standard Messaging
+
+Using Reactive Tasks instead of a traditional message broker (RabbitMQ, Kafka) provides distinct architectural benefits:
+
+1.  **Lean Stack & Simplified DevOps**:
+    -   Eliminates the need to manage, scale, and secure external message brokers.
+    -   **Zero-Config Development**: Local testing requires only the database connection—no extra Docker containers or infrastructure to spin up.
+
+2.  **Transactional Consistency (Solving the "Dual Write" Problem)**:
+    -   *The Problem*: In standard architectures, writing to the database and publishing an event are two separate operations. If the database write succeeds but the message flush fails (network error, crash), your system enters an inconsistent state.
+    -   *The Solution*: With Reactive Tasks, the "event" **is** the database write. The task is triggered electronically by the MongoDB Oplog. This guarantees that **if and only if** data is persisted, the corresponding task will be scheduled—ensuring 100% data consistency without distributed transactions.
+
+3.  **Inspectable State**:
+    -   The task queue is stored in a standard MongoDB collection (`[collection]_tasks`), not in a hidden broker queue.
+    -   You can use standard tools (MongoDB Compass, Atlas Data Explorer, simple queries) to inspect pending jobs, debug failures, and analyze queue distribution without needing specialized queue management interfaces.

package/docs/reactive-tasks/management.md

@@ -0,0 +1,86 @@
+# Task Management & DLQ
+
+You can programmatically manage tasks, investigate failures, and handle Dead Letter Queues (DLQ) using the exported management API.
+
+These functions allow you to build custom admin UIs or automated recovery workflows.
+
+> [!TIP]
+> **Dashboard Available**
+> While this page describes the programmatic API, Mongodash also provides a **[Visual Dashboard](../dashboard.md)** (GUI) which wraps these methods in a user-friendly interface. The dashboard allows you to view task lists, filter by status/error, retry failed tasks, and trigger cron jobs without writing any code.
+
+## Listing Tasks
+
+Use `getReactiveTasks` to inspect the queue. You can filter by task name, status, error message, or properties of the **source document**.
+
+```typescript
+import { getReactiveTasks } from 'mongodash';
+
+// list currently failed tasks
+const failedTasks = await getReactiveTasks({ 
+    task: 'send-welcome-email', 
+    status: 'failed' 
+});
+
+// list with pagination
+const page1 = await getReactiveTasks(
+    { task: 'send-welcome-email' }, 
+    { limit: 50, skip: 0, sort: { scheduledAt: -1 } }
+);
+
+// Advanced: Helper to find task by properties of the SOURCE document
+// This is powerful: "Find the task associated with Order #123"
+const orderTasks = await getReactiveTasks({
+    task: 'sync-order',
+    sourceDocFilter: { _id: 'order-123' }
+});
+
+// Advanced: Find tasks where source document matches complex filter
+// "Find sync tasks for all VIP users"
+const vipTasks = await getReactiveTasks({
+    task: 'sync-order',
+    sourceDocFilter: { isVip: true } 
+});
+```
+
+## Counting Tasks
+
+Use `countReactiveTasks` for metrics or UI badges.
+
+```typescript
+import { countReactiveTasks } from 'mongodash';
+
+const dlqSize = await countReactiveTasks({ 
+    task: 'send-welcome-email', 
+    status: 'failed' 
+});
+```
+
+## Retrying Tasks
+
+Use `retryReactiveTasks` to manually re-trigger tasks. This is useful for DLQ recovery after fixing a bug.
+
+This operation is **concurrency-safe**. If a task is currently `processing`, it will be marked to re-run immediately after the current execution finishes (`processing_dirty`), ensuring no race conditions.
+
+```typescript
+import { retryReactiveTasks } from 'mongodash';
+
+// Retry ALL failed tasks for a specific job
+const result = await retryReactiveTasks({ 
+    task: 'send-welcome-email', 
+    status: 'failed' 
+});
+console.log(`Retried ${result.modifiedCount} tasks.`);
+
+// Retry specific task by Source Document ID
+await retryReactiveTasks({
+    task: 'sync-order',
+    sourceDocFilter: { _id: 'order-123' }
+});
+
+// Bulk Retry: Retry all tasks for "VIP" orders
+// This efficiently finds matching tasks and schedules them for execution.
+await retryReactiveTasks({
+    task: 'sync-order',
+    sourceDocFilter: { isVip: true }
+});
+```

package/docs/reactive-tasks/monitoring.md

@@ -0,0 +1,76 @@
+# Monitoring
+
+Mongodash provides built-in Prometheus metrics to monitor your reactive tasks.
+
+> [!NOTE]
+> **Dependency Required**: You must install `prom-client` yourself to use this feature. It is an optional peer dependency.
+> ```bash
+> npm install prom-client
+> ```
+
+## Configuration
+
+Monitoring is configured in the initialization options under the `monitoring` key:
+
+```typescript
+await mongodash.init({
+    // ...
+    monitoring: {
+        enabled: true,           // Default: true
+        scrapeMode: 'cluster',   // 'cluster' (default) or 'local'
+        pushIntervalMs: 60000,   // How often instances synchronize metrics (default: 1m). Relevant only if scrapeMode is 'cluster'.
+        readPreference: 'secondaryPreferred' // 'primary', 'secondaryPreferred' etc.
+    }
+});
+```
+
+- **scrapeMode**:
+    - `'cluster'` (Default): Returns aggregated system-wide metrics. Any instance can respond to this request (by fetching state from the DB). It aggregates metrics from all other active instances. (Recommended for Load Balancers / Heroku)
+    - `'local'`: Returns local metrics for THIS instance. If this instance is the Leader, it ALSO includes Global System Metrics (Queue Depth, Lag) so they are reported exactly once in the cluster. (Recommended for K8s Pod Monitors)
+
+## Retrieving Metrics
+
+Expose the metrics endpoint (e.g., in Express):
+
+```typescript
+import { getPrometheusMetrics } from 'mongodash';
+
+app.get('/metrics', async (req, res) => {
+    const registry = await getPrometheusMetrics();
+    
+    if (registry) {
+        res.set('Content-Type', registry.contentType);
+        return res.end(await registry.metrics());
+    }
+    
+    res.status(503).send('Monitoring disabled');
+});
+```
+
+## Available Metrics
+
+The system exposes the following metrics with standardized labels:
+
+| Metric Name | Type | Labels | Description |
+| :--- | :--- | :--- | :--- |
+| `reactive_tasks_duration_seconds` | Histogram | `task_name`, `status` | Distribution of task processing time (success/failure). |
+| `reactive_tasks_retries_total` | Counter | `task_name` | Total number of retries attempted. |
+| `reactive_tasks_queue_depth` | Gauge | `task_name`, `status` | Current number of tasks in the queue, grouped by status (`pending`, `processing`, `processing_dirty`, `failed`). |
+| `reactive_tasks_global_lag_seconds` | Gauge | `task_name` | Age of the oldest `pending` task, measured from `initialScheduledAt` (or `scheduledAt` if not deferred). This ensures deferred tasks still reflect their true waiting time. |
+| `reactive_tasks_change_stream_lag_seconds` | Gauge | *none* | Time difference between now and the last processed Change Stream event. |
+| `reactive_tasks_last_reconciliation_timestamp_seconds` | Gauge | *none* | Timestamp when the last full reconciliation (recovery) finished. |
+
+## Grafana Dashboard
+
+A comprehensive **Grafana Dashboard** ("Reactive Tasks - System Overview") is included with the package.
+
+It provides real-time visibility into:
+- System Health & Global Lag
+- Throughput & Latency Heatmaps
+- Queue Depth & Composition
+- Error Rates & Retries
+
+You can find the dashboard JSON file at:
+`node_modules/mongodash/grafana/reactive_tasks.json`
+
+Import this file directly into Grafana to get started.

package/docs/reactive-tasks/policy-cleanup.md

@@ -0,0 +1,70 @@
+# Cleanup Policy
+
+The Cleanup Policy controls automatic deletion of orphaned task records — tasks whose source documents have been deleted or no longer match the configured filter.
+
+## Configuration
+
+```typescript
+cleanupPolicy?: {
+    deleteWhen?: 'sourceDocumentDeleted' | 'sourceDocumentDeletedOrNoLongerMatching' | 'never';
+    keepFor?: string | number;
+}
+```
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `deleteWhen` | `string` | `'sourceDocumentDeleted'` | When to trigger task deletion |
+| `keepFor` | `string \| number` | `'24h'` | Grace period before deletion (e.g., `'1h'`, `'7d'`, or `86400000` ms) |
+
+### Deletion Strategies (`deleteWhen`)
+
+| Strategy | Behavior |
+|----------|----------|
+| `sourceDocumentDeleted` | **Default.** Task deleted only when its source document is deleted from the database. Filter mismatches are ignored. |
+| `sourceDocumentDeletedOrNoLongerMatching` | Task deleted when source document is deleted **OR** when it no longer matches the task's `filter`. Useful for cases the change of document is permament and it is not expected the document could match in the future again and retrigger because of that. Also useful for  `$$NOW`-based or dynamic filters. |
+| `never` | Tasks are never automatically deleted. Use for audit trails or manual cleanup scenarios. |
+
+### Grace Period Calculation
+
+The `keepFor` grace period is measured from `MAX(updatedAt, lastFinalizedAt)`:
+
+- **`updatedAt`**: When the source document's watched fields (`watchProjection`) last changed
+- **`lastFinalizedAt`**: When a worker last completed or failed the task
+
+This ensures tasks are protected if either:
+1. The source data changed recently, OR
+2. A worker processed the task recently
+
+### Example: Dynamic Filter Cleanup
+
+```typescript
+await reactiveTask({
+    task: 'remind-pending-order',
+    collection: 'orders',
+    // Match orders pending for more than 24 hours
+    filter: { $expr: { $gt: ['$$NOW', { $add: ['$createdAt', 24 * 60 * 60 * 1000] }] } },
+    
+    cleanupPolicy: {
+        deleteWhen: 'sourceDocumentDeletedOrNoLongerMatching',
+        keepFor: '1h', // Keep it at least 1 hour after last scheduled matching or finalization
+    },
+    
+    handler: async (order) => { /* Send reminder email */ }
+});
+```
+
+### Scheduler-Level Configuration
+
+Control how often the cleanup runs using `reactiveTaskCleanupInterval` in scheduler options. Cleanup is performed in **batches** (default 1000 items) to ensure stability on large datasets.
+
+```typescript
+await mongodash.init({
+    // ...
+    reactiveTaskCleanupInterval: '12h', // Run cleanup every 12 hours (default: '24h')
+});
+```
+
+Supported formats:
+- Duration string: `'1h'`, `'24h'`, `'7d'`
+- Milliseconds: `86400000`
+- Cron expression: `'CRON 0 3 * * *'` (e.g., daily at 3 AM)

package/docs/reactive-tasks/policy-retry.md

@@ -0,0 +1,60 @@
+# Retry Policy
+
+You can configure the retry behavior using the `retryPolicy` option.
+
+## General Options
+
+| Option | Type | Default | Description |
+| :--- | :--- | :--- | :--- |
+| `type` | `string` | **Required** | `'fixed'`, `'linear'`, `'exponential'`, `'series'`, or `'cron'` |
+| `maxAttempts` | `number` | `5`* | Maximum total attempts (use `-1` for unlimited). |
+| `maxDuration` | `string \| number` | `undefined` | Stop retrying if elapsed time since the **first failure** in the current sequence exceeds this value. |
+| `resetRetriesOnDataChange` | `boolean` | `true` | Reset attempt count if the source document changes. |
+
+*\* If `maxDuration` is specified, `maxAttempts` defaults to unlimited.*
+
+### Policy Specific Settings
+
+| Policy | Property | Default | Description |
+| :--- | :--- | :--- | :--- |
+| **`fixed`** | `interval` | - | Delay between retries (e.g., `'10s'`). |
+| **`linear`** | `interval` | - | Base delay multiplied by `attempt` number. |
+| **`exponential`** | `min` | `'10s'` | Initial delay for the first retry. |
+| **`exponential`** | `max` | `'1d'` | Maximum delay cap for backoff. |
+| **`exponential`** | `factor` | `2` | Multiplication factor per attempt. |
+| **`series`** | `intervals` | - | Array of fixed delays (e.g., `['1m', '5m', '15m']`). |
+| **`cron`** | `expression` | - | Standard cron string for scheduling retries. |
+
+### Examples
+
+```typescript
+// 1. Give up after 24 hours (infinite attempts within that window)
+retryPolicy: {
+    maxDuration: '24h',
+    type: 'exponential',
+    min: '10s',
+    max: '1h'
+}
+
+// 2. Exact retry ladder (try after 1m, then 5m, then 15m, then fail)
+retryPolicy: {
+    maxAttempts: 4, // 1st run + 3 retries
+    type: 'series',
+    intervals: ['1m', '5m', '15m']
+}
+
+// 3. Series with last interval reuse
+// Sequence: 1m, 5m, 5m, 5m ... (last one repeats)
+retryPolicy: {
+    maxAttempts: 10,
+    type: 'series',
+    intervals: ['1m', '5m']
+}
+
+// 4. Permanent retries every hour
+retryPolicy: {
+    maxAttempts: -1,
+    type: 'fixed',
+    interval: '1h'
+}
+```

package/docs/reactive-tasks/reconciliation.md

@@ -0,0 +1,40 @@
+# Reconciliation & Reliability
+
+The system includes a self-healing mechanism called **Reconciliation**.
+
+## What is it?
+
+It is a "full scan" process that ensures the state of your tasks matches the actual data in your collections. It iterates through your source collections (efficiently, fetching only `_id`) and ensures every document has the correct corresponding tasks planned.
+
+## When does it run?
+
+1.  **On Startup (Partial)**: When `startReactiveTasks()` is called, the leader performs a reconciliation only for tasks that have **never been reconciled before**. This ensures that newly added tasks catch up with existing data.
+2.  **On History Loss**: If the MongoDB Change Stream buffer (Oplog) is full and events are lost (Error code 280), the system automatically triggers full reconciliation to ensure consistency is restored.
+3.  **On Trigger Evolution**: When you widen a task filter (e.g. `amount > 100` -> `amount > 50`), the system triggers reconciliation to backfill tasks for existing documents that now match the new filter.
+
+## Resilience
+
+Reconciliation is **persistent and resilient**.
+-   **Checkpoints**: The system saves its progress (`lastId`) periodically to the database (`_mongodash_planner_meta`).
+-   **Resumable**: If the process is interrupted (e.g., deployment, crash), it effectively **resumes** from the last checkpoint upon restart, preventing re-processing of already reconciled documents.
+-   **Invalidation**: If the set of tasks being reconciled changes (e.g., you deploy a version with a NEW task definition for the same collection), the system detects this change, invalidates the checkpoint, and restarts reconciliation from the beginning to ensure the new task is applied to the entire collection.
+
+## Expectations
+
+-   **No Data Loss**: Even if your specific localized Oplog history is lost, the system will eventually process every document.
+-   **Performance**: The scan is optimized (uses batching and projection of `_id` only), but it performs a **full collection scan**. On huge collections (millions of docs), this causes increased database load during startup or recovery.
+-   **Batch Processing**: Both reconciliation and periodic cleanup process documents in batches to avoid overwhelming the database and the application memory.
+
+> [!CAUTION]
+> **Limitations of `$$NOW` in filters**  
+> MongoDB Change Streams only trigger when a document is physically updated. If your `filter` depends on time passing (e.g., `dueAt: { $lte: '$$NOW' }`), the task **will not** trigger automatically just because time passed. It will only be picked up during:
+> 1.  A physical update to the source document.
+> 2.  The next system restart, if the reconciliation is run.
+> 3.  Manual re-triggers via `retryReactiveTasks()`.
+
+## Configuration Matters
+
+Reconciliation respects your `filter` and `watchProjection`.
+-   If a document doesn't match the `filter`, no task is planned.
+-   If the `watchProjection` hasn't changed since the last run (comparing `lastObservedValues`), the task is **not** re-triggered.
+-   **Recommendation**: Carefully configure `filter` and `watchProjection` to minimize unnecessary processing during reconciliation.