@chidchanun/bcp 0.2.9 → 0.2.11

package/docs/api-reference.md

@@ -66,30 +66,38 @@ Related guides: [Authentication](authentication.md), [Auth Session Stores](auth-
 
 ## `bcp/jobs`
 
-Server-only Background Jobs and Job Scheduling APIs.
+Server-only Background Jobs, Job Scheduling and Durable Jobs APIs.
 
-Queue APIs:
+Core queue APIs:
 
 ```ts
 import {
     createJobQueue,
     createMemoryJobQueueAdapter,
     type BackgroundJobQueue,
+    type CleanupJobsOptions,
+    type DeadLetterJobRecord,
     type EnqueueJobOptions,
+    type HeartbeatJobOptions,
     type JobHandler,
     type JobHandlerContext,
     type JobQueueAdapter,
     type JobQueueOptions,
+    type JobQueueStats,
     type JobRecord,
     type JobRetryDelay,
     type JobState,
     type JobWorker,
     type MemoryJobQueueAdapter,
+    type ProcessNextJobOptions,
+    type RecoverStaleJobsOptions,
+    type RequeueDeadLetterOptions,
+    type ReserveJobOptions,
     type StartJobWorkerOptions,
 } from "bcp/jobs";
 ```
 
-Scheduling APIs added in `0.2.9`:
+Scheduling APIs:
 
 ```ts
 import {
@@ -110,15 +118,112 @@ import {
 } from "bcp/jobs";
 ```
 
-`createJobQueue()` provides delayed enqueueing, retry/backoff, cancellation, manual `processNext()` execution and concurrent workers.
+Durable Redis-compatible APIs added in `0.2.10`:
 
-`createJobScheduler()` adds recurring interval and UTC five-field cron schedules. The scheduler leases due records from `JobScheduleStore`, then enqueues normal queue jobs using deterministic occurrence ids.
+```ts
+import {
+    createRedisJobQueueAdapter,
+    createRedisJobScheduleStore,
+    type RedisCommandClient,
+    type RedisJobQueueAdapter,
+    type RedisJobsAdapterOptions,
+    type RedisJobScheduleStore,
+} from "bcp/jobs";
+```
+
+### Queue lifecycle
+
+`createJobQueue()` supports immediate/delayed enqueue, retry/backoff, cancellation, manual `processNext()`, worker concurrency, visibility timeout, heartbeat lease renewal, stale-running recovery, DLQ inspection/requeue, terminal retention cleanup and queue statistics.
+
+Durable adapter methods such as `heartbeat()`, `recoverStale()`, `listDeadLetters()`, `requeueDeadLetter()`, `cleanup()` and `stats()` are optional so earlier `JobQueueAdapter` implementations remain compatible.
+
+`reserve()` receives an optional second `ReserveJobOptions` argument containing `ownerId` and `visibilityTimeoutMs`. Production durable adapters should use it to create an atomic visibility lease.
+
+### Scheduling
+
+`createJobScheduler()` supports recurring interval and UTC five-field cron schedules.
+
+The scheduler leases due records from `JobScheduleStore`, then enqueues normal jobs using deterministic occurrence IDs. `nextCronTime()` and `nextScheduleTime()` are available for tooling and tests.
+
+### Redis-compatible durable adapters
+
+`createRedisJobQueueAdapter()` implements the queue contract on a shared Redis-compatible command client.
+
+`createRedisJobScheduleStore()` implements the schedule-store contract using the same minimal client shape:
+
+```ts
+interface RedisCommandClient {
+    sendCommand(
+        command: string[]
+    ): Promise<unknown>;
+}
+```
+
+BCP does not install a Redis library and does not create the connection. Applications own authentication, TLS, Cluster/Sentinel configuration and connection shutdown.
+
+The reference adapters use atomic Lua state transitions. The default namespace `bcp:{jobs}` uses a Redis Cluster hash tag so all job/schedule keys share one slot.
+
+The processing model is at-least-once. Handlers that perform non-idempotent external side effects should use application-level idempotency protection.
+
+Related guides: [Background Jobs Platform](background-jobs.md), [Job Scheduling Platform](job-scheduling.md), [Durable Jobs Platform](durable-jobs.md), [Observability Platform v2](observability.md).
+
+## `bcp/workflow`
+
+Server-only Workflow Orchestration APIs added in `0.2.11`.
+
+```ts
+import {
+    createMemoryWorkflowStore,
+    createWorkflow,
+    type CancelWorkflowOptions,
+    type MemoryWorkflowStore,
+    type ResumeWorkflowOptions,
+    type StartWorkflowOptions,
+    type Workflow,
+    type WorkflowBuilder,
+    type WorkflowCompensationHandler,
+    type WorkflowOptions,
+    type WorkflowParallelBuilder,
+    type WorkflowRetryDelay,
+    type WorkflowRunRecord,
+    type WorkflowRunState,
+    type WorkflowStepContext,
+    type WorkflowStepHandler,
+    type WorkflowStepKind,
+    type WorkflowStepOptions,
+    type WorkflowStepRecord,
+    type WorkflowStepState,
+    type WorkflowStore,
+} from "bcp/workflow";
+```
+
+`createWorkflow()` defines a persistent server-side workflow with sequential steps, parallel groups, per-step retry policies, persisted delays and compensation handlers.
+
+Workflow controls include:
+
+```text
+start()
+run()
+get()
+list()
+resume()
+retry()
+cancel()
+compensate()
+close()
+```
+
+Without a queue, workflow execution runs in the current process until it succeeds, fails, is cancelled or reaches a persisted delay.
+
+When an existing `BackgroundJobQueue` is passed through `WorkflowOptions.queue`, workflow execution is submitted through `bcp/jobs`. Delay steps enqueue delayed continuation jobs instead of holding a long-running timer.
+
+`WorkflowStore` is the workflow persistence boundary. Its `claim()` / `release()` methods form the run-level lease contract for shared multi-instance stores.
 
-The built-in queue adapter and schedule store are process-local. Durable multi-instance deployments should implement both `JobQueueAdapter` and `JobScheduleStore` against shared infrastructure. `JobQueueAdapter.reserve()` and `JobScheduleStore.acquireDue()` are the respective atomic concurrency boundaries.
+`createMemoryWorkflowStore()` is intended for development and deterministic tests. Production multi-instance applications should implement a shared durable store with atomic `claim()` behavior.
 
-Cron evaluation in `0.2.9` uses UTC. `nextCronTime()` and `nextScheduleTime()` are available for tooling and tests.
+Compensation is saga-style and executes successful compensatable steps in reverse completion order. It does not turn external services into one distributed database transaction.
 
-Related guides: [Background Jobs Platform](background-jobs.md), [Job Scheduling Platform](job-scheduling.md), [Observability Platform v2](observability.md).
+Related guides: [Workflow Orchestration](workflow-orchestration.md), [Durable Jobs Platform](durable-jobs.md), [Observability Platform v2](observability.md).
 
 ## `bcp/observability`
 

package/docs/docs-web-manifest.json

@@ -1,7 +1,7 @@
 {
   "schemaVersion": 1,
   "framework": "bcp",
-  "versionTarget": "0.2.9",
+  "versionTarget": "0.2.11",
   "releaseState": "unreleased",
   "sections": [
     {
@@ -57,7 +57,7 @@
     {
       "id": "runtime",
       "title": "Runtime & Infrastructure",
-      "description": "Middleware, background jobs, scheduling, observability, logging, caching, security and production hardening.",
+      "description": "Middleware, durable background jobs, scheduling, workflow orchestration, observability, logging, caching, security and production hardening.",
       "pages": [
         { "route": "/docs/middleware", "source": "middleware.md", "title": "Middleware" },
         { "route": "/docs/hydration", "source": "hydration.md", "title": "Hydration" },
@@ -65,6 +65,8 @@
         { "route": "/docs/observability", "source": "observability.md", "title": "Observability Platform v2" },
         { "route": "/docs/background-jobs", "source": "background-jobs.md", "title": "Background Jobs Platform" },
         { "route": "/docs/job-scheduling", "source": "job-scheduling.md", "title": "Job Scheduling Platform" },
+        { "route": "/docs/durable-jobs", "source": "durable-jobs.md", "title": "Durable Jobs Platform" },
+        { "route": "/docs/workflow-orchestration", "source": "workflow-orchestration.md", "title": "Workflow Orchestration" },
         { "route": "/docs/caching", "source": "caching.md", "title": "Caching" },
         { "route": "/docs/security", "source": "security.md", "title": "Security" },
         { "route": "/docs/production-hardening", "source": "production-hardening.md", "title": "Production Hardening" }
@@ -110,7 +112,9 @@
     }
   ],
   "releases": [
-    { "route": "/releases/0.2.9", "source": "releases/0.2.9.md", "version": "0.2.9", "state": "unreleased" },
+    { "route": "/releases/0.2.11", "source": "releases/0.2.11.md", "version": "0.2.11", "state": "unreleased" },
+    { "route": "/releases/0.2.10", "source": "releases/0.2.10.md", "version": "0.2.10" },
+    { "route": "/releases/0.2.9", "source": "releases/0.2.9.md", "version": "0.2.9" },
     { "route": "/releases/0.2.8", "source": "releases/0.2.8.md", "version": "0.2.8" },
     { "route": "/releases/0.2.7", "source": "releases/0.2.7.md", "version": "0.2.7" },
     { "route": "/releases/0.2.6", "source": "releases/0.2.6.md", "version": "0.2.6" },

package/docs/durable-jobs.md

@@ -0,0 +1,359 @@
+# Durable Jobs Platform
+
+BCP Framework `0.2.10` extends `bcp/jobs` with production-oriented worker lifecycle primitives, dead-letter handling, retention utilities and Redis-compatible queue/scheduler adapters.
+
+The public entrypoint remains:
+
+```ts
+import {
+    createJobQueue,
+    createJobScheduler,
+} from "bcp/jobs";
+```
+
+`0.2.10` is additive to the queue contract introduced in `0.2.8` and the scheduling contract introduced in `0.2.9`.
+
+## What 0.2.10 adds
+
+```text
+visibility timeout
+worker lease owner
+heartbeat renewal
+stale-running recovery
+dead-letter queue (DLQ)
+dead-letter requeue
+terminal-job cleanup
+queue statistics
+Redis-compatible JobQueueAdapter
+Redis-compatible JobScheduleStore
+```
+
+The built-in memory adapter implements the same lifecycle for development and deterministic tests. Durable multi-process deployments should use a shared adapter such as the Redis-compatible adapter or another implementation of the public contracts.
+
+## Worker visibility timeout
+
+Workers now reserve jobs with a visibility lease:
+
+```ts
+const worker =
+    jobs.startWorker({
+        workerId: "email-worker",
+        concurrency: 4,
+        visibilityTimeoutMs: 30_000,
+        heartbeatIntervalMs: 10_000,
+        pollIntervalMs: 250,
+    });
+```
+
+A durable adapter can persist:
+
+```text
+leaseOwner
+leaseUntil
+heartbeatAt
+```
+
+While a handler is running, BCP renews the lease when the adapter implements `heartbeat()`.
+
+If the process disappears and the lease expires, `recoverStale()` can move the job back to `queued` so another worker can process it.
+
+This is an **at-least-once** model. Handlers that perform non-idempotent side effects should use application-level idempotency keys or transactions.
+
+## Manual stale recovery
+
+```ts
+const recovered =
+    await jobs.recoverStale({
+        limit: 100,
+    });
+```
+
+Workers also request stale-job recovery before reserving work when the adapter implements the capability.
+
+A recovered job keeps its attempt count. It may therefore reach `maxAttempts` after repeated crashes or visibility-timeout recoveries.
+
+## Dead-letter queue
+
+When a handler exhausts `maxAttempts`, the memory and Redis adapters keep the job in terminal `failed` state and also add it to the DLQ index.
+
+```ts
+const failedJobs =
+    await jobs.deadLetters();
+```
+
+Each result includes:
+
+```ts
+{
+    state: "failed",
+    deadLetteredAt: number,
+    // normal JobRecord fields
+}
+```
+
+Requeue a DLQ job:
+
+```ts
+await jobs.requeueDeadLetter(
+    jobId,
+    {
+        delayMs: 5_000,
+        resetAttempts: true,
+    }
+);
+```
+
+`resetAttempts` defaults to `true`.
+
+## Queue statistics
+
+```ts
+const stats =
+    await jobs.stats();
+```
+
+Shape:
+
+```ts
+{
+    total: number;
+    queued: number;
+    running: number;
+    succeeded: number;
+    failed: number;
+    cancelled: number;
+    deadLetters: number;
+}
+```
+
+Adapters may provide an optimized `stats()` implementation. Otherwise BCP derives statistics from `list()` plus the optional DLQ contract.
+
+## Retention cleanup
+
+Terminal jobs can be removed after an application-defined retention window:
+
+```ts
+const sevenDaysAgo =
+    Date.now() -
+    7 * 24 * 60 * 60 * 1000;
+
+await jobs.cleanup({
+    before: sevenDaysAgo,
+    states: [
+        "succeeded",
+        "cancelled",
+    ],
+});
+```
+
+If `states` is omitted, cleanup considers:
+
+```text
+succeeded
+failed
+cancelled
+```
+
+For failed jobs, cleanup also removes the DLQ index entry.
+
+The Redis reference adapter removes at most 1,000 matching terminal records per cleanup call. Large installations can invoke cleanup repeatedly from a scheduled maintenance job.
+
+## Redis-compatible queue adapter
+
+BCP does not install or own a Redis client library. The adapter accepts a minimal command client:
+
+```ts
+interface RedisCommandClient {
+    sendCommand(
+        command: string[]
+    ): Promise<unknown>;
+}
+```
+
+Create the queue adapter:
+
+```ts
+import {
+    createJobQueue,
+    createRedisJobQueueAdapter,
+} from "bcp/jobs";
+
+const adapter =
+    createRedisJobQueueAdapter({
+        client: redisCommandClient,
+        namespace: "my-app:{jobs}",
+    });
+
+export const jobs =
+    createJobQueue({
+        adapter,
+    });
+```
+
+The default namespace is:
+
+```text
+bcp:{jobs}
+```
+
+The `{jobs}` Redis hash tag keeps all adapter keys in the same Redis Cluster slot so Lua operations can remain atomic.
+
+### Connection lifecycle
+
+Applications own the Redis connection. When desired, provide a close hook:
+
+```ts
+const adapter =
+    createRedisJobQueueAdapter({
+        client: redisCommandClient,
+        close: async () => {
+            await redisClient.quit();
+        },
+    });
+```
+
+Then:
+
+```ts
+await jobs.close();
+```
+
+will stop BCP workers and invoke the adapter close hook.
+
+If queue and scheduler share one Redis connection, normally provide the close hook to only one owner or coordinate connection shutdown at the application layer.
+
+## Redis environment configuration
+
+A typical application can use:
+
+```dotenv
+REDIS_URL=redis://localhost:6379
+```
+
+`bcp/jobs` does **not** read `REDIS_URL` automatically. Connection creation remains application-owned so credentials, TLS, Sentinel/Cluster configuration and client-library choice stay explicit.
+
+## Atomic Redis behavior
+
+The reference Redis adapter uses Lua `EVAL` for state-changing operations that require atomicity, including:
+
+```text
+enqueue duplicate protection
+job reservation
+complete/fail/cancel
+heartbeat lease renewal
+stale-running recovery
+DLQ requeue
+retention cleanup
+schedule upsert/remove
+schedule lease acquisition
+schedule completion/release
+```
+
+Queue reservation atomically removes one eligible job from the available index, marks it running and writes the visibility lease before another worker can reserve it.
+
+## Redis-backed scheduler
+
+The same Redis command client can back the scheduling contract:
+
+```ts
+import {
+    createJobScheduler,
+    createRedisJobScheduleStore,
+} from "bcp/jobs";
+
+const scheduleStore =
+    createRedisJobScheduleStore({
+        client: redisCommandClient,
+        namespace: "my-app:{jobs}",
+    });
+
+export const scheduler =
+    createJobScheduler({
+        queue: jobs,
+        store: scheduleStore,
+        ownerId: "scheduler-a",
+    });
+```
+
+`acquireDue()` is implemented with an atomic Redis lease, allowing multiple scheduler processes to share the same schedule store.
+
+## Recommended production topology
+
+```text
+Web/API instances
+      |
+      | enqueue / schedule
+      v
+Redis-compatible shared job state
+      |
+      +-------------------+
+      |                   |
+      v                   v
+Worker A              Worker B
+visibility lease      visibility lease
+heartbeat             heartbeat
+      |
+      +---- failure ----> retry / DLQ
+
+Scheduler A -----+
+                 +---- shared schedule lease
+Scheduler B -----+
+```
+
+Queue workers and schedulers can run in the same Node process for small deployments, but separate worker processes/containers are recommended when job load can affect web latency.
+
+## Adapter compatibility
+
+Existing `0.2.8` `JobQueueAdapter` implementations remain valid. Durable methods are optional:
+
+```ts
+heartbeat?()
+recoverStale?()
+listDeadLetters?()
+requeueDeadLetter?()
+cleanup?()
+stats?()
+```
+
+The `reserve()` method now receives an optional second argument with the worker owner ID and visibility timeout. Existing JavaScript adapters that ignore the optional argument continue to work.
+
+For production durable adapters, implementations should support the lease contract rather than ignoring it.
+
+## Failure model
+
+BCP jobs intentionally use practical at-least-once delivery semantics.
+
+Applications should assume that a job can execute more than once when:
+
+- a worker completes external side effects but crashes before recording success,
+- a visibility lease expires during a long or blocked handler,
+- a DLQ job is manually requeued,
+- infrastructure retries a command after an ambiguous network result.
+
+Use idempotency keys, unique database constraints or transactional outbox/inbox patterns for side effects where duplicates are unsafe.
+
+## Observability
+
+`jobs.stats()` can feed application metrics from `bcp/observability`:
+
+```ts
+const stats =
+    await jobs.stats();
+
+queueDepth.set(
+    stats.queued
+);
+
+dlqDepth.set(
+    stats.deadLetters
+);
+```
+
+For Redis deployments, infrastructure-level Redis monitoring should complement application-level BCP metrics.
+
+## Related guides
+
+- [Background Jobs Platform](background-jobs.md)
+- [Job Scheduling Platform](job-scheduling.md)
+- [Observability Platform v2](observability.md)
+- [Production Hardening](production-hardening.md)

package/docs/platform-manifest.json

@@ -1,9 +1,9 @@
 {
   "schemaVersion": 1,
   "framework": "bcp",
-  "version": "0.2.9",
+  "version": "0.2.11",
   "releaseState": "unreleased",
-  "baseline": "job-scheduling-platform",
+  "baseline": "workflow-orchestration",
   "runtime": {
     "node": ">=24.11.0",
     "react": "19",
@@ -20,6 +20,7 @@
     "bcp/database",
     "bcp/auth",
     "bcp/jobs",
+    "bcp/workflow",
     "bcp/observability",
     "bcp/server",
     "bcp/server-only",
@@ -80,6 +81,25 @@
     "jobScheduleStoreContract": true,
     "schedulerLeases": true,
     "scheduledRunDeduplication": true,
+    "durableJobsPlatform": true,
+    "redisJobQueueAdapter": true,
+    "redisJobScheduleStore": true,
+    "jobVisibilityTimeout": true,
+    "jobHeartbeats": true,
+    "staleJobRecovery": true,
+    "deadLetterQueue": true,
+    "deadLetterRequeue": true,
+    "jobRetentionCleanup": true,
+    "jobQueueStatistics": true,
+    "workflowOrchestration": true,
+    "workflowStoreContract": true,
+    "workflowRunLeases": true,
+    "workflowSequentialSteps": true,
+    "workflowParallelSteps": true,
+    "workflowStepRetries": true,
+    "workflowDelays": true,
+    "workflowCompensation": true,
+    "workflowQueueExecution": true,
     "databaseMigrations": true,
     "databaseAdapterContract": true,
     "databasePostgresql": true,
@@ -119,7 +139,7 @@
     "s3-compatible"
   ],
   "compatibility": {
-    "previousBaseline": "0.2.8",
+    "previousBaseline": "0.2.10",
     "intentionalBreakingChangesFromPreviousBaseline": false,
     "migrationGuide": "migration-0.2.md"
   },
@@ -138,7 +158,9 @@
     "observability": "observability.md",
     "backgroundJobs": "background-jobs.md",
     "jobScheduling": "job-scheduling.md",
+    "durableJobs": "durable-jobs.md",
+    "workflowOrchestration": "workflow-orchestration.md",
     "migrationGuide": "migration-0.2.md",
-    "releaseNotes": "releases/0.2.9.md"
+    "releaseNotes": "releases/0.2.11.md"
   }
 }