@chidchanun/bcp 0.2.8 → 0.2.10

package/docs/durable-jobs.md

@@ -0,0 +1,359 @@
+# Durable Jobs Platform
+
+BCP Framework `0.2.10` extends `bcp/jobs` with production-oriented worker lifecycle primitives, dead-letter handling, retention utilities and Redis-compatible queue/scheduler adapters.
+
+The public entrypoint remains:
+
+```ts
+import {
+    createJobQueue,
+    createJobScheduler,
+} from "bcp/jobs";
+```
+
+`0.2.10` is additive to the queue contract introduced in `0.2.8` and the scheduling contract introduced in `0.2.9`.
+
+## What 0.2.10 adds
+
+```text
+visibility timeout
+worker lease owner
+heartbeat renewal
+stale-running recovery
+dead-letter queue (DLQ)
+dead-letter requeue
+terminal-job cleanup
+queue statistics
+Redis-compatible JobQueueAdapter
+Redis-compatible JobScheduleStore
+```
+
+The built-in memory adapter implements the same lifecycle for development and deterministic tests. Durable multi-process deployments should use a shared adapter such as the Redis-compatible adapter or another implementation of the public contracts.
+
+## Worker visibility timeout
+
+Workers now reserve jobs with a visibility lease:
+
+```ts
+const worker =
+    jobs.startWorker({
+        workerId: "email-worker",
+        concurrency: 4,
+        visibilityTimeoutMs: 30_000,
+        heartbeatIntervalMs: 10_000,
+        pollIntervalMs: 250,
+    });
+```
+
+A durable adapter can persist:
+
+```text
+leaseOwner
+leaseUntil
+heartbeatAt
+```
+
+While a handler is running, BCP renews the lease when the adapter implements `heartbeat()`.
+
+If the process disappears and the lease expires, `recoverStale()` can move the job back to `queued` so another worker can process it.
+
+This is an **at-least-once** model. Handlers that perform non-idempotent side effects should use application-level idempotency keys or transactions.
+
+## Manual stale recovery
+
+```ts
+const recovered =
+    await jobs.recoverStale({
+        limit: 100,
+    });
+```
+
+Workers also request stale-job recovery before reserving work when the adapter implements the capability.
+
+A recovered job keeps its attempt count. It may therefore reach `maxAttempts` after repeated crashes or visibility-timeout recoveries.
+
+## Dead-letter queue
+
+When a handler exhausts `maxAttempts`, the memory and Redis adapters keep the job in terminal `failed` state and also add it to the DLQ index.
+
+```ts
+const failedJobs =
+    await jobs.deadLetters();
+```
+
+Each result includes:
+
+```ts
+{
+    state: "failed",
+    deadLetteredAt: number,
+    // normal JobRecord fields
+}
+```
+
+Requeue a DLQ job:
+
+```ts
+await jobs.requeueDeadLetter(
+    jobId,
+    {
+        delayMs: 5_000,
+        resetAttempts: true,
+    }
+);
+```
+
+`resetAttempts` defaults to `true`.
+
+## Queue statistics
+
+```ts
+const stats =
+    await jobs.stats();
+```
+
+Shape:
+
+```ts
+{
+    total: number;
+    queued: number;
+    running: number;
+    succeeded: number;
+    failed: number;
+    cancelled: number;
+    deadLetters: number;
+}
+```
+
+Adapters may provide an optimized `stats()` implementation. Otherwise BCP derives statistics from `list()` plus the optional DLQ contract.
+
+## Retention cleanup
+
+Terminal jobs can be removed after an application-defined retention window:
+
+```ts
+const sevenDaysAgo =
+    Date.now() -
+    7 * 24 * 60 * 60 * 1000;
+
+await jobs.cleanup({
+    before: sevenDaysAgo,
+    states: [
+        "succeeded",
+        "cancelled",
+    ],
+});
+```
+
+If `states` is omitted, cleanup considers:
+
+```text
+succeeded
+failed
+cancelled
+```
+
+For failed jobs, cleanup also removes the DLQ index entry.
+
+The Redis reference adapter removes at most 1,000 matching terminal records per cleanup call. Large installations can invoke cleanup repeatedly from a scheduled maintenance job.
+
+## Redis-compatible queue adapter
+
+BCP does not install or own a Redis client library. The adapter accepts a minimal command client:
+
+```ts
+interface RedisCommandClient {
+    sendCommand(
+        command: string[]
+    ): Promise<unknown>;
+}
+```
+
+Create the queue adapter:
+
+```ts
+import {
+    createJobQueue,
+    createRedisJobQueueAdapter,
+} from "bcp/jobs";
+
+const adapter =
+    createRedisJobQueueAdapter({
+        client: redisCommandClient,
+        namespace: "my-app:{jobs}",
+    });
+
+export const jobs =
+    createJobQueue({
+        adapter,
+    });
+```
+
+The default namespace is:
+
+```text
+bcp:{jobs}
+```
+
+The `{jobs}` Redis hash tag keeps all adapter keys in the same Redis Cluster slot so Lua operations can remain atomic.
+
+### Connection lifecycle
+
+Applications own the Redis connection. When desired, provide a close hook:
+
+```ts
+const adapter =
+    createRedisJobQueueAdapter({
+        client: redisCommandClient,
+        close: async () => {
+            await redisClient.quit();
+        },
+    });
+```
+
+Then:
+
+```ts
+await jobs.close();
+```
+
+will stop BCP workers and invoke the adapter close hook.
+
+If queue and scheduler share one Redis connection, normally provide the close hook to only one owner or coordinate connection shutdown at the application layer.
+
+## Redis environment configuration
+
+A typical application can use:
+
+```dotenv
+REDIS_URL=redis://localhost:6379
+```
+
+`bcp/jobs` does **not** read `REDIS_URL` automatically. Connection creation remains application-owned so credentials, TLS, Sentinel/Cluster configuration and client-library choice stay explicit.
+
+## Atomic Redis behavior
+
+The reference Redis adapter uses Lua `EVAL` for state-changing operations that require atomicity, including:
+
+```text
+enqueue duplicate protection
+job reservation
+complete/fail/cancel
+heartbeat lease renewal
+stale-running recovery
+DLQ requeue
+retention cleanup
+schedule upsert/remove
+schedule lease acquisition
+schedule completion/release
+```
+
+Queue reservation atomically removes one eligible job from the available index, marks it running and writes the visibility lease before another worker can reserve it.
+
+## Redis-backed scheduler
+
+The same Redis command client can back the scheduling contract:
+
+```ts
+import {
+    createJobScheduler,
+    createRedisJobScheduleStore,
+} from "bcp/jobs";
+
+const scheduleStore =
+    createRedisJobScheduleStore({
+        client: redisCommandClient,
+        namespace: "my-app:{jobs}",
+    });
+
+export const scheduler =
+    createJobScheduler({
+        queue: jobs,
+        store: scheduleStore,
+        ownerId: "scheduler-a",
+    });
+```
+
+`acquireDue()` is implemented with an atomic Redis lease, allowing multiple scheduler processes to share the same schedule store.
+
+## Recommended production topology
+
+```text
+Web/API instances
+      |
+      | enqueue / schedule
+      v
+Redis-compatible shared job state
+      |
+      +-------------------+
+      |                   |
+      v                   v
+Worker A              Worker B
+visibility lease      visibility lease
+heartbeat             heartbeat
+      |
+      +---- failure ----> retry / DLQ
+
+Scheduler A -----+
+                 +---- shared schedule lease
+Scheduler B -----+
+```
+
+Queue workers and schedulers can run in the same Node process for small deployments, but separate worker processes/containers are recommended when job load can affect web latency.
+
+## Adapter compatibility
+
+Existing `0.2.8` `JobQueueAdapter` implementations remain valid. Durable methods are optional:
+
+```ts
+heartbeat?()
+recoverStale?()
+listDeadLetters?()
+requeueDeadLetter?()
+cleanup?()
+stats?()
+```
+
+The `reserve()` method now receives an optional second argument with the worker owner ID and visibility timeout. Existing JavaScript adapters that ignore the optional argument continue to work.
+
+For production durable adapters, implementations should support the lease contract rather than ignoring it.
+
+## Failure model
+
+BCP jobs intentionally use practical at-least-once delivery semantics.
+
+Applications should assume that a job can execute more than once when:
+
+- a worker completes external side effects but crashes before recording success,
+- a visibility lease expires during a long or blocked handler,
+- a DLQ job is manually requeued,
+- infrastructure retries a command after an ambiguous network result.
+
+Use idempotency keys, unique database constraints or transactional outbox/inbox patterns for side effects where duplicates are unsafe.
+
+## Observability
+
+`jobs.stats()` can feed application metrics from `bcp/observability`:
+
+```ts
+const stats =
+    await jobs.stats();
+
+queueDepth.set(
+    stats.queued
+);
+
+dlqDepth.set(
+    stats.deadLetters
+);
+```
+
+For Redis deployments, infrastructure-level Redis monitoring should complement application-level BCP metrics.
+
+## Related guides
+
+- [Background Jobs Platform](background-jobs.md)
+- [Job Scheduling Platform](job-scheduling.md)
+- [Observability Platform v2](observability.md)
+- [Production Hardening](production-hardening.md)

package/docs/job-scheduling.md

@@ -0,0 +1,357 @@
+# Job Scheduling Platform
+
+BCP Framework `0.2.9` extends the Background Jobs Platform with recurring job schedules, dependency-free UTC cron expressions, interval schedules and a schedule-store lease contract for multi-instance deployments.
+
+Import scheduling APIs from the server-only `bcp/jobs` entrypoint:
+
+```ts
+import {
+    createJobQueue,
+    createJobScheduler,
+} from "bcp/jobs";
+```
+
+## Create a scheduler
+
+A scheduler enqueues work into an existing BCP background job queue:
+
+```ts
+import {
+    createJobQueue,
+    createJobScheduler,
+} from "bcp/jobs";
+
+export const jobs =
+    createJobQueue();
+
+export const scheduler =
+    createJobScheduler({
+        queue: jobs,
+    });
+```
+
+The scheduler does not execute application work itself. It creates normal queue jobs when a schedule becomes due; queue workers execute the registered job handlers.
+
+## Interval schedules
+
+Schedule a recurring job every five minutes:
+
+```ts
+await scheduler.schedule(
+    "cache.cleanup",
+    {
+        scope: "expired",
+    },
+    {
+        id: "cache-cleanup",
+        everyMs: 5 * 60 * 1000,
+    }
+);
+```
+
+`everyMs` must be a positive integer.
+
+By default the first run occurs one interval after the schedule is created.
+
+Use `startAt` to select the first due time explicitly:
+
+```ts
+await scheduler.schedule(
+    "report.daily",
+    {},
+    {
+        everyMs:
+            24 * 60 * 60 * 1000,
+        startAt:
+            Date.now() + 10_000,
+    }
+);
+```
+
+`startAt` accepts either a millisecond timestamp or a `Date`.
+
+## Cron schedules
+
+BCP `0.2.9` supports dependency-free five-field cron expressions:
+
+```text
+minute hour day-of-month month day-of-week
+```
+
+Example — every weekday at 09:30 UTC:
+
+```ts
+await scheduler.schedule(
+    "report.weekday",
+    {},
+    {
+        cron: "30 9 * * 1-5",
+    }
+);
+```
+
+Example — every 15 minutes:
+
+```ts
+await scheduler.schedule(
+    "sync.incremental",
+    {},
+    {
+        cron: "*/15 * * * *",
+    }
+);
+```
+
+Supported field syntax:
+
+```text
+*          every allowed value
+*/n        step
+1,5,10     list
+1-5        range
+1-10/2     range with step
+```
+
+Ranges:
+
+```text
+minute        0-59
+hour          0-23
+day-of-month  1-31
+month         1-12
+day-of-week   0-7 (0 and 7 are Sunday)
+```
+
+Cron evaluation is **UTC** in `0.2.9`. This avoids machine-local timezone differences between containers and servers.
+
+When both day-of-month and day-of-week are restricted, BCP follows conventional cron OR semantics: either field may match.
+
+Use `nextCronTime()` when application tooling needs to preview the next UTC occurrence:
+
+```ts
+import {
+    nextCronTime,
+} from "bcp/jobs";
+
+const next =
+    nextCronTime(
+        "0 2 * * *",
+        Date.now()
+    );
+```
+
+## Run due schedules manually
+
+`runDue()` is useful for deterministic tests or externally driven schedulers:
+
+```ts
+const count =
+    await scheduler.runDue();
+```
+
+The return value is the number of newly enqueued jobs.
+
+Options:
+
+```ts
+await scheduler.runDue({
+    limit: 100,
+    leaseMs: 30_000,
+});
+```
+
+Each call claims at most one occurrence per due schedule. If an application was offline for many intervals, later calls may progressively catch up.
+
+## Start a scheduler runner
+
+For a long-running Node.js process:
+
+```ts
+const runner =
+    scheduler.start({
+        pollIntervalMs: 1_000,
+        leaseMs: 30_000,
+        limit: 100,
+        onError(error) {
+            console.error(
+                "scheduler failure",
+                error
+            );
+        },
+    });
+```
+
+Graceful shutdown:
+
+```ts
+await runner.stop();
+await scheduler.close();
+await jobs.close();
+```
+
+Stopping a scheduler does not close the queue automatically because applications may run queue workers independently.
+
+## Schedule records
+
+`JobScheduleRecord` includes:
+
+```text
+id
+jobName
+payload
+schedule
+createdAt
+updatedAt
+nextRunAt
+lastRunAt
+maxAttempts
+leaseOwner
+leaseUntil
+```
+
+Inspect schedules:
+
+```ts
+const schedules =
+    await scheduler.list();
+
+const schedule =
+    await scheduler.get(
+        "cache-cleanup"
+    );
+```
+
+Remove one:
+
+```ts
+await scheduler.remove(
+    "cache-cleanup"
+);
+```
+
+Calling `schedule()` with an existing schedule id is an upsert. Use stable ids for application-owned recurring schedules.
+
+## Retry policy
+
+A schedule may forward a queue retry limit:
+
+```ts
+await scheduler.schedule(
+    "billing.reconcile",
+    {},
+    {
+        cron: "0 * * * *",
+        maxAttempts: 5,
+    }
+);
+```
+
+Retry timing is still owned by `createJobQueue()`:
+
+```ts
+const jobs =
+    createJobQueue({
+        retryDelayMs: attempt =>
+            attempt * 5_000,
+    });
+```
+
+The scheduler determines **when a recurring occurrence is created**. The queue determines **how that occurrence is retried after handler failure**.
+
+## Deterministic scheduled run IDs
+
+Scheduled occurrences receive deterministic queue ids:
+
+```text
+schedule:<schedule-id>:<scheduled-for-timestamp>
+```
+
+This provides an additional duplicate-enqueue defense if a scheduler retries the same occurrence.
+
+Applications should avoid manually creating queue job ids with the `schedule:` prefix.
+
+## Schedule store contract
+
+The built-in schedule store is process-local:
+
+```ts
+import {
+    createMemoryJobScheduleStore,
+} from "bcp/jobs";
+```
+
+It is useful for development, tests and single-process applications.
+
+Production deployments that run multiple scheduler processes or must preserve schedules across restarts should implement `JobScheduleStore` using shared durable infrastructure.
+
+Required operations:
+
+```text
+upsert
+get
+list
+remove
+acquireDue
+complete
+release
+close (optional)
+```
+
+`acquireDue()` is the critical concurrency boundary. A durable implementation should atomically claim due schedules and assign a lease owner/expiry so two scheduler instances do not both own the same occurrence.
+
+Conceptually:
+
+```text
+scheduler A ─┐
+             ├─ shared JobScheduleStore ── atomic acquireDue()
+scheduler B ─┘
+                         │
+                         ▼
+                  shared/durable queue
+```
+
+Examples of possible application adapters include PostgreSQL, Redis, DynamoDB or another transactional/atomic data store. `0.2.9` intentionally does not force one provider.
+
+## Lease behavior
+
+A scheduler runner identifies itself with `ownerId`:
+
+```ts
+const scheduler =
+    createJobScheduler({
+        queue: jobs,
+        store,
+        ownerId:
+            process.env.INSTANCE_ID,
+    });
+```
+
+If `ownerId` is omitted, BCP generates a unique process-local scheduler id.
+
+A due schedule is leased for `leaseMs`. The store releases the lease after the occurrence is committed. If a scheduler dies while holding a lease, another instance may reclaim it after lease expiry.
+
+For durable adapters, choose a lease duration longer than the expected enqueue/store round trip but short enough for acceptable recovery.
+
+## Queue durability remains separate
+
+A durable schedule store does not make the queue durable by itself.
+
+For multi-instance production you will normally need both:
+
+```text
+shared JobScheduleStore
++
+shared JobQueueAdapter
+```
+
+The built-in memory schedule store and memory queue are intentionally process-local.
+
+## Server-only boundary
+
+`bcp/jobs` is server-only. Do not import it from page components, client islands or other modules reachable from browser bundles.
+
+Use job/scheduler calls from server routes, loaders/actions, server startup modules or dedicated worker processes.
+
+## Compatibility
+
+`0.2.9` does not intentionally break `0.2.8` Background Jobs APIs. Existing queue/worker code remains valid; scheduling is additive.