@chidchanun/bcp 0.2.10 → 0.2.12

package/docs/transactional-outbox-events.md

@@ -0,0 +1,465 @@
+# Transactional Outbox & Events
+
+BCP `0.2.12` adds a server-only `bcp/events` entrypoint for storing integration events in the same database transaction as application data and dispatching those events after commit.
+
+## Why an outbox exists
+
+This is unsafe when the database and message system are separate resources:
+
+```ts
+await db.transaction(async tx => {
+    await tx.execute(
+        "INSERT INTO orders ..."
+    );
+
+    await jobs.enqueue(
+        "event.order.created",
+        payload
+    );
+});
+```
+
+The database can commit while the queue publish fails, or the queue can publish before the database transaction ultimately rolls back.
+
+The transactional outbox changes the write path to:
+
+```text
+Database transaction
+    |
+    +-- business row
+    +-- outbox event row
+             |
+          COMMIT
+             |
+             v
+      Outbox Dispatcher
+             |
+      queue / publisher
+```
+
+The business write and outbox write share one database transaction. External delivery happens later.
+
+## Public entrypoint
+
+```ts
+import {
+    createOutboxDispatcher,
+    createOutboxMigrationSql,
+    createSqlOutboxStore,
+    createTransactionalOutbox,
+} from "bcp/events";
+```
+
+`bcp/events` is server-only and must not be imported into page/client bundles.
+
+## Create the outbox table
+
+Generate provider-specific SQL:
+
+```ts
+import {
+    createOutboxMigrationSql,
+} from "bcp/events";
+
+const sql =
+    createOutboxMigrationSql(
+        "postgresql"
+    );
+```
+
+Supported providers match the SQL Database Platform:
+
+```text
+mysql
+postgresql
+sqlite
+```
+
+The default table name is:
+
+```text
+bcp_outbox_events
+```
+
+A custom table name must be a simple SQL identifier.
+
+Use the generated SQL in a normal BCP migration. The outbox table stores event identity/type/payload/metadata, delivery state, retry counters, timestamps and dispatcher lease information.
+
+## Create a SQL outbox
+
+```ts
+import {
+    db,
+} from "bcp/database";
+
+import {
+    createSqlOutboxStore,
+    createTransactionalOutbox,
+} from "bcp/events";
+
+export const outboxStore =
+    createSqlOutboxStore({
+        database: db,
+        driver: "postgresql",
+    });
+
+export const outbox =
+    createTransactionalOutbox({
+        store: outboxStore,
+        defaultMaxAttempts: 5,
+    });
+```
+
+## Publish inside the business transaction
+
+Pass the `TransactionDatabase` supplied by `db.transaction()` directly to `outbox.publish()`:
+
+```ts
+await db.transaction(
+    async tx => {
+        await tx.execute(
+            `INSERT INTO orders (
+                id,
+                user_id,
+                total
+            ) VALUES ($1, $2, $3)`,
+            [
+                order.id,
+                order.userId,
+                order.total,
+            ]
+        );
+
+        await outbox.publish(
+            tx,
+            "order.created",
+            {
+                orderId: order.id,
+                userId: order.userId,
+            },
+            {
+                aggregateId:
+                    String(order.id),
+                correlationId:
+                    requestId,
+            }
+        );
+    }
+);
+```
+
+For MySQL or SQLite, use the placeholder syntax expected by that database in your business SQL. `createSqlOutboxStore()` handles its own provider-specific placeholders internally.
+
+Do not publish to Redis, RabbitMQ, HTTP or another external system from inside this transaction. Only write the outbox row there.
+
+## Event metadata
+
+`publish()` accepts:
+
+```ts
+await outbox.publish(
+    tx,
+    "payment.captured",
+    payload,
+    {
+        id: eventId,
+        metadata: {
+            source: "checkout",
+        },
+        correlationId: requestId,
+        causationId: previousEventId,
+        aggregateId: paymentId,
+        delayMs: 1_000,
+        maxAttempts: 8,
+    }
+);
+```
+
+Fields have these intended uses:
+
+- `id`: stable event/idempotency identifier.
+- `correlationId`: groups work belonging to one request or distributed operation.
+- `causationId`: identifies the event/command that caused this event.
+- `aggregateId`: identifies the domain resource associated with the event.
+- `metadata`: application-defined non-secret event metadata.
+
+## Dispatcher
+
+The dispatcher claims due events using a lease, delivers them, then marks the outbox row published or schedules another attempt.
+
+```ts
+import {
+    createOutboxDispatcher,
+} from "bcp/events";
+
+export const dispatcher =
+    createOutboxDispatcher({
+        store: outboxStore,
+        queue: jobs,
+        ownerId: "outbox-a",
+        leaseMs: 30_000,
+        batchSize: 100,
+        pollIntervalMs: 1_000,
+    });
+```
+
+Start it:
+
+```ts
+const runner =
+    dispatcher.start();
+```
+
+Shutdown:
+
+```ts
+await runner.stop();
+await dispatcher.close();
+```
+
+## Durable Jobs delivery
+
+When `queue` is configured, event type `order.created` is enqueued as:
+
+```text
+event.order.created
+```
+
+Register the consumer normally:
+
+```ts
+jobs.register<{
+    eventId: string;
+    type: string;
+    payload: {
+        orderId: number;
+    };
+}>(
+    "event.order.created",
+    async ({ payload }) => {
+        await notifyOrderCreated(
+            payload.payload.orderId
+        );
+    }
+);
+```
+
+The job payload is an envelope containing:
+
+```text
+eventId
+type
+payload
+metadata
+correlationId
+causationId
+aggregateId
+createdAt
+```
+
+The queue job ID is derived from the outbox event ID:
+
+```text
+outbox:<event-id>
+```
+
+This provides a stable handoff identity. Durable job handlers should still be idempotent because the overall delivery model is at-least-once.
+
+## Custom publisher
+
+A dispatcher can publish to an application-owned broker/API instead of the BCP job queue:
+
+```ts
+const dispatcher =
+    createOutboxDispatcher({
+        store: outboxStore,
+        publish: async event => {
+            await broker.publish(
+                event.type,
+                event
+            );
+        },
+    });
+```
+
+BCP intentionally does not require RabbitMQ, Kafka, NATS or another event-broker dependency.
+
+## In-process event bus
+
+For local application delivery:
+
+```ts
+import {
+    createEventBus,
+} from "bcp/events";
+
+const bus =
+    createEventBus();
+
+bus.on(
+    "account.updated",
+    async ({ payload }) => {
+        await refreshAccountCache(
+            payload
+        );
+    }
+);
+```
+
+Then pass `eventBus: bus` to `createOutboxDispatcher()`.
+
+The in-process bus is not a durable distributed broker. Use it for local handlers, tests or composition, not as a replacement for a shared queue in multi-instance deployments.
+
+## Retry and terminal failure
+
+A claimed event increments `attempts`.
+
+If delivery throws and attempts remain, the dispatcher returns the event to `pending` with a future `availableAt`.
+
+The default delay is exponential with a 60-second cap. Override it:
+
+```ts
+createOutboxDispatcher({
+    store: outboxStore,
+    publish,
+    retryDelayMs:
+        attempt =>
+            attempt * 2_000,
+});
+```
+
+When `maxAttempts` is exhausted the event becomes:
+
+```text
+failed
+```
+
+The record remains available for inspection/retention until cleanup.
+
+## Leases and stale recovery
+
+Claimed events use:
+
+```text
+state = processing
+leaseOwner
+leaseUntil
+```
+
+If a dispatcher crashes before acknowledgement, another dispatcher can recover the expired lease and return the event to `pending`.
+
+Manual recovery:
+
+```ts
+await dispatcher.recoverStale(
+    100
+);
+```
+
+Production SQL stores perform claim/update operations through the configured BCP database transaction boundary. PostgreSQL/MySQL claims also use `FOR UPDATE SKIP LOCKED`; SQLite uses its transactional write serialization and conditional state updates.
+
+## States
+
+```text
+pending
+   |
+   v
+processing
+   | \
+   |  \ delivery error + attempts left
+   |   -----------------> pending
+   |
+   +--------------------> published
+   |
+   + retry exhausted ---> failed
+```
+
+`published` means the configured dispatcher destination accepted the event. For job-queue delivery, it means enqueue succeeded; it does not mean the downstream job handler has completed.
+
+## Statistics
+
+```ts
+const stats =
+    await outboxStore.stats();
+```
+
+Shape:
+
+```ts
+{
+    total: number;
+    pending: number;
+    processing: number;
+    published: number;
+    failed: number;
+}
+```
+
+## Retention cleanup
+
+Remove old terminal records:
+
+```ts
+await outboxStore.cleanup({
+    before:
+        Date.now() -
+        7 * 24 * 60 * 60 * 1000,
+});
+```
+
+Only `published` and `failed` records are eligible. Pending or processing records are preserved.
+
+## Memory store
+
+For tests and single-process development:
+
+```ts
+import {
+    createMemoryOutboxStore,
+} from "bcp/events";
+```
+
+The memory store demonstrates the `OutboxStore` contract but is not transactionally durable. Its `append()` accepts the transaction parameter for API compatibility, but its data lives only in process memory.
+
+Use `createSqlOutboxStore()` when the outbox must participate in the same durable database transaction as application writes.
+
+## Delivery guarantees
+
+The outbox prevents the database-write / external-publish gap, but it does not provide exactly-once side effects.
+
+Treat delivery as:
+
+```text
+at-least-once
+```
+
+Use stable event IDs and consumer-level idempotency for external side effects such as charging cards, sending one-time notifications or calling third-party APIs.
+
+## Production topology
+
+```text
+API / Form Action
+      |
+      v
+Database transaction
+      |
+      +-- business data
+      +-- outbox event
+      |
+    COMMIT
+      |
+      v
+Shared SQL Outbox
+      |
+  +---+---+
+  |       |
+Dispatcher A  Dispatcher B
+      |
+      v
+Durable queue / broker
+      |
+  +---+---+
+Worker A  Worker B
+```
+
+This allows multiple API, dispatcher and worker instances without coupling the business transaction to the external broker availability window.