@outbox-event-bus/sqlite-better-sqlite3-outbox 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md ADDED
@@ -0,0 +1,759 @@
1
+ # SQLite Better-SQLite3 Outbox
2
+
3
+ ![npm version](https://img.shields.io/npm/v/@outbox-event-bus/sqlite-better-sqlite3-outbox?style=flat-square&color=2563eb)
4
+ ![npm downloads](https://img.shields.io/npm/dm/@outbox-event-bus/sqlite-better-sqlite3-outbox?style=flat-square&color=2563eb)
5
+ ![license](https://img.shields.io/npm/l/@outbox-event-bus/sqlite-better-sqlite3-outbox?style=flat-square&color=2563eb)
6
+ ![TypeScript](https://img.shields.io/badge/TypeScript-5.0+-blue?style=flat-square)
7
+
8
+ > **Reliable event storage for single-instance applications—zero external dependencies.**
9
+
10
+ A SQLite adapter for the [Outbox Pattern](https://github.com/dunika/outbox-event-bus#readme), providing transactional event storage with automatic retry, archiving, and WAL-mode concurrency. Perfect for local development, testing, desktop applications, and single-instance deployments.
11
+
12
+ ## When to Use
13
+
14
+ | Feature | SQLite | Redis | PostgreSQL | DynamoDB |
15
+ |---------|--------|-------|------------|----------|
16
+ | **Setup Complexity** | ⭐ Zero config | ⭐⭐ Docker/Cloud | ⭐⭐⭐ Server required | ⭐⭐⭐ AWS setup |
17
+ | **Horizontal Scaling** | ❌ Single instance | ✅ Yes | ✅ Yes | ✅ Yes |
18
+ | **Write Throughput** | ~1K events/sec | ~10K events/sec | ~5K events/sec | ~10K events/sec |
19
+ | **Best For** | Dev, CLI, Desktop | High throughput | ACID guarantees | Cloud-native |
20
+
21
+ **Choose SQLite when:**
22
+ - You're in **local development** or testing
23
+ - You have a **single-instance deployment** (no horizontal scaling)
24
+ - You want **zero external dependencies**
25
+ - You're building a **desktop application** or CLI tool
26
+
27
+ **Consider alternatives when:**
28
+ - You need **horizontal scaling** across multiple servers → use [Redis](https://github.com/dunika/outbox-event-bus/tree/main/adapters/redis-ioredis#readme) or [DynamoDB](https://github.com/dunika/outbox-event-bus/tree/main/adapters/dynamodb-aws-sdk#readme)
29
+ - You require **high write throughput** (>1K events/sec) → SQLite serializes writes
30
+ - You want **cloud-native deployment** → use managed database services
31
+
32
+ ---
33
+
34
+ ## Installation
35
+
36
+ ```bash
37
+ npm install @outbox-event-bus/sqlite-better-sqlite3-outbox outbox-event-bus
38
+ ```
39
+
40
+ ---
41
+
42
+ ## Getting Started
43
+
44
+ ### 1. Create Your First Event Bus
45
+
46
+ ```typescript
47
+ import { SqliteBetterSqlite3Outbox } from '@outbox-event-bus/sqlite-better-sqlite3-outbox';
48
+ import { OutboxEventBus } from 'outbox-event-bus';
49
+
50
+ // 1. Create the outbox
51
+ const outbox = new SqliteBetterSqlite3Outbox({
52
+ dbPath: './events.db'
53
+ });
54
+
55
+ // 2. Create the bus
56
+ const bus = new OutboxEventBus(outbox, console.error);
57
+
58
+ // 3. Listen for events
59
+ bus.on('user.created', async (event) => {
60
+ console.log('New user:', event.payload);
61
+ });
62
+
63
+ // 4. Start processing
64
+ bus.start();
65
+
66
+ // 5. Emit an event
67
+ await bus.emit({
68
+ id: crypto.randomUUID(),
69
+ type: 'user.created',
70
+ payload: { name: 'Alice' }
71
+ });
72
+ ```
73
+
74
+ ### 2. Verify It Works
75
+
76
+ Check your database to see the archived event:
77
+
78
+ ```bash
79
+ sqlite3 events.db "SELECT * FROM outbox_events_archive;"
80
+ ```
81
+
82
+ > [!TIP]
83
+ > Use `:memory:` for blazing-fast in-memory testing without disk I/O:
84
+ > ```typescript
85
+ > const outbox = new SqliteBetterSqlite3Outbox({ dbPath: ':memory:' });
86
+ > ```
87
+
88
+ ---
89
+
90
+ ## Features
91
+
92
+ ### 🔄 Automatic Retry with Exponential Backoff
93
+
94
+ Failed events are automatically retried up to 5 times (configurable) with exponentially increasing delays between attempts.
95
+
96
+ ```typescript
97
+ const outbox = new SqliteBetterSqlite3Outbox({
98
+ maxRetries: 3,
99
+ baseBackoffMs: 1000 // Delays: 1s, 2s, 4s
100
+ });
101
+ ```
102
+
103
+ ### 🔒 ACID Transactions
104
+
105
+ Events are committed atomically with your business data—no partial writes, guaranteed consistency.
106
+
107
+ ```typescript
108
+ db.transaction(() => {
109
+ db.prepare('INSERT INTO users (name) VALUES (?)').run('Alice');
110
+ bus.emit({ type: 'user.created', payload: { name: 'Alice' } });
111
+ })(); // Both committed together or both rolled back
112
+ ```
113
+
114
+ ### 📦 Auto-Archiving
115
+
116
+ Completed events are automatically moved to `outbox_events_archive` for audit trails without bloating the active table.
117
+
118
+ ### 🛡️ Stuck Event Recovery
119
+
120
+ Events that timeout during processing are automatically reclaimed and retried, preventing lost events.
121
+
122
+ ### 🔍 Failed Event Inspection
123
+
124
+ Query and manually retry failed events:
125
+
126
+ ```typescript
127
+ const failed = await outbox.getFailedEvents();
128
+ console.log(failed[0].error); // Last error message
129
+ await outbox.retryEvents(failed.map(e => e.id));
130
+ ```
131
+
132
+ ---
133
+
134
+ ## Concurrency & Locking
135
+
136
+ SQLite is designed for **single-instance deployments** and uses **file-level locking** for concurrency control.
137
+
138
+ - **Single Writer**: Only one write transaction can execute at a time (serialized writes)
139
+ - **WAL Mode**: Enables concurrent reads during writes (readers don't block writers)
140
+ - **No Distributed Locking**: Not suitable for horizontal scaling across multiple servers
141
+ - **Thread-Safe**: Safe for multi-threaded applications within a single process
142
+
143
+ > [!WARNING]
144
+ > Do not run multiple instances of your application with the same SQLite database file. This can lead to database corruption. For multi-instance deployments, use [PostgreSQL](https://github.com/dunika/outbox-event-bus/tree/main/adapters/postgres-prisma#readme), [Redis](https://github.com/dunika/outbox-event-bus/tree/main/adapters/redis-ioredis#readme), or [DynamoDB](https://github.com/dunika/outbox-event-bus/tree/main/adapters/dynamodb-aws-sdk#readme) adapters instead.
145
+
146
+ ### ⚡ WAL Mode for Concurrency
147
+
148
+ Write-Ahead Logging (WAL) enables concurrent reads during writes, improving throughput and reducing lock contention.
149
+
150
+ **Why WAL matters:**
151
+ - ✅ Readers don't block writers
152
+ - ✅ Writers don't block readers
153
+ - ✅ Better performance for write-heavy workloads
154
+ - ✅ Crash recovery without data loss
155
+
156
+ Without WAL, SQLite uses rollback journaling, which blocks all reads during writes.
157
+
158
+ ### 🎨 Custom Table Names
159
+
160
+ Integrate with existing schemas by customizing table names:
161
+
162
+ ```typescript
163
+ const outbox = new SqliteBetterSqlite3Outbox({
164
+ dbPath: './app.db',
165
+ tableName: 'my_events',
166
+ archiveTableName: 'my_events_history'
167
+ });
168
+ ```
169
+
170
+ ### 📘 Full TypeScript Support
171
+
172
+ Complete type definitions for all APIs and configurations with full IntelliSense support.
173
+
174
+ ---
175
+
176
+ ## Transactions
177
+
178
+ ### With AsyncLocalStorage (Recommended)
179
+
180
+ Use `AsyncLocalStorage` to manage SQLite transactions, ensuring outbox events are committed along with your business data.
181
+
182
+ > [!NOTE]
183
+ > better-sqlite3 transactions are synchronous, but `bus.emit()` is async. The recommended pattern is to call `emit()` synchronously within the transaction (it queues the write) and the actual I/O happens immediately since better-sqlite3 is synchronous.
184
+
185
+ ```typescript
186
+ import Database from 'better-sqlite3';
187
+ import { AsyncLocalStorage } from 'node:async_hooks';
188
+
189
+ const als = new AsyncLocalStorage<Database.Database>();
190
+
191
+ const outbox = new SqliteBetterSqlite3Outbox({
192
+ dbPath: './data/events.db',
193
+ getTransaction: () => als.getStore()
194
+ });
195
+
196
+ const bus = new OutboxEventBus(outbox, (error) => console.error(error));
197
+
198
+ async function createUser(user: any) {
199
+ const db = new Database('./data/events.db');
200
+
201
+ // Run the transaction synchronously
202
+ const transaction = db.transaction(() => {
203
+ // Set ALS context for the transaction
204
+ return als.run(db, () => {
205
+ // 1. Save business data
206
+ db.prepare('INSERT INTO users (name) VALUES (?)').run(user.name);
207
+
208
+ // 2. Emit event (synchronously writes to outbox table via ALS)
209
+ void bus.emit({
210
+ id: crypto.randomUUID(),
211
+ type: 'user.created',
212
+ payload: user
213
+ });
214
+ });
215
+ });
216
+
217
+ // Execute the transaction
218
+ transaction();
219
+ }
220
+ ```
221
+
222
+ ### With AsyncLocalStorage Helper
223
+
224
+ Use the provided `withBetterSqlite3Transaction` helper for cleaner async transaction management:
225
+
226
+ ```typescript
227
+ import {
228
+ SqliteBetterSqlite3Outbox,
229
+ withBetterSqlite3Transaction,
230
+ getBetterSqlite3Transaction
231
+ } from '@outbox-event-bus/sqlite-better-sqlite3-outbox';
232
+ import Database from 'better-sqlite3';
233
+
234
+ const db = new Database('./events.db');
235
+
236
+ const outbox = new SqliteBetterSqlite3Outbox({
237
+ db,
238
+ getTransaction: getBetterSqlite3Transaction()
239
+ });
240
+
241
+ const bus = new OutboxEventBus(outbox, console.error);
242
+
243
+ async function createUser(user: any) {
244
+ await withBetterSqlite3Transaction(db, async (tx) => {
245
+ // 1. Save business data
246
+ tx.prepare('INSERT INTO users (name) VALUES (?)').run(user.name);
247
+
248
+ // 2. Emit event (uses transaction from AsyncLocalStorage)
249
+ await bus.emit({
250
+ id: crypto.randomUUID(),
251
+ type: 'user.created',
252
+ payload: user
253
+ });
254
+ });
255
+ }
256
+ ```
257
+
258
+ ### With Explicit Transaction
259
+
260
+ You can also pass the SQLite database instance explicitly to `emit`:
261
+
262
+ ```typescript
263
+ const db = new Database('./data/events.db');
264
+
265
+ const transaction = db.transaction(() => {
266
+ // 1. Save business data
267
+ db.prepare('INSERT INTO users (name) VALUES (?)').run(user.name);
268
+
269
+ // 2. Emit event (passing the db explicitly)
270
+ void bus.emit({
271
+ id: crypto.randomUUID(),
272
+ type: 'user.created',
273
+ payload: user
274
+ }, db);
275
+ });
276
+
277
+ transaction();
278
+ ```
279
+
280
+ ---
281
+
282
+ ## How-to Guides
283
+
284
+ ### Retry Failed Events
285
+
286
+ ```typescript
287
+ // 1. Get all failed events
288
+ const failed = await outbox.getFailedEvents();
289
+
290
+ // 2. Inspect errors
291
+ for (const event of failed) {
292
+ console.log(`Event ${event.id} failed ${event.retryCount} times`);
293
+ console.log(`Last error: ${event.error}`);
294
+ }
295
+
296
+ // 3. Retry specific events
297
+ const retryable = failed.filter(e => e.retryCount < 3);
298
+ await outbox.retryEvents(retryable.map(e => e.id));
299
+ ```
300
+
301
+ ### Debug Stuck Events
302
+
303
+ Find events stuck in `active` state:
304
+
305
+ ```sql
306
+ SELECT * FROM outbox_events
307
+ WHERE status = 'active'
308
+ AND datetime(keep_alive, '+' || expire_in_seconds || ' seconds') < datetime('now');
309
+ ```
310
+
311
+ These events will be automatically reclaimed on the next polling cycle.
312
+
313
+ ### Use an Existing Database Instance
314
+
315
+ ```typescript
316
+ import Database from 'better-sqlite3';
317
+
318
+ const db = new Database('./app.db');
319
+ db.pragma('journal_mode = WAL');
320
+
321
+ const outbox = new SqliteBetterSqlite3Outbox({ db });
322
+ ```
323
+
324
+ ### Graceful Shutdown
325
+
326
+ ```typescript
327
+ process.on('SIGTERM', async () => {
328
+ console.log('Shutting down...');
329
+ await bus.stop(); // Wait for in-flight events to complete
330
+ process.exit(0);
331
+ });
332
+ ```
333
+
334
+ ### Query the Archive Table
335
+
336
+ ```typescript
337
+ import Database from 'better-sqlite3';
338
+
339
+ const db = new Database('./events.db');
340
+
341
+ const archived = db.prepare(`
342
+ SELECT * FROM outbox_events_archive
343
+ WHERE type = ?
344
+ ORDER BY completed_on DESC
345
+ LIMIT 100
346
+ `).all('user.created');
347
+
348
+ console.log(archived);
349
+ ```
350
+
351
+ ---
352
+
353
+ ## Configuration
354
+
355
+ ### SqliteBetterSqlite3OutboxConfig
356
+
357
+ ```typescript
358
+ interface SqliteBetterSqlite3OutboxConfig extends OutboxConfig {
359
+ // SQLite-specific options
360
+ dbPath?: string;
361
+ db?: Database.Database;
362
+ getTransaction?: () => Database.Database | undefined;
363
+ tableName?: string;
364
+ archiveTableName?: string;
365
+ // Inherited from OutboxConfig
366
+ maxRetries?: number;
367
+ baseBackoffMs?: number;
368
+ processingTimeoutMs?: number; // Processing timeout (default: 30000ms)
369
+ pollIntervalMs?: number;
370
+ batchSize?: number;
371
+ maxErrorBackoffMs?: number; // Max polling error backoff (default: 30000ms)
372
+ }
373
+ ```
374
+
375
+ > [!NOTE]
376
+ > All adapters inherit base configuration options from `OutboxConfig`. See the [API Reference](https://github.com/dunika/outbox-event-bus/blob/main/docs/API_REFERENCE.md#base-outbox-configuration) for details on inherited options.
377
+
378
+ | Option | Type | Default | Description |
379
+ |--------|------|---------|-------------|
380
+ | `dbPath` | `string` | - | Path to SQLite file. **Required if `db` not provided.** Use `:memory:` for in-memory. |
381
+ | `db` | `Database` | - | Existing better-sqlite3 instance. If provided, `dbPath` is ignored. WAL mode must be enabled manually. |
382
+ | `getTransaction` | `() => Database \| undefined` | - | Function to retrieve current transaction from AsyncLocalStorage. Use `getBetterSqlite3Transaction()` helper. |
383
+ | `tableName` | `string` | `"outbox_events"` | Name of the outbox table. |
384
+ | `archiveTableName` | `string` | `"outbox_events_archive"` | Name of the archive table. |
385
+ | `maxRetries` | `number` | `5` | Maximum retry attempts before marking event as permanently failed. |
386
+ | `baseBackoffMs` | `number` | `1000` | Base delay for exponential backoff (ms). Retry delays: 1s, 2s, 4s, 8s, 16s. |
387
+ | `processingTimeoutMs` | `number` | `30000` | Timeout for event handler execution (ms). Events exceeding this are marked as stuck. |
388
+ | `pollIntervalMs` | `number` | `1000` | Interval between polling cycles (ms). |
389
+ | `batchSize` | `number` | `50` | Maximum events to process per batch. |
390
+ | `maxErrorBackoffMs` | `number` | `30000` | Maximum backoff delay after polling errors (ms). |
391
+
392
+ > [!WARNING]
393
+ > You must provide either `dbPath` or `db`. If neither is provided, the constructor will throw an error.
394
+
395
+ ---
396
+
397
+ ## API Reference
398
+
399
+ ### `SqliteBetterSqlite3Outbox`
400
+
401
+ #### Constructor
402
+
403
+ ```typescript
404
+ new SqliteBetterSqlite3Outbox(config: SqliteBetterSqlite3OutboxConfig)
405
+ ```
406
+
407
+ Creates a new SQLite outbox adapter. Automatically creates tables and indexes on initialization.
408
+
409
+ **Throws:**
410
+ - `Error` if neither `dbPath` nor `db` is provided
411
+
412
+ **Example:**
413
+ ```typescript
414
+ const outbox = new SqliteBetterSqlite3Outbox({
415
+ dbPath: './events.db',
416
+ maxRetries: 3,
417
+ batchSize: 100
418
+ });
419
+ ```
420
+
421
+ ---
422
+
423
+ #### Methods
424
+
425
+ ##### `publish(events: BusEvent[], transaction?: Database): Promise<void>`
426
+
427
+ Inserts events into the outbox table. Events are inserted with `status = 'created'` and will be picked up by the next polling cycle.
428
+
429
+ **Parameters:**
430
+ - `events` - Array of events to publish
431
+ - `transaction` - Optional better-sqlite3 database instance for transactional writes
432
+
433
+ **Example:**
434
+ ```typescript
435
+ await outbox.publish([
436
+ {
437
+ id: crypto.randomUUID(),
438
+ type: 'user.created',
439
+ payload: { name: 'Alice' },
440
+ occurredAt: new Date()
441
+ }
442
+ ]);
443
+ ```
444
+
445
+ ---
446
+
447
+ ##### `getFailedEvents(): Promise<FailedBusEvent[]>`
448
+
449
+ Retrieves up to 100 failed events, ordered by occurrence time (newest first).
450
+
451
+ **Returns:** Array of `FailedBusEvent` objects with error details and retry count.
452
+
453
+ **Example:**
454
+ ```typescript
455
+ const failed = await outbox.getFailedEvents();
456
+ for (const event of failed) {
457
+ console.log(`Event ${event.id}:`);
458
+ console.log(` Type: ${event.type}`);
459
+ console.log(` Retry Count: ${event.retryCount}`);
460
+ console.log(` Error: ${event.error}`);
461
+ console.log(` Last Attempt: ${event.lastAttemptAt}`);
462
+ }
463
+ ```
464
+
465
+ ---
466
+
467
+ ##### `retryEvents(eventIds: string[]): Promise<void>`
468
+
469
+ Resets failed events to `created` status for retry. Clears retry count, error message, and next retry timestamp.
470
+
471
+ **Parameters:**
472
+ - `eventIds` - Array of event IDs to retry
473
+
474
+ **Example:**
475
+ ```typescript
476
+ await outbox.retryEvents(['event-id-1', 'event-id-2']);
477
+ ```
478
+
479
+ ---
480
+
481
+ ##### `start(handler: (event: BusEvent) => Promise<void>, onError: ErrorHandler): void`
482
+
483
+ Starts polling for events. The handler is called for each event in the batch.
484
+
485
+ **Parameters:**
486
+ - `handler` - Async function to process each event
487
+ - `onError` - Error handler called when event processing fails
488
+
489
+ **Example:**
490
+ ```typescript
491
+ outbox.start(
492
+ async (event) => {
493
+ console.log('Processing:', event);
494
+ // Your event handling logic
495
+ },
496
+ (error: OutboxError) => {
497
+ const event = error.context?.event;
498
+ console.error('Failed to process event:', error, event);
499
+ }
500
+ );
501
+ ```
502
+
503
+ ---
504
+
505
+ ##### `stop(): Promise<void>`
506
+
507
+ Stops polling and waits for in-flight events to complete.
508
+
509
+ **Example:**
510
+ ```typescript
511
+ await outbox.stop();
512
+ ```
513
+
514
+ ---
515
+
516
+ ### Helper Functions
517
+
518
+ #### `withBetterSqlite3Transaction<T>(db: Database, fn: (tx: Database) => Promise<T>): Promise<T>`
519
+
520
+ Executes an async function within a SQLite transaction using AsyncLocalStorage. Supports nested transactions via savepoints.
521
+
522
+ **Parameters:**
523
+ - `db` - better-sqlite3 database instance
524
+ - `fn` - Async function to execute within the transaction
525
+
526
+ **Returns:** Result of the function
527
+
528
+ **Example:**
529
+ ```typescript
530
+ import { withBetterSqlite3Transaction } from '@outbox-event-bus/sqlite-better-sqlite3-outbox';
531
+
532
+ const result = await withBetterSqlite3Transaction(db, async (tx) => {
533
+ tx.prepare('INSERT INTO users (name) VALUES (?)').run('Alice');
534
+ await bus.emit({ type: 'user.created', payload: { name: 'Alice' } });
535
+ return { success: true };
536
+ });
537
+ ```
538
+
539
+ ---
540
+
541
+ #### `getBetterSqlite3Transaction(): () => Database | undefined`
542
+
543
+ Returns a function that retrieves the current transaction from AsyncLocalStorage. Use this with the `getTransaction` config option.
544
+
545
+ **Example:**
546
+ ```typescript
547
+ import { getBetterSqlite3Transaction } from '@outbox-event-bus/sqlite-better-sqlite3-outbox';
548
+
549
+ const outbox = new SqliteBetterSqlite3Outbox({
550
+ db,
551
+ getTransaction: getBetterSqlite3Transaction()
552
+ });
553
+ ```
554
+
555
+ ---
556
+
557
+ #### `betterSqlite3TransactionStorage: AsyncLocalStorage<Database>`
558
+
559
+ The AsyncLocalStorage instance used for transaction management. Exported for advanced use cases.
560
+
561
+ **Example:**
562
+ ```typescript
563
+ import { betterSqlite3TransactionStorage } from '@outbox-event-bus/sqlite-better-sqlite3-outbox';
564
+
565
+ const currentTx = betterSqlite3TransactionStorage.getStore();
566
+ if (currentTx) {
567
+ console.log('Inside a transaction');
568
+ }
569
+ ```
570
+
571
+ ---
572
+
573
+ ## Database Schema
574
+
575
+ ### `outbox_events` Table
576
+
577
+ Stores active and pending events.
578
+
579
+ | Column | Type | Constraints | Description |
580
+ |--------|------|-------------|-------------|
581
+ | `id` | TEXT | PRIMARY KEY | Unique event identifier |
582
+ | `type` | TEXT | NOT NULL | Event type (e.g., `user.created`) |
583
+ | `payload` | TEXT | NOT NULL | JSON-serialized event payload |
584
+ | `occurred_at` | TEXT | NOT NULL | ISO 8601 timestamp of event occurrence |
585
+ | `status` | TEXT | NOT NULL, DEFAULT `'created'` | Event status: `created`, `active`, `failed`, `completed` |
586
+ | `retry_count` | INTEGER | NOT NULL, DEFAULT 0 | Number of retry attempts |
587
+ | `last_error` | TEXT | - | Last error message (if failed) |
588
+ | `next_retry_at` | TEXT | - | ISO 8601 timestamp for next retry attempt |
589
+ | `created_on` | TEXT | NOT NULL, DEFAULT CURRENT_TIMESTAMP | Timestamp when event was inserted |
590
+ | `started_on` | TEXT | - | Timestamp when processing started |
591
+ | `completed_on` | TEXT | - | Timestamp when processing completed |
592
+ | `keep_alive` | TEXT | - | Last keep-alive timestamp for stuck event detection |
593
+ | `expire_in_seconds` | INTEGER | NOT NULL, DEFAULT 30 | Heartbeat timeout (seconds) |
594
+
595
+ **Indexes:**
596
+ - `idx_outbox_events_status_retry` on `(status, next_retry_at)` - Optimizes polling queries
597
+
598
+ ---
599
+
600
+ ### `outbox_events_archive` Table
601
+
602
+ Stores successfully processed events for audit purposes.
603
+
604
+ | Column | Type | Constraints | Description |
605
+ |--------|------|-------------|-------------|
606
+ | `id` | TEXT | PRIMARY KEY | Unique event identifier |
607
+ | `type` | TEXT | NOT NULL | Event type |
608
+ | `payload` | TEXT | NOT NULL | JSON-serialized event payload |
609
+ | `occurred_at` | TEXT | NOT NULL | ISO 8601 timestamp of event occurrence |
610
+ | `status` | TEXT | NOT NULL | Final status (always `completed`) |
611
+ | `retry_count` | INTEGER | NOT NULL | Total retry attempts before success |
612
+ | `last_error` | TEXT | - | Last error before success (if any) |
613
+ | `created_on` | TEXT | NOT NULL | Timestamp when event was inserted |
614
+ | `started_on` | TEXT | - | Timestamp when processing started |
615
+ | `completed_on` | TEXT | NOT NULL | Timestamp when processing completed |
616
+
617
+ ---
618
+
619
+ ## How It Works
620
+
621
+ ### Event Lifecycle
622
+
623
+ ```mermaid
624
+ stateDiagram-v2
625
+ [*] --> created: emit()
626
+ created --> active: polling claims
627
+ active --> completed: handler succeeds
628
+ active --> failed: handler throws
629
+ failed --> active: retry (if retries remaining)
630
+ failed --> [*]: max retries exceeded
631
+ completed --> archived: auto-archive
632
+ ```
633
+
634
+ **State Descriptions:**
635
+ - **created**: Event is queued and waiting to be processed
636
+ - **active**: Event is currently being processed by a handler
637
+ - **failed**: Event processing failed and is waiting for retry
638
+ - **completed**: Event processed successfully (moved to archive immediately)
639
+ - **archived**: Event is in the archive table for audit purposes
640
+
641
+ ### Polling Mechanism
642
+
643
+ 1. **Claim Events**: Select up to `batchSize` events that are:
644
+ - New events (`status = 'created'`)
645
+ - Failed events ready for retry (`status = 'failed'` AND `retry_count < maxRetries` AND `next_retry_at <= now`)
646
+ - Stuck events (`status = 'active'` AND `keep_alive + expire_in_seconds < now`)
647
+
648
+ 2. **Lock Events**: Update claimed events to `status = 'active'` and set `keep_alive = now`
649
+
650
+ 3. **Process Events**: Call the handler for each event
651
+
652
+ 4. **Handle Results**:
653
+ - **Success**: Archive event and delete from active table
654
+ - **Failure**: Increment `retry_count`, calculate next retry time, update `status = 'failed'`
655
+
656
+ 5. **Repeat**: Wait `pollIntervalMs` and poll again
657
+
658
+ ---
659
+
660
+ ## Troubleshooting
661
+
662
+ ### `SQLITE_BUSY: database is locked`
663
+
664
+ **Cause:** High write contention or multiple processes accessing the same file.
665
+
666
+ **Solutions:**
667
+ 1. Ensure WAL mode is enabled (enabled by default when using `dbPath`)
668
+ 2. Reduce `pollIntervalMs` or `batchSize` to minimize lock duration
669
+ 3. Avoid multiple processes accessing the same database file
670
+ 4. If using an existing `db` instance, enable WAL manually:
671
+ ```typescript
672
+ db.pragma('journal_mode = WAL');
673
+ ```
674
+
675
+ ---
676
+
677
+ ### Data Loss on Crash
678
+
679
+ **Cause:** SQLite persistence settings or disk cache.
680
+
681
+ **Solution:** SQLite with WAL mode is highly durable, but ensure:
682
+ - Your `dbPath` is on a stable filesystem (not network-mounted)
683
+ - For critical data, consider a client-server database (PostgreSQL, etc.)
684
+ - Enable synchronous mode for maximum durability (with performance trade-off):
685
+ ```typescript
686
+ db.pragma('synchronous = FULL');
687
+ ```
688
+
689
+ ---
690
+
691
+ ### Events Not Processing
692
+
693
+ **Checklist:**
694
+ 1. Did you call `bus.start()`?
695
+ 2. Is the handler throwing an error? Check `onError` logs
696
+ 3. Are events stuck in `failed` state? Use `getFailedEvents()` to inspect
697
+ 4. Check database permissions and file locks
698
+
699
+ ---
700
+
701
+ ### High Memory Usage
702
+
703
+ **Cause:** Large `batchSize` or large event payloads.
704
+
705
+ **Solutions:**
706
+ 1. Reduce `batchSize` (default: 50)
707
+ 2. Compress large payloads before storing
708
+ 3. Archive old events regularly:
709
+ ```sql
710
+ DELETE FROM outbox_events_archive WHERE completed_on < datetime('now', '-30 days');
711
+ ```
712
+
713
+ ---
714
+
715
+ ## Performance Tuning
716
+
717
+ ### Optimize Batch Size
718
+
719
+ ```typescript
720
+ const outbox = new SqliteBetterSqlite3Outbox({
721
+ dbPath: './events.db',
722
+ batchSize: 100, // Process more events per cycle
723
+ pollIntervalMs: 500 // Poll more frequently
724
+ });
725
+ ```
726
+
727
+ **Guidelines:**
728
+ - **Low throughput** (<100 events/min): `batchSize: 10-25`
729
+ - **Medium throughput** (100-1000 events/min): `batchSize: 50-100`
730
+ - **High throughput** (>1000 events/min): Consider Redis or PostgreSQL
731
+
732
+ ---
733
+
734
+ ### Optimize Polling Interval
735
+
736
+ ```typescript
737
+ const outbox = new SqliteBetterSqlite3Outbox({
738
+ dbPath: './events.db',
739
+ pollIntervalMs: 100 // Poll every 100ms for low latency
740
+ });
741
+ ```
742
+
743
+ **Trade-offs:**
744
+ - **Lower interval**: Lower latency, higher CPU usage
745
+ - **Higher interval**: Lower CPU usage, higher latency
746
+
747
+ ---
748
+
749
+ ## Related Documentation
750
+
751
+ - [Main README](https://github.com/dunika/outbox-event-bus#readme) - Overview of the outbox-event-bus library
752
+ - [API Reference](https://github.com/dunika/outbox-event-bus/blob/main/docs/API_REFERENCE.md) - Complete API documentation
753
+ - [Contributing Guide](https://github.com/dunika/outbox-event-bus/blob/main/docs/CONTRIBUTING.md) - How to contribute
754
+
755
+ ---
756
+
757
+ ## License
758
+
759
+ MIT © [dunika](https://github.com/dunika)