@outbox-event-bus/sqlite-better-sqlite3-outbox 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +759 -0
- package/dist/index.cjs +259 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +75 -0
- package/dist/index.d.cts.map +1 -0
- package/dist/index.d.mts +75 -0
- package/dist/index.d.mts.map +1 -0
- package/dist/index.mjs +228 -0
- package/dist/index.mjs.map +1 -0
- package/package.json +54 -0
- package/src/error-handling.test.ts +145 -0
- package/src/index.ts +2 -0
- package/src/integration.e2e.ts +258 -0
- package/src/schema.sql +30 -0
- package/src/sqlite-better-sqlite3-outbox.ts +291 -0
- package/src/sync-transaction.test.ts +117 -0
- package/src/transaction-storage.ts +42 -0
- package/src/transactions.test.ts +196 -0
package/README.md
ADDED
|
@@ -0,0 +1,759 @@
|
|
|
1
|
+
# SQLite Better-SQLite3 Outbox
|
|
2
|
+
|
|
3
|
+

|
|
4
|
+

|
|
5
|
+

|
|
6
|
+

|
|
7
|
+
|
|
8
|
+
> **Reliable event storage for single-instance applications—zero external dependencies.**
|
|
9
|
+
|
|
10
|
+
A SQLite adapter for the [Outbox Pattern](https://github.com/dunika/outbox-event-bus#readme), providing transactional event storage with automatic retry, archiving, and WAL-mode concurrency. Perfect for local development, testing, desktop applications, and single-instance deployments.
|
|
11
|
+
|
|
12
|
+
## When to Use
|
|
13
|
+
|
|
14
|
+
| Feature | SQLite | Redis | PostgreSQL | DynamoDB |
|
|
15
|
+
|---------|--------|-------|------------|----------|
|
|
16
|
+
| **Setup Complexity** | ⭐ Zero config | ⭐⭐ Docker/Cloud | ⭐⭐⭐ Server required | ⭐⭐⭐ AWS setup |
|
|
17
|
+
| **Horizontal Scaling** | ❌ Single instance | ✅ Yes | ✅ Yes | ✅ Yes |
|
|
18
|
+
| **Write Throughput** | ~1K events/sec | ~10K events/sec | ~5K events/sec | ~10K events/sec |
|
|
19
|
+
| **Best For** | Dev, CLI, Desktop | High throughput | ACID guarantees | Cloud-native |
|
|
20
|
+
|
|
21
|
+
**Choose SQLite when:**
|
|
22
|
+
- You're in **local development** or testing
|
|
23
|
+
- You have a **single-instance deployment** (no horizontal scaling)
|
|
24
|
+
- You want **zero external dependencies**
|
|
25
|
+
- You're building a **desktop application** or CLI tool
|
|
26
|
+
|
|
27
|
+
**Consider alternatives when:**
|
|
28
|
+
- You need **horizontal scaling** across multiple servers → use [Redis](https://github.com/dunika/outbox-event-bus/tree/main/adapters/redis-ioredis#readme) or [DynamoDB](https://github.com/dunika/outbox-event-bus/tree/main/adapters/dynamodb-aws-sdk#readme)
|
|
29
|
+
- You require **high write throughput** (>1K events/sec) → SQLite serializes writes
|
|
30
|
+
- You want **cloud-native deployment** → use managed database services
|
|
31
|
+
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## Installation
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
npm install @outbox-event-bus/sqlite-better-sqlite3-outbox outbox-event-bus
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## Getting Started
|
|
43
|
+
|
|
44
|
+
### 1. Create Your First Event Bus
|
|
45
|
+
|
|
46
|
+
```typescript
|
|
47
|
+
import { SqliteBetterSqlite3Outbox } from '@outbox-event-bus/sqlite-better-sqlite3-outbox';
|
|
48
|
+
import { OutboxEventBus } from 'outbox-event-bus';
|
|
49
|
+
|
|
50
|
+
// 1. Create the outbox
|
|
51
|
+
const outbox = new SqliteBetterSqlite3Outbox({
|
|
52
|
+
dbPath: './events.db'
|
|
53
|
+
});
|
|
54
|
+
|
|
55
|
+
// 2. Create the bus
|
|
56
|
+
const bus = new OutboxEventBus(outbox, console.error);
|
|
57
|
+
|
|
58
|
+
// 3. Listen for events
|
|
59
|
+
bus.on('user.created', async (event) => {
|
|
60
|
+
console.log('New user:', event.payload);
|
|
61
|
+
});
|
|
62
|
+
|
|
63
|
+
// 4. Start processing
|
|
64
|
+
bus.start();
|
|
65
|
+
|
|
66
|
+
// 5. Emit an event
|
|
67
|
+
await bus.emit({
|
|
68
|
+
id: crypto.randomUUID(),
|
|
69
|
+
type: 'user.created',
|
|
70
|
+
payload: { name: 'Alice' }
|
|
71
|
+
});
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
### 2. Verify It Works
|
|
75
|
+
|
|
76
|
+
Check your database to see the archived event:
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
sqlite3 events.db "SELECT * FROM outbox_events_archive;"
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
> [!TIP]
|
|
83
|
+
> Use `:memory:` for blazing-fast in-memory testing without disk I/O:
|
|
84
|
+
> ```typescript
|
|
85
|
+
> const outbox = new SqliteBetterSqlite3Outbox({ dbPath: ':memory:' });
|
|
86
|
+
> ```
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## Features
|
|
91
|
+
|
|
92
|
+
### 🔄 Automatic Retry with Exponential Backoff
|
|
93
|
+
|
|
94
|
+
Failed events are automatically retried up to 5 times (configurable) with exponentially increasing delays between attempts.
|
|
95
|
+
|
|
96
|
+
```typescript
|
|
97
|
+
const outbox = new SqliteBetterSqlite3Outbox({
|
|
98
|
+
maxRetries: 3,
|
|
99
|
+
baseBackoffMs: 1000 // Delays: 1s, 2s, 4s
|
|
100
|
+
});
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
### 🔒 ACID Transactions
|
|
104
|
+
|
|
105
|
+
Events are committed atomically with your business data—no partial writes, guaranteed consistency.
|
|
106
|
+
|
|
107
|
+
```typescript
|
|
108
|
+
db.transaction(() => {
|
|
109
|
+
db.prepare('INSERT INTO users (name) VALUES (?)').run('Alice');
|
|
110
|
+
bus.emit({ type: 'user.created', payload: { name: 'Alice' } });
|
|
111
|
+
})(); // Both committed together or both rolled back
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
### 📦 Auto-Archiving
|
|
115
|
+
|
|
116
|
+
Completed events are automatically moved to `outbox_events_archive` for audit trails without bloating the active table.
|
|
117
|
+
|
|
118
|
+
### 🛡️ Stuck Event Recovery
|
|
119
|
+
|
|
120
|
+
Events that timeout during processing are automatically reclaimed and retried, preventing lost events.
|
|
121
|
+
|
|
122
|
+
### 🔍 Failed Event Inspection
|
|
123
|
+
|
|
124
|
+
Query and manually retry failed events:
|
|
125
|
+
|
|
126
|
+
```typescript
|
|
127
|
+
const failed = await outbox.getFailedEvents();
|
|
128
|
+
console.log(failed[0].error); // Last error message
|
|
129
|
+
await outbox.retryEvents(failed.map(e => e.id));
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
## Concurrency & Locking
|
|
135
|
+
|
|
136
|
+
SQLite is designed for **single-instance deployments** and uses **file-level locking** for concurrency control.
|
|
137
|
+
|
|
138
|
+
- **Single Writer**: Only one write transaction can execute at a time (serialized writes)
|
|
139
|
+
- **WAL Mode**: Enables concurrent reads during writes (readers don't block writers)
|
|
140
|
+
- **No Distributed Locking**: Not suitable for horizontal scaling across multiple servers
|
|
141
|
+
- **Thread-Safe**: Safe for multi-threaded applications within a single process
|
|
142
|
+
|
|
143
|
+
> [!WARNING]
|
|
144
|
+
> Do not run multiple instances of your application with the same SQLite database file. This can lead to database corruption. For multi-instance deployments, use [PostgreSQL](https://github.com/dunika/outbox-event-bus/tree/main/adapters/postgres-prisma#readme), [Redis](https://github.com/dunika/outbox-event-bus/tree/main/adapters/redis-ioredis#readme), or [DynamoDB](https://github.com/dunika/outbox-event-bus/tree/main/adapters/dynamodb-aws-sdk#readme) adapters instead.
|
|
145
|
+
|
|
146
|
+
### ⚡ WAL Mode for Concurrency
|
|
147
|
+
|
|
148
|
+
Write-Ahead Logging (WAL) enables concurrent reads during writes, improving throughput and reducing lock contention.
|
|
149
|
+
|
|
150
|
+
**Why WAL matters:**
|
|
151
|
+
- ✅ Readers don't block writers
|
|
152
|
+
- ✅ Writers don't block readers
|
|
153
|
+
- ✅ Better performance for write-heavy workloads
|
|
154
|
+
- ✅ Crash recovery without data loss
|
|
155
|
+
|
|
156
|
+
Without WAL, SQLite uses rollback journaling, which blocks all reads during writes.
|
|
157
|
+
|
|
158
|
+
### 🎨 Custom Table Names
|
|
159
|
+
|
|
160
|
+
Integrate with existing schemas by customizing table names:
|
|
161
|
+
|
|
162
|
+
```typescript
|
|
163
|
+
const outbox = new SqliteBetterSqlite3Outbox({
|
|
164
|
+
dbPath: './app.db',
|
|
165
|
+
tableName: 'my_events',
|
|
166
|
+
archiveTableName: 'my_events_history'
|
|
167
|
+
});
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
### 📘 Full TypeScript Support
|
|
171
|
+
|
|
172
|
+
Complete type definitions for all APIs and configurations with full IntelliSense support.
|
|
173
|
+
|
|
174
|
+
---
|
|
175
|
+
|
|
176
|
+
## Transactions
|
|
177
|
+
|
|
178
|
+
### With AsyncLocalStorage (Recommended)
|
|
179
|
+
|
|
180
|
+
Use `AsyncLocalStorage` to manage SQLite transactions, ensuring outbox events are committed along with your business data.
|
|
181
|
+
|
|
182
|
+
> [!NOTE]
|
|
183
|
+
> better-sqlite3 transactions are synchronous, but `bus.emit()` is async. The recommended pattern is to call `emit()` synchronously within the transaction (it queues the write) and the actual I/O happens immediately since better-sqlite3 is synchronous.
|
|
184
|
+
|
|
185
|
+
```typescript
|
|
186
|
+
import Database from 'better-sqlite3';
|
|
187
|
+
import { AsyncLocalStorage } from 'node:async_hooks';
|
|
188
|
+
|
|
189
|
+
const als = new AsyncLocalStorage<Database.Database>();
|
|
190
|
+
|
|
191
|
+
const outbox = new SqliteBetterSqlite3Outbox({
|
|
192
|
+
dbPath: './data/events.db',
|
|
193
|
+
getTransaction: () => als.getStore()
|
|
194
|
+
});
|
|
195
|
+
|
|
196
|
+
const bus = new OutboxEventBus(outbox, (error) => console.error(error));
|
|
197
|
+
|
|
198
|
+
async function createUser(user: any) {
|
|
199
|
+
const db = new Database('./data/events.db');
|
|
200
|
+
|
|
201
|
+
// Run the transaction synchronously
|
|
202
|
+
const transaction = db.transaction(() => {
|
|
203
|
+
// Set ALS context for the transaction
|
|
204
|
+
return als.run(db, () => {
|
|
205
|
+
// 1. Save business data
|
|
206
|
+
db.prepare('INSERT INTO users (name) VALUES (?)').run(user.name);
|
|
207
|
+
|
|
208
|
+
// 2. Emit event (synchronously writes to outbox table via ALS)
|
|
209
|
+
void bus.emit({
|
|
210
|
+
id: crypto.randomUUID(),
|
|
211
|
+
type: 'user.created',
|
|
212
|
+
payload: user
|
|
213
|
+
});
|
|
214
|
+
});
|
|
215
|
+
});
|
|
216
|
+
|
|
217
|
+
// Execute the transaction
|
|
218
|
+
transaction();
|
|
219
|
+
}
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
### With AsyncLocalStorage Helper
|
|
223
|
+
|
|
224
|
+
Use the provided `withBetterSqlite3Transaction` helper for cleaner async transaction management:
|
|
225
|
+
|
|
226
|
+
```typescript
|
|
227
|
+
import {
|
|
228
|
+
SqliteBetterSqlite3Outbox,
|
|
229
|
+
withBetterSqlite3Transaction,
|
|
230
|
+
getBetterSqlite3Transaction
|
|
231
|
+
} from '@outbox-event-bus/sqlite-better-sqlite3-outbox';
|
|
232
|
+
import Database from 'better-sqlite3';
|
|
233
|
+
|
|
234
|
+
const db = new Database('./events.db');
|
|
235
|
+
|
|
236
|
+
const outbox = new SqliteBetterSqlite3Outbox({
|
|
237
|
+
db,
|
|
238
|
+
getTransaction: getBetterSqlite3Transaction()
|
|
239
|
+
});
|
|
240
|
+
|
|
241
|
+
const bus = new OutboxEventBus(outbox, console.error);
|
|
242
|
+
|
|
243
|
+
async function createUser(user: any) {
|
|
244
|
+
await withBetterSqlite3Transaction(db, async (tx) => {
|
|
245
|
+
// 1. Save business data
|
|
246
|
+
tx.prepare('INSERT INTO users (name) VALUES (?)').run(user.name);
|
|
247
|
+
|
|
248
|
+
// 2. Emit event (uses transaction from AsyncLocalStorage)
|
|
249
|
+
await bus.emit({
|
|
250
|
+
id: crypto.randomUUID(),
|
|
251
|
+
type: 'user.created',
|
|
252
|
+
payload: user
|
|
253
|
+
});
|
|
254
|
+
});
|
|
255
|
+
}
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
### With Explicit Transaction
|
|
259
|
+
|
|
260
|
+
You can also pass the SQLite database instance explicitly to `emit`:
|
|
261
|
+
|
|
262
|
+
```typescript
|
|
263
|
+
const db = new Database('./data/events.db');
|
|
264
|
+
|
|
265
|
+
const transaction = db.transaction(() => {
|
|
266
|
+
// 1. Save business data
|
|
267
|
+
db.prepare('INSERT INTO users (name) VALUES (?)').run(user.name);
|
|
268
|
+
|
|
269
|
+
// 2. Emit event (passing the db explicitly)
|
|
270
|
+
void bus.emit({
|
|
271
|
+
id: crypto.randomUUID(),
|
|
272
|
+
type: 'user.created',
|
|
273
|
+
payload: user
|
|
274
|
+
}, db);
|
|
275
|
+
});
|
|
276
|
+
|
|
277
|
+
transaction();
|
|
278
|
+
```
|
|
279
|
+
|
|
280
|
+
---
|
|
281
|
+
|
|
282
|
+
## How-to Guides
|
|
283
|
+
|
|
284
|
+
### Retry Failed Events
|
|
285
|
+
|
|
286
|
+
```typescript
|
|
287
|
+
// 1. Get all failed events
|
|
288
|
+
const failed = await outbox.getFailedEvents();
|
|
289
|
+
|
|
290
|
+
// 2. Inspect errors
|
|
291
|
+
for (const event of failed) {
|
|
292
|
+
console.log(`Event ${event.id} failed ${event.retryCount} times`);
|
|
293
|
+
console.log(`Last error: ${event.error}`);
|
|
294
|
+
}
|
|
295
|
+
|
|
296
|
+
// 3. Retry specific events
|
|
297
|
+
const retryable = failed.filter(e => e.retryCount < 3);
|
|
298
|
+
await outbox.retryEvents(retryable.map(e => e.id));
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
### Debug Stuck Events
|
|
302
|
+
|
|
303
|
+
Find events stuck in `active` state:
|
|
304
|
+
|
|
305
|
+
```sql
|
|
306
|
+
SELECT * FROM outbox_events
|
|
307
|
+
WHERE status = 'active'
|
|
308
|
+
AND datetime(keep_alive, '+' || expire_in_seconds || ' seconds') < datetime('now');
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
These events will be automatically reclaimed on the next polling cycle.
|
|
312
|
+
|
|
313
|
+
### Use an Existing Database Instance
|
|
314
|
+
|
|
315
|
+
```typescript
|
|
316
|
+
import Database from 'better-sqlite3';
|
|
317
|
+
|
|
318
|
+
const db = new Database('./app.db');
|
|
319
|
+
db.pragma('journal_mode = WAL');
|
|
320
|
+
|
|
321
|
+
const outbox = new SqliteBetterSqlite3Outbox({ db });
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
### Graceful Shutdown
|
|
325
|
+
|
|
326
|
+
```typescript
|
|
327
|
+
process.on('SIGTERM', async () => {
|
|
328
|
+
console.log('Shutting down...');
|
|
329
|
+
await bus.stop(); // Wait for in-flight events to complete
|
|
330
|
+
process.exit(0);
|
|
331
|
+
});
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
### Query the Archive Table
|
|
335
|
+
|
|
336
|
+
```typescript
|
|
337
|
+
import Database from 'better-sqlite3';
|
|
338
|
+
|
|
339
|
+
const db = new Database('./events.db');
|
|
340
|
+
|
|
341
|
+
const archived = db.prepare(`
|
|
342
|
+
SELECT * FROM outbox_events_archive
|
|
343
|
+
WHERE type = ?
|
|
344
|
+
ORDER BY completed_on DESC
|
|
345
|
+
LIMIT 100
|
|
346
|
+
`).all('user.created');
|
|
347
|
+
|
|
348
|
+
console.log(archived);
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
---
|
|
352
|
+
|
|
353
|
+
## Configuration
|
|
354
|
+
|
|
355
|
+
### SqliteBetterSqlite3OutboxConfig
|
|
356
|
+
|
|
357
|
+
```typescript
|
|
358
|
+
interface SqliteBetterSqlite3OutboxConfig extends OutboxConfig {
|
|
359
|
+
// SQLite-specific options
|
|
360
|
+
dbPath?: string;
|
|
361
|
+
db?: Database.Database;
|
|
362
|
+
getTransaction?: () => Database.Database | undefined;
|
|
363
|
+
tableName?: string;
|
|
364
|
+
archiveTableName?: string;
|
|
365
|
+
// Inherited from OutboxConfig
|
|
366
|
+
maxRetries?: number;
|
|
367
|
+
baseBackoffMs?: number;
|
|
368
|
+
processingTimeoutMs?: number; // Processing timeout (default: 30000ms)
|
|
369
|
+
pollIntervalMs?: number;
|
|
370
|
+
batchSize?: number;
|
|
371
|
+
maxErrorBackoffMs?: number; // Max polling error backoff (default: 30000ms)
|
|
372
|
+
}
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
> [!NOTE]
|
|
376
|
+
> All adapters inherit base configuration options from `OutboxConfig`. See the [API Reference](https://github.com/dunika/outbox-event-bus/blob/main/docs/API_REFERENCE.md#base-outbox-configuration) for details on inherited options.
|
|
377
|
+
|
|
378
|
+
| Option | Type | Default | Description |
|
|
379
|
+
|--------|------|---------|-------------|
|
|
380
|
+
| `dbPath` | `string` | - | Path to SQLite file. **Required if `db` not provided.** Use `:memory:` for in-memory. |
|
|
381
|
+
| `db` | `Database` | - | Existing better-sqlite3 instance. If provided, `dbPath` is ignored. WAL mode must be enabled manually. |
|
|
382
|
+
| `getTransaction` | `() => Database \| undefined` | - | Function to retrieve current transaction from AsyncLocalStorage. Use `getBetterSqlite3Transaction()` helper. |
|
|
383
|
+
| `tableName` | `string` | `"outbox_events"` | Name of the outbox table. |
|
|
384
|
+
| `archiveTableName` | `string` | `"outbox_events_archive"` | Name of the archive table. |
|
|
385
|
+
| `maxRetries` | `number` | `5` | Maximum retry attempts before marking event as permanently failed. |
|
|
386
|
+
| `baseBackoffMs` | `number` | `1000` | Base delay for exponential backoff (ms). Retry delays: 1s, 2s, 4s, 8s, 16s. |
|
|
387
|
+
| `processingTimeoutMs` | `number` | `30000` | Timeout for event handler execution (ms). Events exceeding this are marked as stuck. |
|
|
388
|
+
| `pollIntervalMs` | `number` | `1000` | Interval between polling cycles (ms). |
|
|
389
|
+
| `batchSize` | `number` | `50` | Maximum events to process per batch. |
|
|
390
|
+
| `maxErrorBackoffMs` | `number` | `30000` | Maximum backoff delay after polling errors (ms). |
|
|
391
|
+
|
|
392
|
+
> [!WARNING]
|
|
393
|
+
> You must provide either `dbPath` or `db`. If neither is provided, the constructor will throw an error.
|
|
394
|
+
|
|
395
|
+
---
|
|
396
|
+
|
|
397
|
+
## API Reference
|
|
398
|
+
|
|
399
|
+
### `SqliteBetterSqlite3Outbox`
|
|
400
|
+
|
|
401
|
+
#### Constructor
|
|
402
|
+
|
|
403
|
+
```typescript
|
|
404
|
+
new SqliteBetterSqlite3Outbox(config: SqliteBetterSqlite3OutboxConfig)
|
|
405
|
+
```
|
|
406
|
+
|
|
407
|
+
Creates a new SQLite outbox adapter. Automatically creates tables and indexes on initialization.
|
|
408
|
+
|
|
409
|
+
**Throws:**
|
|
410
|
+
- `Error` if neither `dbPath` nor `db` is provided
|
|
411
|
+
|
|
412
|
+
**Example:**
|
|
413
|
+
```typescript
|
|
414
|
+
const outbox = new SqliteBetterSqlite3Outbox({
|
|
415
|
+
dbPath: './events.db',
|
|
416
|
+
maxRetries: 3,
|
|
417
|
+
batchSize: 100
|
|
418
|
+
});
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
---
|
|
422
|
+
|
|
423
|
+
#### Methods
|
|
424
|
+
|
|
425
|
+
##### `publish(events: BusEvent[], transaction?: Database): Promise<void>`
|
|
426
|
+
|
|
427
|
+
Inserts events into the outbox table. Events are inserted with `status = 'created'` and will be picked up by the next polling cycle.
|
|
428
|
+
|
|
429
|
+
**Parameters:**
|
|
430
|
+
- `events` - Array of events to publish
|
|
431
|
+
- `transaction` - Optional better-sqlite3 database instance for transactional writes
|
|
432
|
+
|
|
433
|
+
**Example:**
|
|
434
|
+
```typescript
|
|
435
|
+
await outbox.publish([
|
|
436
|
+
{
|
|
437
|
+
id: crypto.randomUUID(),
|
|
438
|
+
type: 'user.created',
|
|
439
|
+
payload: { name: 'Alice' },
|
|
440
|
+
occurredAt: new Date()
|
|
441
|
+
}
|
|
442
|
+
]);
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
---
|
|
446
|
+
|
|
447
|
+
##### `getFailedEvents(): Promise<FailedBusEvent[]>`
|
|
448
|
+
|
|
449
|
+
Retrieves up to 100 failed events, ordered by occurrence time (newest first).
|
|
450
|
+
|
|
451
|
+
**Returns:** Array of `FailedBusEvent` objects with error details and retry count.
|
|
452
|
+
|
|
453
|
+
**Example:**
|
|
454
|
+
```typescript
|
|
455
|
+
const failed = await outbox.getFailedEvents();
|
|
456
|
+
for (const event of failed) {
|
|
457
|
+
console.log(`Event ${event.id}:`);
|
|
458
|
+
console.log(` Type: ${event.type}`);
|
|
459
|
+
console.log(` Retry Count: ${event.retryCount}`);
|
|
460
|
+
console.log(` Error: ${event.error}`);
|
|
461
|
+
console.log(` Last Attempt: ${event.lastAttemptAt}`);
|
|
462
|
+
}
|
|
463
|
+
```
|
|
464
|
+
|
|
465
|
+
---
|
|
466
|
+
|
|
467
|
+
##### `retryEvents(eventIds: string[]): Promise<void>`
|
|
468
|
+
|
|
469
|
+
Resets failed events to `created` status for retry. Clears retry count, error message, and next retry timestamp.
|
|
470
|
+
|
|
471
|
+
**Parameters:**
|
|
472
|
+
- `eventIds` - Array of event IDs to retry
|
|
473
|
+
|
|
474
|
+
**Example:**
|
|
475
|
+
```typescript
|
|
476
|
+
await outbox.retryEvents(['event-id-1', 'event-id-2']);
|
|
477
|
+
```
|
|
478
|
+
|
|
479
|
+
---
|
|
480
|
+
|
|
481
|
+
##### `start(handler: (event: BusEvent) => Promise<void>, onError: ErrorHandler): void`
|
|
482
|
+
|
|
483
|
+
Starts polling for events. The handler is called for each event in the batch.
|
|
484
|
+
|
|
485
|
+
**Parameters:**
|
|
486
|
+
- `handler` - Async function to process each event
|
|
487
|
+
- `onError` - Error handler called when event processing fails
|
|
488
|
+
|
|
489
|
+
**Example:**
|
|
490
|
+
```typescript
|
|
491
|
+
outbox.start(
|
|
492
|
+
async (event) => {
|
|
493
|
+
console.log('Processing:', event);
|
|
494
|
+
// Your event handling logic
|
|
495
|
+
},
|
|
496
|
+
(error: OutboxError) => {
|
|
497
|
+
const event = error.context?.event;
|
|
498
|
+
console.error('Failed to process event:', error, event);
|
|
499
|
+
}
|
|
500
|
+
);
|
|
501
|
+
```
|
|
502
|
+
|
|
503
|
+
---
|
|
504
|
+
|
|
505
|
+
##### `stop(): Promise<void>`
|
|
506
|
+
|
|
507
|
+
Stops polling and waits for in-flight events to complete.
|
|
508
|
+
|
|
509
|
+
**Example:**
|
|
510
|
+
```typescript
|
|
511
|
+
await outbox.stop();
|
|
512
|
+
```
|
|
513
|
+
|
|
514
|
+
---
|
|
515
|
+
|
|
516
|
+
### Helper Functions
|
|
517
|
+
|
|
518
|
+
#### `withBetterSqlite3Transaction<T>(db: Database, fn: (tx: Database) => Promise<T>): Promise<T>`
|
|
519
|
+
|
|
520
|
+
Executes an async function within a SQLite transaction using AsyncLocalStorage. Supports nested transactions via savepoints.
|
|
521
|
+
|
|
522
|
+
**Parameters:**
|
|
523
|
+
- `db` - better-sqlite3 database instance
|
|
524
|
+
- `fn` - Async function to execute within the transaction
|
|
525
|
+
|
|
526
|
+
**Returns:** Result of the function
|
|
527
|
+
|
|
528
|
+
**Example:**
|
|
529
|
+
```typescript
|
|
530
|
+
import { withBetterSqlite3Transaction } from '@outbox-event-bus/sqlite-better-sqlite3-outbox';
|
|
531
|
+
|
|
532
|
+
const result = await withBetterSqlite3Transaction(db, async (tx) => {
|
|
533
|
+
tx.prepare('INSERT INTO users (name) VALUES (?)').run('Alice');
|
|
534
|
+
await bus.emit({ type: 'user.created', payload: { name: 'Alice' } });
|
|
535
|
+
return { success: true };
|
|
536
|
+
});
|
|
537
|
+
```
|
|
538
|
+
|
|
539
|
+
---
|
|
540
|
+
|
|
541
|
+
#### `getBetterSqlite3Transaction(): () => Database | undefined`
|
|
542
|
+
|
|
543
|
+
Returns a function that retrieves the current transaction from AsyncLocalStorage. Use this with the `getTransaction` config option.
|
|
544
|
+
|
|
545
|
+
**Example:**
|
|
546
|
+
```typescript
|
|
547
|
+
import { getBetterSqlite3Transaction } from '@outbox-event-bus/sqlite-better-sqlite3-outbox';
|
|
548
|
+
|
|
549
|
+
const outbox = new SqliteBetterSqlite3Outbox({
|
|
550
|
+
db,
|
|
551
|
+
getTransaction: getBetterSqlite3Transaction()
|
|
552
|
+
});
|
|
553
|
+
```
|
|
554
|
+
|
|
555
|
+
---
|
|
556
|
+
|
|
557
|
+
#### `betterSqlite3TransactionStorage: AsyncLocalStorage<Database>`
|
|
558
|
+
|
|
559
|
+
The AsyncLocalStorage instance used for transaction management. Exported for advanced use cases.
|
|
560
|
+
|
|
561
|
+
**Example:**
|
|
562
|
+
```typescript
|
|
563
|
+
import { betterSqlite3TransactionStorage } from '@outbox-event-bus/sqlite-better-sqlite3-outbox';
|
|
564
|
+
|
|
565
|
+
const currentTx = betterSqlite3TransactionStorage.getStore();
|
|
566
|
+
if (currentTx) {
|
|
567
|
+
console.log('Inside a transaction');
|
|
568
|
+
}
|
|
569
|
+
```
|
|
570
|
+
|
|
571
|
+
---
|
|
572
|
+
|
|
573
|
+
## Database Schema
|
|
574
|
+
|
|
575
|
+
### `outbox_events` Table
|
|
576
|
+
|
|
577
|
+
Stores active and pending events.
|
|
578
|
+
|
|
579
|
+
| Column | Type | Constraints | Description |
|
|
580
|
+
|--------|------|-------------|-------------|
|
|
581
|
+
| `id` | TEXT | PRIMARY KEY | Unique event identifier |
|
|
582
|
+
| `type` | TEXT | NOT NULL | Event type (e.g., `user.created`) |
|
|
583
|
+
| `payload` | TEXT | NOT NULL | JSON-serialized event payload |
|
|
584
|
+
| `occurred_at` | TEXT | NOT NULL | ISO 8601 timestamp of event occurrence |
|
|
585
|
+
| `status` | TEXT | NOT NULL, DEFAULT `'created'` | Event status: `created`, `active`, `failed`, `completed` |
|
|
586
|
+
| `retry_count` | INTEGER | NOT NULL, DEFAULT 0 | Number of retry attempts |
|
|
587
|
+
| `last_error` | TEXT | - | Last error message (if failed) |
|
|
588
|
+
| `next_retry_at` | TEXT | - | ISO 8601 timestamp for next retry attempt |
|
|
589
|
+
| `created_on` | TEXT | NOT NULL, DEFAULT CURRENT_TIMESTAMP | Timestamp when event was inserted |
|
|
590
|
+
| `started_on` | TEXT | - | Timestamp when processing started |
|
|
591
|
+
| `completed_on` | TEXT | - | Timestamp when processing completed |
|
|
592
|
+
| `keep_alive` | TEXT | - | Last keep-alive timestamp for stuck event detection |
|
|
593
|
+
| `expire_in_seconds` | INTEGER | NOT NULL, DEFAULT 30 | Heartbeat timeout (seconds) |
|
|
594
|
+
|
|
595
|
+
**Indexes:**
|
|
596
|
+
- `idx_outbox_events_status_retry` on `(status, next_retry_at)` - Optimizes polling queries
|
|
597
|
+
|
|
598
|
+
---
|
|
599
|
+
|
|
600
|
+
### `outbox_events_archive` Table
|
|
601
|
+
|
|
602
|
+
Stores successfully processed events for audit purposes.
|
|
603
|
+
|
|
604
|
+
| Column | Type | Constraints | Description |
|
|
605
|
+
|--------|------|-------------|-------------|
|
|
606
|
+
| `id` | TEXT | PRIMARY KEY | Unique event identifier |
|
|
607
|
+
| `type` | TEXT | NOT NULL | Event type |
|
|
608
|
+
| `payload` | TEXT | NOT NULL | JSON-serialized event payload |
|
|
609
|
+
| `occurred_at` | TEXT | NOT NULL | ISO 8601 timestamp of event occurrence |
|
|
610
|
+
| `status` | TEXT | NOT NULL | Final status (always `completed`) |
|
|
611
|
+
| `retry_count` | INTEGER | NOT NULL | Total retry attempts before success |
|
|
612
|
+
| `last_error` | TEXT | - | Last error before success (if any) |
|
|
613
|
+
| `created_on` | TEXT | NOT NULL | Timestamp when event was inserted |
|
|
614
|
+
| `started_on` | TEXT | - | Timestamp when processing started |
|
|
615
|
+
| `completed_on` | TEXT | NOT NULL | Timestamp when processing completed |
|
|
616
|
+
|
|
617
|
+
---
|
|
618
|
+
|
|
619
|
+
## How It Works
|
|
620
|
+
|
|
621
|
+
### Event Lifecycle
|
|
622
|
+
|
|
623
|
+
```mermaid
|
|
624
|
+
stateDiagram-v2
|
|
625
|
+
[*] --> created: emit()
|
|
626
|
+
created --> active: polling claims
|
|
627
|
+
active --> completed: handler succeeds
|
|
628
|
+
active --> failed: handler throws
|
|
629
|
+
failed --> active: retry (if retries remaining)
|
|
630
|
+
failed --> [*]: max retries exceeded
|
|
631
|
+
completed --> archived: auto-archive
|
|
632
|
+
```
|
|
633
|
+
|
|
634
|
+
**State Descriptions:**
|
|
635
|
+
- **created**: Event is queued and waiting to be processed
|
|
636
|
+
- **active**: Event is currently being processed by a handler
|
|
637
|
+
- **failed**: Event processing failed and is waiting for retry
|
|
638
|
+
- **completed**: Event processed successfully (moved to archive immediately)
|
|
639
|
+
- **archived**: Event is in the archive table for audit purposes
|
|
640
|
+
|
|
641
|
+
### Polling Mechanism
|
|
642
|
+
|
|
643
|
+
1. **Claim Events**: Select up to `batchSize` events that are:
|
|
644
|
+
- New events (`status = 'created'`)
|
|
645
|
+
- Failed events ready for retry (`status = 'failed'` AND `retry_count < maxRetries` AND `next_retry_at <= now`)
|
|
646
|
+
- Stuck events (`status = 'active'` AND `keep_alive + expire_in_seconds < now`)
|
|
647
|
+
|
|
648
|
+
2. **Lock Events**: Update claimed events to `status = 'active'` and set `keep_alive = now`
|
|
649
|
+
|
|
650
|
+
3. **Process Events**: Call the handler for each event
|
|
651
|
+
|
|
652
|
+
4. **Handle Results**:
|
|
653
|
+
- **Success**: Archive event and delete from active table
|
|
654
|
+
- **Failure**: Increment `retry_count`, calculate next retry time, update `status = 'failed'`
|
|
655
|
+
|
|
656
|
+
5. **Repeat**: Wait `pollIntervalMs` and poll again
|
|
657
|
+
|
|
658
|
+
---
|
|
659
|
+
|
|
660
|
+
## Troubleshooting
|
|
661
|
+
|
|
662
|
+
### `SQLITE_BUSY: database is locked`
|
|
663
|
+
|
|
664
|
+
**Cause:** High write contention or multiple processes accessing the same file.
|
|
665
|
+
|
|
666
|
+
**Solutions:**
|
|
667
|
+
1. Ensure WAL mode is enabled (enabled by default when using `dbPath`)
|
|
668
|
+
2. Reduce `pollIntervalMs` or `batchSize` to minimize lock duration
|
|
669
|
+
3. Avoid multiple processes accessing the same database file
|
|
670
|
+
4. If using an existing `db` instance, enable WAL manually:
|
|
671
|
+
```typescript
|
|
672
|
+
db.pragma('journal_mode = WAL');
|
|
673
|
+
```
|
|
674
|
+
|
|
675
|
+
---
|
|
676
|
+
|
|
677
|
+
### Data Loss on Crash
|
|
678
|
+
|
|
679
|
+
**Cause:** SQLite persistence settings or disk cache.
|
|
680
|
+
|
|
681
|
+
**Solution:** SQLite with WAL mode is highly durable, but ensure:
|
|
682
|
+
- Your `dbPath` is on a stable filesystem (not network-mounted)
|
|
683
|
+
- For critical data, consider a client-server database (PostgreSQL, etc.)
|
|
684
|
+
- Enable synchronous mode for maximum durability (with performance trade-off):
|
|
685
|
+
```typescript
|
|
686
|
+
db.pragma('synchronous = FULL');
|
|
687
|
+
```
|
|
688
|
+
|
|
689
|
+
---
|
|
690
|
+
|
|
691
|
+
### Events Not Processing
|
|
692
|
+
|
|
693
|
+
**Checklist:**
|
|
694
|
+
1. Did you call `bus.start()`?
|
|
695
|
+
2. Is the handler throwing an error? Check `onError` logs
|
|
696
|
+
3. Are events stuck in `failed` state? Use `getFailedEvents()` to inspect
|
|
697
|
+
4. Check database permissions and file locks
|
|
698
|
+
|
|
699
|
+
---
|
|
700
|
+
|
|
701
|
+
### High Memory Usage
|
|
702
|
+
|
|
703
|
+
**Cause:** Large `batchSize` or large event payloads.
|
|
704
|
+
|
|
705
|
+
**Solutions:**
|
|
706
|
+
1. Reduce `batchSize` (default: 50)
|
|
707
|
+
2. Compress large payloads before storing
|
|
708
|
+
3. Archive old events regularly:
|
|
709
|
+
```sql
|
|
710
|
+
DELETE FROM outbox_events_archive WHERE completed_on < datetime('now', '-30 days');
|
|
711
|
+
```
|
|
712
|
+
|
|
713
|
+
---
|
|
714
|
+
|
|
715
|
+
## Performance Tuning
|
|
716
|
+
|
|
717
|
+
### Optimize Batch Size
|
|
718
|
+
|
|
719
|
+
```typescript
|
|
720
|
+
const outbox = new SqliteBetterSqlite3Outbox({
|
|
721
|
+
dbPath: './events.db',
|
|
722
|
+
batchSize: 100, // Process more events per cycle
|
|
723
|
+
pollIntervalMs: 500 // Poll more frequently
|
|
724
|
+
});
|
|
725
|
+
```
|
|
726
|
+
|
|
727
|
+
**Guidelines:**
|
|
728
|
+
- **Low throughput** (<100 events/min): `batchSize: 10-25`
|
|
729
|
+
- **Medium throughput** (100-1000 events/min): `batchSize: 50-100`
|
|
730
|
+
- **High throughput** (>1000 events/min): Consider Redis or PostgreSQL
|
|
731
|
+
|
|
732
|
+
---
|
|
733
|
+
|
|
734
|
+
### Optimize Polling Interval
|
|
735
|
+
|
|
736
|
+
```typescript
|
|
737
|
+
const outbox = new SqliteBetterSqlite3Outbox({
|
|
738
|
+
dbPath: './events.db',
|
|
739
|
+
pollIntervalMs: 100 // Poll every 100ms for low latency
|
|
740
|
+
});
|
|
741
|
+
```
|
|
742
|
+
|
|
743
|
+
**Trade-offs:**
|
|
744
|
+
- **Lower interval**: Lower latency, higher CPU usage
|
|
745
|
+
- **Higher interval**: Lower CPU usage, higher latency
|
|
746
|
+
|
|
747
|
+
---
|
|
748
|
+
|
|
749
|
+
## Related Documentation
|
|
750
|
+
|
|
751
|
+
- [Main README](https://github.com/dunika/outbox-event-bus#readme) - Overview of the outbox-event-bus library
|
|
752
|
+
- [API Reference](https://github.com/dunika/outbox-event-bus/blob/main/docs/API_REFERENCE.md) - Complete API documentation
|
|
753
|
+
- [Contributing Guide](https://github.com/dunika/outbox-event-bus/blob/main/docs/CONTRIBUTING.md) - How to contribute
|
|
754
|
+
|
|
755
|
+
---
|
|
756
|
+
|
|
757
|
+
## License
|
|
758
|
+
|
|
759
|
+
MIT © [dunika](https://github.com/dunika)
|