@message-queue-toolkit/sqs 22.3.0 → 22.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2615 -0
- package/dist/index.d.ts +4 -3
- package/dist/index.js +2 -1
- package/dist/index.js.map +1 -1
- package/dist/sqs/AbstractSqsConsumer.d.ts +66 -2
- package/dist/sqs/AbstractSqsConsumer.js +274 -21
- package/dist/sqs/AbstractSqsConsumer.js.map +1 -1
- package/dist/sqs/AbstractSqsPublisher.d.ts +38 -2
- package/dist/sqs/AbstractSqsPublisher.js +44 -1
- package/dist/sqs/AbstractSqsPublisher.js.map +1 -1
- package/dist/sqs/AbstractSqsService.d.ts +10 -0
- package/dist/sqs/AbstractSqsService.js +3 -1
- package/dist/sqs/AbstractSqsService.js.map +1 -1
- package/dist/utils/eventBridgeSchemaBuilder.d.ts +85 -0
- package/dist/utils/eventBridgeSchemaBuilder.js +81 -0
- package/dist/utils/eventBridgeSchemaBuilder.js.map +1 -0
- package/dist/utils/sqsAttributeUtils.d.ts +1 -1
- package/dist/utils/sqsAttributeUtils.js +1 -1
- package/dist/utils/sqsInitter.d.ts +1 -1
- package/dist/utils/sqsInitter.js +7 -3
- package/dist/utils/sqsInitter.js.map +1 -1
- package/dist/utils/sqsUtils.d.ts +19 -1
- package/dist/utils/sqsUtils.js +62 -2
- package/dist/utils/sqsUtils.js.map +1 -1
- package/package.json +2 -2
package/README.md
ADDED
|
@@ -0,0 +1,2615 @@
|
|
|
1
|
+
# @message-queue-toolkit/sqs
|
|
2
|
+
|
|
3
|
+
AWS SQS (Simple Queue Service) implementation for the message-queue-toolkit. Provides a robust, type-safe abstraction for publishing and consuming messages from both standard and FIFO SQS queues.
|
|
4
|
+
|
|
5
|
+
## Table of Contents
|
|
6
|
+
|
|
7
|
+
- [Installation](#installation)
|
|
8
|
+
- [Features](#features)
|
|
9
|
+
- [Core Concepts](#core-concepts)
|
|
10
|
+
- [Quick Start](#quick-start)
|
|
11
|
+
- [Standard Queue Publisher](#standard-queue-publisher)
|
|
12
|
+
- [Standard Queue Consumer](#standard-queue-consumer)
|
|
13
|
+
- [FIFO Queue Publisher](#fifo-queue-publisher)
|
|
14
|
+
- [FIFO Queue Consumer](#fifo-queue-consumer)
|
|
15
|
+
- [Configuration](#configuration)
|
|
16
|
+
- [Queue Creation](#queue-creation)
|
|
17
|
+
- [Queue Locator](#queue-locator)
|
|
18
|
+
- [Publisher Options](#publisher-options)
|
|
19
|
+
- [Consumer Options](#consumer-options)
|
|
20
|
+
- [Advanced Features](#advanced-features)
|
|
21
|
+
- [Custom Message Field Names](#custom-message-field-names)
|
|
22
|
+
- [Dead Letter Queue (DLQ)](#dead-letter-queue-dlq)
|
|
23
|
+
- [Message Retry Logic](#message-retry-logic)
|
|
24
|
+
- [Message Deduplication](#message-deduplication)
|
|
25
|
+
- [Payload Offloading](#payload-offloading)
|
|
26
|
+
- [Message Handlers](#message-handlers)
|
|
27
|
+
- [Pre-handlers and Barriers](#pre-handlers-and-barriers)
|
|
28
|
+
- [Handler Spies](#handler-spies)
|
|
29
|
+
- [Non-Standard Message Formats](#non-standard-message-formats)
|
|
30
|
+
- [EventBridge Events](#eventbridge-events)
|
|
31
|
+
- [Custom Message Structures](#custom-message-structures)
|
|
32
|
+
- [FIFO Queues](#fifo-queues)
|
|
33
|
+
- [FIFO Queue Requirements](#fifo-queue-requirements)
|
|
34
|
+
- [Message Ordering](#message-ordering)
|
|
35
|
+
- [Message Groups](#message-groups)
|
|
36
|
+
- [FIFO-Specific Configuration](#fifo-specific-configuration)
|
|
37
|
+
- [Policy Configuration](#policy-configuration)
|
|
38
|
+
- [Testing](#testing)
|
|
39
|
+
- [API Reference](#api-reference)
|
|
40
|
+
|
|
41
|
+
## Installation
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
npm install @message-queue-toolkit/sqs @message-queue-toolkit/core
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
**Peer Dependencies:**
|
|
48
|
+
- `@aws-sdk/client-sqs` - AWS SDK for SQS
|
|
49
|
+
- `zod` - Schema validation
|
|
50
|
+
|
|
51
|
+
## Features
|
|
52
|
+
|
|
53
|
+
- ✅ **Type-safe message handling** with Zod schema validation
|
|
54
|
+
- ✅ **Standard and FIFO queue support**
|
|
55
|
+
- ✅ **Automatic retry logic** with exponential backoff
|
|
56
|
+
- ✅ **Dead Letter Queue (DLQ)** support
|
|
57
|
+
- ✅ **Message deduplication** (publisher and consumer level)
|
|
58
|
+
- ✅ **Payload offloading** for large messages (S3 integration)
|
|
59
|
+
- ✅ **Concurrent consumers** for high throughput
|
|
60
|
+
- ✅ **Policy-based access control**
|
|
61
|
+
- ✅ **Handler spies** for testing
|
|
62
|
+
- ✅ **Pre-handlers and barriers** for complex message processing
|
|
63
|
+
- ✅ **Automatic queue creation** with validation
|
|
64
|
+
|
|
65
|
+
## Core Concepts
|
|
66
|
+
|
|
67
|
+
### Publishers
|
|
68
|
+
|
|
69
|
+
Publishers send messages to SQS queues. They handle:
|
|
70
|
+
- Message validation against Zod schemas
|
|
71
|
+
- Automatic serialization
|
|
72
|
+
- Optional deduplication (preventing duplicate sends)
|
|
73
|
+
- Optional payload offloading (for messages > 256KB)
|
|
74
|
+
- FIFO-specific concerns (MessageGroupId, MessageDeduplicationId)
|
|
75
|
+
|
|
76
|
+
### Consumers
|
|
77
|
+
|
|
78
|
+
Consumers receive and process messages from SQS queues. They handle:
|
|
79
|
+
- Message deserialization and validation
|
|
80
|
+
- Routing to appropriate handlers based on message type
|
|
81
|
+
- Automatic retry with exponential backoff
|
|
82
|
+
- Dead letter queue integration
|
|
83
|
+
- Optional deduplication (preventing duplicate processing)
|
|
84
|
+
- FIFO ordering guarantees
|
|
85
|
+
|
|
86
|
+
### Message Schemas
|
|
87
|
+
|
|
88
|
+
Messages are validated using Zod schemas. Each message must have:
|
|
89
|
+
- A unique message type field (discriminator for routing) - configurable via `messageTypeField` (required)
|
|
90
|
+
- A message ID field (for tracking and deduplication) - configurable via `messageIdField` (default: `'id'`)
|
|
91
|
+
- A timestamp field (added automatically if missing) - configurable via `messageTimestampField` (default: `'timestamp'`)
|
|
92
|
+
|
|
93
|
+
**Note:** All field names are configurable, allowing you to adapt the library to your existing message schemas without modification.
|
|
94
|
+
|
|
95
|
+
## Quick Start
|
|
96
|
+
|
|
97
|
+
### Standard Queue Publisher
|
|
98
|
+
|
|
99
|
+
```typescript
|
|
100
|
+
import { AbstractSqsPublisher } from '@message-queue-toolkit/sqs'
|
|
101
|
+
import { SQSClient } from '@aws-sdk/client-sqs'
|
|
102
|
+
import z from 'zod'
|
|
103
|
+
|
|
104
|
+
// Define your message schemas
|
|
105
|
+
const UserCreatedSchema = z.object({
|
|
106
|
+
id: z.string(),
|
|
107
|
+
messageType: z.literal('user.created'),
|
|
108
|
+
userId: z.string(),
|
|
109
|
+
email: z.string().email(),
|
|
110
|
+
timestamp: z.string().optional(),
|
|
111
|
+
})
|
|
112
|
+
|
|
113
|
+
const UserUpdatedSchema = z.object({
|
|
114
|
+
id: z.string(),
|
|
115
|
+
messageType: z.literal('user.updated'),
|
|
116
|
+
userId: z.string(),
|
|
117
|
+
changes: z.record(z.unknown()),
|
|
118
|
+
timestamp: z.string().optional(),
|
|
119
|
+
})
|
|
120
|
+
|
|
121
|
+
type UserCreated = z.infer<typeof UserCreatedSchema>
|
|
122
|
+
type UserUpdated = z.infer<typeof UserUpdatedSchema>
|
|
123
|
+
type SupportedMessages = UserCreated | UserUpdated
|
|
124
|
+
|
|
125
|
+
// Create your publisher class
|
|
126
|
+
class UserEventsPublisher extends AbstractSqsPublisher<SupportedMessages> {
|
|
127
|
+
constructor(sqsClient: SQSClient) {
|
|
128
|
+
super(
|
|
129
|
+
{
|
|
130
|
+
sqsClient,
|
|
131
|
+
logger: console,
|
|
132
|
+
errorReporter: { report: (error) => console.error(error) },
|
|
133
|
+
},
|
|
134
|
+
{
|
|
135
|
+
messageSchemas: [UserCreatedSchema, UserUpdatedSchema],
|
|
136
|
+
messageTypeField: 'messageType',
|
|
137
|
+
creationConfig: {
|
|
138
|
+
queue: {
|
|
139
|
+
QueueName: 'user-events-queue',
|
|
140
|
+
},
|
|
141
|
+
},
|
|
142
|
+
deletionConfig: {
|
|
143
|
+
deleteIfExists: false,
|
|
144
|
+
},
|
|
145
|
+
}
|
|
146
|
+
)
|
|
147
|
+
}
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
// Use the publisher
|
|
151
|
+
const sqsClient = new SQSClient({ region: 'us-east-1' })
|
|
152
|
+
const publisher = new UserEventsPublisher(sqsClient)
|
|
153
|
+
|
|
154
|
+
await publisher.init()
|
|
155
|
+
|
|
156
|
+
await publisher.publish({
|
|
157
|
+
id: '123',
|
|
158
|
+
messageType: 'user.created',
|
|
159
|
+
userId: 'user-456',
|
|
160
|
+
email: 'user@example.com',
|
|
161
|
+
})
|
|
162
|
+
|
|
163
|
+
await publisher.close()
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
### Standard Queue Consumer
|
|
167
|
+
|
|
168
|
+
```typescript
|
|
169
|
+
import { AbstractSqsConsumer } from '@message-queue-toolkit/sqs'
|
|
170
|
+
import { MessageHandlerConfigBuilder } from '@message-queue-toolkit/core'
|
|
171
|
+
import type { Either } from '@lokalise/node-core'
|
|
172
|
+
|
|
173
|
+
type ExecutionContext = {
|
|
174
|
+
userService: UserService
|
|
175
|
+
}
|
|
176
|
+
|
|
177
|
+
class UserEventsConsumer extends AbstractSqsConsumer<
|
|
178
|
+
SupportedMessages,
|
|
179
|
+
ExecutionContext
|
|
180
|
+
> {
|
|
181
|
+
constructor(sqsClient: SQSClient, userService: UserService) {
|
|
182
|
+
super(
|
|
183
|
+
{
|
|
184
|
+
sqsClient,
|
|
185
|
+
logger: console,
|
|
186
|
+
errorReporter: { report: (error) => console.error(error) },
|
|
187
|
+
consumerErrorResolver: {
|
|
188
|
+
resolveError: () => ({ resolve: 'retryLater' as const }),
|
|
189
|
+
},
|
|
190
|
+
transactionObservabilityManager: {
|
|
191
|
+
start: () => {},
|
|
192
|
+
stop: () => {},
|
|
193
|
+
},
|
|
194
|
+
},
|
|
195
|
+
{
|
|
196
|
+
messageTypeField: 'messageType',
|
|
197
|
+
handlers: new MessageHandlerConfigBuilder<SupportedMessages, ExecutionContext>()
|
|
198
|
+
.addConfig(
|
|
199
|
+
UserCreatedSchema,
|
|
200
|
+
async (message, context): Promise<Either<'retryLater', 'success'>> => {
|
|
201
|
+
await context.userService.createUser(message.userId, message.email)
|
|
202
|
+
return { result: 'success' }
|
|
203
|
+
}
|
|
204
|
+
)
|
|
205
|
+
.addConfig(
|
|
206
|
+
UserUpdatedSchema,
|
|
207
|
+
async (message, context): Promise<Either<'retryLater', 'success'>> => {
|
|
208
|
+
await context.userService.updateUser(message.userId, message.changes)
|
|
209
|
+
return { result: 'success' }
|
|
210
|
+
}
|
|
211
|
+
)
|
|
212
|
+
.build(),
|
|
213
|
+
creationConfig: {
|
|
214
|
+
queue: {
|
|
215
|
+
QueueName: 'user-events-queue',
|
|
216
|
+
},
|
|
217
|
+
},
|
|
218
|
+
deletionConfig: {
|
|
219
|
+
deleteIfExists: false,
|
|
220
|
+
},
|
|
221
|
+
},
|
|
222
|
+
{ userService } // Execution context
|
|
223
|
+
)
|
|
224
|
+
}
|
|
225
|
+
}
|
|
226
|
+
|
|
227
|
+
// Use the consumer
|
|
228
|
+
const consumer = new UserEventsConsumer(sqsClient, userService)
|
|
229
|
+
await consumer.start() // Initializes and starts consuming
|
|
230
|
+
|
|
231
|
+
// Later, to stop
|
|
232
|
+
await consumer.close()
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
### FIFO Queue Publisher
|
|
236
|
+
|
|
237
|
+
FIFO (First-In-First-Out) queues guarantee that messages are processed exactly once and in order within a message group.
|
|
238
|
+
|
|
239
|
+
```typescript
|
|
240
|
+
class UserEventsFifoPublisher extends AbstractSqsPublisher<SupportedMessages> {
|
|
241
|
+
constructor(sqsClient: SQSClient) {
|
|
242
|
+
super(
|
|
243
|
+
{
|
|
244
|
+
sqsClient,
|
|
245
|
+
logger: console,
|
|
246
|
+
errorReporter: { report: (error) => console.error(error) },
|
|
247
|
+
},
|
|
248
|
+
{
|
|
249
|
+
messageSchemas: [UserCreatedSchema, UserUpdatedSchema],
|
|
250
|
+
messageTypeField: 'messageType',
|
|
251
|
+
fifoQueue: true, // Enable FIFO mode
|
|
252
|
+
|
|
253
|
+
// Option 1: Use a field from the message as MessageGroupId
|
|
254
|
+
messageGroupIdField: 'userId',
|
|
255
|
+
|
|
256
|
+
// Option 2: Use a default MessageGroupId for all messages
|
|
257
|
+
// defaultMessageGroupId: 'user-events',
|
|
258
|
+
|
|
259
|
+
creationConfig: {
|
|
260
|
+
queue: {
|
|
261
|
+
QueueName: 'user-events-queue.fifo', // Must end with .fifo
|
|
262
|
+
Attributes: {
|
|
263
|
+
FifoQueue: 'true',
|
|
264
|
+
ContentBasedDeduplication: 'false', // or 'true' for automatic deduplication
|
|
265
|
+
},
|
|
266
|
+
},
|
|
267
|
+
},
|
|
268
|
+
}
|
|
269
|
+
)
|
|
270
|
+
}
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
// Publishing to FIFO queue
|
|
274
|
+
const fifoPublisher = new UserEventsFifoPublisher(sqsClient)
|
|
275
|
+
await fifoPublisher.init()
|
|
276
|
+
|
|
277
|
+
// Messages with the same userId will be processed in order
|
|
278
|
+
await fifoPublisher.publish({
|
|
279
|
+
id: '123',
|
|
280
|
+
messageType: 'user.created',
|
|
281
|
+
userId: 'user-456', // Used as MessageGroupId
|
|
282
|
+
email: 'user@example.com',
|
|
283
|
+
})
|
|
284
|
+
|
|
285
|
+
await fifoPublisher.publish({
|
|
286
|
+
id: '124',
|
|
287
|
+
messageType: 'user.updated',
|
|
288
|
+
userId: 'user-456', // Same group - processed after the first message
|
|
289
|
+
changes: { name: 'John Doe' },
|
|
290
|
+
})
|
|
291
|
+
|
|
292
|
+
// You can also explicitly provide MessageGroupId
|
|
293
|
+
await fifoPublisher.publish(
|
|
294
|
+
{
|
|
295
|
+
id: '125',
|
|
296
|
+
messageType: 'user.created',
|
|
297
|
+
userId: 'user-789',
|
|
298
|
+
email: 'other@example.com',
|
|
299
|
+
},
|
|
300
|
+
{
|
|
301
|
+
MessageGroupId: 'custom-group-id',
|
|
302
|
+
MessageDeduplicationId: 'unique-dedup-id', // Optional
|
|
303
|
+
}
|
|
304
|
+
)
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
### FIFO Queue Consumer
|
|
308
|
+
|
|
309
|
+
```typescript
|
|
310
|
+
class UserEventsFifoConsumer extends AbstractSqsConsumer<
|
|
311
|
+
SupportedMessages,
|
|
312
|
+
ExecutionContext
|
|
313
|
+
> {
|
|
314
|
+
constructor(sqsClient: SQSClient, userService: UserService) {
|
|
315
|
+
super(
|
|
316
|
+
{
|
|
317
|
+
sqsClient,
|
|
318
|
+
logger: console,
|
|
319
|
+
errorReporter: { report: (error) => console.error(error) },
|
|
320
|
+
consumerErrorResolver: {
|
|
321
|
+
resolveError: () => ({ resolve: 'retryLater' as const }),
|
|
322
|
+
},
|
|
323
|
+
transactionObservabilityManager: {
|
|
324
|
+
start: () => {},
|
|
325
|
+
stop: () => {},
|
|
326
|
+
},
|
|
327
|
+
},
|
|
328
|
+
{
|
|
329
|
+
fifoQueue: true, // Enable FIFO mode
|
|
330
|
+
messageTypeField: 'messageType',
|
|
331
|
+
handlers: new MessageHandlerConfigBuilder<SupportedMessages, ExecutionContext>()
|
|
332
|
+
.addConfig(UserCreatedSchema, handleUserCreated)
|
|
333
|
+
.addConfig(UserUpdatedSchema, handleUserUpdated)
|
|
334
|
+
.build(),
|
|
335
|
+
creationConfig: {
|
|
336
|
+
queue: {
|
|
337
|
+
QueueName: 'user-events-queue.fifo',
|
|
338
|
+
Attributes: {
|
|
339
|
+
FifoQueue: 'true',
|
|
340
|
+
ContentBasedDeduplication: 'false',
|
|
341
|
+
VisibilityTimeout: '30',
|
|
342
|
+
},
|
|
343
|
+
},
|
|
344
|
+
},
|
|
345
|
+
// Optional: Configure concurrent consumers for parallel processing of different groups
|
|
346
|
+
concurrentConsumersAmount: 3, // Process 3 different message groups in parallel
|
|
347
|
+
},
|
|
348
|
+
{ userService }
|
|
349
|
+
)
|
|
350
|
+
}
|
|
351
|
+
}
|
|
352
|
+
```
|
|
353
|
+
|
|
354
|
+
## Configuration
|
|
355
|
+
|
|
356
|
+
### Queue Creation
|
|
357
|
+
|
|
358
|
+
When using `creationConfig`, the queue will be created automatically if it doesn't exist:
|
|
359
|
+
|
|
360
|
+
```typescript
|
|
361
|
+
{
|
|
362
|
+
creationConfig: {
|
|
363
|
+
queue: {
|
|
364
|
+
QueueName: 'my-queue',
|
|
365
|
+
Attributes: {
|
|
366
|
+
// Standard Queue attributes
|
|
367
|
+
VisibilityTimeout: '30', // Seconds a message is invisible after being received
|
|
368
|
+
MessageRetentionPeriod: '345600', // 4 days (in seconds)
|
|
369
|
+
ReceiveMessageWaitTimeSeconds: '20', // Long polling duration
|
|
370
|
+
|
|
371
|
+
// FIFO Queue attributes (only for .fifo queues)
|
|
372
|
+
FifoQueue: 'true', // Must be 'true' for FIFO queues
|
|
373
|
+
ContentBasedDeduplication: 'false', // Automatic deduplication based on message body
|
|
374
|
+
DeduplicationScope: 'queue', // 'queue' or 'messageGroup'
|
|
375
|
+
FifoThroughputLimit: 'perQueue', // 'perQueue' or 'perMessageGroupId'
|
|
376
|
+
|
|
377
|
+
// Encryption
|
|
378
|
+
KmsMasterKeyId: 'alias/aws/sqs', // KMS key for encryption
|
|
379
|
+
|
|
380
|
+
// Other attributes
|
|
381
|
+
DelaySeconds: '0', // Default delay for all messages
|
|
382
|
+
MaximumMessageSize: '262144', // 256 KB (default maximum)
|
|
383
|
+
},
|
|
384
|
+
tags: {
|
|
385
|
+
Environment: 'production',
|
|
386
|
+
Team: 'backend',
|
|
387
|
+
},
|
|
388
|
+
},
|
|
389
|
+
updateAttributesIfExists: true, // Update attributes if queue exists
|
|
390
|
+
forceTagUpdate: false, // Force tag update even if unchanged
|
|
391
|
+
|
|
392
|
+
// Policy configuration (see Policy Configuration section)
|
|
393
|
+
policyConfig: {
|
|
394
|
+
resource: 'arn:aws:sqs:us-east-1:123456789012:my-queue',
|
|
395
|
+
statements: [
|
|
396
|
+
{
|
|
397
|
+
Effect: 'Allow',
|
|
398
|
+
Principal: '*',
|
|
399
|
+
Action: ['sqs:SendMessage'],
|
|
400
|
+
},
|
|
401
|
+
],
|
|
402
|
+
},
|
|
403
|
+
},
|
|
404
|
+
}
|
|
405
|
+
```
|
|
406
|
+
|
|
407
|
+
### Queue Locator
|
|
408
|
+
|
|
409
|
+
When using `locatorConfig`, you connect to an existing queue without creating it:
|
|
410
|
+
|
|
411
|
+
```typescript
|
|
412
|
+
{
|
|
413
|
+
locatorConfig: {
|
|
414
|
+
// Option 1: By queue URL
|
|
415
|
+
queueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue',
|
|
416
|
+
|
|
417
|
+
// Option 2: By queue name (URL will be resolved)
|
|
418
|
+
// queueName: 'my-queue',
|
|
419
|
+
},
|
|
420
|
+
}
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
### Publisher Options
|
|
424
|
+
|
|
425
|
+
```typescript
|
|
426
|
+
{
|
|
427
|
+
// Required - Message Schema Configuration
|
|
428
|
+
messageSchemas: [Schema1, Schema2], // Array of Zod schemas
|
|
429
|
+
messageTypeField: 'messageType', // Field containing message type discriminator
|
|
430
|
+
|
|
431
|
+
// Queue Configuration (one of these required)
|
|
432
|
+
creationConfig: { /* ... */ }, // Create queue if doesn't exist
|
|
433
|
+
locatorConfig: { /* ... */ }, // Use existing queue
|
|
434
|
+
|
|
435
|
+
// Optional - FIFO Configuration
|
|
436
|
+
fifoQueue: false, // Set to true for FIFO queues
|
|
437
|
+
messageGroupIdField: 'userId', // Field to use as MessageGroupId
|
|
438
|
+
defaultMessageGroupId: 'default', // Default MessageGroupId if field not present
|
|
439
|
+
|
|
440
|
+
// Optional - Message Field Configuration
|
|
441
|
+
messageIdField: 'id', // Field containing message ID (default: 'id')
|
|
442
|
+
messageTimestampField: 'timestamp', // Field containing timestamp (default: 'timestamp')
|
|
443
|
+
messageDeduplicationIdField: 'deduplicationId', // Field for deduplication ID (default: 'deduplicationId')
|
|
444
|
+
messageDeduplicationOptionsField: 'deduplicationOptions', // Field for deduplication options (default: 'deduplicationOptions')
|
|
445
|
+
|
|
446
|
+
// Optional - Features
|
|
447
|
+
logMessages: false, // Log all published messages
|
|
448
|
+
handlerSpy: true, // Enable handler spy for testing
|
|
449
|
+
|
|
450
|
+
// Optional - Deduplication
|
|
451
|
+
enablePublisherDeduplication: false, // Enable store-based deduplication
|
|
452
|
+
messageDeduplicationConfig: {
|
|
453
|
+
deduplicationStore: redisStore, // Redis-based deduplication store
|
|
454
|
+
},
|
|
455
|
+
|
|
456
|
+
// Optional - Payload Offloading
|
|
457
|
+
payloadStoreConfig: {
|
|
458
|
+
payloadStore: s3Store, // S3-based payload store
|
|
459
|
+
maxPayloadSize: 256 * 1024, // 256 KB
|
|
460
|
+
},
|
|
461
|
+
|
|
462
|
+
// Optional - Deletion
|
|
463
|
+
deletionConfig: {
|
|
464
|
+
deleteIfExists: false, // Delete queue on init
|
|
465
|
+
waitForConfirmation: true, // Wait for deletion to complete
|
|
466
|
+
forceDeleteInProduction: false, // Allow deletion in production
|
|
467
|
+
},
|
|
468
|
+
}
|
|
469
|
+
```
|
|
470
|
+
|
|
471
|
+
### Consumer Options
|
|
472
|
+
|
|
473
|
+
```typescript
|
|
474
|
+
{
|
|
475
|
+
// Required - Message Handling Configuration
|
|
476
|
+
handlers: MessageHandlerConfigBuilder.build(), // Message handlers configuration
|
|
477
|
+
messageTypeField: 'messageType', // Field containing message type discriminator
|
|
478
|
+
|
|
479
|
+
// Queue Configuration (one of these required)
|
|
480
|
+
creationConfig: { /* ... */ },
|
|
481
|
+
locatorConfig: { /* ... */ },
|
|
482
|
+
|
|
483
|
+
// Optional - FIFO Configuration
|
|
484
|
+
fifoQueue: false, // Set to true for FIFO queues
|
|
485
|
+
|
|
486
|
+
// Optional - Message Field Configuration
|
|
487
|
+
messageIdField: 'id', // Field containing message ID (default: 'id')
|
|
488
|
+
messageTimestampField: 'timestamp', // Field containing timestamp (default: 'timestamp')
|
|
489
|
+
messageDeduplicationIdField: 'deduplicationId', // Field for deduplication ID (default: 'deduplicationId')
|
|
490
|
+
messageDeduplicationOptionsField: 'deduplicationOptions', // Field for deduplication options (default: 'deduplicationOptions')
|
|
491
|
+
|
|
492
|
+
// Optional - Concurrency
|
|
493
|
+
concurrentConsumersAmount: 1, // Number of concurrent consumer instances
|
|
494
|
+
|
|
495
|
+
// Optional - Retry Configuration
|
|
496
|
+
maxRetryDuration: 345600, // 4 days in seconds (default)
|
|
497
|
+
|
|
498
|
+
// Optional - Dead Letter Queue
|
|
499
|
+
deadLetterQueue: {
|
|
500
|
+
creationConfig: {
|
|
501
|
+
queue: {
|
|
502
|
+
QueueName: 'my-queue-dlq',
|
|
503
|
+
// For FIFO queues, DLQ must also be FIFO
|
|
504
|
+
Attributes: {
|
|
505
|
+
FifoQueue: 'true', // Match source queue type
|
|
506
|
+
},
|
|
507
|
+
},
|
|
508
|
+
},
|
|
509
|
+
redrivePolicy: {
|
|
510
|
+
maxReceiveCount: 3, // Move to DLQ after 3 receive attempts
|
|
511
|
+
},
|
|
512
|
+
},
|
|
513
|
+
|
|
514
|
+
// Optional - Consumer Behavior
|
|
515
|
+
consumerOverrides: {
|
|
516
|
+
batchSize: 10, // Messages per receive (1-10)
|
|
517
|
+
pollingWaitTimeMs: 0, // Time between polls
|
|
518
|
+
terminateVisibilityTimeout: true, // Reset visibility on error
|
|
519
|
+
heartbeatInterval: 300, // Heartbeat interval in seconds
|
|
520
|
+
},
|
|
521
|
+
|
|
522
|
+
// Optional - Deduplication
|
|
523
|
+
enableConsumerDeduplication: false,
|
|
524
|
+
messageDeduplicationConfig: {
|
|
525
|
+
deduplicationStore: redisStore,
|
|
526
|
+
},
|
|
527
|
+
|
|
528
|
+
// Optional - Payload Offloading
|
|
529
|
+
payloadStoreConfig: {
|
|
530
|
+
payloadStore: s3Store,
|
|
531
|
+
},
|
|
532
|
+
|
|
533
|
+
// Optional - Other
|
|
534
|
+
logMessages: false,
|
|
535
|
+
handlerSpy: true,
|
|
536
|
+
deletionConfig: { /* ... */ },
|
|
537
|
+
}
|
|
538
|
+
```
|
|
539
|
+
|
|
540
|
+
## Advanced Features
|
|
541
|
+
|
|
542
|
+
### Custom Message Field Names
|
|
543
|
+
|
|
544
|
+
All message field names are configurable, allowing you to adapt the library to your existing message schemas:
|
|
545
|
+
|
|
546
|
+
```typescript
|
|
547
|
+
// Your existing message schema with custom field names
|
|
548
|
+
const CustomMessageSchema = z.object({
|
|
549
|
+
messageId: z.string(), // Custom ID field
|
|
550
|
+
eventType: z.literal('order.created'), // Custom type field
|
|
551
|
+
createdAt: z.string(), // Custom timestamp field
|
|
552
|
+
txId: z.string(), // Custom deduplication ID
|
|
553
|
+
txOptions: z.object({ // Custom deduplication options
|
|
554
|
+
deduplicationWindowSeconds: z.number().optional(),
|
|
555
|
+
}).optional(),
|
|
556
|
+
orderId: z.string(),
|
|
557
|
+
amount: z.number(),
|
|
558
|
+
})
|
|
559
|
+
|
|
560
|
+
// Configure the publisher to use your custom field names
|
|
561
|
+
class OrderPublisher extends AbstractSqsPublisher<CustomMessage> {
|
|
562
|
+
constructor(sqsClient: SQSClient) {
|
|
563
|
+
super(
|
|
564
|
+
{ sqsClient, logger: console, errorReporter: { report: console.error } },
|
|
565
|
+
{
|
|
566
|
+
messageSchemas: [CustomMessageSchema],
|
|
567
|
+
|
|
568
|
+
// Map library's internal fields to your custom fields
|
|
569
|
+
messageIdField: 'messageId', // Default: 'id'
|
|
570
|
+
messageTypeField: 'eventType', // Required
|
|
571
|
+
messageTimestampField: 'createdAt', // Default: 'timestamp'
|
|
572
|
+
messageDeduplicationIdField: 'txId', // Default: 'deduplicationId'
|
|
573
|
+
messageDeduplicationOptionsField: 'txOptions', // Default: 'deduplicationOptions'
|
|
574
|
+
|
|
575
|
+
creationConfig: {
|
|
576
|
+
queue: { QueueName: 'orders-queue' },
|
|
577
|
+
},
|
|
578
|
+
}
|
|
579
|
+
)
|
|
580
|
+
}
|
|
581
|
+
}
|
|
582
|
+
|
|
583
|
+
// Use with your custom schema
|
|
584
|
+
await publisher.publish({
|
|
585
|
+
messageId: 'msg-123', // Library will use this for tracking
|
|
586
|
+
eventType: 'order.created', // Library will use this for routing
|
|
587
|
+
createdAt: new Date().toISOString(), // Library will use this for retry tracking
|
|
588
|
+
txId: 'tx-456', // Library will use this for deduplication
|
|
589
|
+
orderId: 'order-789',
|
|
590
|
+
amount: 99.99,
|
|
591
|
+
})
|
|
592
|
+
```
|
|
593
|
+
|
|
594
|
+
**Benefits:**
|
|
595
|
+
- ✅ No need to modify existing message schemas
|
|
596
|
+
- ✅ Maintain consistency with your domain model
|
|
597
|
+
- ✅ Gradual migration from legacy systems
|
|
598
|
+
- ✅ Works with all features (retry, deduplication, offloading)
|
|
599
|
+
|
|
600
|
+
### Dead Letter Queue (DLQ)
|
|
601
|
+
|
|
602
|
+
Dead Letter Queues capture messages that cannot be processed after multiple attempts:
|
|
603
|
+
|
|
604
|
+
```typescript
|
|
605
|
+
{
|
|
606
|
+
deadLetterQueue: {
|
|
607
|
+
creationConfig: {
|
|
608
|
+
queue: {
|
|
609
|
+
QueueName: 'my-queue-dlq',
|
|
610
|
+
// For FIFO source queues, DLQ must also be FIFO
|
|
611
|
+
Attributes: {
|
|
612
|
+
FifoQueue: 'true', // Match source queue type
|
|
613
|
+
MessageRetentionPeriod: '1209600', // 14 days
|
|
614
|
+
},
|
|
615
|
+
},
|
|
616
|
+
},
|
|
617
|
+
redrivePolicy: {
|
|
618
|
+
maxReceiveCount: 3, // Send to DLQ after 3 failed attempts
|
|
619
|
+
},
|
|
620
|
+
},
|
|
621
|
+
}
|
|
622
|
+
```
|
|
623
|
+
|
|
624
|
+
**How it works:**
|
|
625
|
+
1. Message fails processing (handler returns error or throws)
|
|
626
|
+
2. Message becomes visible again (visibility timeout expires)
|
|
627
|
+
3. Consumer receives message again (receive count increments)
|
|
628
|
+
4. After `maxReceiveCount` attempts, SQS automatically moves message to DLQ
|
|
629
|
+
5. DLQ messages can be inspected, reprocessed, or deleted
|
|
630
|
+
|
|
631
|
+
### Message Retry Logic
|
|
632
|
+
|
|
633
|
+
The library implements intelligent retry logic with exponential backoff:
|
|
634
|
+
|
|
635
|
+
```typescript
|
|
636
|
+
{
|
|
637
|
+
maxRetryDuration: 345600, // 4 days in seconds (default)
|
|
638
|
+
}
|
|
639
|
+
```
|
|
640
|
+
|
|
641
|
+
**Retry Flow:**
|
|
642
|
+
|
|
643
|
+
1. **Handler returns `{ error: 'retryLater' }`** or **throws an error**
|
|
644
|
+
2. Consumer checks if message should be retried:
|
|
645
|
+
- Calculates how long the message has been retrying
|
|
646
|
+
- If within `maxRetryDuration`, re-queues message
|
|
647
|
+
- If exceeded, sends to DLQ (if configured) or marks as failed
|
|
648
|
+
|
|
649
|
+
3. **Exponential Backoff (Standard Queues):**
|
|
650
|
+
```
|
|
651
|
+
Attempt 1: 2^0 = 1 second delay
|
|
652
|
+
Attempt 2: 2^1 = 2 seconds delay
|
|
653
|
+
Attempt 3: 2^2 = 4 seconds delay
|
|
654
|
+
Attempt 4: 2^3 = 8 seconds delay
|
|
655
|
+
...
|
|
656
|
+
Max: 900 seconds (15 minutes) per AWS limits
|
|
657
|
+
```
|
|
658
|
+
|
|
659
|
+
4. **FIFO Queues:**
|
|
660
|
+
- No delay support (AWS limitation)
|
|
661
|
+
- Messages retry immediately
|
|
662
|
+
- Order preserved within message group
|
|
663
|
+
|
|
664
|
+
**Handler Return Types:**
|
|
665
|
+
|
|
666
|
+
```typescript
|
|
667
|
+
type HandlerResult = Either<'retryLater', 'success'>
|
|
668
|
+
|
|
669
|
+
// Success - message is deleted from queue
|
|
670
|
+
return { result: 'success' }
|
|
671
|
+
|
|
672
|
+
// Retry - message is re-queued with delay
|
|
673
|
+
return { error: 'retryLater' }
|
|
674
|
+
|
|
675
|
+
// Error thrown - automatically retries
|
|
676
|
+
throw new Error('Database connection failed')
|
|
677
|
+
```
|
|
678
|
+
|
|
679
|
+
### Message Deduplication
|
|
680
|
+
|
|
681
|
+
Prevent duplicate message publishing or processing:
|
|
682
|
+
|
|
683
|
+
#### Publisher-Level Deduplication
|
|
684
|
+
|
|
685
|
+
Prevents sending the same message multiple times:
|
|
686
|
+
|
|
687
|
+
```typescript
|
|
688
|
+
import { InMemoryDeduplicationStore } from '@message-queue-toolkit/core'
|
|
689
|
+
// or
|
|
690
|
+
import { RedisMessageDeduplicationStore } from '@message-queue-toolkit/redis-message-deduplication-store'
|
|
691
|
+
|
|
692
|
+
const deduplicationStore = new RedisMessageDeduplicationStore(redisClient)
|
|
693
|
+
|
|
694
|
+
// Publisher configuration
|
|
695
|
+
{
|
|
696
|
+
enablePublisherDeduplication: true,
|
|
697
|
+
messageDeduplicationIdField: 'deduplicationId',
|
|
698
|
+
messageDeduplicationConfig: {
|
|
699
|
+
deduplicationStore,
|
|
700
|
+
},
|
|
701
|
+
}
|
|
702
|
+
|
|
703
|
+
// Publishing with deduplication
|
|
704
|
+
await publisher.publish({
|
|
705
|
+
id: '123',
|
|
706
|
+
messageType: 'user.created',
|
|
707
|
+
deduplicationId: 'user-456-creation', // Unique key for deduplication
|
|
708
|
+
deduplicationOptions: {
|
|
709
|
+
deduplicationWindowSeconds: 60, // Prevent duplicates for 60 seconds
|
|
710
|
+
},
|
|
711
|
+
})
|
|
712
|
+
|
|
713
|
+
// Second publish with same deduplicationId within 60s is skipped
|
|
714
|
+
await publisher.publish({
|
|
715
|
+
id: '124',
|
|
716
|
+
messageType: 'user.created',
|
|
717
|
+
deduplicationId: 'user-456-creation', // Duplicate - won't be sent
|
|
718
|
+
})
|
|
719
|
+
```
|
|
720
|
+
|
|
721
|
+
#### Consumer-Level Deduplication
|
|
722
|
+
|
|
723
|
+
Prevents processing the same message multiple times:
|
|
724
|
+
|
|
725
|
+
```typescript
|
|
726
|
+
{
|
|
727
|
+
enableConsumerDeduplication: true,
|
|
728
|
+
messageDeduplicationIdField: 'deduplicationId',
|
|
729
|
+
messageDeduplicationConfig: {
|
|
730
|
+
deduplicationStore,
|
|
731
|
+
},
|
|
732
|
+
}
|
|
733
|
+
|
|
734
|
+
// Message configuration
|
|
735
|
+
{
|
|
736
|
+
deduplicationId: 'unique-operation-id',
|
|
737
|
+
deduplicationOptions: {
|
|
738
|
+
deduplicationWindowSeconds: 3600, // 1 hour
|
|
739
|
+
lockTimeoutSeconds: 20, // Lock duration while processing
|
|
740
|
+
acquireTimeoutSeconds: 20, // Max wait time to acquire lock
|
|
741
|
+
refreshIntervalSeconds: 10, // Lock refresh interval
|
|
742
|
+
},
|
|
743
|
+
}
|
|
744
|
+
```
|
|
745
|
+
|
|
746
|
+
**How it works:**
|
|
747
|
+
1. Consumer receives message
|
|
748
|
+
2. Checks deduplication store for duplicate
|
|
749
|
+
3. If duplicate found (within window), skips processing
|
|
750
|
+
4. If not duplicate, acquires exclusive lock
|
|
751
|
+
5. Processes message
|
|
752
|
+
6. Releases lock and marks as processed
|
|
753
|
+
7. Subsequent messages with same ID are skipped
|
|
754
|
+
|
|
755
|
+
### Payload Offloading
|
|
756
|
+
|
|
757
|
+
For messages larger than 256 KB, store the payload externally (e.g., S3):
|
|
758
|
+
|
|
759
|
+
```typescript
|
|
760
|
+
import { S3PayloadStore } from '@message-queue-toolkit/s3-payload-store'
|
|
761
|
+
|
|
762
|
+
const payloadStore = new S3PayloadStore({
|
|
763
|
+
s3Client,
|
|
764
|
+
bucketName: 'my-message-payloads',
|
|
765
|
+
})
|
|
766
|
+
|
|
767
|
+
// Publisher configuration
|
|
768
|
+
{
|
|
769
|
+
payloadStoreConfig: {
|
|
770
|
+
payloadStore,
|
|
771
|
+
maxPayloadSize: 256 * 1024, // 256 KB threshold
|
|
772
|
+
},
|
|
773
|
+
}
|
|
774
|
+
|
|
775
|
+
// Large message is automatically offloaded
|
|
776
|
+
await publisher.publish({
|
|
777
|
+
id: '123',
|
|
778
|
+
messageType: 'document.processed',
|
|
779
|
+
largeData: hugeArrayOfData, // If total size > 256 KB, stored in S3
|
|
780
|
+
})
|
|
781
|
+
```
|
|
782
|
+
|
|
783
|
+
**How it works:**
|
|
784
|
+
1. Publisher checks message size before sending
|
|
785
|
+
2. If size exceeds `maxPayloadSize`, stores payload in S3
|
|
786
|
+
3. Replaces payload with pointer: `{ _offloadedPayload: { bucketName, key, size } }`
|
|
787
|
+
4. Sends pointer message to SQS
|
|
788
|
+
5. Consumer detects pointer, fetches payload from S3
|
|
789
|
+
6. Processes message with full payload
|
|
790
|
+
|
|
791
|
+
**Note:** Payload cleanup is the responsibility of the store (e.g., S3 lifecycle policies).
|
|
792
|
+
|
|
793
|
+
### Message Handlers
|
|
794
|
+
|
|
795
|
+
Handlers process messages based on their type. Messages are routed to the appropriate handler using the discriminator field (configurable via `messageTypeField`):
|
|
796
|
+
|
|
797
|
+
```typescript
|
|
798
|
+
import { MessageHandlerConfigBuilder } from '@message-queue-toolkit/core'
|
|
799
|
+
|
|
800
|
+
const handlers = new MessageHandlerConfigBuilder<
|
|
801
|
+
SupportedMessages,
|
|
802
|
+
ExecutionContext,
|
|
803
|
+
PrehandlerOutput
|
|
804
|
+
>()
|
|
805
|
+
.addConfig(
|
|
806
|
+
UserCreatedSchema,
|
|
807
|
+
async (message, context, preHandlingOutputs) => {
|
|
808
|
+
// Access execution context
|
|
809
|
+
await context.userService.createUser(message.userId)
|
|
810
|
+
|
|
811
|
+
// Access pre-handler outputs
|
|
812
|
+
console.log('Pre-handler result:', preHandlingOutputs.preHandlerOutput)
|
|
813
|
+
console.log('Barrier result:', preHandlingOutputs.barrierOutput)
|
|
814
|
+
|
|
815
|
+
return { result: 'success' }
|
|
816
|
+
},
|
|
817
|
+
{
|
|
818
|
+
// Optional: Pre-handlers (run before main handler)
|
|
819
|
+
preHandlers: [
|
|
820
|
+
(message, context, output, next) => {
|
|
821
|
+
console.log('Pre-processing message:', message.id)
|
|
822
|
+
output.processedAt = Date.now()
|
|
823
|
+
next({ result: 'success' })
|
|
824
|
+
},
|
|
825
|
+
],
|
|
826
|
+
|
|
827
|
+
// Optional: Barrier (controls whether message should be processed)
|
|
828
|
+
preHandlerBarrier: async (message, context, preHandlerOutput) => {
|
|
829
|
+
const isReady = await context.userService.isSystemReady()
|
|
830
|
+
return {
|
|
831
|
+
isPassing: isReady,
|
|
832
|
+
output: { systemStatus: 'ready' },
|
|
833
|
+
}
|
|
834
|
+
},
|
|
835
|
+
|
|
836
|
+
// Optional: Custom message log formatter
|
|
837
|
+
messageLogFormatter: (message) => ({
|
|
838
|
+
userId: message.userId,
|
|
839
|
+
action: 'create',
|
|
840
|
+
}),
|
|
841
|
+
}
|
|
842
|
+
)
|
|
843
|
+
.addConfig(UserUpdatedSchema, handleUserUpdated)
|
|
844
|
+
.build()
|
|
845
|
+
```
|
|
846
|
+
|
|
847
|
+
### Pre-handlers and Barriers
|
|
848
|
+
|
|
849
|
+
#### Pre-handlers
|
|
850
|
+
|
|
851
|
+
Pre-handlers are middleware functions that run before the main message handler, allowing you to:
|
|
852
|
+
- Enrich the execution context with additional data
|
|
853
|
+
- Set up scoped resources (child loggers, database transactions)
|
|
854
|
+
- Validate prerequisites
|
|
855
|
+
- Transform message data
|
|
856
|
+
- Implement cross-cutting concerns (logging, metrics, caching)
|
|
857
|
+
|
|
858
|
+
The output from pre-handlers is passed to both the barrier and the main handler, enabling a powerful data flow pattern.
|
|
859
|
+
|
|
860
|
+
**Type Signature:**
|
|
861
|
+
|
|
862
|
+
```typescript
|
|
863
|
+
type Prehandler<Message, Context, Output> = (
|
|
864
|
+
message: Message,
|
|
865
|
+
context: Context,
|
|
866
|
+
output: Output,
|
|
867
|
+
next: (result: PrehandlerResult) => void
|
|
868
|
+
) => void
|
|
869
|
+
```
|
|
870
|
+
|
|
871
|
+
**Common Use Cases:**
|
|
872
|
+
|
|
873
|
+
##### 1. Child Logger Resolution
|
|
874
|
+
|
|
875
|
+
Create message-specific loggers with contextual information:
|
|
876
|
+
|
|
877
|
+
```typescript
|
|
878
|
+
type PrehandlerOutput = {
|
|
879
|
+
logger: Logger
|
|
880
|
+
}
|
|
881
|
+
|
|
882
|
+
const preHandlers: Prehandler<UserMessage, ExecutionContext, PrehandlerOutput>[] = [
|
|
883
|
+
(message, context, output, next) => {
|
|
884
|
+
// Create child logger with message context
|
|
885
|
+
output.logger = context.logger.child({
|
|
886
|
+
messageId: message.id,
|
|
887
|
+
messageType: message.messageType,
|
|
888
|
+
userId: message.userId,
|
|
889
|
+
correlationId: message.correlationId,
|
|
890
|
+
})
|
|
891
|
+
|
|
892
|
+
output.logger.info('Message processing started')
|
|
893
|
+
next({ result: 'success' })
|
|
894
|
+
},
|
|
895
|
+
]
|
|
896
|
+
|
|
897
|
+
// In your handler
|
|
898
|
+
const handler = async (message, context, preHandlingOutputs) => {
|
|
899
|
+
const logger = preHandlingOutputs.preHandlerOutput.logger
|
|
900
|
+
|
|
901
|
+
logger.info('Processing user update') // Automatically includes message context
|
|
902
|
+
logger.error({ error: someError }, 'Failed to update user')
|
|
903
|
+
|
|
904
|
+
return { result: 'success' }
|
|
905
|
+
}
|
|
906
|
+
```
|
|
907
|
+
|
|
908
|
+
##### 2. User Data and Permissions Resolution
|
|
909
|
+
|
|
910
|
+
Fetch and cache user information needed by the handler:
|
|
911
|
+
|
|
912
|
+
```typescript
|
|
913
|
+
type PrehandlerOutput = {
|
|
914
|
+
user: User
|
|
915
|
+
permissions: string[]
|
|
916
|
+
organizationId: string
|
|
917
|
+
}
|
|
918
|
+
|
|
919
|
+
const preHandlers: Prehandler<OrderMessage, ExecutionContext, PrehandlerOutput>[] = [
|
|
920
|
+
// Fetch user data
|
|
921
|
+
async (message, context, output, next) => {
|
|
922
|
+
try {
|
|
923
|
+
const user = await context.userRepository.findById(message.userId)
|
|
924
|
+
if (!user) {
|
|
925
|
+
next({ error: new Error(`User ${message.userId} not found`) })
|
|
926
|
+
return
|
|
927
|
+
}
|
|
928
|
+
output.user = user
|
|
929
|
+
next({ result: 'success' })
|
|
930
|
+
} catch (error) {
|
|
931
|
+
next({ error })
|
|
932
|
+
}
|
|
933
|
+
},
|
|
934
|
+
|
|
935
|
+
// Resolve permissions
|
|
936
|
+
async (message, context, output, next) => {
|
|
937
|
+
try {
|
|
938
|
+
output.permissions = await context.permissionService.getPermissions(output.user.id)
|
|
939
|
+
output.organizationId = output.user.organizationId
|
|
940
|
+
next({ result: 'success' })
|
|
941
|
+
} catch (error) {
|
|
942
|
+
next({ error })
|
|
943
|
+
}
|
|
944
|
+
},
|
|
945
|
+
]
|
|
946
|
+
|
|
947
|
+
// In your handler - user data is already fetched
|
|
948
|
+
const handler = async (message, context, preHandlingOutputs) => {
|
|
949
|
+
const { user, permissions, organizationId } = preHandlingOutputs.preHandlerOutput
|
|
950
|
+
|
|
951
|
+
// Check permissions
|
|
952
|
+
if (!permissions.includes('orders:create')) {
|
|
953
|
+
throw new Error('Insufficient permissions')
|
|
954
|
+
}
|
|
955
|
+
|
|
956
|
+
// Use pre-fetched data
|
|
957
|
+
await context.orderService.createOrder({
|
|
958
|
+
orderId: message.orderId,
|
|
959
|
+
userId: user.id,
|
|
960
|
+
organizationId,
|
|
961
|
+
userEmail: user.email, // Already available, no need to fetch again
|
|
962
|
+
})
|
|
963
|
+
|
|
964
|
+
return { result: 'success' }
|
|
965
|
+
}
|
|
966
|
+
```
|
|
967
|
+
|
|
968
|
+
##### 3. Database Transaction Management
|
|
969
|
+
|
|
970
|
+
Set up scoped database transactions:
|
|
971
|
+
|
|
972
|
+
```typescript
|
|
973
|
+
type PrehandlerOutput = {
|
|
974
|
+
transaction: DatabaseTransaction
|
|
975
|
+
}
|
|
976
|
+
|
|
977
|
+
const preHandlers = [
|
|
978
|
+
async (message, context, output, next) => {
|
|
979
|
+
const transaction = await context.database.beginTransaction()
|
|
980
|
+
output.transaction = transaction
|
|
981
|
+
|
|
982
|
+
try {
|
|
983
|
+
next({ result: 'success' })
|
|
984
|
+
} catch (error) {
|
|
985
|
+
await transaction.rollback()
|
|
986
|
+
throw error
|
|
987
|
+
}
|
|
988
|
+
},
|
|
989
|
+
]
|
|
990
|
+
|
|
991
|
+
const handler = async (message, context, preHandlingOutputs) => {
|
|
992
|
+
const { transaction } = preHandlingOutputs.preHandlerOutput
|
|
993
|
+
|
|
994
|
+
try {
|
|
995
|
+
await context.userRepository.create(message.userData, { transaction })
|
|
996
|
+
await context.auditRepository.log(message.action, { transaction })
|
|
997
|
+
|
|
998
|
+
await transaction.commit()
|
|
999
|
+
return { result: 'success' }
|
|
1000
|
+
} catch (error) {
|
|
1001
|
+
await transaction.rollback()
|
|
1002
|
+
throw error
|
|
1003
|
+
}
|
|
1004
|
+
}
|
|
1005
|
+
```
|
|
1006
|
+
|
|
1007
|
+
##### 4. Caching and Deduplication
|
|
1008
|
+
|
|
1009
|
+
Implement custom caching logic:
|
|
1010
|
+
|
|
1011
|
+
```typescript
|
|
1012
|
+
type PrehandlerOutput = {
|
|
1013
|
+
cachedData?: ProductData
|
|
1014
|
+
cacheHit: boolean
|
|
1015
|
+
}
|
|
1016
|
+
|
|
1017
|
+
const preHandlers = [
|
|
1018
|
+
async (message, context, output, next) => {
|
|
1019
|
+
const cacheKey = `product:${message.productId}`
|
|
1020
|
+
const cached = await context.cache.get(cacheKey)
|
|
1021
|
+
|
|
1022
|
+
if (cached) {
|
|
1023
|
+
output.cachedData = cached
|
|
1024
|
+
output.cacheHit = true
|
|
1025
|
+
context.logger.info('Cache hit', { productId: message.productId })
|
|
1026
|
+
} else {
|
|
1027
|
+
output.cacheHit = false
|
|
1028
|
+
}
|
|
1029
|
+
|
|
1030
|
+
next({ result: 'success' })
|
|
1031
|
+
},
|
|
1032
|
+
]
|
|
1033
|
+
|
|
1034
|
+
const handler = async (message, context, preHandlingOutputs) => {
|
|
1035
|
+
const { cachedData, cacheHit } = preHandlingOutputs.preHandlerOutput
|
|
1036
|
+
|
|
1037
|
+
if (cacheHit && cachedData) {
|
|
1038
|
+
// Use cached data
|
|
1039
|
+
return processWithCache(cachedData)
|
|
1040
|
+
}
|
|
1041
|
+
|
|
1042
|
+
// Fetch fresh data
|
|
1043
|
+
const data = await context.productService.fetch(message.productId)
|
|
1044
|
+
await context.cache.set(`product:${message.productId}`, data, { ttl: 3600 })
|
|
1045
|
+
|
|
1046
|
+
return processWithCache(data)
|
|
1047
|
+
}
|
|
1048
|
+
```
|
|
1049
|
+
|
|
1050
|
+
##### 5. Metrics and Monitoring
|
|
1051
|
+
|
|
1052
|
+
Track message processing metrics:
|
|
1053
|
+
|
|
1054
|
+
```typescript
|
|
1055
|
+
type PrehandlerOutput = {
|
|
1056
|
+
startTime: number
|
|
1057
|
+
metricsLabels: Record<string, string>
|
|
1058
|
+
}
|
|
1059
|
+
|
|
1060
|
+
const preHandlers = [
|
|
1061
|
+
(message, context, output, next) => {
|
|
1062
|
+
output.startTime = Date.now()
|
|
1063
|
+
output.metricsLabels = {
|
|
1064
|
+
messageType: message.messageType,
|
|
1065
|
+
userId: message.userId,
|
|
1066
|
+
source: message.source || 'unknown',
|
|
1067
|
+
}
|
|
1068
|
+
|
|
1069
|
+
context.metrics.increment('messages.received', output.metricsLabels)
|
|
1070
|
+
next({ result: 'success' })
|
|
1071
|
+
},
|
|
1072
|
+
]
|
|
1073
|
+
|
|
1074
|
+
const handler = async (message, context, preHandlingOutputs) => {
|
|
1075
|
+
const { startTime, metricsLabels } = preHandlingOutputs.preHandlerOutput
|
|
1076
|
+
|
|
1077
|
+
try {
|
|
1078
|
+
await processMessage(message)
|
|
1079
|
+
|
|
1080
|
+
const duration = Date.now() - startTime
|
|
1081
|
+
context.metrics.histogram('message.processing.duration', duration, metricsLabels)
|
|
1082
|
+
context.metrics.increment('messages.processed', { ...metricsLabels, status: 'success' })
|
|
1083
|
+
|
|
1084
|
+
return { result: 'success' }
|
|
1085
|
+
} catch (error) {
|
|
1086
|
+
context.metrics.increment('messages.processed', { ...metricsLabels, status: 'error' })
|
|
1087
|
+
throw error
|
|
1088
|
+
}
|
|
1089
|
+
}
|
|
1090
|
+
```
|
|
1091
|
+
|
|
1092
|
+
**Configuration:**
|
|
1093
|
+
|
|
1094
|
+
```typescript
|
|
1095
|
+
new MessageHandlerConfigBuilder<SupportedMessages, ExecutionContext, PrehandlerOutput>()
|
|
1096
|
+
.addConfig(
|
|
1097
|
+
MessageSchema,
|
|
1098
|
+
handler,
|
|
1099
|
+
{
|
|
1100
|
+
preHandlers: [
|
|
1101
|
+
loggerPreHandler,
|
|
1102
|
+
userDataPreHandler,
|
|
1103
|
+
permissionsPreHandler,
|
|
1104
|
+
],
|
|
1105
|
+
}
|
|
1106
|
+
)
|
|
1107
|
+
.build()
|
|
1108
|
+
```
|
|
1109
|
+
|
|
1110
|
+
#### Barriers
|
|
1111
|
+
|
|
1112
|
+
Barriers are async functions that determine whether a message should be processed immediately or retried later. They are essential for handling message dependencies and ensuring prerequisites are met.
|
|
1113
|
+
|
|
1114
|
+
**Type Signature:**
|
|
1115
|
+
|
|
1116
|
+
```typescript
|
|
1117
|
+
type BarrierCallback<Message, Context, PrehandlerOutput, BarrierOutput> = (
|
|
1118
|
+
message: Message,
|
|
1119
|
+
context: Context,
|
|
1120
|
+
preHandlerOutput: PrehandlerOutput
|
|
1121
|
+
) => Promise<BarrierResult<BarrierOutput>>
|
|
1122
|
+
|
|
1123
|
+
type BarrierResult<Output> = {
|
|
1124
|
+
isPassing: boolean // true = process now, false = retry later
|
|
1125
|
+
output: Output // Additional data passed to the handler
|
|
1126
|
+
}
|
|
1127
|
+
```
|
|
1128
|
+
|
|
1129
|
+
**Common Use Cases:**
|
|
1130
|
+
|
|
1131
|
+
##### 1. Message Ordering Dependencies
|
|
1132
|
+
|
|
1133
|
+
Ensure messages are processed in the correct order when they arrive out of sequence:
|
|
1134
|
+
|
|
1135
|
+
```typescript
|
|
1136
|
+
// Scenario: Process order.updated only after order.created
|
|
1137
|
+
const preHandlerBarrier = async (message: OrderUpdatedMessage, context, preHandlerOutput) => {
|
|
1138
|
+
// Check if the order exists (created event was processed)
|
|
1139
|
+
const orderExists = await context.orderRepository.exists(message.orderId)
|
|
1140
|
+
|
|
1141
|
+
if (!orderExists) {
|
|
1142
|
+
context.logger.warn('Order not found, retrying later', {
|
|
1143
|
+
orderId: message.orderId,
|
|
1144
|
+
messageId: message.id,
|
|
1145
|
+
})
|
|
1146
|
+
|
|
1147
|
+
return {
|
|
1148
|
+
isPassing: false,
|
|
1149
|
+
output: { reason: 'order_not_created_yet' },
|
|
1150
|
+
}
|
|
1151
|
+
}
|
|
1152
|
+
|
|
1153
|
+
return {
|
|
1154
|
+
isPassing: true,
|
|
1155
|
+
output: { orderExists: true },
|
|
1156
|
+
}
|
|
1157
|
+
}
|
|
1158
|
+
|
|
1159
|
+
// Message will be automatically retried until order.created is processed
|
|
1160
|
+
```
|
|
1161
|
+
|
|
1162
|
+
##### 2. External Resource Availability
|
|
1163
|
+
|
|
1164
|
+
Wait for external systems to be ready:
|
|
1165
|
+
|
|
1166
|
+
```typescript
|
|
1167
|
+
// Scenario: Process message only when third-party API is available
|
|
1168
|
+
const preHandlerBarrier = async (message, context, preHandlerOutput) => {
|
|
1169
|
+
try {
|
|
1170
|
+
// Check if external service is healthy
|
|
1171
|
+
const isHealthy = await context.externalApiClient.healthCheck()
|
|
1172
|
+
|
|
1173
|
+
if (!isHealthy) {
|
|
1174
|
+
context.logger.info('External API unhealthy, retrying later')
|
|
1175
|
+
return {
|
|
1176
|
+
isPassing: false,
|
|
1177
|
+
output: { reason: 'external_api_unavailable' },
|
|
1178
|
+
}
|
|
1179
|
+
}
|
|
1180
|
+
|
|
1181
|
+
// Check rate limit
|
|
1182
|
+
const rateLimitOk = await context.rateLimiter.checkLimit(message.userId)
|
|
1183
|
+
if (!rateLimitOk) {
|
|
1184
|
+
context.logger.info('Rate limit exceeded, retrying later')
|
|
1185
|
+
return {
|
|
1186
|
+
isPassing: false,
|
|
1187
|
+
output: { reason: 'rate_limit_exceeded' },
|
|
1188
|
+
}
|
|
1189
|
+
}
|
|
1190
|
+
|
|
1191
|
+
return {
|
|
1192
|
+
isPassing: true,
|
|
1193
|
+
output: { apiAvailable: true },
|
|
1194
|
+
}
|
|
1195
|
+
} catch (error) {
|
|
1196
|
+
context.logger.error({ error }, 'Barrier check failed')
|
|
1197
|
+
return {
|
|
1198
|
+
isPassing: false,
|
|
1199
|
+
output: { reason: 'barrier_error', error },
|
|
1200
|
+
}
|
|
1201
|
+
}
|
|
1202
|
+
}
|
|
1203
|
+
```
|
|
1204
|
+
|
|
1205
|
+
##### 3. Business Workflow Prerequisites
|
|
1206
|
+
|
|
1207
|
+
Implement complex business logic gates:
|
|
1208
|
+
|
|
1209
|
+
```typescript
|
|
1210
|
+
// Scenario: Process payment only after KYC verification is complete
|
|
1211
|
+
const preHandlerBarrier = async (
|
|
1212
|
+
message: PaymentMessage,
|
|
1213
|
+
context,
|
|
1214
|
+
preHandlerOutput
|
|
1215
|
+
) => {
|
|
1216
|
+
const { user } = preHandlerOutput // From pre-handler
|
|
1217
|
+
|
|
1218
|
+
// Check KYC status
|
|
1219
|
+
const kycStatus = await context.kycService.getStatus(user.id)
|
|
1220
|
+
|
|
1221
|
+
if (kycStatus !== 'approved') {
|
|
1222
|
+
context.logger.info('KYC not approved, retrying later', {
|
|
1223
|
+
userId: user.id,
|
|
1224
|
+
kycStatus,
|
|
1225
|
+
})
|
|
1226
|
+
|
|
1227
|
+
return {
|
|
1228
|
+
isPassing: false,
|
|
1229
|
+
output: {
|
|
1230
|
+
reason: 'kyc_pending',
|
|
1231
|
+
kycStatus,
|
|
1232
|
+
retriedAt: new Date(),
|
|
1233
|
+
},
|
|
1234
|
+
}
|
|
1235
|
+
}
|
|
1236
|
+
|
|
1237
|
+
// Check account balance
|
|
1238
|
+
const balance = await context.accountService.getBalance(user.id)
|
|
1239
|
+
if (balance < message.amount) {
|
|
1240
|
+
context.logger.info('Insufficient balance, retrying later', {
|
|
1241
|
+
userId: user.id,
|
|
1242
|
+
balance,
|
|
1243
|
+
required: message.amount,
|
|
1244
|
+
})
|
|
1245
|
+
|
|
1246
|
+
return {
|
|
1247
|
+
isPassing: false,
|
|
1248
|
+
output: {
|
|
1249
|
+
reason: 'insufficient_balance',
|
|
1250
|
+
balance,
|
|
1251
|
+
required: message.amount,
|
|
1252
|
+
},
|
|
1253
|
+
}
|
|
1254
|
+
}
|
|
1255
|
+
|
|
1256
|
+
return {
|
|
1257
|
+
isPassing: true,
|
|
1258
|
+
output: {
|
|
1259
|
+
kycApproved: true,
|
|
1260
|
+
currentBalance: balance,
|
|
1261
|
+
},
|
|
1262
|
+
}
|
|
1263
|
+
}
|
|
1264
|
+
|
|
1265
|
+
const handler = async (message, context, preHandlingOutputs) => {
|
|
1266
|
+
const { kycApproved, currentBalance } = preHandlingOutputs.barrierOutput
|
|
1267
|
+
|
|
1268
|
+
// Safe to process payment - all prerequisites met
|
|
1269
|
+
await context.paymentService.processPayment({
|
|
1270
|
+
userId: message.userId,
|
|
1271
|
+
amount: message.amount,
|
|
1272
|
+
currentBalance, // From barrier
|
|
1273
|
+
})
|
|
1274
|
+
|
|
1275
|
+
return { result: 'success' }
|
|
1276
|
+
}
|
|
1277
|
+
```
|
|
1278
|
+
|
|
1279
|
+
##### 4. Multi-Message Dependencies
|
|
1280
|
+
|
|
1281
|
+
Wait for multiple related messages to be processed:
|
|
1282
|
+
|
|
1283
|
+
```typescript
|
|
1284
|
+
// Scenario: Process shipment only after all items are packed
|
|
1285
|
+
const preHandlerBarrier = async (
|
|
1286
|
+
message: ShipmentMessage,
|
|
1287
|
+
context,
|
|
1288
|
+
preHandlerOutput
|
|
1289
|
+
) => {
|
|
1290
|
+
const orderId = message.orderId
|
|
1291
|
+
|
|
1292
|
+
// Check if all items are packed
|
|
1293
|
+
const orderItems = await context.orderRepository.getItems(orderId)
|
|
1294
|
+
const packedItems = await context.packingRepository.getPackedItems(orderId)
|
|
1295
|
+
|
|
1296
|
+
const allItemsPacked = orderItems.every(item =>
|
|
1297
|
+
packedItems.some(packed => packed.itemId === item.id)
|
|
1298
|
+
)
|
|
1299
|
+
|
|
1300
|
+
if (!allItemsPacked) {
|
|
1301
|
+
const pendingItems = orderItems.filter(item =>
|
|
1302
|
+
!packedItems.some(packed => packed.itemId === item.id)
|
|
1303
|
+
)
|
|
1304
|
+
|
|
1305
|
+
context.logger.info('Not all items packed, retrying later', {
|
|
1306
|
+
orderId,
|
|
1307
|
+
totalItems: orderItems.length,
|
|
1308
|
+
packedItems: packedItems.length,
|
|
1309
|
+
pendingItems: pendingItems.map(i => i.id),
|
|
1310
|
+
})
|
|
1311
|
+
|
|
1312
|
+
return {
|
|
1313
|
+
isPassing: false,
|
|
1314
|
+
output: {
|
|
1315
|
+
reason: 'items_not_packed',
|
|
1316
|
+
pendingItemsCount: pendingItems.length,
|
|
1317
|
+
},
|
|
1318
|
+
}
|
|
1319
|
+
}
|
|
1320
|
+
|
|
1321
|
+
return {
|
|
1322
|
+
isPassing: true,
|
|
1323
|
+
output: {
|
|
1324
|
+
allItemsPacked: true,
|
|
1325
|
+
totalWeight: packedItems.reduce((sum, item) => sum + item.weight, 0),
|
|
1326
|
+
},
|
|
1327
|
+
}
|
|
1328
|
+
}
|
|
1329
|
+
```
|
|
1330
|
+
|
|
1331
|
+
##### 5. Time-Based Gating
|
|
1332
|
+
|
|
1333
|
+
Delay processing until a specific time:
|
|
1334
|
+
|
|
1335
|
+
```typescript
|
|
1336
|
+
// Scenario: Process scheduled messages only after their scheduled time
|
|
1337
|
+
const preHandlerBarrier = async (message: ScheduledMessage, context, preHandlerOutput) => {
|
|
1338
|
+
const scheduledTime = new Date(message.scheduledFor)
|
|
1339
|
+
const now = new Date()
|
|
1340
|
+
|
|
1341
|
+
if (now < scheduledTime) {
|
|
1342
|
+
const delayMs = scheduledTime.getTime() - now.getTime()
|
|
1343
|
+
context.logger.info('Message scheduled for future, retrying later', {
|
|
1344
|
+
messageId: message.id,
|
|
1345
|
+
scheduledFor: scheduledTime,
|
|
1346
|
+
delayMs,
|
|
1347
|
+
})
|
|
1348
|
+
|
|
1349
|
+
return {
|
|
1350
|
+
isPassing: false,
|
|
1351
|
+
output: {
|
|
1352
|
+
reason: 'scheduled_for_future',
|
|
1353
|
+
scheduledFor: scheduledTime,
|
|
1354
|
+
},
|
|
1355
|
+
}
|
|
1356
|
+
}
|
|
1357
|
+
|
|
1358
|
+
return {
|
|
1359
|
+
isPassing: true,
|
|
1360
|
+
output: {
|
|
1361
|
+
scheduledFor: scheduledTime,
|
|
1362
|
+
actualProcessingTime: now,
|
|
1363
|
+
},
|
|
1364
|
+
}
|
|
1365
|
+
}
|
|
1366
|
+
```
|
|
1367
|
+
|
|
1368
|
+
**Configuration:**
|
|
1369
|
+
|
|
1370
|
+
```typescript
|
|
1371
|
+
new MessageHandlerConfigBuilder<SupportedMessages, ExecutionContext, PrehandlerOutput>()
|
|
1372
|
+
.addConfig(
|
|
1373
|
+
MessageSchema,
|
|
1374
|
+
handler,
|
|
1375
|
+
{
|
|
1376
|
+
preHandlers: [userDataPreHandler, permissionsPreHandler],
|
|
1377
|
+
preHandlerBarrier: orderDependencyBarrier,
|
|
1378
|
+
}
|
|
1379
|
+
)
|
|
1380
|
+
.build()
|
|
1381
|
+
```
|
|
1382
|
+
|
|
1383
|
+
**Important Notes:**
|
|
1384
|
+
|
|
1385
|
+
- **Barriers return `isPassing: false`** → Message is automatically retried with exponential backoff
|
|
1386
|
+
- **Barriers throw errors** → Message follows normal error handling (retry or DLQ)
|
|
1387
|
+
- **Barrier output** → Available in handler via `preHandlingOutputs.barrierOutput`
|
|
1388
|
+
- **Retry limits apply** → Messages exceeding `maxRetryDuration` will be sent to DLQ even if barrier keeps returning false
|
|
1389
|
+
- **FIFO queues** → Barriers are especially important for FIFO queues to handle out-of-order delivery within message groups
|
|
1390
|
+
|
|
1391
|
+
### Handler Spies
|
|
1392
|
+
|
|
1393
|
+
Handler spies solve the fundamental challenge of testing asynchronous message-based systems.
|
|
1394
|
+
|
|
1395
|
+
**The Problem:**
|
|
1396
|
+
|
|
1397
|
+
Testing message queues is complex because:
|
|
1398
|
+
1. **Asynchronous processing** - Messages are published and consumed asynchronously with unpredictable timing
|
|
1399
|
+
2. **Indirect interactions** - Business logic may trigger message publishing without explicit calls to the publisher
|
|
1400
|
+
3. **Non-deterministic order** - Messages may be processed in different orders across test runs
|
|
1401
|
+
4. **Hard to verify** - Traditional mocking/stubbing doesn't work well for async pub/sub patterns
|
|
1402
|
+
|
|
1403
|
+
**The Solution:**
|
|
1404
|
+
|
|
1405
|
+
Handler spies provide a way to wait for and inspect messages during tests without having to:
|
|
1406
|
+
- Poll the queue directly
|
|
1407
|
+
- Add artificial delays (`setTimeout`)
|
|
1408
|
+
- Mock the entire message infrastructure
|
|
1409
|
+
- Modify production code for testing
|
|
1410
|
+
|
|
1411
|
+
#### Configuration
|
|
1412
|
+
|
|
1413
|
+
```typescript
|
|
1414
|
+
// Enable handler spy for publisher and/or consumer
|
|
1415
|
+
const publisher = new UserEventsPublisher(sqsClient, {
|
|
1416
|
+
handlerSpy: true, // Track published messages
|
|
1417
|
+
})
|
|
1418
|
+
|
|
1419
|
+
const consumer = new UserEventsConsumer(sqsClient, {
|
|
1420
|
+
handlerSpy: true, // Track consumed messages
|
|
1421
|
+
})
|
|
1422
|
+
```
|
|
1423
|
+
|
|
1424
|
+
#### Example 1: Testing Direct Message Publishing
|
|
1425
|
+
|
|
1426
|
+
```typescript
|
|
1427
|
+
import { describe, it, expect, beforeEach } from 'vitest'
|
|
1428
|
+
|
|
1429
|
+
describe('UserEventsPublisher', () => {
|
|
1430
|
+
let publisher: UserEventsPublisher
|
|
1431
|
+
|
|
1432
|
+
beforeEach(async () => {
|
|
1433
|
+
publisher = new UserEventsPublisher(sqsClient, { handlerSpy: true })
|
|
1434
|
+
await publisher.init()
|
|
1435
|
+
})
|
|
1436
|
+
|
|
1437
|
+
it('publishes user.created event', async () => {
|
|
1438
|
+
// Act: Publish message
|
|
1439
|
+
await publisher.publish({
|
|
1440
|
+
id: 'msg-123',
|
|
1441
|
+
messageType: 'user.created',
|
|
1442
|
+
userId: 'user-456',
|
|
1443
|
+
email: 'test@example.com',
|
|
1444
|
+
})
|
|
1445
|
+
|
|
1446
|
+
// Assert: Wait for message to be tracked by publisher spy
|
|
1447
|
+
const publishedMessage = await publisher.handlerSpy.waitForMessageWithId(
|
|
1448
|
+
'msg-123',
|
|
1449
|
+
'published',
|
|
1450
|
+
5000 // 5 second timeout
|
|
1451
|
+
)
|
|
1452
|
+
|
|
1453
|
+
expect(publishedMessage).toMatchObject({
|
|
1454
|
+
id: 'msg-123',
|
|
1455
|
+
userId: 'user-456',
|
|
1456
|
+
email: 'test@example.com',
|
|
1457
|
+
})
|
|
1458
|
+
})
|
|
1459
|
+
})
|
|
1460
|
+
```
|
|
1461
|
+
|
|
1462
|
+
#### Example 2: Testing Indirect Message Publishing via API
|
|
1463
|
+
|
|
1464
|
+
This example demonstrates testing business logic that publishes messages internally:
|
|
1465
|
+
|
|
1466
|
+
```typescript
|
|
1467
|
+
import { describe, it, expect, beforeEach } from 'vitest'
|
|
1468
|
+
import request from 'supertest'
|
|
1469
|
+
|
|
1470
|
+
// Your API endpoint that creates a user and publishes an event
|
|
1471
|
+
class UserController {
|
|
1472
|
+
constructor(
|
|
1473
|
+
private userRepository: UserRepository,
|
|
1474
|
+
private eventPublisher: UserEventsPublisher
|
|
1475
|
+
) {}
|
|
1476
|
+
|
|
1477
|
+
async createUser(req, res) {
|
|
1478
|
+
const user = await this.userRepository.create(req.body)
|
|
1479
|
+
|
|
1480
|
+
// Publish event internally - not directly exposed to the test
|
|
1481
|
+
await this.eventPublisher.publish({
|
|
1482
|
+
id: crypto.randomUUID(),
|
|
1483
|
+
messageType: 'user.created',
|
|
1484
|
+
userId: user.id,
|
|
1485
|
+
email: user.email,
|
|
1486
|
+
})
|
|
1487
|
+
|
|
1488
|
+
res.status(201).json(user)
|
|
1489
|
+
}
|
|
1490
|
+
}
|
|
1491
|
+
|
|
1492
|
+
describe('User Creation Flow', () => {
|
|
1493
|
+
let app: Express
|
|
1494
|
+
let publisher: UserEventsPublisher
|
|
1495
|
+
let consumer: UserEventsConsumer
|
|
1496
|
+
|
|
1497
|
+
beforeEach(async () => {
|
|
1498
|
+
// Set up publisher with handler spy
|
|
1499
|
+
publisher = new UserEventsPublisher(sqsClient, { handlerSpy: true })
|
|
1500
|
+
await publisher.init()
|
|
1501
|
+
|
|
1502
|
+
// Set up consumer with handler spy
|
|
1503
|
+
consumer = new UserEventsConsumer(sqsClient, userService, { handlerSpy: true })
|
|
1504
|
+
await consumer.start()
|
|
1505
|
+
|
|
1506
|
+
// Create API with real publisher
|
|
1507
|
+
app = createApp({ eventPublisher: publisher })
|
|
1508
|
+
})
|
|
1509
|
+
|
|
1510
|
+
it('publishes event when user is created via API', async () => {
|
|
1511
|
+
// Act: Make API call (no direct interaction with publisher)
|
|
1512
|
+
const response = await request(app)
|
|
1513
|
+
.post('/api/users')
|
|
1514
|
+
.send({
|
|
1515
|
+
email: 'newuser@example.com',
|
|
1516
|
+
name: 'John Doe',
|
|
1517
|
+
})
|
|
1518
|
+
|
|
1519
|
+
expect(response.status).toBe(201)
|
|
1520
|
+
const createdUserId = response.body.id
|
|
1521
|
+
|
|
1522
|
+
// Assert: Wait for message to be published (by internal business logic)
|
|
1523
|
+
const publishedMessage = await publisher.handlerSpy.waitForMessage(
|
|
1524
|
+
(msg) => msg.userId === createdUserId && msg.messageType === 'user.created',
|
|
1525
|
+
'published',
|
|
1526
|
+
5000
|
|
1527
|
+
)
|
|
1528
|
+
|
|
1529
|
+
expect(publishedMessage).toMatchObject({
|
|
1530
|
+
messageType: 'user.created',
|
|
1531
|
+
userId: createdUserId,
|
|
1532
|
+
email: 'newuser@example.com',
|
|
1533
|
+
})
|
|
1534
|
+
|
|
1535
|
+
// Assert: Wait for message to be consumed and processed
|
|
1536
|
+
const consumedMessage = await consumer.handlerSpy.waitForMessage(
|
|
1537
|
+
(msg) => msg.userId === createdUserId,
|
|
1538
|
+
'consumed',
|
|
1539
|
+
10000 // Allow more time for async processing
|
|
1540
|
+
)
|
|
1541
|
+
|
|
1542
|
+
expect(consumedMessage.userId).toBe(createdUserId)
|
|
1543
|
+
|
|
1544
|
+
// Verify side effects in your user service
|
|
1545
|
+
expect(userService.onUserCreated).toHaveBeenCalledWith(createdUserId)
|
|
1546
|
+
})
|
|
1547
|
+
|
|
1548
|
+
it('handles complex multi-step workflows', async () => {
|
|
1549
|
+
// Create user via API
|
|
1550
|
+
const createResponse = await request(app)
|
|
1551
|
+
.post('/api/users')
|
|
1552
|
+
.send({ email: 'user@example.com', name: 'Jane Doe' })
|
|
1553
|
+
|
|
1554
|
+
const userId = createResponse.body.id
|
|
1555
|
+
|
|
1556
|
+
// Wait for user.created event
|
|
1557
|
+
await consumer.handlerSpy.waitForMessage(
|
|
1558
|
+
(msg) => msg.userId === userId && msg.messageType === 'user.created',
|
|
1559
|
+
'consumed'
|
|
1560
|
+
)
|
|
1561
|
+
|
|
1562
|
+
// Update user via API (triggers another event)
|
|
1563
|
+
await request(app)
|
|
1564
|
+
.patch(`/api/users/${userId}`)
|
|
1565
|
+
.send({ name: 'Jane Smith' })
|
|
1566
|
+
|
|
1567
|
+
// Wait for user.updated event
|
|
1568
|
+
const updatedMessage = await consumer.handlerSpy.waitForMessage(
|
|
1569
|
+
(msg) => msg.userId === userId && msg.messageType === 'user.updated',
|
|
1570
|
+
'consumed',
|
|
1571
|
+
5000
|
|
1572
|
+
)
|
|
1573
|
+
|
|
1574
|
+
expect(updatedMessage.changes).toMatchObject({ name: 'Jane Smith' })
|
|
1575
|
+
})
|
|
1576
|
+
})
|
|
1577
|
+
```
|
|
1578
|
+
|
|
1579
|
+
#### Example 3: Non-Waiting Checks
|
|
1580
|
+
|
|
1581
|
+
For scenarios where you don't want to wait:
|
|
1582
|
+
|
|
1583
|
+
```typescript
|
|
1584
|
+
it('checks message without waiting', async () => {
|
|
1585
|
+
await publisher.publish({
|
|
1586
|
+
id: 'msg-789',
|
|
1587
|
+
messageType: 'user.deleted',
|
|
1588
|
+
userId: 'user-123',
|
|
1589
|
+
})
|
|
1590
|
+
|
|
1591
|
+
// Wait briefly for async processing
|
|
1592
|
+
await new Promise(resolve => setTimeout(resolve, 100))
|
|
1593
|
+
|
|
1594
|
+
// Check without waiting
|
|
1595
|
+
const result = consumer.handlerSpy.checkMessage(
|
|
1596
|
+
(msg) => msg.id === 'msg-789'
|
|
1597
|
+
)
|
|
1598
|
+
|
|
1599
|
+
if (result) {
|
|
1600
|
+
expect(result.message.userId).toBe('user-123')
|
|
1601
|
+
expect(result.processingResult.status).toBe('consumed')
|
|
1602
|
+
} else {
|
|
1603
|
+
throw new Error('Message not found')
|
|
1604
|
+
}
|
|
1605
|
+
})
|
|
1606
|
+
```
|
|
1607
|
+
|
|
1608
|
+
#### Example 4: Inspecting All Messages
|
|
1609
|
+
|
|
1610
|
+
Useful for debugging or verifying batch operations:
|
|
1611
|
+
|
|
1612
|
+
```typescript
|
|
1613
|
+
it('processes batch of user events', async () => {
|
|
1614
|
+
// Publish multiple messages
|
|
1615
|
+
for (let i = 0; i < 10; i++) {
|
|
1616
|
+
await publisher.publish({
|
|
1617
|
+
id: `msg-${i}`,
|
|
1618
|
+
messageType: 'user.created',
|
|
1619
|
+
userId: `user-${i}`,
|
|
1620
|
+
email: `user${i}@example.com`,
|
|
1621
|
+
})
|
|
1622
|
+
}
|
|
1623
|
+
|
|
1624
|
+
// Wait for the last message
|
|
1625
|
+
await consumer.handlerSpy.waitForMessageWithId('msg-9', 'consumed')
|
|
1626
|
+
|
|
1627
|
+
// Inspect all processed messages
|
|
1628
|
+
const allMessages = consumer.handlerSpy.getAllMessages()
|
|
1629
|
+
|
|
1630
|
+
expect(allMessages.length).toBeGreaterThanOrEqual(10)
|
|
1631
|
+
|
|
1632
|
+
const successfulMessages = allMessages.filter(
|
|
1633
|
+
({ processingResult }) => processingResult.status === 'consumed'
|
|
1634
|
+
)
|
|
1635
|
+
|
|
1636
|
+
expect(successfulMessages.length).toBe(10)
|
|
1637
|
+
})
|
|
1638
|
+
```
|
|
1639
|
+
|
|
1640
|
+
#### Handler Spy API Reference
|
|
1641
|
+
|
|
1642
|
+
```typescript
|
|
1643
|
+
interface HandlerSpy<Message> {
|
|
1644
|
+
// Wait for message by ID (with timeout)
|
|
1645
|
+
waitForMessageWithId(
|
|
1646
|
+
messageId: string,
|
|
1647
|
+
state: 'consumed' | 'published' | 'retryLater',
|
|
1648
|
+
timeout?: number // Default: 15000ms
|
|
1649
|
+
): Promise<Message>
|
|
1650
|
+
|
|
1651
|
+
// Wait for message matching predicate (with timeout)
|
|
1652
|
+
waitForMessage(
|
|
1653
|
+
predicate: (message: Message) => boolean,
|
|
1654
|
+
state: 'consumed' | 'published' | 'retryLater',
|
|
1655
|
+
timeout?: number // Default: 15000ms
|
|
1656
|
+
): Promise<Message>
|
|
1657
|
+
|
|
1658
|
+
// Check if message exists without waiting
|
|
1659
|
+
checkMessage(
|
|
1660
|
+
predicate: (message: Message) => boolean
|
|
1661
|
+
): { message: Message; processingResult: ProcessingResult } | undefined
|
|
1662
|
+
|
|
1663
|
+
// Get all tracked messages (circular buffer, limited size)
|
|
1664
|
+
getAllMessages(): Array<{ message: Message; processingResult: ProcessingResult }>
|
|
1665
|
+
}
|
|
1666
|
+
```
|
|
1667
|
+
|
|
1668
|
+
**Best Practices:**
|
|
1669
|
+
|
|
1670
|
+
1. **Always set timeouts** - Tests can hang indefinitely if messages don't arrive
|
|
1671
|
+
2. **Use specific predicates** - Avoid overly broad matchers that could match wrong messages
|
|
1672
|
+
3. **Clean up between tests** - Reset handler spies or recreate publishers/consumers
|
|
1673
|
+
4. **Use in integration tests** - Handler spies are most valuable for integration tests, not unit tests
|
|
1674
|
+
5. **Don't use in production** - Handler spies add memory overhead (circular buffer of messages)
|
|
1675
|
+
|
|
1676
|
+
## Non-Standard Message Formats
|
|
1677
|
+
|
|
1678
|
+
The toolkit supports consuming messages that don't follow the standard message structure (with `type`, `id`, and `timestamp` fields at the root level). This is particularly useful for consuming events from external systems like AWS EventBridge, CloudWatch Events, or custom event sources.
|
|
1679
|
+
|
|
1680
|
+
### EventBridge Events
|
|
1681
|
+
|
|
1682
|
+
AWS EventBridge events have a different structure than standard toolkit messages. They use:
|
|
1683
|
+
- `detail-type` instead of `type` (routing field at root level)
|
|
1684
|
+
- `time` instead of `timestamp`
|
|
1685
|
+
- `detail` for the actual payload (nested)
|
|
1686
|
+
- `id` for the message ID (same as default)
|
|
1687
|
+
|
|
1688
|
+
Handlers receive the full EventBridge envelope and can access the nested `detail` field directly.
|
|
1689
|
+
|
|
1690
|
+
#### Using the EventBridge Schema Builder
|
|
1691
|
+
|
|
1692
|
+
The toolkit provides helper functions to create properly typed EventBridge schemas:
|
|
1693
|
+
|
|
1694
|
+
```typescript
|
|
1695
|
+
import {
|
|
1696
|
+
AbstractSqsConsumer, createEventBridgeSchema,
|
|
1697
|
+
} from '@message-queue-toolkit/sqs'
|
|
1698
|
+
import { MessageHandlerConfigBuilder } from '@message-queue-toolkit/core'
|
|
1699
|
+
import z from 'zod'
|
|
1700
|
+
|
|
1701
|
+
// Step 1: Define your detail (payload) schema
|
|
1702
|
+
const USER_PRESENCE_DETAIL = z.object({
|
|
1703
|
+
userId: z.string(),
|
|
1704
|
+
status: z.enum(['AVAILABLE', 'AWAY', 'BUSY', 'OFFLINE']),
|
|
1705
|
+
timestamp: z.string(),
|
|
1706
|
+
})
|
|
1707
|
+
|
|
1708
|
+
// Step 2: Create envelope schema with literal detail-type for routing
|
|
1709
|
+
const USER_PRESENCE_ENVELOPE = createEventBridgeSchema(
|
|
1710
|
+
USER_PRESENCE_DETAIL,
|
|
1711
|
+
'user.presence.changed' // Literal value for routing
|
|
1712
|
+
)
|
|
1713
|
+
|
|
1714
|
+
// Step 3: Infer the envelope type (what handlers receive)
|
|
1715
|
+
type UserPresenceEvent = z.infer<typeof USER_PRESENCE_ENVELOPE>
|
|
1716
|
+
// {
|
|
1717
|
+
// version: string
|
|
1718
|
+
// id: string
|
|
1719
|
+
// 'detail-type': 'user.presence.changed' // Literal type for routing
|
|
1720
|
+
// source: string
|
|
1721
|
+
// account: string
|
|
1722
|
+
// time: string
|
|
1723
|
+
// region: string
|
|
1724
|
+
// resources: string[]
|
|
1725
|
+
// detail: {
|
|
1726
|
+
// userId: string
|
|
1727
|
+
// status: 'AVAILABLE' | 'AWAY' | 'BUSY' | 'OFFLINE'
|
|
1728
|
+
// timestamp: string
|
|
1729
|
+
// }
|
|
1730
|
+
// }
|
|
1731
|
+
```
|
|
1732
|
+
|
|
1733
|
+
#### Creating an EventBridge Consumer
|
|
1734
|
+
|
|
1735
|
+
```typescript
|
|
1736
|
+
class EventBridgeConsumer extends AbstractSqsConsumer<UserPresenceEvent, ExecutionContext> {
|
|
1737
|
+
constructor(dependencies: SQSConsumerDependencies) {
|
|
1738
|
+
super(dependencies, {
|
|
1739
|
+
creationConfig: {
|
|
1740
|
+
queue: { QueueName: 'eventbridge-events' },
|
|
1741
|
+
},
|
|
1742
|
+
|
|
1743
|
+
// Configure field mappings for EventBridge
|
|
1744
|
+
messageTypeField: 'detail-type', // EventBridge uses 'detail-type'
|
|
1745
|
+
messageIdField: 'id', // Standard, same as default
|
|
1746
|
+
messageTimestampField: 'time', // EventBridge uses 'time'
|
|
1747
|
+
|
|
1748
|
+
// Handlers receive the full EventBridge envelope
|
|
1749
|
+
handlers: new MessageHandlerConfigBuilder<UserPresenceEvent, ExecutionContext>()
|
|
1750
|
+
.addConfig(
|
|
1751
|
+
USER_PRESENCE_ENVELOPE, // Schema validates full envelope
|
|
1752
|
+
async (message, context) => {
|
|
1753
|
+
// message is the full EventBridge envelope
|
|
1754
|
+
// Access the detail field directly
|
|
1755
|
+
console.log('User status changed:', message.detail.userId, message.detail.status)
|
|
1756
|
+
return { result: 'success' as const }
|
|
1757
|
+
},
|
|
1758
|
+
)
|
|
1759
|
+
.build(),
|
|
1760
|
+
})
|
|
1761
|
+
}
|
|
1762
|
+
}
|
|
1763
|
+
```
|
|
1764
|
+
|
|
1765
|
+
**Key Points:**
|
|
1766
|
+
- The envelope schema validates the full EventBridge message structure
|
|
1767
|
+
- The literal `detail-type` value in the schema is used for routing
|
|
1768
|
+
- Handlers receive the complete validated envelope
|
|
1769
|
+
- Access nested payload via `message.detail`
|
|
1770
|
+
|
|
1771
|
+
#### Multiple EventBridge Event Types
|
|
1772
|
+
|
|
1773
|
+
Handle multiple EventBridge event types with distinct detail-type values:
|
|
1774
|
+
|
|
1775
|
+
```typescript
|
|
1776
|
+
import { createEventBridgeSchema } from '@message-queue-toolkit/sqs'
|
|
1777
|
+
|
|
1778
|
+
// Step 1: Define detail (payload) schemas
|
|
1779
|
+
const USER_PRESENCE_DETAIL = z.object({
|
|
1780
|
+
userId: z.string(),
|
|
1781
|
+
status: z.string(),
|
|
1782
|
+
timestamp: z.string(),
|
|
1783
|
+
})
|
|
1784
|
+
|
|
1785
|
+
const USER_ROUTING_STATUS_DETAIL = z.object({
|
|
1786
|
+
userId: z.string(),
|
|
1787
|
+
routingStatus: z.object({
|
|
1788
|
+
id: z.string(),
|
|
1789
|
+
status: z.string(),
|
|
1790
|
+
}),
|
|
1791
|
+
timestamp: z.string(),
|
|
1792
|
+
})
|
|
1793
|
+
|
|
1794
|
+
// Step 2: Create envelope schemas with literal detail-type values
|
|
1795
|
+
const USER_PRESENCE_ENVELOPE = createEventBridgeSchema(
|
|
1796
|
+
USER_PRESENCE_DETAIL,
|
|
1797
|
+
'v2.users.{id}.presence' // Literal routing value
|
|
1798
|
+
)
|
|
1799
|
+
|
|
1800
|
+
const USER_ROUTING_STATUS_ENVELOPE = createEventBridgeSchema(
|
|
1801
|
+
USER_ROUTING_STATUS_DETAIL,
|
|
1802
|
+
'v2.users.{id}.routing.status' // Literal routing value
|
|
1803
|
+
)
|
|
1804
|
+
|
|
1805
|
+
// Step 3: Union type for all event envelopes
|
|
1806
|
+
type SupportedEventBridgeEvents =
|
|
1807
|
+
| z.infer<typeof USER_PRESENCE_ENVELOPE>
|
|
1808
|
+
| z.infer<typeof USER_ROUTING_STATUS_ENVELOPE>
|
|
1809
|
+
|
|
1810
|
+
// Step 4: Consumer with multiple handlers
|
|
1811
|
+
class MultiEventConsumer extends AbstractSqsConsumer<SupportedEventBridgeEvents, ExecutionContext> {
|
|
1812
|
+
constructor(dependencies: SQSConsumerDependencies) {
|
|
1813
|
+
super(dependencies, {
|
|
1814
|
+
creationConfig: { queue: { QueueName: 'multi-event-queue' } },
|
|
1815
|
+
|
|
1816
|
+
messageTypeField: 'detail-type',
|
|
1817
|
+
messageTimestampField: 'time',
|
|
1818
|
+
|
|
1819
|
+
handlers: new MessageHandlerConfigBuilder<SupportedEventBridgeEvents, ExecutionContext>()
|
|
1820
|
+
.addConfig(
|
|
1821
|
+
USER_PRESENCE_ENVELOPE,
|
|
1822
|
+
async (message, context) => {
|
|
1823
|
+
// message.detail is typed as UserPresenceDetail
|
|
1824
|
+
console.log(`User ${message.detail.userId} status: ${message.detail.status}`)
|
|
1825
|
+
return { result: 'success' as const }
|
|
1826
|
+
}
|
|
1827
|
+
)
|
|
1828
|
+
.addConfig(
|
|
1829
|
+
USER_ROUTING_STATUS_ENVELOPE,
|
|
1830
|
+
async (message, context) => {
|
|
1831
|
+
// message.detail is typed as UserRoutingStatusDetail
|
|
1832
|
+
console.log(`User ${message.detail.userId} routing: ${message.detail.routingStatus.status}`)
|
|
1833
|
+
return { result: 'success' as const }
|
|
1834
|
+
}
|
|
1835
|
+
)
|
|
1836
|
+
.build(),
|
|
1837
|
+
})
|
|
1838
|
+
}
|
|
1839
|
+
}
|
|
1840
|
+
```
|
|
1841
|
+
|
|
1842
|
+
#### Complete EventBridge Example
|
|
1843
|
+
|
|
1844
|
+
```typescript
|
|
1845
|
+
import { SQSClient } from '@aws-sdk/client-sqs'
|
|
1846
|
+
import {
|
|
1847
|
+
AbstractSqsConsumer,
|
|
1848
|
+
createEventBridgeSchema,
|
|
1849
|
+
SqsConsumerErrorResolver,
|
|
1850
|
+
type SQSConsumerDependencies
|
|
1851
|
+
} from '@message-queue-toolkit/sqs'
|
|
1852
|
+
import { MessageHandlerConfigBuilder } from '@message-queue-toolkit/core'
|
|
1853
|
+
import z from 'zod'
|
|
1854
|
+
|
|
1855
|
+
// 1. Define the detail payload schema
|
|
1856
|
+
const USER_CREATED_DETAIL = z.object({
|
|
1857
|
+
userId: z.string(),
|
|
1858
|
+
email: z.string().email(),
|
|
1859
|
+
name: z.string(),
|
|
1860
|
+
createdAt: z.string(),
|
|
1861
|
+
})
|
|
1862
|
+
|
|
1863
|
+
// 2. Create envelope schema with literal detail-type for routing
|
|
1864
|
+
const USER_CREATED_ENVELOPE = createEventBridgeSchema(
|
|
1865
|
+
USER_CREATED_DETAIL,
|
|
1866
|
+
'user.created' // Literal value for routing
|
|
1867
|
+
)
|
|
1868
|
+
|
|
1869
|
+
// 3. Type for the event envelope (what handlers receive)
|
|
1870
|
+
type UserCreatedEvent = z.infer<typeof USER_CREATED_ENVELOPE>
|
|
1871
|
+
|
|
1872
|
+
// 4. Create consumer
|
|
1873
|
+
class UserEventConsumer extends AbstractSqsConsumer<UserCreatedEvent> {
|
|
1874
|
+
constructor(dependencies: SQSConsumerDependencies) {
|
|
1875
|
+
super(dependencies, {
|
|
1876
|
+
creationConfig: {
|
|
1877
|
+
queue: { QueueName: 'user-events' },
|
|
1878
|
+
},
|
|
1879
|
+
|
|
1880
|
+
// EventBridge field mappings
|
|
1881
|
+
messageTypeField: 'detail-type',
|
|
1882
|
+
messageTimestampField: 'time',
|
|
1883
|
+
|
|
1884
|
+
handlers: new MessageHandlerConfigBuilder<UserCreatedEvent>()
|
|
1885
|
+
.addConfig(
|
|
1886
|
+
USER_CREATED_ENVELOPE,
|
|
1887
|
+
async (message) => {
|
|
1888
|
+
// message is the full EventBridge envelope
|
|
1889
|
+
// Access the detail field directly
|
|
1890
|
+
console.log(`New user created: ${message.detail.name} (${message.detail.email})`)
|
|
1891
|
+
await saveUserToDatabase(message.detail)
|
|
1892
|
+
return { result: 'success' as const }
|
|
1893
|
+
},
|
|
1894
|
+
)
|
|
1895
|
+
.build(),
|
|
1896
|
+
})
|
|
1897
|
+
}
|
|
1898
|
+
}
|
|
1899
|
+
|
|
1900
|
+
// 5. Use the consumer
|
|
1901
|
+
const sqsClient = new SQSClient({ region: 'us-east-1' })
|
|
1902
|
+
const consumer = new UserEventConsumer({
|
|
1903
|
+
sqsClient,
|
|
1904
|
+
consumerErrorResolver: new SqsConsumerErrorResolver(),
|
|
1905
|
+
errorReporter: { report: (error) => console.error(error) },
|
|
1906
|
+
logger: console,
|
|
1907
|
+
transactionObservabilityManager: undefined,
|
|
1908
|
+
})
|
|
1909
|
+
|
|
1910
|
+
await consumer.start()
|
|
1911
|
+
|
|
1912
|
+
// Example EventBridge event structure (for testing):
|
|
1913
|
+
const exampleEvent = {
|
|
1914
|
+
version: '0',
|
|
1915
|
+
id: '12345678-1234-1234-1234-123456789012',
|
|
1916
|
+
'detail-type': 'user.created',
|
|
1917
|
+
source: 'my.application',
|
|
1918
|
+
account: '123456789012',
|
|
1919
|
+
time: '2025-01-15T12:00:00Z',
|
|
1920
|
+
region: 'us-east-1',
|
|
1921
|
+
resources: [],
|
|
1922
|
+
detail: {
|
|
1923
|
+
userId: 'user-123',
|
|
1924
|
+
email: 'user@example.com',
|
|
1925
|
+
name: 'John Doe',
|
|
1926
|
+
createdAt: '2025-01-15T12:00:00Z',
|
|
1927
|
+
},
|
|
1928
|
+
} satisfies UserCreatedEvent
|
|
1929
|
+
```
|
|
1930
|
+
|
|
1931
|
+
### Custom Message Structures
|
|
1932
|
+
|
|
1933
|
+
For other non-standard message formats, you can configure the field mappings:
|
|
1934
|
+
|
|
1935
|
+
```typescript
|
|
1936
|
+
// Define your custom message schema
|
|
1937
|
+
const CUSTOM_MESSAGE_SCHEMA = z.object({
|
|
1938
|
+
eventType: z.literal('order.created'), // Routing field
|
|
1939
|
+
correlationId: z.string(),
|
|
1940
|
+
occurredAt: z.string(),
|
|
1941
|
+
data: z.object({
|
|
1942
|
+
orderId: z.string(),
|
|
1943
|
+
amount: z.number(),
|
|
1944
|
+
}),
|
|
1945
|
+
})
|
|
1946
|
+
|
|
1947
|
+
type CustomMessage = z.infer<typeof CUSTOM_MESSAGE_SCHEMA>
|
|
1948
|
+
|
|
1949
|
+
class CustomConsumer extends AbstractSqsConsumer<CustomMessage> {
|
|
1950
|
+
constructor(dependencies: SQSConsumerDependencies) {
|
|
1951
|
+
super(dependencies, {
|
|
1952
|
+
creationConfig: { queue: { QueueName: 'custom-queue' } },
|
|
1953
|
+
|
|
1954
|
+
// Map your custom field names
|
|
1955
|
+
messageTypeField: 'eventType', // Instead of 'type'
|
|
1956
|
+
messageIdField: 'correlationId', // Instead of 'id'
|
|
1957
|
+
messageTimestampField: 'occurredAt', // Instead of 'timestamp'
|
|
1958
|
+
|
|
1959
|
+
handlers: new MessageHandlerConfigBuilder<CustomMessage>()
|
|
1960
|
+
.addConfig(CUSTOM_MESSAGE_SCHEMA, async (message) => {
|
|
1961
|
+
// Handler receives the full validated message
|
|
1962
|
+
// Access nested data via message.data
|
|
1963
|
+
console.log(`Order ${message.data.orderId}: $${message.data.amount}`)
|
|
1964
|
+
return { result: 'success' as const }
|
|
1965
|
+
})
|
|
1966
|
+
.build(),
|
|
1967
|
+
})
|
|
1968
|
+
}
|
|
1969
|
+
}
|
|
1970
|
+
```
|
|
1971
|
+
|
|
1972
|
+
**Configuration Options:**
|
|
1973
|
+
|
|
1974
|
+
- `messageTypeField` (required) - Field name containing the message type for routing
|
|
1975
|
+
- `messageIdField` (optional, default: `'id'`) - Field name containing the message ID
|
|
1976
|
+
- `messageTimestampField` (optional, default: `'timestamp'`) - Field name containing the timestamp
|
|
1977
|
+
|
|
1978
|
+
## FIFO Queues
|
|
1979
|
+
|
|
1980
|
+
FIFO (First-In-First-Out) queues provide message ordering and exactly-once processing.
|
|
1981
|
+
|
|
1982
|
+
### FIFO Queue Requirements
|
|
1983
|
+
|
|
1984
|
+
1. **Queue name must end with `.fifo`**
|
|
1985
|
+
```typescript
|
|
1986
|
+
QueueName: 'my-queue.fifo' // ✅ Valid
|
|
1987
|
+
QueueName: 'my-queue' // ❌ Invalid for FIFO
|
|
1988
|
+
```
|
|
1989
|
+
|
|
1990
|
+
2. **FifoQueue attribute must be 'true'**
|
|
1991
|
+
```typescript
|
|
1992
|
+
Attributes: {
|
|
1993
|
+
FifoQueue: 'true',
|
|
1994
|
+
}
|
|
1995
|
+
```
|
|
1996
|
+
|
|
1997
|
+
3. **MessageGroupId required for all messages**
|
|
1998
|
+
```typescript
|
|
1999
|
+
// Option 1: From message field
|
|
2000
|
+
messageGroupIdField: 'userId'
|
|
2001
|
+
|
|
2002
|
+
// Option 2: Default value
|
|
2003
|
+
defaultMessageGroupId: 'default-group'
|
|
2004
|
+
|
|
2005
|
+
// Option 3: Explicit in publish call
|
|
2006
|
+
await publisher.publish(message, {
|
|
2007
|
+
MessageGroupId: 'custom-group',
|
|
2008
|
+
})
|
|
2009
|
+
```
|
|
2010
|
+
|
|
2011
|
+
4. **DLQ must also be FIFO**
|
|
2012
|
+
```typescript
|
|
2013
|
+
deadLetterQueue: {
|
|
2014
|
+
creationConfig: {
|
|
2015
|
+
queue: {
|
|
2016
|
+
QueueName: 'my-queue-dlq.fifo', // Must be FIFO
|
|
2017
|
+
Attributes: {
|
|
2018
|
+
FifoQueue: 'true',
|
|
2019
|
+
},
|
|
2020
|
+
},
|
|
2021
|
+
},
|
|
2022
|
+
}
|
|
2023
|
+
```
|
|
2024
|
+
|
|
2025
|
+
### Message Ordering
|
|
2026
|
+
|
|
2027
|
+
**Ordering guarantees:**
|
|
2028
|
+
- ✅ Messages within the same MessageGroupId are delivered in order
|
|
2029
|
+
- ✅ Messages are processed exactly once (no duplicates)
|
|
2030
|
+
- ❌ No ordering guarantee across different MessageGroupIds
|
|
2031
|
+
|
|
2032
|
+
**Example:**
|
|
2033
|
+
|
|
2034
|
+
```typescript
|
|
2035
|
+
// All messages for user-123 are processed in order
|
|
2036
|
+
await publisher.publish({ id: '1', userId: 'user-123', action: 'create' })
|
|
2037
|
+
await publisher.publish({ id: '2', userId: 'user-123', action: 'update' })
|
|
2038
|
+
await publisher.publish({ id: '3', userId: 'user-123', action: 'delete' })
|
|
2039
|
+
|
|
2040
|
+
// Messages for user-456 can be processed in parallel
|
|
2041
|
+
await publisher.publish({ id: '4', userId: 'user-456', action: 'create' })
|
|
2042
|
+
```
|
|
2043
|
+
|
|
2044
|
+
**Processing order:**
|
|
2045
|
+
```
|
|
2046
|
+
Message 1 (user-123) → Message 2 (user-123) → Message 3 (user-123)
|
|
2047
|
+
Message 4 (user-456) can process in parallel with above
|
|
2048
|
+
```
|
|
2049
|
+
|
|
2050
|
+
### FIFO Retry Behavior
|
|
2051
|
+
|
|
2052
|
+
**IMPORTANT**: FIFO queues have special retry behavior to preserve message ordering.
|
|
2053
|
+
|
|
2054
|
+
When a handler returns `{ error: 'retryLater' }` for a FIFO queue:
|
|
2055
|
+
|
|
2056
|
+
1. **The consumer throws an error** instead of republishing the message
|
|
2057
|
+
2. **AWS SQS handles the retry** using its native retry mechanism
|
|
2058
|
+
3. **Message order is preserved** within the MessageGroupId
|
|
2059
|
+
4. The message returns to the **same position** in the queue after visibility timeout
|
|
2060
|
+
|
|
2061
|
+
**Why this matters:**
|
|
2062
|
+
|
|
2063
|
+
```typescript
|
|
2064
|
+
// ❌ Standard queue: Republishes message to back of queue (breaks order)
|
|
2065
|
+
// Message 1 → Message 2 → Message 3
|
|
2066
|
+
// If Message 1 retries: Message 2 → Message 3 → Message 1 (wrong!)
|
|
2067
|
+
|
|
2068
|
+
// ✅ FIFO queue: Throws error, AWS retries in place (preserves order)
|
|
2069
|
+
// Message 1 → Message 2 → Message 3
|
|
2070
|
+
// If Message 1 retries: Message 1 (retry) → Message 2 → Message 3 (correct!)
|
|
2071
|
+
```
|
|
2072
|
+
|
|
2073
|
+
**Configuration:**
|
|
2074
|
+
|
|
2075
|
+
```typescript
|
|
2076
|
+
class MyFifoConsumer extends AbstractSqsConsumer<MessageType, Context> {
|
|
2077
|
+
constructor(dependencies: SQSConsumerDependencies) {
|
|
2078
|
+
super(dependencies, {
|
|
2079
|
+
creationConfig: {
|
|
2080
|
+
queue: {
|
|
2081
|
+
QueueName: 'my-queue.fifo',
|
|
2082
|
+
Attributes: {
|
|
2083
|
+
FifoQueue: 'true',
|
|
2084
|
+
VisibilityTimeout: '30', // Retry delay
|
|
2085
|
+
},
|
|
2086
|
+
},
|
|
2087
|
+
},
|
|
2088
|
+
handlers: new MessageHandlerConfigBuilder()
|
|
2089
|
+
.addConfig(SCHEMA, async (message) => {
|
|
2090
|
+
// Return retryLater to trigger AWS SQS native retry
|
|
2091
|
+
if (shouldRetry) {
|
|
2092
|
+
return { error: 'retryLater' }
|
|
2093
|
+
}
|
|
2094
|
+
return { result: 'success' }
|
|
2095
|
+
})
|
|
2096
|
+
.build(),
|
|
2097
|
+
})
|
|
2098
|
+
}
|
|
2099
|
+
}
|
|
2100
|
+
```
|
|
2101
|
+
|
|
2102
|
+
**Barrier Sleep Mechanism (FIFO Queues):**
|
|
2103
|
+
|
|
2104
|
+
When using barriers with FIFO queues, the consumer implements a smart sleep-and-check mechanism to avoid unnecessary AWS retries:
|
|
2105
|
+
|
|
2106
|
+
1. **Handler or barrier returns `{ error: 'retryLater' }`**
|
|
2107
|
+
2. **Consumer enters sleep loop** instead of immediately throwing error
|
|
2108
|
+
3. **Periodically re-checks barrier** (every `barrierSleepCheckIntervalInMsecs`, default: 5 seconds)
|
|
2109
|
+
4. **If barrier passes**: Process message successfully and exit
|
|
2110
|
+
5. **If max retry duration exceeded** (`maxRetryDuration`, default: 4 days): Send message to DLQ
|
|
2111
|
+
6. **Visibility timeout extended** every 30 seconds during sleep to prevent message reclaim
|
|
2112
|
+
|
|
2113
|
+
**Configuration with barrier sleep:**
|
|
2114
|
+
|
|
2115
|
+
```typescript
|
|
2116
|
+
class MyFifoConsumer extends AbstractSqsConsumer<MessageType, Context> {
|
|
2117
|
+
constructor(dependencies: SQSConsumerDependencies) {
|
|
2118
|
+
super(dependencies, {
|
|
2119
|
+
creationConfig: {
|
|
2120
|
+
queue: {
|
|
2121
|
+
QueueName: 'my-queue.fifo',
|
|
2122
|
+
Attributes: {
|
|
2123
|
+
FifoQueue: 'true',
|
|
2124
|
+
VisibilityTimeout: '3600', // 1 hour - must be greater than heartbeatInterval
|
|
2125
|
+
},
|
|
2126
|
+
},
|
|
2127
|
+
},
|
|
2128
|
+
maxRetryDuration: 345600, // 4 days in seconds (default)
|
|
2129
|
+
barrierSleepCheckIntervalInMsecs: 5000, // 5 seconds (default)
|
|
2130
|
+
|
|
2131
|
+
// Consumer configuration - controls visibility timeout management
|
|
2132
|
+
consumerOverrides: {
|
|
2133
|
+
heartbeatInterval: 300, // Extends visibility every 5 minutes (default)
|
|
2134
|
+
// Must be less than VisibilityTimeout
|
|
2135
|
+
// Prevents AWS from reclaiming the message during long processing or barrier sleep
|
|
2136
|
+
},
|
|
2137
|
+
|
|
2138
|
+
handlers: new MessageHandlerConfigBuilder()
|
|
2139
|
+
.addConfig(
|
|
2140
|
+
SCHEMA,
|
|
2141
|
+
async (message) => {
|
|
2142
|
+
return { result: 'success' }
|
|
2143
|
+
},
|
|
2144
|
+
{
|
|
2145
|
+
preHandlerBarrier: async (message, context) => {
|
|
2146
|
+
// Check if prerequisite is met
|
|
2147
|
+
const prerequisiteMet = await checkPrerequisite(message)
|
|
2148
|
+
return { isPassing: prerequisiteMet }
|
|
2149
|
+
}
|
|
2150
|
+
}
|
|
2151
|
+
)
|
|
2152
|
+
.build(),
|
|
2153
|
+
})
|
|
2154
|
+
}
|
|
2155
|
+
}
|
|
2156
|
+
```
|
|
2157
|
+
|
|
2158
|
+
**Retry flow for FIFO queues (with barrier sleep):**
|
|
2159
|
+
1. Handler or barrier returns `{ error: 'retryLater' }`
|
|
2160
|
+
2. Consumer enters sleep loop for up to `maxRetryDuration` (default: 4 days)
|
|
2161
|
+
3. Every `barrierSleepCheckIntervalInMsecs` (default: 5 sec), consumer:
|
|
2162
|
+
- Re-checks barrier
|
|
2163
|
+
- Extends visibility timeout (every 30 seconds)
|
|
2164
|
+
4. If barrier passes: Process message and mark as consumed
|
|
2165
|
+
5. If `maxRetryDuration` exceeded: Send message to DLQ with error `"FIFO queue: Retry duration exceeded. Moving message to DLQ."`
|
|
2166
|
+
|
|
2167
|
+
**Retry flow for FIFO queues (without barrier - immediate retry):**
|
|
2168
|
+
1. Handler returns `{ error: 'retryLater' }` but no barrier configured
|
|
2169
|
+
2. Consumer immediately throws error: `"FIFO queue: Triggering AWS SQS retry to preserve message order"`
|
|
2170
|
+
3. AWS SQS makes message invisible for `VisibilityTimeout` duration
|
|
2171
|
+
4. After timeout, message becomes visible again **at the same position**
|
|
2172
|
+
5. Consumer processes it again
|
|
2173
|
+
|
|
2174
|
+
**Retry duration limit:**
|
|
2175
|
+
- The `maxRetryDuration` setting still applies (default: 4 days)
|
|
2176
|
+
- After exceeding retry duration, message moves to DLQ
|
|
2177
|
+
- Error message: `"FIFO queue: Retry duration exceeded. Moving message to DLQ"`
|
|
2178
|
+
|
|
2179
|
+
**Understanding VisibilityTimeout and heartbeatInterval:**
|
|
2180
|
+
|
|
2181
|
+
The visibility timeout mechanism prevents multiple consumers from processing the same message:
|
|
2182
|
+
|
|
2183
|
+
- **`VisibilityTimeout` (queue attribute)**: Initial time (in seconds) that a message is hidden from other consumers after being received. This is the starting timeout when a message is first picked up.
|
|
2184
|
+
|
|
2185
|
+
- **`heartbeatInterval` (consumerOverrides option)**: Interval (in seconds) at which the consumer automatically extends the visibility timeout using `ChangeMessageVisibilityCommand`. This keeps the message "locked" to the current consumer during long processing.
|
|
2186
|
+
|
|
2187
|
+
**How they work together:**
|
|
2188
|
+
|
|
2189
|
+
1. **Message received**: AWS SQS hides message for `VisibilityTimeout` seconds
|
|
2190
|
+
2. **During processing**: Consumer automatically extends timeout every `heartbeatInterval` seconds
|
|
2191
|
+
3. **If processing exceeds VisibilityTimeout without extension**: AWS makes message visible again (allowing retry)
|
|
2192
|
+
4. **Requirement**: `heartbeatInterval` < `VisibilityTimeout` (otherwise extension happens after message already became visible)
|
|
2193
|
+
|
|
2194
|
+
**For FIFO queues with barrier sleep:**
|
|
2195
|
+
- The barrier sleep mechanism can explicitly extend visibility timeout every 30 seconds
|
|
2196
|
+
- Extensions set visibility to 90 seconds **from now** (not cumulative) before each sleep chunk
|
|
2197
|
+
- Extension happens BEFORE sleeping (not after) to prevent first-check vulnerability
|
|
2198
|
+
- The 90-second value (3x the check interval) provides a 60-second safety buffer against timing drift
|
|
2199
|
+
- However, it smartly defers to `heartbeatInterval` when set to avoid redundant API calls:
|
|
2200
|
+
- If `heartbeatInterval` <= 30 seconds: rely on sqs-consumer's automatic extension (no explicit extension)
|
|
2201
|
+
- If `heartbeatInterval` > 30 seconds or not set: explicitly extend every 30 seconds during sleep
|
|
2202
|
+
- This optimization prevents unnecessary ChangeMessageVisibility API calls
|
|
2203
|
+
- Set `VisibilityTimeout` high enough for expected barrier wait time (e.g., 3600 seconds / 1 hour)
|
|
2204
|
+
|
|
2205
|
+
**Best practices:**
|
|
2206
|
+
- Set `VisibilityTimeout` high enough to accommodate the barrier sleep period (consider `maxRetryDuration`)
|
|
2207
|
+
- Use `heartbeatInterval` < `VisibilityTimeout` (default: 300 seconds / 5 minutes)
|
|
2208
|
+
- For long-running barriers: increase `VisibilityTimeout` (not `heartbeatInterval`)
|
|
2209
|
+
- Use `maxRetryDuration` to control total retry time for both FIFO and standard queues
|
|
2210
|
+
- Adjust `barrierSleepCheckIntervalInMsecs` based on how quickly prerequisites typically become available
|
|
2211
|
+
- For FIFO queues: barriers are rechecked periodically instead of republishing messages
|
|
2212
|
+
- For standard queues: messages are republished with exponential backoff delay
|
|
2213
|
+
|
|
2214
|
+
### Message Groups
|
|
2215
|
+
|
|
2216
|
+
Message groups enable parallel processing while maintaining order:
|
|
2217
|
+
|
|
2218
|
+
**Scenario: E-commerce orders**
|
|
2219
|
+
|
|
2220
|
+
```typescript
|
|
2221
|
+
// Each customer's orders are processed in order
|
|
2222
|
+
await publisher.publish({
|
|
2223
|
+
orderId: 'order-1',
|
|
2224
|
+
customerId: 'customer-A',
|
|
2225
|
+
action: 'created',
|
|
2226
|
+
}, {
|
|
2227
|
+
MessageGroupId: 'customer-A', // Group by customer
|
|
2228
|
+
})
|
|
2229
|
+
|
|
2230
|
+
await publisher.publish({
|
|
2231
|
+
orderId: 'order-2',
|
|
2232
|
+
customerId: 'customer-A',
|
|
2233
|
+
action: 'paid',
|
|
2234
|
+
}, {
|
|
2235
|
+
MessageGroupId: 'customer-A', // Same group - processed in order
|
|
2236
|
+
})
|
|
2237
|
+
|
|
2238
|
+
await publisher.publish({
|
|
2239
|
+
orderId: 'order-3',
|
|
2240
|
+
customerId: 'customer-B',
|
|
2241
|
+
action: 'created',
|
|
2242
|
+
}, {
|
|
2243
|
+
MessageGroupId: 'customer-B', // Different group - parallel processing
|
|
2244
|
+
})
|
|
2245
|
+
```
|
|
2246
|
+
|
|
2247
|
+
**Group Assignment:**
|
|
2248
|
+
- AWS SQS automatically assigns groups to consumers
|
|
2249
|
+
- Assignment is dynamic (groups can move between consumers)
|
|
2250
|
+
- Assignment is sticky (SQS prefers consistency)
|
|
2251
|
+
- You cannot control which consumer processes which group
|
|
2252
|
+
|
|
2253
|
+
**Parallel Processing:**
|
|
2254
|
+
|
|
2255
|
+
```typescript
|
|
2256
|
+
{
|
|
2257
|
+
concurrentConsumersAmount: 5, // 5 consumers can process 5 groups in parallel
|
|
2258
|
+
}
|
|
2259
|
+
```
|
|
2260
|
+
|
|
2261
|
+
**Best Practices:**
|
|
2262
|
+
|
|
2263
|
+
1. **Design groups for parallelism**
|
|
2264
|
+
```typescript
|
|
2265
|
+
// ✅ Good: Many groups = high parallelism
|
|
2266
|
+
MessageGroupId: `customer-${customerId}`
|
|
2267
|
+
|
|
2268
|
+
// ❌ Bad: One group = no parallelism
|
|
2269
|
+
MessageGroupId: 'all-customers'
|
|
2270
|
+
```
|
|
2271
|
+
|
|
2272
|
+
2. **Balance group sizes**
|
|
2273
|
+
- Avoid one group having 90% of messages
|
|
2274
|
+
- Aim for relatively balanced message distribution
|
|
2275
|
+
|
|
2276
|
+
3. **Size concurrent consumers appropriately**
|
|
2277
|
+
```typescript
|
|
2278
|
+
// If you have 10 active customer groups at peak
|
|
2279
|
+
concurrentConsumersAmount: 10
|
|
2280
|
+
```
|
|
2281
|
+
|
|
2282
|
+
### FIFO-Specific Configuration
|
|
2283
|
+
|
|
2284
|
+
```typescript
|
|
2285
|
+
// Publisher
|
|
2286
|
+
{
|
|
2287
|
+
fifoQueue: true,
|
|
2288
|
+
|
|
2289
|
+
// Choose one or more:
|
|
2290
|
+
messageGroupIdField: 'userId', // Use field from message
|
|
2291
|
+
defaultMessageGroupId: 'default', // Fallback value
|
|
2292
|
+
// or provide in publish call
|
|
2293
|
+
}
|
|
2294
|
+
|
|
2295
|
+
// Consumer
|
|
2296
|
+
{
|
|
2297
|
+
fifoQueue: true,
|
|
2298
|
+
concurrentConsumersAmount: 3, // Process 3 groups in parallel
|
|
2299
|
+
|
|
2300
|
+
// Note: Retry behavior is different for FIFO
|
|
2301
|
+
// - No DelaySeconds support (AWS limitation)
|
|
2302
|
+
// - Messages retry immediately
|
|
2303
|
+
// - Order is preserved
|
|
2304
|
+
}
|
|
2305
|
+
|
|
2306
|
+
// Queue Configuration
|
|
2307
|
+
{
|
|
2308
|
+
creationConfig: {
|
|
2309
|
+
queue: {
|
|
2310
|
+
QueueName: 'my-queue.fifo',
|
|
2311
|
+
Attributes: {
|
|
2312
|
+
FifoQueue: 'true',
|
|
2313
|
+
|
|
2314
|
+
// Optional: Automatic deduplication based on message body
|
|
2315
|
+
ContentBasedDeduplication: 'false', // or 'true'
|
|
2316
|
+
|
|
2317
|
+
// Optional: Deduplication scope
|
|
2318
|
+
DeduplicationScope: 'queue', // or 'messageGroup'
|
|
2319
|
+
|
|
2320
|
+
// Optional: Throughput limit
|
|
2321
|
+
FifoThroughputLimit: 'perQueue', // or 'perMessageGroupId'
|
|
2322
|
+
},
|
|
2323
|
+
},
|
|
2324
|
+
},
|
|
2325
|
+
}
|
|
2326
|
+
```
|
|
2327
|
+
|
|
2328
|
+
**FIFO vs Standard Queues:**
|
|
2329
|
+
|
|
2330
|
+
| Feature | Standard Queue | FIFO Queue |
|
|
2331
|
+
|---------|---------------|------------|
|
|
2332
|
+
| Ordering | Best-effort | Guaranteed within group |
|
|
2333
|
+
| Delivery | At-least-once | Exactly-once |
|
|
2334
|
+
| Throughput | Unlimited | 3,000 msg/s (per queue) or 300 msg/s (per group) |
|
|
2335
|
+
| Retry Delay | Exponential backoff | Immediate (no delay) |
|
|
2336
|
+
| Message Groups | N/A | Required (MessageGroupId) |
|
|
2337
|
+
| Naming | Any name | Must end with `.fifo` |
|
|
2338
|
+
|
|
2339
|
+
## Policy Configuration
|
|
2340
|
+
|
|
2341
|
+
Configure queue access policies for SNS topic subscriptions or cross-account access:
|
|
2342
|
+
|
|
2343
|
+
### Allow SNS Topic to Send Messages
|
|
2344
|
+
|
|
2345
|
+
```typescript
|
|
2346
|
+
import { SQS_RESOURCE_CURRENT_QUEUE } from '@message-queue-toolkit/sqs'
|
|
2347
|
+
|
|
2348
|
+
{
|
|
2349
|
+
creationConfig: {
|
|
2350
|
+
queue: { QueueName: 'my-queue' },
|
|
2351
|
+
policyConfig: {
|
|
2352
|
+
resource: SQS_RESOURCE_CURRENT_QUEUE, // Use queue's ARN
|
|
2353
|
+
statements: {
|
|
2354
|
+
Effect: 'Allow',
|
|
2355
|
+
Principal: '*',
|
|
2356
|
+
Action: ['sqs:SendMessage'],
|
|
2357
|
+
},
|
|
2358
|
+
},
|
|
2359
|
+
},
|
|
2360
|
+
}
|
|
2361
|
+
```
|
|
2362
|
+
|
|
2363
|
+
### Allow Specific SNS Topics
|
|
2364
|
+
|
|
2365
|
+
```typescript
|
|
2366
|
+
{
|
|
2367
|
+
creationConfig: {
|
|
2368
|
+
queue: { QueueName: 'my-queue' },
|
|
2369
|
+
topicArnsWithPublishPermissionsPrefix: 'arn:aws:sns:us-east-1:123456789012:my-topic',
|
|
2370
|
+
},
|
|
2371
|
+
}
|
|
2372
|
+
```
|
|
2373
|
+
|
|
2374
|
+
### Custom Policy
|
|
2375
|
+
|
|
2376
|
+
```typescript
|
|
2377
|
+
{
|
|
2378
|
+
creationConfig: {
|
|
2379
|
+
queue: { QueueName: 'my-queue' },
|
|
2380
|
+
policyConfig: {
|
|
2381
|
+
resource: 'arn:aws:sqs:us-east-1:123456789012:my-queue',
|
|
2382
|
+
statements: [
|
|
2383
|
+
{
|
|
2384
|
+
Effect: 'Allow',
|
|
2385
|
+
Principal: { AWS: 'arn:aws:iam::111111111111:root' },
|
|
2386
|
+
Action: ['sqs:SendMessage', 'sqs:ReceiveMessage'],
|
|
2387
|
+
},
|
|
2388
|
+
{
|
|
2389
|
+
Effect: 'Deny',
|
|
2390
|
+
Principal: '*',
|
|
2391
|
+
Action: ['sqs:DeleteQueue'],
|
|
2392
|
+
},
|
|
2393
|
+
],
|
|
2394
|
+
},
|
|
2395
|
+
},
|
|
2396
|
+
}
|
|
2397
|
+
```
|
|
2398
|
+
|
|
2399
|
+
## Testing
|
|
2400
|
+
|
|
2401
|
+
The library is designed to be testable:
|
|
2402
|
+
|
|
2403
|
+
### Integration Tests
|
|
2404
|
+
|
|
2405
|
+
```typescript
|
|
2406
|
+
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
|
2407
|
+
import { SQSClient } from '@aws-sdk/client-sqs'
|
|
2408
|
+
|
|
2409
|
+
describe('UserEventsConsumer', () => {
|
|
2410
|
+
let sqsClient: SQSClient
|
|
2411
|
+
let publisher: UserEventsPublisher
|
|
2412
|
+
let consumer: UserEventsConsumer
|
|
2413
|
+
|
|
2414
|
+
beforeEach(async () => {
|
|
2415
|
+
sqsClient = new SQSClient({
|
|
2416
|
+
endpoint: 'http://localhost:4566', // LocalStack
|
|
2417
|
+
region: 'us-east-1',
|
|
2418
|
+
})
|
|
2419
|
+
|
|
2420
|
+
publisher = new UserEventsPublisher(sqsClient)
|
|
2421
|
+
consumer = new UserEventsConsumer(sqsClient, userService)
|
|
2422
|
+
|
|
2423
|
+
await publisher.init()
|
|
2424
|
+
await consumer.start()
|
|
2425
|
+
})
|
|
2426
|
+
|
|
2427
|
+
afterEach(async () => {
|
|
2428
|
+
await consumer.close()
|
|
2429
|
+
await publisher.close()
|
|
2430
|
+
})
|
|
2431
|
+
|
|
2432
|
+
it('processes user.created message', async () => {
|
|
2433
|
+
await publisher.publish({
|
|
2434
|
+
id: '123',
|
|
2435
|
+
messageType: 'user.created',
|
|
2436
|
+
userId: 'user-456',
|
|
2437
|
+
email: 'test@example.com',
|
|
2438
|
+
})
|
|
2439
|
+
|
|
2440
|
+
// Wait for message to be processed
|
|
2441
|
+
await consumer.handlerSpy.waitForMessageWithId('123', 'consumed')
|
|
2442
|
+
|
|
2443
|
+
// Verify side effects
|
|
2444
|
+
expect(userService.createUser).toHaveBeenCalledWith('user-456', 'test@example.com')
|
|
2445
|
+
})
|
|
2446
|
+
|
|
2447
|
+
it('retries failed messages', async () => {
|
|
2448
|
+
let attempts = 0
|
|
2449
|
+
userService.createUser.mockImplementation(() => {
|
|
2450
|
+
attempts++
|
|
2451
|
+
if (attempts < 3) throw new Error('Temporary failure')
|
|
2452
|
+
return Promise.resolve()
|
|
2453
|
+
})
|
|
2454
|
+
|
|
2455
|
+
await publisher.publish({
|
|
2456
|
+
id: '124',
|
|
2457
|
+
messageType: 'user.created',
|
|
2458
|
+
userId: 'user-789',
|
|
2459
|
+
email: 'test2@example.com',
|
|
2460
|
+
})
|
|
2461
|
+
|
|
2462
|
+
await consumer.handlerSpy.waitForMessageWithId('124', 'consumed')
|
|
2463
|
+
|
|
2464
|
+
expect(attempts).toBe(3)
|
|
2465
|
+
})
|
|
2466
|
+
})
|
|
2467
|
+
```
|
|
2468
|
+
|
|
2469
|
+
### Unit Tests with Handler Spies
|
|
2470
|
+
|
|
2471
|
+
```typescript
|
|
2472
|
+
it('publishes message', async () => {
|
|
2473
|
+
await publisher.publish({
|
|
2474
|
+
id: '123',
|
|
2475
|
+
messageType: 'user.created',
|
|
2476
|
+
userId: 'user-456',
|
|
2477
|
+
email: 'test@example.com',
|
|
2478
|
+
})
|
|
2479
|
+
|
|
2480
|
+
const publishedMessage = await publisher.handlerSpy.waitForMessageWithId('123', 'published')
|
|
2481
|
+
|
|
2482
|
+
expect(publishedMessage).toMatchObject({
|
|
2483
|
+
id: '123',
|
|
2484
|
+
userId: 'user-456',
|
|
2485
|
+
})
|
|
2486
|
+
})
|
|
2487
|
+
```
|
|
2488
|
+
|
|
2489
|
+
## API Reference
|
|
2490
|
+
|
|
2491
|
+
### AbstractSqsPublisher
|
|
2492
|
+
|
|
2493
|
+
```typescript
|
|
2494
|
+
class AbstractSqsPublisher<MessagePayloadType extends object> {
|
|
2495
|
+
constructor(
|
|
2496
|
+
dependencies: SQSDependencies,
|
|
2497
|
+
options: SQSPublisherOptions<MessagePayloadType>
|
|
2498
|
+
)
|
|
2499
|
+
|
|
2500
|
+
async init(): Promise<void>
|
|
2501
|
+
async close(): Promise<void>
|
|
2502
|
+
|
|
2503
|
+
async publish(
|
|
2504
|
+
message: MessagePayloadType,
|
|
2505
|
+
options?: SQSMessageOptions
|
|
2506
|
+
): Promise<void>
|
|
2507
|
+
|
|
2508
|
+
readonly handlerSpy: HandlerSpy<MessagePayloadType>
|
|
2509
|
+
}
|
|
2510
|
+
```
|
|
2511
|
+
|
|
2512
|
+
### AbstractSqsConsumer
|
|
2513
|
+
|
|
2514
|
+
```typescript
|
|
2515
|
+
class AbstractSqsConsumer<
|
|
2516
|
+
MessagePayloadType extends object,
|
|
2517
|
+
ExecutionContext,
|
|
2518
|
+
PrehandlerOutput = undefined
|
|
2519
|
+
> {
|
|
2520
|
+
constructor(
|
|
2521
|
+
dependencies: SQSConsumerDependencies,
|
|
2522
|
+
options: SQSConsumerOptions<MessagePayloadType, ExecutionContext, PrehandlerOutput>,
|
|
2523
|
+
executionContext: ExecutionContext
|
|
2524
|
+
)
|
|
2525
|
+
|
|
2526
|
+
async init(): Promise<void>
|
|
2527
|
+
async start(): Promise<void>
|
|
2528
|
+
async close(abort?: boolean): Promise<void>
|
|
2529
|
+
|
|
2530
|
+
readonly handlerSpy: HandlerSpy<MessagePayloadType>
|
|
2531
|
+
}
|
|
2532
|
+
```
|
|
2533
|
+
|
|
2534
|
+
### Types
|
|
2535
|
+
|
|
2536
|
+
```typescript
|
|
2537
|
+
// Message options for publishing
|
|
2538
|
+
type SQSMessageOptions = {
|
|
2539
|
+
MessageGroupId?: string // Required for FIFO queues
|
|
2540
|
+
MessageDeduplicationId?: string // Optional for FIFO queues
|
|
2541
|
+
}
|
|
2542
|
+
|
|
2543
|
+
// Handler result
|
|
2544
|
+
type HandlerResult = Either<'retryLater', 'success'>
|
|
2545
|
+
|
|
2546
|
+
// Dependencies
|
|
2547
|
+
type SQSDependencies = {
|
|
2548
|
+
sqsClient: SQSClient
|
|
2549
|
+
logger: Logger
|
|
2550
|
+
errorReporter: ErrorReporter
|
|
2551
|
+
messageMetricsManager?: MessageMetricsManager
|
|
2552
|
+
}
|
|
2553
|
+
|
|
2554
|
+
// Consumer dependencies (extends SQSDependencies)
|
|
2555
|
+
type SQSConsumerDependencies = SQSDependencies & {
|
|
2556
|
+
consumerErrorResolver: ErrorResolver
|
|
2557
|
+
transactionObservabilityManager: TransactionObservabilityManager
|
|
2558
|
+
}
|
|
2559
|
+
```
|
|
2560
|
+
|
|
2561
|
+
### Utility Functions
|
|
2562
|
+
|
|
2563
|
+
```typescript
|
|
2564
|
+
// Queue validation
|
|
2565
|
+
function isFifoQueueName(queueName: string): boolean
|
|
2566
|
+
function validateFifoQueueName(queueName: string, isFifoQueue: boolean): void
|
|
2567
|
+
function validateFifoQueueConfiguration(
|
|
2568
|
+
queueName: string,
|
|
2569
|
+
attributes?: Record<string, string>,
|
|
2570
|
+
isFifoQueue?: boolean
|
|
2571
|
+
): void
|
|
2572
|
+
|
|
2573
|
+
// Queue operations
|
|
2574
|
+
async function assertQueue(
|
|
2575
|
+
sqsClient: SQSClient,
|
|
2576
|
+
queueConfig: CreateQueueCommandInput,
|
|
2577
|
+
extraParams?: ExtraSQSCreationParams,
|
|
2578
|
+
isFifoQueue?: boolean
|
|
2579
|
+
): Promise<{ queueUrl: string; queueArn: string; queueName: string }>
|
|
2580
|
+
|
|
2581
|
+
async function deleteQueue(
|
|
2582
|
+
client: SQSClient,
|
|
2583
|
+
queueName: string,
|
|
2584
|
+
waitForConfirmation?: boolean
|
|
2585
|
+
): Promise<void>
|
|
2586
|
+
|
|
2587
|
+
async function getQueueUrl(
|
|
2588
|
+
sqsClient: SQSClient,
|
|
2589
|
+
queueName: string
|
|
2590
|
+
): Promise<Either<'not_found', string>>
|
|
2591
|
+
|
|
2592
|
+
async function getQueueAttributes(
|
|
2593
|
+
sqsClient: SQSClient,
|
|
2594
|
+
queueUrl: string,
|
|
2595
|
+
attributeNames?: QueueAttributeName[]
|
|
2596
|
+
): Promise<Either<'not_found', { attributes?: Record<string, string> }>>
|
|
2597
|
+
|
|
2598
|
+
// Message size calculation
|
|
2599
|
+
function calculateOutgoingMessageSize(message: unknown): number
|
|
2600
|
+
```
|
|
2601
|
+
|
|
2602
|
+
## License
|
|
2603
|
+
|
|
2604
|
+
MIT
|
|
2605
|
+
|
|
2606
|
+
## Contributing
|
|
2607
|
+
|
|
2608
|
+
Contributions are welcome! Please see the main repository for guidelines.
|
|
2609
|
+
|
|
2610
|
+
## Links
|
|
2611
|
+
|
|
2612
|
+
- [Main Repository](https://github.com/kibertoad/message-queue-toolkit)
|
|
2613
|
+
- [Core Package](https://www.npmjs.com/package/@message-queue-toolkit/core)
|
|
2614
|
+
- [AWS SQS Documentation](https://docs.aws.amazon.com/sqs/)
|
|
2615
|
+
- [FIFO Queue Documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html)
|