@message-queue-toolkit/sns 23.1.4 → 24.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1206 -0
- package/dist/errors/SnsConsumerErrorResolver.js +2 -0
- package/dist/errors/SnsConsumerErrorResolver.js.map +1 -1
- package/dist/index.d.ts +1 -1
- package/dist/index.js +1 -1
- package/dist/index.js.map +1 -1
- package/dist/sns/AbstractSnsPublisher.d.ts +43 -1
- package/dist/sns/AbstractSnsPublisher.js +60 -2
- package/dist/sns/AbstractSnsPublisher.js.map +1 -1
- package/dist/utils/snsUtils.d.ts +8 -0
- package/dist/utils/snsUtils.js +18 -0
- package/dist/utils/snsUtils.js.map +1 -1
- package/package.json +10 -10
package/README.md
ADDED
|
@@ -0,0 +1,1206 @@
|
|
|
1
|
+
# @message-queue-toolkit/sns
|
|
2
|
+
|
|
3
|
+
AWS SNS (Simple Notification Service) implementation for the message-queue-toolkit. Provides a robust, type-safe abstraction for publishing messages to SNS topics and consuming them via SQS queue subscriptions, with support for both standard and FIFO topics.
|
|
4
|
+
|
|
5
|
+
## Table of Contents
|
|
6
|
+
|
|
7
|
+
- [Installation](#installation)
|
|
8
|
+
- [Features](#features)
|
|
9
|
+
- [Core Concepts](#core-concepts)
|
|
10
|
+
- [Quick Start](#quick-start)
|
|
11
|
+
- [Standard Topic Publisher](#standard-topic-publisher)
|
|
12
|
+
- [SNS-to-SQS Consumer](#sns-to-sqs-consumer)
|
|
13
|
+
- [FIFO Topic Publisher](#fifo-topic-publisher)
|
|
14
|
+
- [FIFO Topic Consumer](#fifo-topic-consumer)
|
|
15
|
+
- [Configuration](#configuration)
|
|
16
|
+
- [Topic Creation](#topic-creation)
|
|
17
|
+
- [Topic Locator](#topic-locator)
|
|
18
|
+
- [Publisher Options](#publisher-options)
|
|
19
|
+
- [Consumer Options](#consumer-options)
|
|
20
|
+
- [SNS-Specific Features](#sns-specific-features)
|
|
21
|
+
- [Topic Subscriptions](#topic-subscriptions)
|
|
22
|
+
- [Fan-Out Pattern](#fan-out-pattern)
|
|
23
|
+
- [Message Filtering](#message-filtering)
|
|
24
|
+
- [Cross-Account Publishing](#cross-account-publishing)
|
|
25
|
+
- [Advanced Features](#advanced-features)
|
|
26
|
+
- [FIFO Topics](#fifo-topics)
|
|
27
|
+
- [FIFO Topic Requirements](#fifo-topic-requirements)
|
|
28
|
+
- [Message Groups](#message-groups)
|
|
29
|
+
- [FIFO-Specific Configuration](#fifo-specific-configuration)
|
|
30
|
+
- [Testing](#testing)
|
|
31
|
+
- [API Reference](#api-reference)
|
|
32
|
+
|
|
33
|
+
## Installation
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
npm install @message-queue-toolkit/sns @message-queue-toolkit/sqs @message-queue-toolkit/core
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
**Peer Dependencies:**
|
|
40
|
+
- `@aws-sdk/client-sns` - AWS SDK for SNS
|
|
41
|
+
- `@aws-sdk/client-sqs` - AWS SDK for SQS (required for consumers)
|
|
42
|
+
- `@aws-sdk/client-sts` - AWS SDK for STS (for ARN resolution)
|
|
43
|
+
- `zod` - Schema validation
|
|
44
|
+
|
|
45
|
+
## Features
|
|
46
|
+
|
|
47
|
+
- ✅ **Type-safe message handling** with Zod schema validation
|
|
48
|
+
- ✅ **Standard and FIFO topic support**
|
|
49
|
+
- ✅ **Automatic topic and subscription creation**
|
|
50
|
+
- ✅ **Fan-out pattern** - publish once, consume from multiple queues
|
|
51
|
+
- ✅ **Message filtering** - subscribe to specific message types
|
|
52
|
+
- ✅ **Message deduplication** (publisher and consumer level)
|
|
53
|
+
- ✅ **Payload offloading** for large messages (S3 integration)
|
|
54
|
+
- ✅ **Automatic retry logic** with exponential backoff (inherited from SQS consumer)
|
|
55
|
+
- ✅ **Dead Letter Queue (DLQ)** support
|
|
56
|
+
- ✅ **Handler spies** for testing
|
|
57
|
+
- ✅ **Pre-handlers and barriers** for complex message processing
|
|
58
|
+
- ✅ **Cross-account and cross-region publishing**
|
|
59
|
+
|
|
60
|
+
## Core Concepts
|
|
61
|
+
|
|
62
|
+
### Publishers
|
|
63
|
+
|
|
64
|
+
SNS publishers send messages to **topics** (not queues). A topic is a logical access point that acts as a communication channel. Publishers are responsible for:
|
|
65
|
+
- Message validation against Zod schemas
|
|
66
|
+
- Automatic serialization
|
|
67
|
+
- Optional deduplication (preventing duplicate sends)
|
|
68
|
+
- Optional payload offloading (for messages > 256KB)
|
|
69
|
+
- FIFO-specific concerns (MessageGroupId, MessageDeduplicationId)
|
|
70
|
+
|
|
71
|
+
### Consumers
|
|
72
|
+
|
|
73
|
+
SNS consumers subscribe to topics via **SQS queues** (SNS-to-SQS pattern). This provides:
|
|
74
|
+
- **Decoupling** - topics don't know about subscribers
|
|
75
|
+
- **Fan-out** - one message published to multiple queues
|
|
76
|
+
- **Persistence** - messages queued until processed
|
|
77
|
+
- **Retry logic** - inherited from SQS consumer capabilities
|
|
78
|
+
- **Message filtering** - only receive relevant messages
|
|
79
|
+
|
|
80
|
+
Consumers are actually `AbstractSnsSqsConsumer` which extends `AbstractSqsConsumer` from the SQS package, inheriting all SQS consumer capabilities.
|
|
81
|
+
|
|
82
|
+
### Message Flow
|
|
83
|
+
|
|
84
|
+
```text
|
|
85
|
+
Publisher → SNS Topic → [Subscriptions] → SQS Queues → Consumers
|
|
86
|
+
↓
|
|
87
|
+
(optional filtering)
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
**Key Difference from SQS:**
|
|
91
|
+
- **SQS**: Direct queue-to-queue communication (1:1)
|
|
92
|
+
- **SNS**: Pub/Sub pattern with fan-out (1:N)
|
|
93
|
+
|
|
94
|
+
### Message Schemas
|
|
95
|
+
|
|
96
|
+
Messages use the same schema requirements as SQS. Each message must have:
|
|
97
|
+
- A unique message type field (discriminator for routing) - configurable via `messageTypeField` (required)
|
|
98
|
+
- A message ID field (for tracking and deduplication) - configurable via `messageIdField` (default: `'id'`)
|
|
99
|
+
- A timestamp field (added automatically if missing) - configurable via `messageTimestampField` (default: `'timestamp'`)
|
|
100
|
+
|
|
101
|
+
See the [SQS README - Message Schemas section](../sqs/README.md#message-schemas) for full details.
|
|
102
|
+
|
|
103
|
+
## Quick Start
|
|
104
|
+
|
|
105
|
+
### Standard Topic Publisher
|
|
106
|
+
|
|
107
|
+
```typescript
|
|
108
|
+
import { AbstractSnsPublisher } from '@message-queue-toolkit/sns'
|
|
109
|
+
import { SNSClient } from '@aws-sdk/client-sns'
|
|
110
|
+
import { STSClient } from '@aws-sdk/client-sts'
|
|
111
|
+
import z from 'zod'
|
|
112
|
+
|
|
113
|
+
// Define your message schemas
|
|
114
|
+
const UserCreatedSchema = z.object({
|
|
115
|
+
id: z.string(),
|
|
116
|
+
messageType: z.literal('user.created'),
|
|
117
|
+
userId: z.string(),
|
|
118
|
+
email: z.string().email(),
|
|
119
|
+
timestamp: z.string().optional(),
|
|
120
|
+
})
|
|
121
|
+
|
|
122
|
+
const UserUpdatedSchema = z.object({
|
|
123
|
+
id: z.string(),
|
|
124
|
+
messageType: z.literal('user.updated'),
|
|
125
|
+
userId: z.string(),
|
|
126
|
+
changes: z.record(z.unknown()),
|
|
127
|
+
timestamp: z.string().optional(),
|
|
128
|
+
})
|
|
129
|
+
|
|
130
|
+
type UserCreated = z.infer<typeof UserCreatedSchema>
|
|
131
|
+
type UserUpdated = z.infer<typeof UserUpdatedSchema>
|
|
132
|
+
type SupportedMessages = UserCreated | UserUpdated
|
|
133
|
+
|
|
134
|
+
// Create your publisher class
|
|
135
|
+
class UserEventsPublisher extends AbstractSnsPublisher<SupportedMessages> {
|
|
136
|
+
constructor(snsClient: SNSClient, stsClient: STSClient) {
|
|
137
|
+
super(
|
|
138
|
+
{
|
|
139
|
+
snsClient,
|
|
140
|
+
stsClient,
|
|
141
|
+
logger: console,
|
|
142
|
+
errorReporter: { report: (error) => console.error(error) },
|
|
143
|
+
},
|
|
144
|
+
{
|
|
145
|
+
messageSchemas: [UserCreatedSchema, UserUpdatedSchema],
|
|
146
|
+
messageTypeField: 'messageType',
|
|
147
|
+
creationConfig: {
|
|
148
|
+
topic: {
|
|
149
|
+
Name: 'user-events-topic',
|
|
150
|
+
},
|
|
151
|
+
},
|
|
152
|
+
deletionConfig: {
|
|
153
|
+
deleteIfExists: false,
|
|
154
|
+
},
|
|
155
|
+
}
|
|
156
|
+
)
|
|
157
|
+
}
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
// Use the publisher
|
|
161
|
+
const snsClient = new SNSClient({ region: 'us-east-1' })
|
|
162
|
+
const stsClient = new STSClient({ region: 'us-east-1' })
|
|
163
|
+
const publisher = new UserEventsPublisher(snsClient, stsClient)
|
|
164
|
+
|
|
165
|
+
await publisher.init()
|
|
166
|
+
|
|
167
|
+
// Publish to the topic - all subscribers will receive this message
|
|
168
|
+
await publisher.publish({
|
|
169
|
+
id: '123',
|
|
170
|
+
messageType: 'user.created',
|
|
171
|
+
userId: 'user-456',
|
|
172
|
+
email: 'user@example.com',
|
|
173
|
+
})
|
|
174
|
+
|
|
175
|
+
await publisher.close()
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
### SNS-to-SQS Consumer
|
|
179
|
+
|
|
180
|
+
Consumers subscribe to SNS topics via SQS queues:
|
|
181
|
+
|
|
182
|
+
```typescript
|
|
183
|
+
import { AbstractSnsSqsConsumer } from '@message-queue-toolkit/sns'
|
|
184
|
+
import { MessageHandlerConfigBuilder } from '@message-queue-toolkit/core'
|
|
185
|
+
import type { Either } from '@lokalise/node-core'
|
|
186
|
+
|
|
187
|
+
type ExecutionContext = {
|
|
188
|
+
userService: UserService
|
|
189
|
+
}
|
|
190
|
+
|
|
191
|
+
class UserEventsConsumer extends AbstractSnsSqsConsumer<
|
|
192
|
+
SupportedMessages,
|
|
193
|
+
ExecutionContext
|
|
194
|
+
> {
|
|
195
|
+
constructor(
|
|
196
|
+
snsClient: SNSClient,
|
|
197
|
+
sqsClient: SQSClient,
|
|
198
|
+
stsClient: STSClient,
|
|
199
|
+
userService: UserService
|
|
200
|
+
) {
|
|
201
|
+
super(
|
|
202
|
+
{
|
|
203
|
+
snsClient,
|
|
204
|
+
sqsClient,
|
|
205
|
+
stsClient,
|
|
206
|
+
logger: console,
|
|
207
|
+
errorReporter: { report: (error) => console.error(error) },
|
|
208
|
+
consumerErrorResolver: {
|
|
209
|
+
resolveError: () => ({ resolve: 'retryLater' as const }),
|
|
210
|
+
},
|
|
211
|
+
transactionObservabilityManager: {
|
|
212
|
+
start: () => {},
|
|
213
|
+
stop: () => {},
|
|
214
|
+
},
|
|
215
|
+
},
|
|
216
|
+
{
|
|
217
|
+
messageTypeField: 'messageType',
|
|
218
|
+
handlers: new MessageHandlerConfigBuilder<SupportedMessages, ExecutionContext>()
|
|
219
|
+
.addConfig(
|
|
220
|
+
UserCreatedSchema,
|
|
221
|
+
async (message, context): Promise<Either<'retryLater', 'success'>> => {
|
|
222
|
+
await context.userService.createUser(message.userId, message.email)
|
|
223
|
+
return { result: 'success' }
|
|
224
|
+
}
|
|
225
|
+
)
|
|
226
|
+
.addConfig(
|
|
227
|
+
UserUpdatedSchema,
|
|
228
|
+
async (message, context): Promise<Either<'retryLater', 'success'>> => {
|
|
229
|
+
await context.userService.updateUser(message.userId, message.changes)
|
|
230
|
+
return { result: 'success' }
|
|
231
|
+
}
|
|
232
|
+
)
|
|
233
|
+
.build(),
|
|
234
|
+
|
|
235
|
+
// Queue configuration (SQS queue that subscribes to SNS topic)
|
|
236
|
+
creationConfig: {
|
|
237
|
+
queue: {
|
|
238
|
+
QueueName: 'user-events-consumer-queue',
|
|
239
|
+
},
|
|
240
|
+
topic: {
|
|
241
|
+
Name: 'user-events-topic', // Must match publisher's topic name
|
|
242
|
+
},
|
|
243
|
+
},
|
|
244
|
+
|
|
245
|
+
// Subscription configuration
|
|
246
|
+
subscriptionConfig: {
|
|
247
|
+
updateAttributesIfExists: false,
|
|
248
|
+
},
|
|
249
|
+
|
|
250
|
+
deletionConfig: {
|
|
251
|
+
deleteIfExists: false,
|
|
252
|
+
},
|
|
253
|
+
},
|
|
254
|
+
{ userService } // Execution context
|
|
255
|
+
)
|
|
256
|
+
}
|
|
257
|
+
}
|
|
258
|
+
|
|
259
|
+
// Use the consumer
|
|
260
|
+
const consumer = new UserEventsConsumer(snsClient, sqsClient, stsClient, userService)
|
|
261
|
+
await consumer.start() // Creates queue, subscribes to topic, starts consuming
|
|
262
|
+
|
|
263
|
+
// Later, to stop
|
|
264
|
+
await consumer.close()
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
**What happens during `start()`:**
|
|
268
|
+
1. Creates SNS topic (if using `creationConfig`)
|
|
269
|
+
2. Creates SQS queue (if using `creationConfig`)
|
|
270
|
+
3. Subscribes queue to topic
|
|
271
|
+
4. Configures queue permissions to allow SNS to publish
|
|
272
|
+
5. Starts consuming messages from the queue
|
|
273
|
+
|
|
274
|
+
### FIFO Topic Publisher
|
|
275
|
+
|
|
276
|
+
FIFO topics guarantee message ordering and exactly-once delivery:
|
|
277
|
+
|
|
278
|
+
```typescript
|
|
279
|
+
class UserEventsFifoPublisher extends AbstractSnsPublisher<SupportedMessages> {
|
|
280
|
+
constructor(snsClient: SNSClient, stsClient: STSClient) {
|
|
281
|
+
super(
|
|
282
|
+
{
|
|
283
|
+
snsClient,
|
|
284
|
+
stsClient,
|
|
285
|
+
logger: console,
|
|
286
|
+
errorReporter: { report: (error) => console.error(error) },
|
|
287
|
+
},
|
|
288
|
+
{
|
|
289
|
+
messageSchemas: [UserCreatedSchema, UserUpdatedSchema],
|
|
290
|
+
messageTypeField: 'messageType',
|
|
291
|
+
fifoTopic: true, // Enable FIFO mode
|
|
292
|
+
|
|
293
|
+
// Option 1: Use a field from the message as MessageGroupId
|
|
294
|
+
messageGroupIdField: 'userId',
|
|
295
|
+
|
|
296
|
+
// Option 2: Use a default MessageGroupId for all messages
|
|
297
|
+
// defaultMessageGroupId: 'user-events',
|
|
298
|
+
|
|
299
|
+
creationConfig: {
|
|
300
|
+
topic: {
|
|
301
|
+
Name: 'user-events-topic.fifo', // Must end with .fifo
|
|
302
|
+
Attributes: {
|
|
303
|
+
FifoTopic: 'true',
|
|
304
|
+
ContentBasedDeduplication: 'false', // or 'true' for automatic deduplication
|
|
305
|
+
},
|
|
306
|
+
},
|
|
307
|
+
},
|
|
308
|
+
}
|
|
309
|
+
)
|
|
310
|
+
}
|
|
311
|
+
}
|
|
312
|
+
|
|
313
|
+
// Publishing to FIFO topic
|
|
314
|
+
const fifoPublisher = new UserEventsFifoPublisher(snsClient, stsClient)
|
|
315
|
+
await fifoPublisher.init()
|
|
316
|
+
|
|
317
|
+
// Messages with the same userId will be processed in order
|
|
318
|
+
await fifoPublisher.publish({
|
|
319
|
+
id: '123',
|
|
320
|
+
messageType: 'user.created',
|
|
321
|
+
userId: 'user-456', // Used as MessageGroupId
|
|
322
|
+
email: 'user@example.com',
|
|
323
|
+
})
|
|
324
|
+
|
|
325
|
+
await fifoPublisher.publish({
|
|
326
|
+
id: '124',
|
|
327
|
+
messageType: 'user.updated',
|
|
328
|
+
userId: 'user-456', // Same group - processed after the first message
|
|
329
|
+
changes: { name: 'John Doe' },
|
|
330
|
+
})
|
|
331
|
+
|
|
332
|
+
// You can also explicitly provide MessageGroupId
|
|
333
|
+
await fifoPublisher.publish(
|
|
334
|
+
{
|
|
335
|
+
id: '125',
|
|
336
|
+
messageType: 'user.created',
|
|
337
|
+
userId: 'user-789',
|
|
338
|
+
email: 'other@example.com',
|
|
339
|
+
},
|
|
340
|
+
{
|
|
341
|
+
MessageGroupId: 'custom-group-id',
|
|
342
|
+
MessageDeduplicationId: 'unique-dedup-id', // Optional
|
|
343
|
+
}
|
|
344
|
+
)
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
### FIFO Topic Consumer
|
|
348
|
+
|
|
349
|
+
```typescript
|
|
350
|
+
class UserEventsFifoConsumer extends AbstractSnsSqsConsumer<
|
|
351
|
+
SupportedMessages,
|
|
352
|
+
ExecutionContext
|
|
353
|
+
> {
|
|
354
|
+
constructor(
|
|
355
|
+
snsClient: SNSClient,
|
|
356
|
+
sqsClient: SQSClient,
|
|
357
|
+
stsClient: STSClient,
|
|
358
|
+
userService: UserService
|
|
359
|
+
) {
|
|
360
|
+
super(
|
|
361
|
+
{
|
|
362
|
+
snsClient,
|
|
363
|
+
sqsClient,
|
|
364
|
+
stsClient,
|
|
365
|
+
logger: console,
|
|
366
|
+
errorReporter: { report: (error) => console.error(error) },
|
|
367
|
+
consumerErrorResolver: {
|
|
368
|
+
resolveError: () => ({ resolve: 'retryLater' as const }),
|
|
369
|
+
},
|
|
370
|
+
transactionObservabilityManager: {
|
|
371
|
+
start: () => {},
|
|
372
|
+
stop: () => {},
|
|
373
|
+
},
|
|
374
|
+
},
|
|
375
|
+
{
|
|
376
|
+
fifoQueue: true, // Enable FIFO mode for SQS queue
|
|
377
|
+
messageTypeField: 'messageType',
|
|
378
|
+
handlers: new MessageHandlerConfigBuilder<SupportedMessages, ExecutionContext>()
|
|
379
|
+
.addConfig(UserCreatedSchema, handleUserCreated)
|
|
380
|
+
.addConfig(UserUpdatedSchema, handleUserUpdated)
|
|
381
|
+
.build(),
|
|
382
|
+
|
|
383
|
+
creationConfig: {
|
|
384
|
+
queue: {
|
|
385
|
+
QueueName: 'user-events-consumer-queue.fifo', // Must end with .fifo
|
|
386
|
+
Attributes: {
|
|
387
|
+
FifoQueue: 'true',
|
|
388
|
+
ContentBasedDeduplication: 'false',
|
|
389
|
+
},
|
|
390
|
+
},
|
|
391
|
+
topic: {
|
|
392
|
+
Name: 'user-events-topic.fifo', // Must match publisher's FIFO topic
|
|
393
|
+
Attributes: {
|
|
394
|
+
FifoTopic: 'true',
|
|
395
|
+
ContentBasedDeduplication: 'false',
|
|
396
|
+
},
|
|
397
|
+
},
|
|
398
|
+
},
|
|
399
|
+
|
|
400
|
+
subscriptionConfig: {
|
|
401
|
+
updateAttributesIfExists: false,
|
|
402
|
+
},
|
|
403
|
+
|
|
404
|
+
// Optional: Configure concurrent consumers for parallel processing of different groups
|
|
405
|
+
concurrentConsumersAmount: 3, // Process 3 different message groups in parallel
|
|
406
|
+
},
|
|
407
|
+
{ userService }
|
|
408
|
+
)
|
|
409
|
+
}
|
|
410
|
+
}
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
## Configuration
|
|
414
|
+
|
|
415
|
+
### Topic Creation
|
|
416
|
+
|
|
417
|
+
When using `creationConfig`, the topic will be created automatically if it doesn't exist:
|
|
418
|
+
|
|
419
|
+
```typescript
|
|
420
|
+
{
|
|
421
|
+
creationConfig: {
|
|
422
|
+
topic: {
|
|
423
|
+
Name: 'my-topic',
|
|
424
|
+
Attributes: {
|
|
425
|
+
// Standard Topic attributes
|
|
426
|
+
DisplayName: 'My Notification Topic',
|
|
427
|
+
|
|
428
|
+
// FIFO Topic attributes (only for .fifo topics)
|
|
429
|
+
FifoTopic: 'true', // Must be 'true' for FIFO topics
|
|
430
|
+
ContentBasedDeduplication: 'false', // Automatic deduplication based on message body
|
|
431
|
+
|
|
432
|
+
// Encryption
|
|
433
|
+
KmsMasterKeyId: 'alias/aws/sns', // KMS key for encryption
|
|
434
|
+
|
|
435
|
+
// Delivery policy
|
|
436
|
+
DeliveryPolicy: JSON.stringify({
|
|
437
|
+
healthyRetryPolicy: {
|
|
438
|
+
minDelayTarget: 20,
|
|
439
|
+
maxDelayTarget: 20,
|
|
440
|
+
numRetries: 3,
|
|
441
|
+
numMaxDelayRetries: 0,
|
|
442
|
+
numNoDelayRetries: 0,
|
|
443
|
+
numMinDelayRetries: 0,
|
|
444
|
+
backoffFunction: 'linear'
|
|
445
|
+
}
|
|
446
|
+
}),
|
|
447
|
+
},
|
|
448
|
+
Tags: [
|
|
449
|
+
{ Key: 'Environment', Value: 'production' },
|
|
450
|
+
{ Key: 'Team', Value: 'backend' },
|
|
451
|
+
],
|
|
452
|
+
},
|
|
453
|
+
|
|
454
|
+
queue: {
|
|
455
|
+
QueueName: 'my-consumer-queue',
|
|
456
|
+
// See SQS README for full queue configuration options
|
|
457
|
+
},
|
|
458
|
+
|
|
459
|
+
updateAttributesIfExists: true, // Update attributes if topic/queue exists
|
|
460
|
+
forceTagUpdate: false, // Force tag update even if unchanged
|
|
461
|
+
|
|
462
|
+
// Queue permissions for SNS publishing (automatically configured)
|
|
463
|
+
queueUrlsWithSubscribePermissionsPrefix: 'https://sqs.us-east-1.amazonaws.com/123456789012/',
|
|
464
|
+
},
|
|
465
|
+
}
|
|
466
|
+
```
|
|
467
|
+
|
|
468
|
+
### Topic Locator
|
|
469
|
+
|
|
470
|
+
When using `locatorConfig`, you connect to an existing topic without creating it:
|
|
471
|
+
|
|
472
|
+
```typescript
|
|
473
|
+
{
|
|
474
|
+
locatorConfig: {
|
|
475
|
+
// Option 1: By topic ARN
|
|
476
|
+
topicArn: 'arn:aws:sns:us-east-1:123456789012:my-topic',
|
|
477
|
+
|
|
478
|
+
// Option 2: By topic name (ARN will be resolved using STS)
|
|
479
|
+
// topicName: 'my-topic',
|
|
480
|
+
|
|
481
|
+
// Optional: Existing queue URL or name
|
|
482
|
+
queueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue',
|
|
483
|
+
// or
|
|
484
|
+
// queueName: 'my-queue',
|
|
485
|
+
|
|
486
|
+
// Optional: Existing subscription ARN
|
|
487
|
+
subscriptionArn: 'arn:aws:sns:us-east-1:123456789012:my-topic:uuid',
|
|
488
|
+
},
|
|
489
|
+
}
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
### Publisher Options
|
|
493
|
+
|
|
494
|
+
```typescript
|
|
495
|
+
{
|
|
496
|
+
// Required - Message Schema Configuration
|
|
497
|
+
messageSchemas: [Schema1, Schema2], // Array of Zod schemas
|
|
498
|
+
messageTypeField: 'messageType', // Field containing message type discriminator
|
|
499
|
+
|
|
500
|
+
// Topic Configuration (one of these required)
|
|
501
|
+
creationConfig: { /* ... */ }, // Create topic if doesn't exist
|
|
502
|
+
locatorConfig: { /* ... */ }, // Use existing topic
|
|
503
|
+
|
|
504
|
+
// Optional - FIFO Configuration
|
|
505
|
+
fifoTopic: false, // Set to true for FIFO topics
|
|
506
|
+
messageGroupIdField: 'userId', // Field to use as MessageGroupId
|
|
507
|
+
defaultMessageGroupId: 'default', // Default MessageGroupId if field not present
|
|
508
|
+
|
|
509
|
+
// Optional - Message Field Configuration (same as SQS)
|
|
510
|
+
messageIdField: 'id', // Default: 'id'
|
|
511
|
+
messageTimestampField: 'timestamp', // Default: 'timestamp'
|
|
512
|
+
messageDeduplicationIdField: 'deduplicationId', // Default: 'deduplicationId'
|
|
513
|
+
messageDeduplicationOptionsField: 'deduplicationOptions', // Default: 'deduplicationOptions'
|
|
514
|
+
|
|
515
|
+
// Optional - Features
|
|
516
|
+
logMessages: false, // Log all published messages
|
|
517
|
+
handlerSpy: true, // Enable handler spy for testing
|
|
518
|
+
|
|
519
|
+
// Optional - Deduplication (same as SQS)
|
|
520
|
+
enablePublisherDeduplication: false,
|
|
521
|
+
messageDeduplicationConfig: { /* ... */ },
|
|
522
|
+
|
|
523
|
+
// Optional - Payload Offloading (same as SQS)
|
|
524
|
+
payloadStoreConfig: { /* ... */ },
|
|
525
|
+
|
|
526
|
+
// Optional - Deletion
|
|
527
|
+
deletionConfig: { /* ... */ },
|
|
528
|
+
}
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
See the [SQS README - Publisher Options section](../sqs/README.md#publisher-options) for full details on shared options.
|
|
532
|
+
|
|
533
|
+
### Consumer Options
|
|
534
|
+
|
|
535
|
+
SNS consumers use the same options as SQS consumers, plus SNS-specific subscription configuration:
|
|
536
|
+
|
|
537
|
+
```typescript
|
|
538
|
+
{
|
|
539
|
+
// All SQS consumer options are supported
|
|
540
|
+
// See SQS README for full consumer options
|
|
541
|
+
|
|
542
|
+
// Required - Message Handling Configuration
|
|
543
|
+
handlers: MessageHandlerConfigBuilder.build(),
|
|
544
|
+
messageTypeField: 'messageType',
|
|
545
|
+
|
|
546
|
+
// Topic & Queue Configuration
|
|
547
|
+
creationConfig: {
|
|
548
|
+
topic: { /* SNS topic config */ },
|
|
549
|
+
queue: { /* SQS queue config */ },
|
|
550
|
+
},
|
|
551
|
+
// or
|
|
552
|
+
locatorConfig: {
|
|
553
|
+
topicArn: 'arn:aws:sns:...',
|
|
554
|
+
queueUrl: 'https://sqs...',
|
|
555
|
+
subscriptionArn: 'arn:aws:sns:...',
|
|
556
|
+
},
|
|
557
|
+
|
|
558
|
+
// SNS-Specific - Subscription Configuration
|
|
559
|
+
subscriptionConfig: {
|
|
560
|
+
updateAttributesIfExists: false, // Update subscription attributes if exists
|
|
561
|
+
|
|
562
|
+
// Optional: Message filtering
|
|
563
|
+
filterPolicy: {
|
|
564
|
+
messageType: ['user.created', 'user.updated'], // Only receive these types
|
|
565
|
+
},
|
|
566
|
+
|
|
567
|
+
// Optional: Raw message delivery (disable SNS envelope)
|
|
568
|
+
rawMessageDelivery: false,
|
|
569
|
+
|
|
570
|
+
// Optional: Redrive policy (DLQ for undeliverable messages)
|
|
571
|
+
redrivePolicy: {
|
|
572
|
+
deadLetterTargetArn: 'arn:aws:sqs:us-east-1:123456789012:my-dlq',
|
|
573
|
+
},
|
|
574
|
+
},
|
|
575
|
+
|
|
576
|
+
// Optional - FIFO Configuration
|
|
577
|
+
fifoQueue: false,
|
|
578
|
+
|
|
579
|
+
// Optional - Other options inherited from SQS
|
|
580
|
+
concurrentConsumersAmount: 1,
|
|
581
|
+
maxRetryDuration: 345600, // 4 days
|
|
582
|
+
deadLetterQueue: { /* ... */ },
|
|
583
|
+
consumerOverrides: { /* ... */ },
|
|
584
|
+
// ... see SQS README for full list
|
|
585
|
+
}
|
|
586
|
+
```
|
|
587
|
+
|
|
588
|
+
See the [SQS README - Consumer Options section](../sqs/README.md#consumer-options) for full details on shared options.
|
|
589
|
+
|
|
590
|
+
## SNS-Specific Features
|
|
591
|
+
|
|
592
|
+
### Topic Subscriptions
|
|
593
|
+
|
|
594
|
+
SNS topics can have multiple subscribers. Each subscriber receives a copy of every message published to the topic:
|
|
595
|
+
|
|
596
|
+
```typescript
|
|
597
|
+
// Publisher publishes to one topic
|
|
598
|
+
class NotificationPublisher extends AbstractSnsPublisher<NotificationMessage> {
|
|
599
|
+
constructor(snsClient: SNSClient, stsClient: STSClient) {
|
|
600
|
+
super(dependencies, {
|
|
601
|
+
messageSchemas: [NotificationSchema],
|
|
602
|
+
creationConfig: {
|
|
603
|
+
topic: { Name: 'notifications' },
|
|
604
|
+
},
|
|
605
|
+
})
|
|
606
|
+
}
|
|
607
|
+
}
|
|
608
|
+
|
|
609
|
+
// Multiple consumers subscribe to the same topic
|
|
610
|
+
class EmailConsumer extends AbstractSnsSqsConsumer<NotificationMessage, EmailContext> {
|
|
611
|
+
constructor(deps) {
|
|
612
|
+
super(deps, {
|
|
613
|
+
handlers: buildHandlers(sendEmail),
|
|
614
|
+
creationConfig: {
|
|
615
|
+
queue: { QueueName: 'email-notifications' },
|
|
616
|
+
topic: { Name: 'notifications' }, // Same topic
|
|
617
|
+
},
|
|
618
|
+
})
|
|
619
|
+
}
|
|
620
|
+
}
|
|
621
|
+
|
|
622
|
+
class SMSConsumer extends AbstractSnsSqsConsumer<NotificationMessage, SMSContext> {
|
|
623
|
+
constructor(deps) {
|
|
624
|
+
super(deps, {
|
|
625
|
+
handlers: buildHandlers(sendSMS),
|
|
626
|
+
creationConfig: {
|
|
627
|
+
queue: { QueueName: 'sms-notifications' },
|
|
628
|
+
topic: { Name: 'notifications' }, // Same topic
|
|
629
|
+
},
|
|
630
|
+
})
|
|
631
|
+
}
|
|
632
|
+
}
|
|
633
|
+
|
|
634
|
+
class PushConsumer extends AbstractSnsSqsConsumer<NotificationMessage, PushContext> {
|
|
635
|
+
constructor(deps) {
|
|
636
|
+
super(deps, {
|
|
637
|
+
handlers: buildHandlers(sendPush),
|
|
638
|
+
creationConfig: {
|
|
639
|
+
queue: { QueueName: 'push-notifications' },
|
|
640
|
+
topic: { Name: 'notifications' }, // Same topic
|
|
641
|
+
},
|
|
642
|
+
})
|
|
643
|
+
}
|
|
644
|
+
}
|
|
645
|
+
|
|
646
|
+
// One publish reaches all three consumers
|
|
647
|
+
await publisher.publish({
|
|
648
|
+
id: '123',
|
|
649
|
+
messageType: 'notification.created',
|
|
650
|
+
userId: 'user-456',
|
|
651
|
+
message: 'Your order has shipped!',
|
|
652
|
+
})
|
|
653
|
+
// → Email sent
|
|
654
|
+
// → SMS sent
|
|
655
|
+
// → Push notification sent
|
|
656
|
+
```
|
|
657
|
+
|
|
658
|
+
### Fan-Out Pattern
|
|
659
|
+
|
|
660
|
+
The fan-out pattern enables broadcasting messages to multiple independent processing pipelines:
|
|
661
|
+
|
|
662
|
+
```typescript
|
|
663
|
+
// Single event publisher
|
|
664
|
+
class OrderEventPublisher extends AbstractSnsPublisher<OrderEvent> {
|
|
665
|
+
constructor(snsClient: SNSClient, stsClient: STSClient) {
|
|
666
|
+
super(dependencies, {
|
|
667
|
+
messageSchemas: [OrderCreatedSchema, OrderUpdatedSchema],
|
|
668
|
+
creationConfig: {
|
|
669
|
+
topic: { Name: 'order-events' },
|
|
670
|
+
},
|
|
671
|
+
})
|
|
672
|
+
}
|
|
673
|
+
}
|
|
674
|
+
|
|
675
|
+
// Different services consume the same events independently
|
|
676
|
+
|
|
677
|
+
// Inventory service - updates stock levels
|
|
678
|
+
class InventoryConsumer extends AbstractSnsSqsConsumer<OrderEvent, InventoryContext> {
|
|
679
|
+
constructor(deps) {
|
|
680
|
+
super(deps, {
|
|
681
|
+
handlers: buildInventoryHandlers(),
|
|
682
|
+
creationConfig: {
|
|
683
|
+
queue: { QueueName: 'inventory-order-events' },
|
|
684
|
+
topic: { Name: 'order-events' },
|
|
685
|
+
},
|
|
686
|
+
})
|
|
687
|
+
}
|
|
688
|
+
}
|
|
689
|
+
|
|
690
|
+
// Analytics service - tracks order metrics
|
|
691
|
+
class AnalyticsConsumer extends AbstractSnsSqsConsumer<OrderEvent, AnalyticsContext> {
|
|
692
|
+
constructor(deps) {
|
|
693
|
+
super(deps, {
|
|
694
|
+
handlers: buildAnalyticsHandlers(),
|
|
695
|
+
creationConfig: {
|
|
696
|
+
queue: { QueueName: 'analytics-order-events' },
|
|
697
|
+
topic: { Name: 'order-events' },
|
|
698
|
+
},
|
|
699
|
+
})
|
|
700
|
+
}
|
|
701
|
+
}
|
|
702
|
+
|
|
703
|
+
// Shipping service - prepares shipments
|
|
704
|
+
class ShippingConsumer extends AbstractSnsSqsConsumer<OrderEvent, ShippingContext> {
|
|
705
|
+
constructor(deps) {
|
|
706
|
+
super(deps, {
|
|
707
|
+
handlers: buildShippingHandlers(),
|
|
708
|
+
creationConfig: {
|
|
709
|
+
queue: { QueueName: 'shipping-order-events' },
|
|
710
|
+
topic: { Name: 'order-events' },
|
|
711
|
+
},
|
|
712
|
+
})
|
|
713
|
+
}
|
|
714
|
+
}
|
|
715
|
+
|
|
716
|
+
// Benefits:
|
|
717
|
+
// - Each service processes independently
|
|
718
|
+
// - Failure in one doesn't affect others
|
|
719
|
+
// - Easy to add new consumers without changing publisher
|
|
720
|
+
// - Each consumer can have its own retry/DLQ configuration
|
|
721
|
+
```
|
|
722
|
+
|
|
723
|
+
### Message Filtering
|
|
724
|
+
|
|
725
|
+
Subscribers can filter messages to receive only specific types:
|
|
726
|
+
|
|
727
|
+
```typescript
|
|
728
|
+
class UserCreatedConsumer extends AbstractSnsSqsConsumer<UserEvent, UserContext> {
|
|
729
|
+
constructor(deps) {
|
|
730
|
+
super(deps, {
|
|
731
|
+
handlers: buildHandlers(),
|
|
732
|
+
creationConfig: {
|
|
733
|
+
queue: { QueueName: 'user-created-processor' },
|
|
734
|
+
topic: { Name: 'user-events' },
|
|
735
|
+
},
|
|
736
|
+
subscriptionConfig: {
|
|
737
|
+
// Only receive user.created events
|
|
738
|
+
filterPolicy: {
|
|
739
|
+
messageType: ['user.created'],
|
|
740
|
+
},
|
|
741
|
+
},
|
|
742
|
+
})
|
|
743
|
+
}
|
|
744
|
+
}
|
|
745
|
+
|
|
746
|
+
class UserModificationConsumer extends AbstractSnsSqsConsumer<UserEvent, UserContext> {
|
|
747
|
+
constructor(deps) {
|
|
748
|
+
super(deps, {
|
|
749
|
+
handlers: buildHandlers(),
|
|
750
|
+
creationConfig: {
|
|
751
|
+
queue: { QueueName: 'user-modification-processor' },
|
|
752
|
+
topic: { Name: 'user-events' },
|
|
753
|
+
},
|
|
754
|
+
subscriptionConfig: {
|
|
755
|
+
// Only receive update and delete events
|
|
756
|
+
filterPolicy: {
|
|
757
|
+
messageType: ['user.updated', 'user.deleted'],
|
|
758
|
+
},
|
|
759
|
+
},
|
|
760
|
+
})
|
|
761
|
+
}
|
|
762
|
+
}
|
|
763
|
+
|
|
764
|
+
// Advanced filtering with message attributes
|
|
765
|
+
class HighPriorityConsumer extends AbstractSnsSqsConsumer<OrderEvent, OrderContext> {
|
|
766
|
+
constructor(deps) {
|
|
767
|
+
super(deps, {
|
|
768
|
+
handlers: buildHandlers(),
|
|
769
|
+
creationConfig: {
|
|
770
|
+
queue: { QueueName: 'high-priority-orders' },
|
|
771
|
+
topic: { Name: 'order-events' },
|
|
772
|
+
},
|
|
773
|
+
subscriptionConfig: {
|
|
774
|
+
filterPolicy: {
|
|
775
|
+
messageType: ['order.created'],
|
|
776
|
+
priority: ['high', 'critical'], // Filters on message attribute
|
|
777
|
+
},
|
|
778
|
+
},
|
|
779
|
+
})
|
|
780
|
+
}
|
|
781
|
+
}
|
|
782
|
+
```
|
|
783
|
+
|
|
784
|
+
**Benefits:**
|
|
785
|
+
- Reduces unnecessary message processing
|
|
786
|
+
- Lowers SQS costs (fewer messages received)
|
|
787
|
+
- Filtering happens at SNS level (before queuing)
|
|
788
|
+
- Each subscriber can have different filters
|
|
789
|
+
|
|
790
|
+
### Cross-Account Publishing
|
|
791
|
+
|
|
792
|
+
SNS supports publishing from one AWS account to topics in another:
|
|
793
|
+
|
|
794
|
+
```typescript
|
|
795
|
+
// Account A - Publisher
|
|
796
|
+
class CrossAccountPublisher extends AbstractSnsPublisher<Message> {
|
|
797
|
+
constructor(snsClient: SNSClient, stsClient: STSClient) {
|
|
798
|
+
super(dependencies, {
|
|
799
|
+
messageSchemas: [MessageSchema],
|
|
800
|
+
locatorConfig: {
|
|
801
|
+
// Topic in Account B
|
|
802
|
+
topicArn: 'arn:aws:sns:us-east-1:222222222222:shared-topic',
|
|
803
|
+
},
|
|
804
|
+
})
|
|
805
|
+
}
|
|
806
|
+
}
|
|
807
|
+
|
|
808
|
+
// Account B - Consumer (topic owner)
|
|
809
|
+
// Topic policy must allow Account A to publish:
|
|
810
|
+
{
|
|
811
|
+
creationConfig: {
|
|
812
|
+
topic: {
|
|
813
|
+
Name: 'shared-topic',
|
|
814
|
+
Attributes: {
|
|
815
|
+
Policy: JSON.stringify({
|
|
816
|
+
Version: '2012-10-17',
|
|
817
|
+
Statement: [
|
|
818
|
+
{
|
|
819
|
+
Effect: 'Allow',
|
|
820
|
+
Principal: {
|
|
821
|
+
AWS: 'arn:aws:iam::111111111111:root', // Account A
|
|
822
|
+
},
|
|
823
|
+
Action: 'SNS:Publish',
|
|
824
|
+
Resource: 'arn:aws:sns:us-east-1:222222222222:shared-topic',
|
|
825
|
+
},
|
|
826
|
+
],
|
|
827
|
+
}),
|
|
828
|
+
},
|
|
829
|
+
},
|
|
830
|
+
},
|
|
831
|
+
}
|
|
832
|
+
```
|
|
833
|
+
|
|
834
|
+
## Advanced Features
|
|
835
|
+
|
|
836
|
+
SNS consumers inherit all advanced features from SQS consumers. See the SQS README for detailed documentation on:
|
|
837
|
+
|
|
838
|
+
- **[Custom Message Field Names](../sqs/README.md#custom-message-field-names)** - Adapt to existing schemas
|
|
839
|
+
- **[Dead Letter Queue (DLQ)](../sqs/README.md#dead-letter-queue-dlq)** - Handle permanently failing messages
|
|
840
|
+
- **[Message Retry Logic](../sqs/README.md#message-retry-logic)** - Exponential backoff and retry limits
|
|
841
|
+
- **[Message Deduplication](../sqs/README.md#message-deduplication)** - Publisher and consumer-level deduplication
|
|
842
|
+
- **[Payload Offloading](../sqs/README.md#payload-offloading)** - S3 storage for large messages
|
|
843
|
+
- **[Message Handlers](../sqs/README.md#message-handlers)** - Type-safe handler configuration
|
|
844
|
+
- **[Pre-handlers and Barriers](../sqs/README.md#pre-handlers-and-barriers)** - Middleware and message dependencies
|
|
845
|
+
- **[Handler Spies](../sqs/README.md#handler-spies)** - Testing async message flows
|
|
846
|
+
|
|
847
|
+
All these features work identically for SNS consumers since they extend the SQS consumer implementation.
|
|
848
|
+
|
|
849
|
+
## FIFO Topics
|
|
850
|
+
|
|
851
|
+
FIFO (First-In-First-Out) topics provide message ordering and exactly-once delivery, similar to FIFO queues.
|
|
852
|
+
|
|
853
|
+
### FIFO Topic Requirements
|
|
854
|
+
|
|
855
|
+
1. **Topic name must end with `.fifo`**
|
|
856
|
+
```typescript
|
|
857
|
+
Name: 'my-topic.fifo' // ✅ Valid
|
|
858
|
+
Name: 'my-topic' // ❌ Invalid for FIFO
|
|
859
|
+
```
|
|
860
|
+
|
|
861
|
+
2. **FifoTopic attribute must be 'true'**
|
|
862
|
+
```typescript
|
|
863
|
+
Attributes: {
|
|
864
|
+
FifoTopic: 'true',
|
|
865
|
+
}
|
|
866
|
+
```
|
|
867
|
+
|
|
868
|
+
3. **Subscribed queues must also be FIFO**
|
|
869
|
+
```typescript
|
|
870
|
+
creationConfig: {
|
|
871
|
+
topic: {
|
|
872
|
+
Name: 'events.fifo',
|
|
873
|
+
Attributes: { FifoTopic: 'true' },
|
|
874
|
+
},
|
|
875
|
+
queue: {
|
|
876
|
+
QueueName: 'consumer.fifo', // Must be FIFO
|
|
877
|
+
Attributes: { FifoQueue: 'true' },
|
|
878
|
+
},
|
|
879
|
+
}
|
|
880
|
+
```
|
|
881
|
+
|
|
882
|
+
4. **MessageGroupId required for all messages**
|
|
883
|
+
```typescript
|
|
884
|
+
// Option 1: From message field
|
|
885
|
+
messageGroupIdField: 'userId'
|
|
886
|
+
|
|
887
|
+
// Option 2: Default value
|
|
888
|
+
defaultMessageGroupId: 'default-group'
|
|
889
|
+
|
|
890
|
+
// Option 3: Explicit in publish call
|
|
891
|
+
await publisher.publish(message, {
|
|
892
|
+
MessageGroupId: 'custom-group',
|
|
893
|
+
})
|
|
894
|
+
```
|
|
895
|
+
|
|
896
|
+
5. **DLQ must also be FIFO**
|
|
897
|
+
```typescript
|
|
898
|
+
deadLetterQueue: {
|
|
899
|
+
creationConfig: {
|
|
900
|
+
queue: {
|
|
901
|
+
QueueName: 'my-dlq.fifo', // Must be FIFO
|
|
902
|
+
Attributes: { FifoQueue: 'true' },
|
|
903
|
+
},
|
|
904
|
+
},
|
|
905
|
+
}
|
|
906
|
+
```
|
|
907
|
+
|
|
908
|
+
### Message Groups
|
|
909
|
+
|
|
910
|
+
Message groups work the same way as in FIFO queues. See the [SQS README - Message Groups section](../sqs/README.md#message-groups) for detailed information on:
|
|
911
|
+
- Group assignment and parallelism
|
|
912
|
+
- Best practices for group design
|
|
913
|
+
- Balancing group sizes
|
|
914
|
+
- Sizing concurrent consumers
|
|
915
|
+
|
|
916
|
+
**SNS FIFO Fan-Out Example:**
|
|
917
|
+
|
|
918
|
+
```typescript
|
|
919
|
+
// FIFO publisher publishes to FIFO topic
|
|
920
|
+
const fifoPublisher = new OrderEventsFifoPublisher(snsClient, stsClient)
|
|
921
|
+
await fifoPublisher.publish({
|
|
922
|
+
id: '123',
|
|
923
|
+
messageType: 'order.created',
|
|
924
|
+
customerId: 'customer-A',
|
|
925
|
+
orderId: 'order-1',
|
|
926
|
+
}, {
|
|
927
|
+
MessageGroupId: 'customer-A', // All messages for customer-A ordered
|
|
928
|
+
})
|
|
929
|
+
|
|
930
|
+
// Multiple FIFO consumers subscribe to the same FIFO topic
|
|
931
|
+
const inventoryConsumer = new InventoryFifoConsumer(deps)
|
|
932
|
+
const analyticsConsumer = new AnalyticsFifoConsumer(deps)
|
|
933
|
+
const shippingConsumer = new ShippingFifoConsumer(deps)
|
|
934
|
+
|
|
935
|
+
// Each consumer receives messages in order within each group
|
|
936
|
+
// Different consumers can process different groups in parallel
|
|
937
|
+
```
|
|
938
|
+
|
|
939
|
+
### FIFO-Specific Configuration
|
|
940
|
+
|
|
941
|
+
```typescript
|
|
942
|
+
// Publisher
|
|
943
|
+
{
|
|
944
|
+
fifoTopic: true,
|
|
945
|
+
|
|
946
|
+
// Choose one or more:
|
|
947
|
+
messageGroupIdField: 'userId', // Use field from message
|
|
948
|
+
defaultMessageGroupId: 'default', // Fallback value
|
|
949
|
+
// or provide in publish call
|
|
950
|
+
}
|
|
951
|
+
|
|
952
|
+
// Consumer
|
|
953
|
+
{
|
|
954
|
+
fifoQueue: true,
|
|
955
|
+
concurrentConsumersAmount: 3, // Process 3 groups in parallel
|
|
956
|
+
|
|
957
|
+
// Note: Retry behavior inherits from SQS FIFO queues
|
|
958
|
+
// - No DelaySeconds support (AWS limitation)
|
|
959
|
+
// - Messages retry immediately
|
|
960
|
+
// - Order is preserved
|
|
961
|
+
}
|
|
962
|
+
|
|
963
|
+
// Topic Configuration
|
|
964
|
+
{
|
|
965
|
+
creationConfig: {
|
|
966
|
+
topic: {
|
|
967
|
+
Name: 'my-topic.fifo',
|
|
968
|
+
Attributes: {
|
|
969
|
+
FifoTopic: 'true',
|
|
970
|
+
|
|
971
|
+
// Optional: Automatic deduplication based on message body
|
|
972
|
+
ContentBasedDeduplication: 'false', // or 'true'
|
|
973
|
+
},
|
|
974
|
+
},
|
|
975
|
+
},
|
|
976
|
+
}
|
|
977
|
+
|
|
978
|
+
// Queue Configuration
|
|
979
|
+
{
|
|
980
|
+
creationConfig: {
|
|
981
|
+
queue: {
|
|
982
|
+
QueueName: 'my-queue.fifo',
|
|
983
|
+
Attributes: {
|
|
984
|
+
FifoQueue: 'true',
|
|
985
|
+
|
|
986
|
+
// Optional: Queue-level deduplication settings
|
|
987
|
+
ContentBasedDeduplication: 'false',
|
|
988
|
+
DeduplicationScope: 'queue', // or 'messageGroup'
|
|
989
|
+
FifoThroughputLimit: 'perQueue', // or 'perMessageGroupId'
|
|
990
|
+
},
|
|
991
|
+
},
|
|
992
|
+
},
|
|
993
|
+
}
|
|
994
|
+
```
|
|
995
|
+
|
|
996
|
+
**FIFO Limitations:**
|
|
997
|
+
- **Throughput**: 3,000 messages/second per topic (or 300/second per message group)
|
|
998
|
+
- **No delays**: FIFO queues don't support `DelaySeconds`
|
|
999
|
+
- **Strict ordering**: Within a message group, messages are delivered in exact order
|
|
1000
|
+
- **FIFO-to-FIFO only**: FIFO topics can only fan out to FIFO queues
|
|
1001
|
+
|
|
1002
|
+
See the [SQS README - FIFO Queues section](../sqs/README.md#fifo-queues) for comprehensive FIFO documentation.
|
|
1003
|
+
|
|
1004
|
+
## Testing
|
|
1005
|
+
|
|
1006
|
+
SNS testing works the same as SQS testing. Handler spies enable testing of async pub/sub flows:
|
|
1007
|
+
|
|
1008
|
+
```typescript
|
|
1009
|
+
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
|
1010
|
+
|
|
1011
|
+
describe('SNS Publisher and Consumer', () => {
|
|
1012
|
+
let publisher: UserEventsPublisher
|
|
1013
|
+
let consumer: UserEventsConsumer
|
|
1014
|
+
|
|
1015
|
+
beforeEach(async () => {
|
|
1016
|
+
publisher = new UserEventsPublisher(snsClient, stsClient, { handlerSpy: true })
|
|
1017
|
+
consumer = new UserEventsConsumer(snsClient, sqsClient, stsClient, userService, {
|
|
1018
|
+
handlerSpy: true
|
|
1019
|
+
})
|
|
1020
|
+
|
|
1021
|
+
await publisher.init()
|
|
1022
|
+
await consumer.start()
|
|
1023
|
+
})
|
|
1024
|
+
|
|
1025
|
+
afterEach(async () => {
|
|
1026
|
+
await consumer.close()
|
|
1027
|
+
await publisher.close()
|
|
1028
|
+
})
|
|
1029
|
+
|
|
1030
|
+
it('publishes to topic and consumes from subscribed queue', async () => {
|
|
1031
|
+
// Publish to SNS topic
|
|
1032
|
+
await publisher.publish({
|
|
1033
|
+
id: '123',
|
|
1034
|
+
messageType: 'user.created',
|
|
1035
|
+
userId: 'user-456',
|
|
1036
|
+
email: 'test@example.com',
|
|
1037
|
+
})
|
|
1038
|
+
|
|
1039
|
+
// Wait for publisher spy
|
|
1040
|
+
await publisher.handlerSpy.waitForMessageWithId('123', 'published')
|
|
1041
|
+
|
|
1042
|
+
// Wait for consumer spy
|
|
1043
|
+
const consumedMessage = await consumer.handlerSpy.waitForMessageWithId('123', 'consumed')
|
|
1044
|
+
|
|
1045
|
+
expect(consumedMessage.userId).toBe('user-456')
|
|
1046
|
+
expect(userService.createUser).toHaveBeenCalledWith('user-456', 'test@example.com')
|
|
1047
|
+
})
|
|
1048
|
+
})
|
|
1049
|
+
```
|
|
1050
|
+
|
|
1051
|
+
See the [SQS README - Testing section](../sqs/README.md#testing) for comprehensive testing documentation including:
|
|
1052
|
+
- Integration tests with LocalStack
|
|
1053
|
+
- Unit tests with handler spies
|
|
1054
|
+
- Testing indirect message publishing
|
|
1055
|
+
- Complex workflow testing
|
|
1056
|
+
|
|
1057
|
+
## API Reference
|
|
1058
|
+
|
|
1059
|
+
### AbstractSnsPublisher
|
|
1060
|
+
|
|
1061
|
+
```typescript
|
|
1062
|
+
class AbstractSnsPublisher<MessagePayloadType extends object> {
|
|
1063
|
+
constructor(
|
|
1064
|
+
dependencies: SNSDependencies,
|
|
1065
|
+
options: SNSPublisherOptions<MessagePayloadType>
|
|
1066
|
+
)
|
|
1067
|
+
|
|
1068
|
+
async init(): Promise<void>
|
|
1069
|
+
async close(): Promise<void>
|
|
1070
|
+
|
|
1071
|
+
async publish(
|
|
1072
|
+
message: MessagePayloadType,
|
|
1073
|
+
options?: SNSMessageOptions
|
|
1074
|
+
): Promise<void>
|
|
1075
|
+
|
|
1076
|
+
readonly handlerSpy: HandlerSpy<MessagePayloadType>
|
|
1077
|
+
readonly topicArn: string
|
|
1078
|
+
}
|
|
1079
|
+
```
|
|
1080
|
+
|
|
1081
|
+
### AbstractSnsSqsConsumer
|
|
1082
|
+
|
|
1083
|
+
```typescript
|
|
1084
|
+
class AbstractSnsSqsConsumer<
|
|
1085
|
+
MessagePayloadType extends object,
|
|
1086
|
+
ExecutionContext,
|
|
1087
|
+
PrehandlerOutput = undefined
|
|
1088
|
+
> extends AbstractSqsConsumer<...> {
|
|
1089
|
+
constructor(
|
|
1090
|
+
dependencies: SNSSQSConsumerDependencies,
|
|
1091
|
+
options: SNSSQSConsumerOptions<MessagePayloadType, ExecutionContext, PrehandlerOutput>,
|
|
1092
|
+
executionContext: ExecutionContext
|
|
1093
|
+
)
|
|
1094
|
+
|
|
1095
|
+
async init(): Promise<void>
|
|
1096
|
+
async start(): Promise<void>
|
|
1097
|
+
async close(abort?: boolean): Promise<void>
|
|
1098
|
+
|
|
1099
|
+
readonly handlerSpy: HandlerSpy<MessagePayloadType>
|
|
1100
|
+
readonly topicArn: string
|
|
1101
|
+
readonly subscriptionArn: string
|
|
1102
|
+
readonly queueUrl: string
|
|
1103
|
+
readonly queueName: string
|
|
1104
|
+
}
|
|
1105
|
+
```
|
|
1106
|
+
|
|
1107
|
+
### Types
|
|
1108
|
+
|
|
1109
|
+
```typescript
|
|
1110
|
+
// Message options for publishing
|
|
1111
|
+
type SNSMessageOptions = {
|
|
1112
|
+
MessageGroupId?: string // Required for FIFO topics
|
|
1113
|
+
MessageDeduplicationId?: string // Optional for FIFO topics
|
|
1114
|
+
}
|
|
1115
|
+
|
|
1116
|
+
// Dependencies
|
|
1117
|
+
type SNSDependencies = {
|
|
1118
|
+
snsClient: SNSClient
|
|
1119
|
+
stsClient: STSClient
|
|
1120
|
+
logger: Logger
|
|
1121
|
+
errorReporter: ErrorReporter
|
|
1122
|
+
messageMetricsManager?: MessageMetricsManager
|
|
1123
|
+
}
|
|
1124
|
+
|
|
1125
|
+
// Consumer dependencies (extends SNSDependencies and SQSDependencies)
|
|
1126
|
+
type SNSSQSConsumerDependencies = SNSDependencies & SQSDependencies & {
|
|
1127
|
+
consumerErrorResolver: ErrorResolver
|
|
1128
|
+
transactionObservabilityManager: TransactionObservabilityManager
|
|
1129
|
+
}
|
|
1130
|
+
|
|
1131
|
+
// Subscription options
|
|
1132
|
+
type SNSSubscriptionOptions = {
|
|
1133
|
+
updateAttributesIfExists?: boolean
|
|
1134
|
+
filterPolicy?: Record<string, string[]>
|
|
1135
|
+
rawMessageDelivery?: boolean
|
|
1136
|
+
redrivePolicy?: {
|
|
1137
|
+
deadLetterTargetArn: string
|
|
1138
|
+
}
|
|
1139
|
+
}
|
|
1140
|
+
```
|
|
1141
|
+
|
|
1142
|
+
### Utility Functions
|
|
1143
|
+
|
|
1144
|
+
```typescript
|
|
1145
|
+
// Topic validation
|
|
1146
|
+
function isFifoTopicName(topicName: string): boolean
|
|
1147
|
+
function validateFifoTopicName(topicName: string, isFifoTopic: boolean): void
|
|
1148
|
+
|
|
1149
|
+
// Topic operations
|
|
1150
|
+
async function assertTopic(
|
|
1151
|
+
snsClient: SNSClient,
|
|
1152
|
+
stsClient: STSClient,
|
|
1153
|
+
topicOptions: CreateTopicCommandInput,
|
|
1154
|
+
extraParams?: ExtraSNSCreationParams
|
|
1155
|
+
): Promise<string> // Returns topicArn
|
|
1156
|
+
|
|
1157
|
+
async function deleteTopic(
|
|
1158
|
+
snsClient: SNSClient,
|
|
1159
|
+
stsClient: STSClient,
|
|
1160
|
+
topicName: string
|
|
1161
|
+
): Promise<void>
|
|
1162
|
+
|
|
1163
|
+
async function getTopicAttributes(
|
|
1164
|
+
snsClient: SNSClient,
|
|
1165
|
+
topicArn: string
|
|
1166
|
+
): Promise<Either<'not_found', { attributes?: Record<string, string> }>>
|
|
1167
|
+
|
|
1168
|
+
// Subscription operations
|
|
1169
|
+
async function subscribeToTopic(
|
|
1170
|
+
snsClient: SNSClient,
|
|
1171
|
+
topicArn: string,
|
|
1172
|
+
queueArn: string,
|
|
1173
|
+
options?: SNSSubscriptionOptions
|
|
1174
|
+
): Promise<string> // Returns subscriptionArn
|
|
1175
|
+
|
|
1176
|
+
async function findSubscriptionByTopicAndQueue(
|
|
1177
|
+
snsClient: SNSClient,
|
|
1178
|
+
topicArn: string,
|
|
1179
|
+
queueArn: string
|
|
1180
|
+
): Promise<Subscription | undefined>
|
|
1181
|
+
|
|
1182
|
+
// Message reading
|
|
1183
|
+
function deserializeSNSMessage(
|
|
1184
|
+
message: SQSMessage
|
|
1185
|
+
): Either<MessageInvalidFormatError, SNSMessageBody>
|
|
1186
|
+
|
|
1187
|
+
// Message size calculation (same as SQS)
|
|
1188
|
+
function calculateOutgoingMessageSize(message: unknown): number
|
|
1189
|
+
```
|
|
1190
|
+
|
|
1191
|
+
## License
|
|
1192
|
+
|
|
1193
|
+
MIT
|
|
1194
|
+
|
|
1195
|
+
## Contributing
|
|
1196
|
+
|
|
1197
|
+
Contributions are welcome! Please see the main repository for guidelines.
|
|
1198
|
+
|
|
1199
|
+
## Links
|
|
1200
|
+
|
|
1201
|
+
- [Main Repository](https://github.com/kibertoad/message-queue-toolkit)
|
|
1202
|
+
- [Core Package](https://www.npmjs.com/package/@message-queue-toolkit/core)
|
|
1203
|
+
- [SQS Package](https://www.npmjs.com/package/@message-queue-toolkit/sqs) - SNS consumers extend SQS consumers
|
|
1204
|
+
- [AWS SNS Documentation](https://docs.aws.amazon.com/sns/)
|
|
1205
|
+
- [FIFO Topic Documentation](https://docs.aws.amazon.com/sns/latest/dg/sns-fifo-topics.html)
|
|
1206
|
+
- [SNS Message Filtering](https://docs.aws.amazon.com/sns/latest/dg/sns-message-filtering.html)
|