cyclone_lariat 0.3.10 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +6 -0
- data/.rubocop.yml +26 -1
- data/CHANGELOG.md +11 -1
- data/Gemfile.lock +26 -21
- data/README.md +466 -91
- data/Rakefile +2 -5
- data/bin/cyclone_lariat +174 -0
- data/config/initializers/sequel.rb +7 -0
- data/cyclone_lariat.gemspec +4 -0
- data/db/migrate/03_add_versions.rb +9 -0
- data/docs/_imgs/graphviz_01.png +0 -0
- data/docs/_imgs/graphviz_02.png +0 -0
- data/docs/_imgs/graphviz_03.png +0 -0
- data/docs/_imgs/logic.png +0 -0
- data/docs/_imgs/sqs_sns_diagram.png +0 -0
- data/lib/cyclone_lariat/abstract/client.rb +20 -14
- data/lib/cyclone_lariat/abstract/message.rb +20 -5
- data/lib/cyclone_lariat/configure.rb +1 -1
- data/lib/cyclone_lariat/errors.rb +22 -0
- data/lib/cyclone_lariat/middleware.rb +2 -1
- data/lib/cyclone_lariat/migration.rb +214 -0
- data/lib/cyclone_lariat/queue.rb +147 -0
- data/lib/cyclone_lariat/sns_client.rb +125 -14
- data/lib/cyclone_lariat/sqs_client.rb +69 -15
- data/lib/cyclone_lariat/topic.rb +113 -0
- data/lib/cyclone_lariat/version.rb +1 -1
- data/lib/cyclone_lariat.rb +1 -0
- data/lib/tasks/console.rake +13 -0
- data/lib/tasks/cyclone_lariat.rake +44 -0
- data/lib/tasks/db.rake +1 -1
- metadata +45 -4
- data/docs/_imgs/diagram.png +0 -0
data/README.md
CHANGED
@@ -1,85 +1,142 @@
|
|
1
1
|
# Cyclone lariat
|
2
2
|
|
3
|
-
This
|
3
|
+
This gem work in few scenarios:
|
4
|
+
- As middleware for [shoryuken](https://github.com/ruby-shoryuken/shoryuken). It saves all events to the database and also catches and throws all exceptions.
|
5
|
+
- As a middleware, it can log all incoming messages.
|
6
|
+
- As a client that can send messages to SNS topics and SQS queues.
|
7
|
+
- Also it can help you with CI\CD for theme, queue and subscription management like database migration.
|
4
8
|
|
5
9
|

|
6
10
|
|
7
11
|
|
12
|
+
# Client / Publisher
|
13
|
+
At first lets understand what the difference between SQS and SNS:
|
14
|
+
- Amazon Simple Queue Service (SQS) lets you send, store, and receive messages between software components at any
|
15
|
+
volume, without losing messages or requiring other services to be available.
|
16
|
+
- Amazon Simple Notification Service (SNS) sends notifications two ways Application2Person (like send sms).
|
17
|
+
And the second way is Application2Application, that's way more important for us. In this way you case use
|
18
|
+
SNS service like fanout.
|
19
|
+
|
20
|
+

|
21
|
+
|
22
|
+
For use **cyclone_lariat** as _Publisher_ lets make install CycloneLariat.
|
23
|
+
|
24
|
+
## Install cyclone_lariat
|
25
|
+
Edit Gemfile:
|
8
26
|
```ruby
|
9
27
|
# Gemfile
|
28
|
+
gem 'sequel'
|
29
|
+
gem 'cyclone_lariat'
|
30
|
+
```
|
31
|
+
And run in console:
|
32
|
+
```bash
|
33
|
+
$ bundle install
|
34
|
+
$ cyclone_lariat install
|
35
|
+
```
|
10
36
|
|
11
|
-
|
12
|
-
|
37
|
+
Last command will create 2 files:
|
38
|
+
- ./lib/tasks/cyclone_lariat.rake - Rake tasks, for management migrations
|
39
|
+
- ./config/initializers/cyclone_lariat.rb - Configuration default values for cyclone lariat usage
|
13
40
|
|
14
|
-
|
15
|
-
|
41
|
+
## Configuration
|
42
|
+
```ruby
|
43
|
+
# frozen_string_literal: true
|
44
|
+
|
45
|
+
CycloneLariat.tap do |cl|
|
46
|
+
cl.default_version = 1 # api version
|
47
|
+
cl.aws_key = ENV['AWS_KEY'] # aws key
|
48
|
+
cl.aws_account_id = ENV['AWS_ACCOUNT_ID'] # aws account id
|
49
|
+
cl.aws_secret_key = ENV['AWS_SECRET_KEY'] # aws secret
|
50
|
+
cl.aws_default_region = ENV['AWS_REGION'] # aws default region
|
51
|
+
cl.publisher = ENV['APP_NAME'] # name of your publishers, usually name of your application
|
52
|
+
cl.default_instance = ENV['INSTANCE'] # stage, production, test
|
53
|
+
cl.events_dataset = DB[:events] # sequel dataset for store income messages, for receiver
|
54
|
+
cl.versions_dataset = DB[:lariat_versions] # sequel dataset for migrations, for publisher
|
55
|
+
end
|
56
|
+
```
|
57
|
+
If you are only using your application as a publisher, you may not need to set the `events_dataset`
|
58
|
+
parameter.
|
59
|
+
|
60
|
+
Before creating the first migration, let's explain what `CycloneLariat::Message` is.
|
61
|
+
|
62
|
+
## Messages
|
63
|
+
Message in Amazon SQS\SNS service it's a
|
64
|
+
[object](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-attributes)
|
65
|
+
that has several attributes. The main attributes are the **body**, which consists of the published
|
66
|
+
data. The body is a _String_, but we can use it as a _JSON_ object. **Cyclone_lariat** use this scheme:
|
67
|
+
|
68
|
+
```json
|
69
|
+
{
|
70
|
+
"uuid": "f2ce3813-0905-4d81-a60e-f289f2431f50", // Uniq message identificator
|
71
|
+
"publisher": "sample_app", // Publisher application name
|
72
|
+
"request_id": "51285005-8a06-4181-b5fd-bf29f3b1a45a", // Optional: X-Request-Id
|
73
|
+
"type": "event_note_created", // Type of Event or Command
|
74
|
+
"version": 1, // Version of data structure
|
75
|
+
"data": {
|
76
|
+
"id": 12,
|
77
|
+
"text": "Sample of published data",
|
78
|
+
"attributes": ["one", "two", "three"]
|
79
|
+
},
|
80
|
+
"sent_at": "2022-11-09T11:42:18.203+01:00" // Time when message was sended in ISO8601 Standard
|
81
|
+
}
|
16
82
|
```
|
17
83
|
|
18
|
-
|
84
|
+
Idea about X-Request-Id you can see at
|
85
|
+
[StackOverflow](https://stackoverflow.com/questions/25433258/what-is-the-x-request-id-http-header).
|
86
|
+
|
87
|
+
As you see, type has prefix 'event_' in cyclone lariat you has two kinds of messages - _Event_ and _Command_.
|
19
88
|
|
20
|
-
|
89
|
+
### Command vs Event
|
90
|
+
Commands and events are both simple domain structures that contain solely data for reading. That means
|
91
|
+
they contain no behaviour or business logic.
|
21
92
|
|
22
|
-
|
23
|
-
|
24
|
-
|
93
|
+
A command is an object that is sent to the domain for a state change which is handled by a command
|
94
|
+
handler. They should be named with a verb in an imperative mood plus the aggregate name which it
|
95
|
+
operates on. Such request can be rejected due to the data the command holds being invalid/inconsistent.
|
96
|
+
There should be exactly 1 handler for each command. Once the command has been executed, the consumer
|
97
|
+
can then carry out whatever the task is depending on the output of the command.
|
25
98
|
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
99
|
+
An event is a statement of fact about what change has been made to the domain state. They are named
|
100
|
+
with the aggregate name where the change took place plus the verb past-participle. An event happens off
|
101
|
+
the back of a command.
|
102
|
+
A command can emit any number of events. The sender of the event does not care who receives it or
|
103
|
+
whether it has been received at all.
|
31
104
|
|
32
|
-
|
33
|
-
|
34
|
-
A command can emit any number of events. The sender of the event does not care who receives it or whether it has been
|
35
|
-
received at all.
|
105
|
+
### Publish
|
106
|
+
For publishing _Event_ or _Commands_, you have two ways, send _Message_ directly:
|
36
107
|
|
37
|
-
## Configure
|
38
108
|
```ruby
|
39
|
-
# 'config/initializers/cyclone_lariat.rb'
|
40
109
|
CycloneLariat.tap do |cl|
|
41
|
-
|
42
|
-
cl.aws_key = # aws key
|
43
|
-
cl.aws_secret_key = # aws secret
|
44
|
-
cl.aws_client_id = # aws client id
|
45
|
-
cl.aws_default_region = # aws default region
|
46
|
-
cl.publisher = 'auth' # name of your publishers, usually name of your application
|
47
|
-
cl.default_instance = APP_INSTANCE # stage, production, test
|
110
|
+
# Config app here
|
48
111
|
end
|
49
|
-
```
|
50
112
|
|
51
|
-
|
52
|
-
You can use client directly
|
113
|
+
client = CycloneLariat::SnsClient.new(instance: 'auth', version: 2)
|
53
114
|
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
secret_key: APP_CONF.aws.secret_key,
|
60
|
-
region: APP_CONF.aws.region,
|
61
|
-
version: 1, # at default 1
|
62
|
-
publisher: 'pilot',
|
63
|
-
instance: INSTANCE # at default :prod
|
115
|
+
client.publish_command('register_user', data: {
|
116
|
+
first_name: 'John',
|
117
|
+
last_name: 'Doe',
|
118
|
+
mail: 'john.doe@example.com'
|
119
|
+
}, fifo: false
|
64
120
|
)
|
65
121
|
```
|
66
122
|
|
67
|
-
|
68
|
-
```
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
123
|
+
That's call, will generate a message body:
|
124
|
+
```json
|
125
|
+
{
|
126
|
+
"uuid": "f2ce3813-0905-4d81-a60e-f289f2431f50",
|
127
|
+
"publisher": "auth",
|
128
|
+
"type": "command_register_user",
|
129
|
+
"version": 2,
|
130
|
+
"data": {
|
131
|
+
"first_name": "John",
|
132
|
+
"last_name": "Doe",
|
133
|
+
"mail": "john.doe@example.com"
|
134
|
+
},
|
135
|
+
"sent_at": "2022-11-09T11:42:18.203+01:00" // Time when message was sended in ISO8601 Standard
|
136
|
+
}
|
80
137
|
```
|
81
138
|
|
82
|
-
Or
|
139
|
+
Or is it better to make your own client, like a [Repository](https://deviq.com/design-patterns/repository-pattern) pattern.
|
83
140
|
|
84
141
|
```ruby
|
85
142
|
require 'cyclone_lariat/sns_client' # If require: false in Gemfile
|
@@ -90,30 +147,21 @@ class YourClient < CycloneLariat::SnsClient
|
|
90
147
|
instance 'stage'
|
91
148
|
|
92
149
|
def email_is_created(mail)
|
93
|
-
publish event('email_is_created',
|
94
|
-
data: { mail: mail }
|
95
|
-
),
|
96
|
-
to: APP_CONF.aws.fanout.emails
|
150
|
+
publish event('email_is_created', data: { mail: mail }), fifo: true
|
97
151
|
end
|
98
152
|
|
99
153
|
def email_is_removed(mail)
|
100
|
-
publish event('email_is_removed',
|
101
|
-
data: { mail: mail }
|
102
|
-
),
|
103
|
-
to: APP_CONF.aws.fanout.email
|
154
|
+
publish event('email_is_removed', data: { mail: mail }), fifo: true
|
104
155
|
end
|
105
156
|
|
106
157
|
|
107
158
|
def delete_user(mail)
|
108
|
-
publish command('delete_user',
|
109
|
-
data: { mail: mail }
|
110
|
-
),
|
111
|
-
to: APP_CONF.aws.fanout.email
|
159
|
+
publish command('delete_user', data: { mail: mail }), fifo: false
|
112
160
|
end
|
113
161
|
end
|
114
162
|
|
115
163
|
# Init repo
|
116
|
-
client = YourClient.new
|
164
|
+
client = YourClient.new
|
117
165
|
|
118
166
|
# And send topics
|
119
167
|
client.email_is_created 'john.doe@example.com'
|
@@ -121,76 +169,394 @@ client.email_is_removed 'john.doe@example.com'
|
|
121
169
|
client.delete_user 'john.doe@example.com'
|
122
170
|
```
|
123
171
|
|
172
|
+
### Topics and Queue
|
173
|
+
An Amazon SNS topic and SQS queue is a logical access point that acts as a communication channel. Both
|
174
|
+
of them has specific address ARN.
|
175
|
+
|
176
|
+
```
|
177
|
+
# Topic example
|
178
|
+
arn:aws:sns:eu-west-1:247602342345:test-event-fanout-cyclone_lariat-note_added.fifo
|
179
|
+
|
180
|
+
# Queue example
|
181
|
+
arn:aws:sqs:eu-west-1:247602342345:test-event-queue-cyclone_lariat-note_added-notifier.fifo
|
182
|
+
```
|
183
|
+
|
184
|
+
Split ARN:
|
185
|
+
- `arn:aws:sns` - Prefix for SNS Topics
|
186
|
+
- `arn:aws:sqs` - Prefix for SQS Queues
|
187
|
+
- `eu-west-1` -
|
188
|
+
[AWS Region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions)
|
189
|
+
- `247602342345` - [AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html)
|
190
|
+
- `test-event-fanout-cyclone_lariat-note_added` - Topic \ Queue name
|
191
|
+
- `.fifo` - if Topic or queue is
|
192
|
+
[FIFO](https://aws.amazon.com/blogs/aws/introducing-amazon-sns-fifo-first-in-first-out-pub-sub-messaging/), they must
|
193
|
+
has that suffix.
|
194
|
+
|
195
|
+
Region and client_id usually set using the **cyclone_lariat** [configuration](#Configuration).
|
196
|
+
|
197
|
+
## Declaration for topic and queues name
|
198
|
+
In **cyclone_lariat** we have a declaration for defining topic and queue names.
|
199
|
+
This can help in organizing the order.
|
200
|
+
|
201
|
+
```ruby
|
202
|
+
CycloneLariat.tap do |cfg|
|
203
|
+
cfg.instance = 'test'
|
204
|
+
cfg.publisher = 'cyclone_lariat'
|
205
|
+
# ...
|
206
|
+
end
|
207
|
+
|
208
|
+
CycloneLariat::SnsClient.new.publish_command('register_user', data: {
|
209
|
+
first_name: 'John',
|
210
|
+
last_name: 'Doe',
|
211
|
+
mail: 'john.doe@example.com'
|
212
|
+
}, fifo: true
|
213
|
+
)
|
214
|
+
|
215
|
+
# or in repository-like style:
|
216
|
+
|
217
|
+
class YourClient < CycloneLariat::SnsClient
|
218
|
+
def register_user(first:, last:, mail:)
|
219
|
+
publish command('register_user', data: { mail: mail }), fifo: true
|
220
|
+
end
|
221
|
+
end
|
222
|
+
```
|
223
|
+
|
224
|
+
|
225
|
+
We will publish a message on this topic: `test-command-fanout-cyclone_lariat-register_user`.
|
124
226
|
|
125
|
-
|
126
|
-
|
227
|
+
Let's split the topic title:
|
228
|
+
- `test` - instance;
|
229
|
+
- `command` - kind - [event or command](#command-vs-event);
|
230
|
+
- `fanount` - resource type - fanout for SNS topics;
|
231
|
+
- `cyclone_lariat` - publisher name;
|
232
|
+
- `regiser_user` - message type.
|
127
233
|
|
234
|
+
For queues you also can define destination.
|
128
235
|
```ruby
|
129
|
-
|
236
|
+
CycloneLariat::SqsClient.new.publish_event(
|
237
|
+
'register_user', data: { mail: 'john.doe@example.com' },
|
238
|
+
dest: :mailer, fifo: true
|
239
|
+
)
|
240
|
+
|
241
|
+
|
242
|
+
# or in repository-like style:
|
243
|
+
|
244
|
+
class YourClient < CycloneLariat::SnsClient
|
245
|
+
# ...
|
246
|
+
|
247
|
+
def register_user(first:, last:, mail:)
|
248
|
+
publish event('register_user', data: { mail: mail }), fifo: true
|
249
|
+
end
|
250
|
+
end
|
251
|
+
```
|
252
|
+
|
253
|
+
We will publish a message on this queue: `test-event-queue-cyclone_lariat-register_user-mailer`.
|
130
254
|
|
131
|
-
|
132
|
-
|
133
|
-
|
134
|
-
|
135
|
-
|
136
|
-
|
137
|
-
|
255
|
+
Let's split the queue title:
|
256
|
+
- `test` - instance;
|
257
|
+
- `event` - kind - [event or command](#command-vs-event);
|
258
|
+
- `queue` - resource type - queue for SQS;
|
259
|
+
- `cyclone_lariat` - publisher name;
|
260
|
+
- `regiser_user` - message type.
|
261
|
+
- `mailer` - destination
|
262
|
+
|
263
|
+
You also can sent message to queue with custom name. But this way does not recommended.
|
264
|
+
|
265
|
+
```ruby
|
266
|
+
# Directly
|
267
|
+
CycloneLariat::SqsClient.new.publish_event(
|
268
|
+
'register_user', data: { mail: 'john.doe@example.com' },
|
269
|
+
dest: :mailer, topic: 'custom_topic_name.fifo', fifo: true
|
138
270
|
)
|
271
|
+
|
272
|
+
# Repository
|
273
|
+
class YourClient < CycloneLariat::SnsClient
|
274
|
+
# ...
|
275
|
+
|
276
|
+
def register_user(first:, last:, mail:)
|
277
|
+
publish event('register_user', data: { mail: mail }),
|
278
|
+
topic: 'custom_topic_name.fifo', fifo: true
|
279
|
+
end
|
280
|
+
end
|
139
281
|
```
|
282
|
+
Will publish message on queue: `custom_topic` with fifo suffix.
|
140
283
|
|
141
|
-
|
142
|
-
|
284
|
+
# Migrations
|
285
|
+
|
286
|
+
With **cyclone_lariat** you can use migrations that can create, delete, and subscribe to your queues and topics, just like database migrations do.
|
287
|
+
To store versions of **cyclone_lariat** migrations, you need to create a database table.
|
143
288
|
|
144
289
|
```ruby
|
145
|
-
|
146
|
-
|
290
|
+
# frozen_string_literal: true
|
291
|
+
|
292
|
+
Sequel.migration do
|
293
|
+
change do
|
294
|
+
create_table :lariat_versions do
|
295
|
+
Integer :version, null: false, unique: true
|
296
|
+
end
|
297
|
+
end
|
298
|
+
end
|
147
299
|
```
|
300
|
+
After migrate this database migration, create **cyclone_lariat** migration.
|
301
|
+
|
302
|
+
```bash
|
303
|
+
$ cyclone_lariat generate migration user_created
|
304
|
+
```
|
305
|
+
|
306
|
+
This command should create a migration file, let's edit it.
|
148
307
|
|
149
|
-
Or you can define topic directly:
|
150
308
|
```ruby
|
151
|
-
|
309
|
+
# ./lariat/migrate/1668097991_user_created_queue.rb
|
310
|
+
|
311
|
+
# frozen_string_literal: true
|
312
|
+
|
313
|
+
class UserCreatedQueue < CycloneLariat::Migration
|
314
|
+
def up
|
315
|
+
create queue(:user_created, dest: :mailer, fifo: true)
|
316
|
+
end
|
317
|
+
|
318
|
+
def down
|
319
|
+
delete queue(:user_created, dest: :mailer, fifo: true)
|
320
|
+
end
|
321
|
+
end
|
322
|
+
```
|
323
|
+
|
324
|
+
To apply migration use:
|
325
|
+
```bash
|
326
|
+
$ rake cyclone_lariat:migrate
|
327
|
+
```
|
328
|
+
|
329
|
+
To decline migration use:
|
330
|
+
```bash
|
331
|
+
$ rake cyclone_lariat:rollback
|
152
332
|
```
|
153
333
|
|
334
|
+
Since the SNS\SQS management does not support an ACID transaction (in the sense of a database),
|
335
|
+
I highly recommend using the atomic schema:
|
336
|
+
|
337
|
+
```ruby
|
338
|
+
# BAD:
|
339
|
+
class UserCreated < CycloneLariat::Migration
|
340
|
+
def up
|
341
|
+
create queue(:user_created, dest: :mailer, fifo: true)
|
342
|
+
create topic(:user_created, fifo: true)
|
343
|
+
|
344
|
+
subscribe topic: topic(:user_created, fifo: true),
|
345
|
+
endpoint: queue(:user_created, dest: :mailer, fifo: true)
|
346
|
+
end
|
347
|
+
|
348
|
+
def down
|
349
|
+
unsubscribe topic: topic(:user_created, fifo: true),
|
350
|
+
endpoint: queue(:user_created, dest: :mailer, fifo: true)
|
351
|
+
|
352
|
+
delete topic(:user_created, fifo: true)
|
353
|
+
delete queue(:user_created, dest: :mailer, fifo: true)
|
354
|
+
end
|
355
|
+
end
|
356
|
+
|
357
|
+
# GOOD:
|
358
|
+
class UserCreatedQueue < CycloneLariat::Migration
|
359
|
+
def up
|
360
|
+
create queue(:user_created, dest: :mailer, fifo: true)
|
361
|
+
end
|
362
|
+
|
363
|
+
def down
|
364
|
+
delete queue(:user_created, dest: :mailer, fifo: true)
|
365
|
+
end
|
366
|
+
end
|
367
|
+
|
368
|
+
class UserCreatedTopic < CycloneLariat::Migration
|
369
|
+
def up
|
370
|
+
create topic(:user_created, fifo: true)
|
371
|
+
end
|
372
|
+
|
373
|
+
def down
|
374
|
+
delete topic(:user_created, fifo: true)
|
375
|
+
end
|
376
|
+
end
|
377
|
+
|
378
|
+
class UserCreatedSubscription < CycloneLariat::Migration
|
379
|
+
def up
|
380
|
+
subscribe topic: topic(:user_created, fifo: true),
|
381
|
+
endpoint: queue(:user_created, dest: :mailer, fifo: true)
|
382
|
+
end
|
383
|
+
|
384
|
+
def down
|
385
|
+
unsubscribe topic: topic(:user_created, fifo: true),
|
386
|
+
endpoint: queue(:user_created, dest: :mailer, fifo: true)
|
387
|
+
end
|
388
|
+
end
|
389
|
+
```
|
390
|
+
|
391
|
+
|
392
|
+
#### Example: one-to-many
|
393
|
+
|
394
|
+
The first example is when your _registration_ service creates new user. You also have two services:
|
395
|
+
_mailer_ - sending a welcome email, and _statistics_ service.
|
396
|
+
|
397
|
+
```ruby
|
398
|
+
create topic(:user_created, fifo: true)
|
399
|
+
create queue(:user_created, dest: :mailer, fifo: true)
|
400
|
+
create queue(:user_created, dest: :stat, fifo: true)
|
401
|
+
|
402
|
+
subscribe topic: topic(:user_created, fifo: true),
|
403
|
+
endpoint: queue(:user_created, dest: :mailer, fifo: true)
|
404
|
+
|
405
|
+
|
406
|
+
subscribe topic: topic(:user_created, fifo: true),
|
407
|
+
endpoint: queue(:user_created, dest: :statistic, fifo: true)
|
408
|
+
```
|
409
|
+

|
410
|
+
|
411
|
+
#### Example: many-to-one
|
412
|
+
|
413
|
+
The second example is when you have three services: _registration_ - creates new users, _order_
|
414
|
+
service - allows you to create new orders, _statistics_ service collects all statistics.
|
415
|
+
|
416
|
+
```ruby
|
417
|
+
create topic(:user_created, fifo: false)
|
418
|
+
create topic(:order_created, fifo: false)
|
419
|
+
create queue(publisher: :any, dest: :statistic, fifo: false)
|
420
|
+
|
421
|
+
subscribe topic: topic(:user_created, fifo: false),
|
422
|
+
endpoint: queue(publisher: :any, dest: :statistic, fifo: false)
|
423
|
+
|
424
|
+
subscribe topic: topic(:order_created, fifo: false),
|
425
|
+
endpoint: queue(publisher: :any, dest: :statistic, fifo: false)
|
426
|
+
```
|
427
|
+

|
428
|
+
|
429
|
+
If queue receives messages from multiple sources you must specify publisher as `:any`. If the
|
430
|
+
subscriber receives messages with different types, `cyclone_lariat` uses a specific keyword - `all`.
|
431
|
+
|
432
|
+
#### Example fanout-to-fanout
|
433
|
+
|
434
|
+
For better organisation you can subscribe topic on topic. For example, you have _management_panel_
|
435
|
+
and _client_panel_ services. Each of these services can register a user with predefined roles.
|
436
|
+
And you want to send this information to the _mailer_ and _statistics_ services.
|
437
|
+
|
438
|
+
```ruby
|
439
|
+
create topic(:client_created, fifo: false)
|
440
|
+
create topic(:manager_created, fifo: false)
|
441
|
+
create topic(:user_created, publisher: :any, fifo: false)
|
442
|
+
create queue(:user_created, publisher: :any, dest: :mailer, fifo: false)
|
443
|
+
create queue(:user_created, publisher: :any, dest: :stat, fifo: false)
|
444
|
+
|
445
|
+
subscribe topic: topic(:user_created, fifo: false),
|
446
|
+
endpoint: topic(:user_created, publisher: :any, fifo: false)
|
447
|
+
|
448
|
+
subscribe topic: topic(:manager_created, fifo: false),
|
449
|
+
endpoint: topic(:user_created, publisher: :any, fifo: false)
|
450
|
+
|
451
|
+
subscribe topic: topic(:user_created, publisher: :any, fifo: false),
|
452
|
+
endpoint: queue(:user_created, publisher: :any, dest: :mailer, fifo: false)
|
453
|
+
|
454
|
+
subscribe topic: topic(:user_created, publisher: :any, fifo: false),
|
455
|
+
endpoint: queue(:user_created, publisher: :any, dest: :stat, fifo: false)
|
456
|
+
```
|
457
|
+
|
458
|
+

|
459
|
+
|
460
|
+
### Create and remove custom Topics and Queues
|
461
|
+
|
462
|
+
You can create Topic and Queues with custom names. That way recommended for:
|
463
|
+
- Remove old resources
|
464
|
+
- Receive messages from external sources
|
465
|
+
|
466
|
+
```ruby
|
467
|
+
create custom_topic('custom_topic_name')
|
468
|
+
delete custom_queue('custom_topic_name')
|
469
|
+
```
|
470
|
+
|
471
|
+
### Where should the migration be?
|
472
|
+
|
473
|
+
We recommend locate migration on:
|
474
|
+
- **topic** - on Publisher side;
|
475
|
+
- **queue** - on Subscriber side;
|
476
|
+
- **subscription** - on Subscriber side.
|
477
|
+
|
478
|
+
# Console tasks
|
479
|
+
|
480
|
+
```bash
|
481
|
+
$ cyclone_lariat install - install cyclone_lariat
|
482
|
+
$ cyclone_lariat generate migration - generate new migration
|
483
|
+
|
484
|
+
$ rake cyclone_lariat:list:queues # List all queues
|
485
|
+
$ rake cyclone_lariat:list:subscriptions # List all subscriptions
|
486
|
+
$ rake cyclone_lariat:list:topics # List all topics
|
487
|
+
$ rake cyclone_lariat:migrate # Migrate topics for SQS/SNS
|
488
|
+
$ rake cyclone_lariat:rollback[version] # Rollback topics for SQS/SNS
|
489
|
+
$ rake cyclone_lariat:graph # Make graph
|
490
|
+
```
|
491
|
+
|
492
|
+
Graph generated in [grpahviz](https://graphviz.org/) format for the entry scheme. You should install
|
493
|
+
it on your system. For convert it in png use:
|
494
|
+
```bash
|
495
|
+
$ rake cyclone_lariat:list:subscriptions | dot -Tpng -o foo.png
|
496
|
+
```
|
497
|
+
|
498
|
+
# Subscriber
|
499
|
+
|
500
|
+
This is gem work like middleware for [shoryuken](https://github.com/ruby-shoryuken/shoryuken). It save all events to
|
501
|
+
database. And catch and produce all exceptions.
|
502
|
+
|
503
|
+
The logic of lariat as a subscriber. Imagine that you are working with an http server. And it gives you various response
|
504
|
+
codes. You have the following processing:
|
505
|
+
|
506
|
+
- 2xx - success, we process the page.
|
507
|
+
- 4хх - Logic error send the error to the developer and wait until he fixes it
|
508
|
+
- 5xx - Send an error and try again
|
509
|
+
|
510
|
+
|
511
|
+

|
154
512
|
|
155
513
|
# Middleware
|
156
514
|
If you use middleware:
|
157
515
|
- Store all events to dataset
|
158
516
|
- Notify every input sqs message
|
159
|
-
- Notify every error
|
517
|
+
- Notify every error
|
160
518
|
|
161
519
|
```ruby
|
162
520
|
require 'cyclone_lariat/middleware' # If require: false in Gemfile
|
163
|
-
|
521
|
+
require 'cyclone_lariat/sqs_client' # If you want use queue name helper
|
164
522
|
|
165
523
|
class Receiver
|
166
524
|
include Shoryuken::Worker
|
167
|
-
|
525
|
+
|
168
526
|
DB = Sequel.connect(host: 'localhost', user: 'ruby')
|
169
527
|
|
170
528
|
shoryuken_options auto_delete: true,
|
171
529
|
body_parser: ->(sqs_msg) {
|
172
530
|
JSON.parse(sqs_msg.body, symbolize_names: true)
|
173
531
|
},
|
174
|
-
queue: 'your_sqs_queue_name'
|
532
|
+
queue: 'your_sqs_queue_name.fifo'
|
533
|
+
# or
|
534
|
+
# queue: CycloneLariat::SqsClient.new.queue('user_added', fifo: true).name
|
175
535
|
|
176
536
|
server_middleware do |chain|
|
177
537
|
# Options dataset, errors_notifier and message_notifier is optionals.
|
178
538
|
# If you dont define notifiers - middleware does not notify
|
179
|
-
# If you dont define dataset - middleware does store events in db
|
539
|
+
# If you dont define dataset - middleware does not store events in db
|
180
540
|
chain.add CycloneLariat::Middleware,
|
181
541
|
dataset: DB[:events],
|
182
542
|
errors_notifier: LunaPark::Notifiers::Sentry.new,
|
183
543
|
message_notifier: LunaPark::Notifiers::Log.new(min_lvl: :debug, format: :pretty_json)
|
184
544
|
end
|
185
545
|
|
546
|
+
class UserIsNotRegistered < LunaPark::Errors::Business
|
547
|
+
end
|
548
|
+
|
186
549
|
def perform(sqs_message, sqs_message_body)
|
187
550
|
# Your logic here
|
551
|
+
|
552
|
+
# If you want to raise business error
|
553
|
+
raise UserIsNotRegistered.new(first_name: 'John', last_name: 'Doe')
|
188
554
|
end
|
189
555
|
end
|
190
556
|
```
|
191
557
|
|
192
558
|
## Migrations
|
193
|
-
Before
|
559
|
+
Before using the event store, add and apply these two migrations:
|
194
560
|
|
195
561
|
```ruby
|
196
562
|
|
@@ -229,9 +595,18 @@ Sequel.migration do
|
|
229
595
|
end
|
230
596
|
```
|
231
597
|
|
598
|
+
And don't forget to add it to the config file:
|
599
|
+
|
600
|
+
```ruby
|
601
|
+
# 'config/initializers/cyclone_lariat.rb'
|
602
|
+
CycloneLariat.tap do |cl|
|
603
|
+
cl.events_dataset = DB[:async_messages]
|
604
|
+
end
|
605
|
+
```
|
606
|
+
|
232
607
|
### Rake tasks
|
233
608
|
|
234
|
-
For simplify write some Rake tasks you can use CycloneLariat::Repo
|
609
|
+
For simplify write some Rake tasks you can use `CycloneLariat::Repo`.
|
235
610
|
|
236
611
|
```ruby
|
237
612
|
# For retry all unprocessed
|
data/Rakefile
CHANGED
@@ -7,9 +7,6 @@ require 'rspec/core/rake_task'
|
|
7
7
|
RSpec::Core::RakeTask.new(:spec)
|
8
8
|
|
9
9
|
# tasks from lib directory
|
10
|
-
|
11
|
-
print "#{entity} : "
|
12
|
-
puts load entity
|
13
|
-
end
|
10
|
+
Rake.add_rakelib 'lib/tasks'
|
14
11
|
|
15
|
-
task default: %i[spec]
|
12
|
+
task default: %i[spec]
|