cyclone_lariat 0.3.10 → 0.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitignore +6 -0
- data/.rubocop.yml +26 -1
- data/CHANGELOG.md +11 -1
- data/Gemfile.lock +26 -21
- data/README.md +466 -91
- data/Rakefile +2 -5
- data/bin/cyclone_lariat +174 -0
- data/config/initializers/sequel.rb +7 -0
- data/cyclone_lariat.gemspec +4 -0
- data/db/migrate/03_add_versions.rb +9 -0
- data/docs/_imgs/graphviz_01.png +0 -0
- data/docs/_imgs/graphviz_02.png +0 -0
- data/docs/_imgs/graphviz_03.png +0 -0
- data/docs/_imgs/logic.png +0 -0
- data/docs/_imgs/sqs_sns_diagram.png +0 -0
- data/lib/cyclone_lariat/abstract/client.rb +20 -14
- data/lib/cyclone_lariat/abstract/message.rb +20 -5
- data/lib/cyclone_lariat/configure.rb +1 -1
- data/lib/cyclone_lariat/errors.rb +22 -0
- data/lib/cyclone_lariat/middleware.rb +2 -1
- data/lib/cyclone_lariat/migration.rb +214 -0
- data/lib/cyclone_lariat/queue.rb +147 -0
- data/lib/cyclone_lariat/sns_client.rb +125 -14
- data/lib/cyclone_lariat/sqs_client.rb +69 -15
- data/lib/cyclone_lariat/topic.rb +113 -0
- data/lib/cyclone_lariat/version.rb +1 -1
- data/lib/cyclone_lariat.rb +1 -0
- data/lib/tasks/console.rake +13 -0
- data/lib/tasks/cyclone_lariat.rake +44 -0
- data/lib/tasks/db.rake +1 -1
- metadata +45 -4
- data/docs/_imgs/diagram.png +0 -0
data/README.md
CHANGED
@@ -1,85 +1,142 @@
|
|
1
1
|
# Cyclone lariat
|
2
2
|
|
3
|
-
This
|
3
|
+
This gem work in few scenarios:
|
4
|
+
- As middleware for [shoryuken](https://github.com/ruby-shoryuken/shoryuken). It saves all events to the database and also catches and throws all exceptions.
|
5
|
+
- As a middleware, it can log all incoming messages.
|
6
|
+
- As a client that can send messages to SNS topics and SQS queues.
|
7
|
+
- Also it can help you with CI\CD for theme, queue and subscription management like database migration.
|
4
8
|
|
5
9
|
![Cyclone lariat](docs/_imgs/lariat.jpg)
|
6
10
|
|
7
11
|
|
12
|
+
# Client / Publisher
|
13
|
+
At first lets understand what the difference between SQS and SNS:
|
14
|
+
- Amazon Simple Queue Service (SQS) lets you send, store, and receive messages between software components at any
|
15
|
+
volume, without losing messages or requiring other services to be available.
|
16
|
+
- Amazon Simple Notification Service (SNS) sends notifications two ways Application2Person (like send sms).
|
17
|
+
And the second way is Application2Application, that's way more important for us. In this way you case use
|
18
|
+
SNS service like fanout.
|
19
|
+
|
20
|
+
![SQS/SNS](docs/_imgs/sqs_sns_diagram.png)
|
21
|
+
|
22
|
+
For use **cyclone_lariat** as _Publisher_ lets make install CycloneLariat.
|
23
|
+
|
24
|
+
## Install cyclone_lariat
|
25
|
+
Edit Gemfile:
|
8
26
|
```ruby
|
9
27
|
# Gemfile
|
28
|
+
gem 'sequel'
|
29
|
+
gem 'cyclone_lariat'
|
30
|
+
```
|
31
|
+
And run in console:
|
32
|
+
```bash
|
33
|
+
$ bundle install
|
34
|
+
$ cyclone_lariat install
|
35
|
+
```
|
10
36
|
|
11
|
-
|
12
|
-
|
37
|
+
Last command will create 2 files:
|
38
|
+
- ./lib/tasks/cyclone_lariat.rake - Rake tasks, for management migrations
|
39
|
+
- ./config/initializers/cyclone_lariat.rb - Configuration default values for cyclone lariat usage
|
13
40
|
|
14
|
-
|
15
|
-
|
41
|
+
## Configuration
|
42
|
+
```ruby
|
43
|
+
# frozen_string_literal: true
|
44
|
+
|
45
|
+
CycloneLariat.tap do |cl|
|
46
|
+
cl.default_version = 1 # api version
|
47
|
+
cl.aws_key = ENV['AWS_KEY'] # aws key
|
48
|
+
cl.aws_account_id = ENV['AWS_ACCOUNT_ID'] # aws account id
|
49
|
+
cl.aws_secret_key = ENV['AWS_SECRET_KEY'] # aws secret
|
50
|
+
cl.aws_default_region = ENV['AWS_REGION'] # aws default region
|
51
|
+
cl.publisher = ENV['APP_NAME'] # name of your publishers, usually name of your application
|
52
|
+
cl.default_instance = ENV['INSTANCE'] # stage, production, test
|
53
|
+
cl.events_dataset = DB[:events] # sequel dataset for store income messages, for receiver
|
54
|
+
cl.versions_dataset = DB[:lariat_versions] # sequel dataset for migrations, for publisher
|
55
|
+
end
|
56
|
+
```
|
57
|
+
If you are only using your application as a publisher, you may not need to set the `events_dataset`
|
58
|
+
parameter.
|
59
|
+
|
60
|
+
Before creating the first migration, let's explain what `CycloneLariat::Message` is.
|
61
|
+
|
62
|
+
## Messages
|
63
|
+
Message in Amazon SQS\SNS service it's a
|
64
|
+
[object](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-attributes)
|
65
|
+
that has several attributes. The main attributes are the **body**, which consists of the published
|
66
|
+
data. The body is a _String_, but we can use it as a _JSON_ object. **Cyclone_lariat** use this scheme:
|
67
|
+
|
68
|
+
```json
|
69
|
+
{
|
70
|
+
"uuid": "f2ce3813-0905-4d81-a60e-f289f2431f50", // Uniq message identificator
|
71
|
+
"publisher": "sample_app", // Publisher application name
|
72
|
+
"request_id": "51285005-8a06-4181-b5fd-bf29f3b1a45a", // Optional: X-Request-Id
|
73
|
+
"type": "event_note_created", // Type of Event or Command
|
74
|
+
"version": 1, // Version of data structure
|
75
|
+
"data": {
|
76
|
+
"id": 12,
|
77
|
+
"text": "Sample of published data",
|
78
|
+
"attributes": ["one", "two", "three"]
|
79
|
+
},
|
80
|
+
"sent_at": "2022-11-09T11:42:18.203+01:00" // Time when message was sended in ISO8601 Standard
|
81
|
+
}
|
16
82
|
```
|
17
83
|
|
18
|
-
|
84
|
+
Idea about X-Request-Id you can see at
|
85
|
+
[StackOverflow](https://stackoverflow.com/questions/25433258/what-is-the-x-request-id-http-header).
|
86
|
+
|
87
|
+
As you see, type has prefix 'event_' in cyclone lariat you has two kinds of messages - _Event_ and _Command_.
|
19
88
|
|
20
|
-
|
89
|
+
### Command vs Event
|
90
|
+
Commands and events are both simple domain structures that contain solely data for reading. That means
|
91
|
+
they contain no behaviour or business logic.
|
21
92
|
|
22
|
-
|
23
|
-
|
24
|
-
|
93
|
+
A command is an object that is sent to the domain for a state change which is handled by a command
|
94
|
+
handler. They should be named with a verb in an imperative mood plus the aggregate name which it
|
95
|
+
operates on. Such request can be rejected due to the data the command holds being invalid/inconsistent.
|
96
|
+
There should be exactly 1 handler for each command. Once the command has been executed, the consumer
|
97
|
+
can then carry out whatever the task is depending on the output of the command.
|
25
98
|
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
99
|
+
An event is a statement of fact about what change has been made to the domain state. They are named
|
100
|
+
with the aggregate name where the change took place plus the verb past-participle. An event happens off
|
101
|
+
the back of a command.
|
102
|
+
A command can emit any number of events. The sender of the event does not care who receives it or
|
103
|
+
whether it has been received at all.
|
31
104
|
|
32
|
-
|
33
|
-
|
34
|
-
A command can emit any number of events. The sender of the event does not care who receives it or whether it has been
|
35
|
-
received at all.
|
105
|
+
### Publish
|
106
|
+
For publishing _Event_ or _Commands_, you have two ways, send _Message_ directly:
|
36
107
|
|
37
|
-
## Configure
|
38
108
|
```ruby
|
39
|
-
# 'config/initializers/cyclone_lariat.rb'
|
40
109
|
CycloneLariat.tap do |cl|
|
41
|
-
|
42
|
-
cl.aws_key = # aws key
|
43
|
-
cl.aws_secret_key = # aws secret
|
44
|
-
cl.aws_client_id = # aws client id
|
45
|
-
cl.aws_default_region = # aws default region
|
46
|
-
cl.publisher = 'auth' # name of your publishers, usually name of your application
|
47
|
-
cl.default_instance = APP_INSTANCE # stage, production, test
|
110
|
+
# Config app here
|
48
111
|
end
|
49
|
-
```
|
50
112
|
|
51
|
-
|
52
|
-
You can use client directly
|
113
|
+
client = CycloneLariat::SnsClient.new(instance: 'auth', version: 2)
|
53
114
|
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
secret_key: APP_CONF.aws.secret_key,
|
60
|
-
region: APP_CONF.aws.region,
|
61
|
-
version: 1, # at default 1
|
62
|
-
publisher: 'pilot',
|
63
|
-
instance: INSTANCE # at default :prod
|
115
|
+
client.publish_command('register_user', data: {
|
116
|
+
first_name: 'John',
|
117
|
+
last_name: 'Doe',
|
118
|
+
mail: 'john.doe@example.com'
|
119
|
+
}, fifo: false
|
64
120
|
)
|
65
121
|
```
|
66
122
|
|
67
|
-
|
68
|
-
```
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
123
|
+
That's call, will generate a message body:
|
124
|
+
```json
|
125
|
+
{
|
126
|
+
"uuid": "f2ce3813-0905-4d81-a60e-f289f2431f50",
|
127
|
+
"publisher": "auth",
|
128
|
+
"type": "command_register_user",
|
129
|
+
"version": 2,
|
130
|
+
"data": {
|
131
|
+
"first_name": "John",
|
132
|
+
"last_name": "Doe",
|
133
|
+
"mail": "john.doe@example.com"
|
134
|
+
},
|
135
|
+
"sent_at": "2022-11-09T11:42:18.203+01:00" // Time when message was sended in ISO8601 Standard
|
136
|
+
}
|
80
137
|
```
|
81
138
|
|
82
|
-
Or
|
139
|
+
Or is it better to make your own client, like a [Repository](https://deviq.com/design-patterns/repository-pattern) pattern.
|
83
140
|
|
84
141
|
```ruby
|
85
142
|
require 'cyclone_lariat/sns_client' # If require: false in Gemfile
|
@@ -90,30 +147,21 @@ class YourClient < CycloneLariat::SnsClient
|
|
90
147
|
instance 'stage'
|
91
148
|
|
92
149
|
def email_is_created(mail)
|
93
|
-
publish event('email_is_created',
|
94
|
-
data: { mail: mail }
|
95
|
-
),
|
96
|
-
to: APP_CONF.aws.fanout.emails
|
150
|
+
publish event('email_is_created', data: { mail: mail }), fifo: true
|
97
151
|
end
|
98
152
|
|
99
153
|
def email_is_removed(mail)
|
100
|
-
publish event('email_is_removed',
|
101
|
-
data: { mail: mail }
|
102
|
-
),
|
103
|
-
to: APP_CONF.aws.fanout.email
|
154
|
+
publish event('email_is_removed', data: { mail: mail }), fifo: true
|
104
155
|
end
|
105
156
|
|
106
157
|
|
107
158
|
def delete_user(mail)
|
108
|
-
publish command('delete_user',
|
109
|
-
data: { mail: mail }
|
110
|
-
),
|
111
|
-
to: APP_CONF.aws.fanout.email
|
159
|
+
publish command('delete_user', data: { mail: mail }), fifo: false
|
112
160
|
end
|
113
161
|
end
|
114
162
|
|
115
163
|
# Init repo
|
116
|
-
client = YourClient.new
|
164
|
+
client = YourClient.new
|
117
165
|
|
118
166
|
# And send topics
|
119
167
|
client.email_is_created 'john.doe@example.com'
|
@@ -121,76 +169,394 @@ client.email_is_removed 'john.doe@example.com'
|
|
121
169
|
client.delete_user 'john.doe@example.com'
|
122
170
|
```
|
123
171
|
|
172
|
+
### Topics and Queue
|
173
|
+
An Amazon SNS topic and SQS queue is a logical access point that acts as a communication channel. Both
|
174
|
+
of them has specific address ARN.
|
175
|
+
|
176
|
+
```
|
177
|
+
# Topic example
|
178
|
+
arn:aws:sns:eu-west-1:247602342345:test-event-fanout-cyclone_lariat-note_added.fifo
|
179
|
+
|
180
|
+
# Queue example
|
181
|
+
arn:aws:sqs:eu-west-1:247602342345:test-event-queue-cyclone_lariat-note_added-notifier.fifo
|
182
|
+
```
|
183
|
+
|
184
|
+
Split ARN:
|
185
|
+
- `arn:aws:sns` - Prefix for SNS Topics
|
186
|
+
- `arn:aws:sqs` - Prefix for SQS Queues
|
187
|
+
- `eu-west-1` -
|
188
|
+
[AWS Region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions)
|
189
|
+
- `247602342345` - [AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html)
|
190
|
+
- `test-event-fanout-cyclone_lariat-note_added` - Topic \ Queue name
|
191
|
+
- `.fifo` - if Topic or queue is
|
192
|
+
[FIFO](https://aws.amazon.com/blogs/aws/introducing-amazon-sns-fifo-first-in-first-out-pub-sub-messaging/), they must
|
193
|
+
has that suffix.
|
194
|
+
|
195
|
+
Region and client_id usually set using the **cyclone_lariat** [configuration](#Configuration).
|
196
|
+
|
197
|
+
## Declaration for topic and queues name
|
198
|
+
In **cyclone_lariat** we have a declaration for defining topic and queue names.
|
199
|
+
This can help in organizing the order.
|
200
|
+
|
201
|
+
```ruby
|
202
|
+
CycloneLariat.tap do |cfg|
|
203
|
+
cfg.instance = 'test'
|
204
|
+
cfg.publisher = 'cyclone_lariat'
|
205
|
+
# ...
|
206
|
+
end
|
207
|
+
|
208
|
+
CycloneLariat::SnsClient.new.publish_command('register_user', data: {
|
209
|
+
first_name: 'John',
|
210
|
+
last_name: 'Doe',
|
211
|
+
mail: 'john.doe@example.com'
|
212
|
+
}, fifo: true
|
213
|
+
)
|
214
|
+
|
215
|
+
# or in repository-like style:
|
216
|
+
|
217
|
+
class YourClient < CycloneLariat::SnsClient
|
218
|
+
def register_user(first:, last:, mail:)
|
219
|
+
publish command('register_user', data: { mail: mail }), fifo: true
|
220
|
+
end
|
221
|
+
end
|
222
|
+
```
|
223
|
+
|
224
|
+
|
225
|
+
We will publish a message on this topic: `test-command-fanout-cyclone_lariat-register_user`.
|
124
226
|
|
125
|
-
|
126
|
-
|
227
|
+
Let's split the topic title:
|
228
|
+
- `test` - instance;
|
229
|
+
- `command` - kind - [event or command](#command-vs-event);
|
230
|
+
- `fanount` - resource type - fanout for SNS topics;
|
231
|
+
- `cyclone_lariat` - publisher name;
|
232
|
+
- `regiser_user` - message type.
|
127
233
|
|
234
|
+
For queues you also can define destination.
|
128
235
|
```ruby
|
129
|
-
|
236
|
+
CycloneLariat::SqsClient.new.publish_event(
|
237
|
+
'register_user', data: { mail: 'john.doe@example.com' },
|
238
|
+
dest: :mailer, fifo: true
|
239
|
+
)
|
240
|
+
|
241
|
+
|
242
|
+
# or in repository-like style:
|
243
|
+
|
244
|
+
class YourClient < CycloneLariat::SnsClient
|
245
|
+
# ...
|
246
|
+
|
247
|
+
def register_user(first:, last:, mail:)
|
248
|
+
publish event('register_user', data: { mail: mail }), fifo: true
|
249
|
+
end
|
250
|
+
end
|
251
|
+
```
|
252
|
+
|
253
|
+
We will publish a message on this queue: `test-event-queue-cyclone_lariat-register_user-mailer`.
|
130
254
|
|
131
|
-
|
132
|
-
|
133
|
-
|
134
|
-
|
135
|
-
|
136
|
-
|
137
|
-
|
255
|
+
Let's split the queue title:
|
256
|
+
- `test` - instance;
|
257
|
+
- `event` - kind - [event or command](#command-vs-event);
|
258
|
+
- `queue` - resource type - queue for SQS;
|
259
|
+
- `cyclone_lariat` - publisher name;
|
260
|
+
- `regiser_user` - message type.
|
261
|
+
- `mailer` - destination
|
262
|
+
|
263
|
+
You also can sent message to queue with custom name. But this way does not recommended.
|
264
|
+
|
265
|
+
```ruby
|
266
|
+
# Directly
|
267
|
+
CycloneLariat::SqsClient.new.publish_event(
|
268
|
+
'register_user', data: { mail: 'john.doe@example.com' },
|
269
|
+
dest: :mailer, topic: 'custom_topic_name.fifo', fifo: true
|
138
270
|
)
|
271
|
+
|
272
|
+
# Repository
|
273
|
+
class YourClient < CycloneLariat::SnsClient
|
274
|
+
# ...
|
275
|
+
|
276
|
+
def register_user(first:, last:, mail:)
|
277
|
+
publish event('register_user', data: { mail: mail }),
|
278
|
+
topic: 'custom_topic_name.fifo', fifo: true
|
279
|
+
end
|
280
|
+
end
|
139
281
|
```
|
282
|
+
Will publish message on queue: `custom_topic` with fifo suffix.
|
140
283
|
|
141
|
-
|
142
|
-
|
284
|
+
# Migrations
|
285
|
+
|
286
|
+
With **cyclone_lariat** you can use migrations that can create, delete, and subscribe to your queues and topics, just like database migrations do.
|
287
|
+
To store versions of **cyclone_lariat** migrations, you need to create a database table.
|
143
288
|
|
144
289
|
```ruby
|
145
|
-
|
146
|
-
|
290
|
+
# frozen_string_literal: true
|
291
|
+
|
292
|
+
Sequel.migration do
|
293
|
+
change do
|
294
|
+
create_table :lariat_versions do
|
295
|
+
Integer :version, null: false, unique: true
|
296
|
+
end
|
297
|
+
end
|
298
|
+
end
|
147
299
|
```
|
300
|
+
After migrate this database migration, create **cyclone_lariat** migration.
|
301
|
+
|
302
|
+
```bash
|
303
|
+
$ cyclone_lariat generate migration user_created
|
304
|
+
```
|
305
|
+
|
306
|
+
This command should create a migration file, let's edit it.
|
148
307
|
|
149
|
-
Or you can define topic directly:
|
150
308
|
```ruby
|
151
|
-
|
309
|
+
# ./lariat/migrate/1668097991_user_created_queue.rb
|
310
|
+
|
311
|
+
# frozen_string_literal: true
|
312
|
+
|
313
|
+
class UserCreatedQueue < CycloneLariat::Migration
|
314
|
+
def up
|
315
|
+
create queue(:user_created, dest: :mailer, fifo: true)
|
316
|
+
end
|
317
|
+
|
318
|
+
def down
|
319
|
+
delete queue(:user_created, dest: :mailer, fifo: true)
|
320
|
+
end
|
321
|
+
end
|
322
|
+
```
|
323
|
+
|
324
|
+
To apply migration use:
|
325
|
+
```bash
|
326
|
+
$ rake cyclone_lariat:migrate
|
327
|
+
```
|
328
|
+
|
329
|
+
To decline migration use:
|
330
|
+
```bash
|
331
|
+
$ rake cyclone_lariat:rollback
|
152
332
|
```
|
153
333
|
|
334
|
+
Since the SNS\SQS management does not support an ACID transaction (in the sense of a database),
|
335
|
+
I highly recommend using the atomic schema:
|
336
|
+
|
337
|
+
```ruby
|
338
|
+
# BAD:
|
339
|
+
class UserCreated < CycloneLariat::Migration
|
340
|
+
def up
|
341
|
+
create queue(:user_created, dest: :mailer, fifo: true)
|
342
|
+
create topic(:user_created, fifo: true)
|
343
|
+
|
344
|
+
subscribe topic: topic(:user_created, fifo: true),
|
345
|
+
endpoint: queue(:user_created, dest: :mailer, fifo: true)
|
346
|
+
end
|
347
|
+
|
348
|
+
def down
|
349
|
+
unsubscribe topic: topic(:user_created, fifo: true),
|
350
|
+
endpoint: queue(:user_created, dest: :mailer, fifo: true)
|
351
|
+
|
352
|
+
delete topic(:user_created, fifo: true)
|
353
|
+
delete queue(:user_created, dest: :mailer, fifo: true)
|
354
|
+
end
|
355
|
+
end
|
356
|
+
|
357
|
+
# GOOD:
|
358
|
+
class UserCreatedQueue < CycloneLariat::Migration
|
359
|
+
def up
|
360
|
+
create queue(:user_created, dest: :mailer, fifo: true)
|
361
|
+
end
|
362
|
+
|
363
|
+
def down
|
364
|
+
delete queue(:user_created, dest: :mailer, fifo: true)
|
365
|
+
end
|
366
|
+
end
|
367
|
+
|
368
|
+
class UserCreatedTopic < CycloneLariat::Migration
|
369
|
+
def up
|
370
|
+
create topic(:user_created, fifo: true)
|
371
|
+
end
|
372
|
+
|
373
|
+
def down
|
374
|
+
delete topic(:user_created, fifo: true)
|
375
|
+
end
|
376
|
+
end
|
377
|
+
|
378
|
+
class UserCreatedSubscription < CycloneLariat::Migration
|
379
|
+
def up
|
380
|
+
subscribe topic: topic(:user_created, fifo: true),
|
381
|
+
endpoint: queue(:user_created, dest: :mailer, fifo: true)
|
382
|
+
end
|
383
|
+
|
384
|
+
def down
|
385
|
+
unsubscribe topic: topic(:user_created, fifo: true),
|
386
|
+
endpoint: queue(:user_created, dest: :mailer, fifo: true)
|
387
|
+
end
|
388
|
+
end
|
389
|
+
```
|
390
|
+
|
391
|
+
|
392
|
+
#### Example: one-to-many
|
393
|
+
|
394
|
+
The first example is when your _registration_ service creates new user. You also have two services:
|
395
|
+
_mailer_ - sending a welcome email, and _statistics_ service.
|
396
|
+
|
397
|
+
```ruby
|
398
|
+
create topic(:user_created, fifo: true)
|
399
|
+
create queue(:user_created, dest: :mailer, fifo: true)
|
400
|
+
create queue(:user_created, dest: :stat, fifo: true)
|
401
|
+
|
402
|
+
subscribe topic: topic(:user_created, fifo: true),
|
403
|
+
endpoint: queue(:user_created, dest: :mailer, fifo: true)
|
404
|
+
|
405
|
+
|
406
|
+
subscribe topic: topic(:user_created, fifo: true),
|
407
|
+
endpoint: queue(:user_created, dest: :statistic, fifo: true)
|
408
|
+
```
|
409
|
+
![one2many](docs/_imgs/graphviz_01.png)
|
410
|
+
|
411
|
+
#### Example: many-to-one
|
412
|
+
|
413
|
+
The second example is when you have three services: _registration_ - creates new users, _order_
|
414
|
+
service - allows you to create new orders, _statistics_ service collects all statistics.
|
415
|
+
|
416
|
+
```ruby
|
417
|
+
create topic(:user_created, fifo: false)
|
418
|
+
create topic(:order_created, fifo: false)
|
419
|
+
create queue(publisher: :any, dest: :statistic, fifo: false)
|
420
|
+
|
421
|
+
subscribe topic: topic(:user_created, fifo: false),
|
422
|
+
endpoint: queue(publisher: :any, dest: :statistic, fifo: false)
|
423
|
+
|
424
|
+
subscribe topic: topic(:order_created, fifo: false),
|
425
|
+
endpoint: queue(publisher: :any, dest: :statistic, fifo: false)
|
426
|
+
```
|
427
|
+
![one2many](docs/_imgs/graphviz_02.png)
|
428
|
+
|
429
|
+
If queue receives messages from multiple sources you must specify publisher as `:any`. If the
|
430
|
+
subscriber receives messages with different types, `cyclone_lariat` uses a specific keyword - `all`.
|
431
|
+
|
432
|
+
#### Example fanout-to-fanout
|
433
|
+
|
434
|
+
For better organisation you can subscribe topic on topic. For example, you have _management_panel_
|
435
|
+
and _client_panel_ services. Each of these services can register a user with predefined roles.
|
436
|
+
And you want to send this information to the _mailer_ and _statistics_ services.
|
437
|
+
|
438
|
+
```ruby
|
439
|
+
create topic(:client_created, fifo: false)
|
440
|
+
create topic(:manager_created, fifo: false)
|
441
|
+
create topic(:user_created, publisher: :any, fifo: false)
|
442
|
+
create queue(:user_created, publisher: :any, dest: :mailer, fifo: false)
|
443
|
+
create queue(:user_created, publisher: :any, dest: :stat, fifo: false)
|
444
|
+
|
445
|
+
subscribe topic: topic(:user_created, fifo: false),
|
446
|
+
endpoint: topic(:user_created, publisher: :any, fifo: false)
|
447
|
+
|
448
|
+
subscribe topic: topic(:manager_created, fifo: false),
|
449
|
+
endpoint: topic(:user_created, publisher: :any, fifo: false)
|
450
|
+
|
451
|
+
subscribe topic: topic(:user_created, publisher: :any, fifo: false),
|
452
|
+
endpoint: queue(:user_created, publisher: :any, dest: :mailer, fifo: false)
|
453
|
+
|
454
|
+
subscribe topic: topic(:user_created, publisher: :any, fifo: false),
|
455
|
+
endpoint: queue(:user_created, publisher: :any, dest: :stat, fifo: false)
|
456
|
+
```
|
457
|
+
|
458
|
+
![one2many](docs/_imgs/graphviz_03.png)
|
459
|
+
|
460
|
+
### Create and remove custom Topics and Queues
|
461
|
+
|
462
|
+
You can create Topic and Queues with custom names. That way recommended for:
|
463
|
+
- Remove old resources
|
464
|
+
- Receive messages from external sources
|
465
|
+
|
466
|
+
```ruby
|
467
|
+
create custom_topic('custom_topic_name')
|
468
|
+
delete custom_queue('custom_topic_name')
|
469
|
+
```
|
470
|
+
|
471
|
+
### Where should the migration be?
|
472
|
+
|
473
|
+
We recommend locate migration on:
|
474
|
+
- **topic** - on Publisher side;
|
475
|
+
- **queue** - on Subscriber side;
|
476
|
+
- **subscription** - on Subscriber side.
|
477
|
+
|
478
|
+
# Console tasks
|
479
|
+
|
480
|
+
```bash
|
481
|
+
$ cyclone_lariat install - install cyclone_lariat
|
482
|
+
$ cyclone_lariat generate migration - generate new migration
|
483
|
+
|
484
|
+
$ rake cyclone_lariat:list:queues # List all queues
|
485
|
+
$ rake cyclone_lariat:list:subscriptions # List all subscriptions
|
486
|
+
$ rake cyclone_lariat:list:topics # List all topics
|
487
|
+
$ rake cyclone_lariat:migrate # Migrate topics for SQS/SNS
|
488
|
+
$ rake cyclone_lariat:rollback[version] # Rollback topics for SQS/SNS
|
489
|
+
$ rake cyclone_lariat:graph # Make graph
|
490
|
+
```
|
491
|
+
|
492
|
+
Graph generated in [grpahviz](https://graphviz.org/) format for the entry scheme. You should install
|
493
|
+
it on your system. For convert it in png use:
|
494
|
+
```bash
|
495
|
+
$ rake cyclone_lariat:list:subscriptions | dot -Tpng -o foo.png
|
496
|
+
```
|
497
|
+
|
498
|
+
# Subscriber
|
499
|
+
|
500
|
+
This is gem work like middleware for [shoryuken](https://github.com/ruby-shoryuken/shoryuken). It save all events to
|
501
|
+
database. And catch and produce all exceptions.
|
502
|
+
|
503
|
+
The logic of lariat as a subscriber. Imagine that you are working with an http server. And it gives you various response
|
504
|
+
codes. You have the following processing:
|
505
|
+
|
506
|
+
- 2xx - success, we process the page.
|
507
|
+
- 4хх - Logic error send the error to the developer and wait until he fixes it
|
508
|
+
- 5xx - Send an error and try again
|
509
|
+
|
510
|
+
|
511
|
+
![diagram](docs/_imgs/logic.png)
|
154
512
|
|
155
513
|
# Middleware
|
156
514
|
If you use middleware:
|
157
515
|
- Store all events to dataset
|
158
516
|
- Notify every input sqs message
|
159
|
-
- Notify every error
|
517
|
+
- Notify every error
|
160
518
|
|
161
519
|
```ruby
|
162
520
|
require 'cyclone_lariat/middleware' # If require: false in Gemfile
|
163
|
-
|
521
|
+
require 'cyclone_lariat/sqs_client' # If you want use queue name helper
|
164
522
|
|
165
523
|
class Receiver
|
166
524
|
include Shoryuken::Worker
|
167
|
-
|
525
|
+
|
168
526
|
DB = Sequel.connect(host: 'localhost', user: 'ruby')
|
169
527
|
|
170
528
|
shoryuken_options auto_delete: true,
|
171
529
|
body_parser: ->(sqs_msg) {
|
172
530
|
JSON.parse(sqs_msg.body, symbolize_names: true)
|
173
531
|
},
|
174
|
-
queue: 'your_sqs_queue_name'
|
532
|
+
queue: 'your_sqs_queue_name.fifo'
|
533
|
+
# or
|
534
|
+
# queue: CycloneLariat::SqsClient.new.queue('user_added', fifo: true).name
|
175
535
|
|
176
536
|
server_middleware do |chain|
|
177
537
|
# Options dataset, errors_notifier and message_notifier is optionals.
|
178
538
|
# If you dont define notifiers - middleware does not notify
|
179
|
-
# If you dont define dataset - middleware does store events in db
|
539
|
+
# If you dont define dataset - middleware does not store events in db
|
180
540
|
chain.add CycloneLariat::Middleware,
|
181
541
|
dataset: DB[:events],
|
182
542
|
errors_notifier: LunaPark::Notifiers::Sentry.new,
|
183
543
|
message_notifier: LunaPark::Notifiers::Log.new(min_lvl: :debug, format: :pretty_json)
|
184
544
|
end
|
185
545
|
|
546
|
+
class UserIsNotRegistered < LunaPark::Errors::Business
|
547
|
+
end
|
548
|
+
|
186
549
|
def perform(sqs_message, sqs_message_body)
|
187
550
|
# Your logic here
|
551
|
+
|
552
|
+
# If you want to raise business error
|
553
|
+
raise UserIsNotRegistered.new(first_name: 'John', last_name: 'Doe')
|
188
554
|
end
|
189
555
|
end
|
190
556
|
```
|
191
557
|
|
192
558
|
## Migrations
|
193
|
-
Before
|
559
|
+
Before using the event store, add and apply these two migrations:
|
194
560
|
|
195
561
|
```ruby
|
196
562
|
|
@@ -229,9 +595,18 @@ Sequel.migration do
|
|
229
595
|
end
|
230
596
|
```
|
231
597
|
|
598
|
+
And don't forget to add it to the config file:
|
599
|
+
|
600
|
+
```ruby
|
601
|
+
# 'config/initializers/cyclone_lariat.rb'
|
602
|
+
CycloneLariat.tap do |cl|
|
603
|
+
cl.events_dataset = DB[:async_messages]
|
604
|
+
end
|
605
|
+
```
|
606
|
+
|
232
607
|
### Rake tasks
|
233
608
|
|
234
|
-
For simplify write some Rake tasks you can use CycloneLariat::Repo
|
609
|
+
For simplify write some Rake tasks you can use `CycloneLariat::Repo`.
|
235
610
|
|
236
611
|
```ruby
|
237
612
|
# For retry all unprocessed
|
data/Rakefile
CHANGED
@@ -7,9 +7,6 @@ require 'rspec/core/rake_task'
|
|
7
7
|
RSpec::Core::RakeTask.new(:spec)
|
8
8
|
|
9
9
|
# tasks from lib directory
|
10
|
-
|
11
|
-
print "#{entity} : "
|
12
|
-
puts load entity
|
13
|
-
end
|
10
|
+
Rake.add_rakelib 'lib/tasks'
|
14
11
|
|
15
|
-
task default: %i[spec]
|
12
|
+
task default: %i[spec]
|