tobox 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: ecf0e9576a8a59834eb6b84eed5c648a2fda3e50b4be49fba068f2bcfc785978
4
- data.tar.gz: fb8dc2819591dfe63a177dc1f27b9a947712db00bf451a8f3633d305ee12172b
3
+ metadata.gz: 33a40c07b2d7f36933540d390287aeeb7d4e0aa39a7d19a439eaca6a2f8fc23f
4
+ data.tar.gz: 939699d801afdb03ed85c0048eee3b00c50a786a345b0731c6dba910fafb51ad
5
5
  SHA512:
6
- metadata.gz: 4b0d7eae10175ce51332e32ad89613196f7b56d01f54670bd78a36af0d411081fda20af93f7c2eee0fd032da84f7c098ad662b6ce00e01b8ab603b6f1f06cc4f
7
- data.tar.gz: 6db739a849af63cee915970ea1abc7df4a9d4aff91edb4c76cc4c78273e7231750771638ff0dc477bbb586fc24f0dfab1de793d875cd22bb395c5ff1d3772641
6
+ metadata.gz: 5f5567a9f8e43fbc0f22e10465a0026a4984932183ae7d70c6fa95cdcb092995cbedd44422fa8a65b6c2d321be3ae15a0b72e905edd43c96427fbaac7cc73e34
7
+ data.tar.gz: f7823d88490ab6edc02ed6a8fd846f7e6cf40ce042fb046e502e3024c26f24ced5ee143de80669ffad49d5077dc97e338c363eda1e0ea1d3196479447c6aa7f6
data/CHANGELOG.md CHANGED
@@ -1,5 +1,32 @@
1
1
  ## [Unreleased]
2
2
 
3
+ ## [0.3.0] - 2022-12-12
4
+
5
+ ### Features
6
+
7
+ #### Inbox
8
+
9
+ Implementation of the "inbox pattern", which ensures that events are processed to completion only once.
10
+
11
+ ```ruby
12
+ # create an inbox table and reference it
13
+ create_table(:inbox) do
14
+ column :id, :varchar, null: true, primary_key: true
15
+ # ...
16
+ create_table(:outbox) do
17
+ column :inbox_id, :varchar
18
+ foreign_key :inbox_id, :inbox
19
+ # ...
20
+
21
+ # tobox.rb
22
+ inbox_table :inbox
23
+ inbox_column :inbox_id
24
+
25
+ # event production
26
+ DB[:outbox].insert(event_type: "order_created", inbox_id: "order_created_#{order.id}", ....
27
+ DB[:outbox].insert(event_type: "billing_event_started", inbox_id: "billing_event_started_#{order.id}", ....
28
+ ```
29
+
3
30
  ## [0.2.0] - 2022-12-05
4
31
 
5
32
  ### Features
data/README.md CHANGED
@@ -15,6 +15,7 @@ Simple, data-first events processing framework based on the [transactional outbo
15
15
  - [Event](#event)
16
16
  - [Features](#features)
17
17
  - [Ordered event processing](#ordered-event-processing)
18
+ - [Inbox](#inbox)
18
19
  - [Plugins](#plugins)
19
20
  - [Zeitwerk](#zeitwerk)
20
21
  - [Sentry](#sentry)
@@ -299,6 +300,14 @@ Overrides the default log level ("info" when in "production" environment, "debug
299
300
 
300
301
  Defines the column to be used for event grouping, when [ordered processing of events is a requirement](#ordered-event-processing).
301
302
 
303
+ ### inbox table
304
+
305
+ Defines the name of the table to be used for inbox, when [inbox usage is a requirement](#inbox).
306
+
307
+ ### inbox column
308
+
309
+ Defines the column in the outbox table which references the inbox table, when one is set.
310
+
302
311
  <a id="markdown-event" name="event"></a>
303
312
  ## Event
304
313
 
@@ -361,6 +370,59 @@ end
361
370
  # "order_created" will be processed first
362
371
  # "billing_event_created" will only start processing once "order_created" finishes
363
372
  ```
373
+ <a id="inbox" name="inbox"></a>
374
+ ### Inbox
375
+
376
+ `tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether).
377
+
378
+ In order to do so, you'll have to:
379
+
380
+ 1. add an "inbox" table in the database
381
+
382
+ ```ruby
383
+ create_table(:inbox) do
384
+ column :inbox_id, :varchar, null: true, primary_key: true # it can also be a uuid, you decide
385
+ column :created_at, "timestamp without time zone", null: false, default: Sequel::CURRENT_TIMESTAMP
386
+ end
387
+ ```
388
+
389
+ 2. add the unique id reference in the outbox table:
390
+
391
+ ```ruby
392
+ create_table(:outbox) do
393
+ primary_key :id
394
+ column :type, :varchar, null: false
395
+ column :inbox_id, :varchar, null: true
396
+ # ...
397
+ foreign_key :inbox_id, :inbox
398
+ ```
399
+
400
+ 3. reference them in the configuration
401
+
402
+ ```ruby
403
+ # tobox.rb
404
+ inbox_table :inbox
405
+ inbox_column :inbox_id
406
+ ```
407
+
408
+ 4. insert related outbox events with an inbox id
409
+
410
+ ```ruby
411
+ order = Order.new(
412
+ item_id: item.id,
413
+ price: 20_20,
414
+ currency: "EUR"
415
+ )
416
+ DB.transaction do
417
+ order.save
418
+ DB[:outbox].insert(event_type: "order_created", inbox_id: "ord_crt_#{order.id}", data_after: order.to_hash)
419
+ DB[:outbox].insert(event_type: "billing_event_started", inbox_id: "bil_evt_std_#{order.id}", data_after: order.to_hash)
420
+ end
421
+
422
+ # assuming this bit above runs two times in two separate workers, each will be processed by tobox only once.
423
+ ```
424
+
425
+ **NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again.
364
426
 
365
427
  <a id="markdown-plugins" name="plugins"></a>
366
428
  ## Plugins
@@ -18,6 +18,8 @@ module Tobox
18
18
  database_uri: nil,
19
19
  table: :outbox,
20
20
  group_column: nil,
21
+ inbox_table: nil,
22
+ inbox_column: nil,
21
23
  max_attempts: 10,
22
24
  exponential_retry_factor: 4,
23
25
  wait_for_events_delay: 5,
data/lib/tobox/fetcher.rb CHANGED
@@ -24,6 +24,9 @@ module Tobox
24
24
 
25
25
  max_attempts = configuration[:max_attempts]
26
26
 
27
+ @inbox_table = configuration[:inbox_table]
28
+ @inbox_column = configuration[:inbox_column]
29
+
27
30
  @ds = @db[@table]
28
31
 
29
32
  run_at_conds = [
@@ -80,14 +83,17 @@ module Tobox
80
83
  if blk
81
84
  num_events = events.size
82
85
 
83
- events.each do |ev|
84
- ev[:metadata] = try_json_parse(ev[:metadata])
85
- handle_before_event(ev)
86
- yield(to_message(ev))
86
+ events.map! do |ev|
87
+ try_insert_inbox(ev) do
88
+ ev[:metadata] = try_json_parse(ev[:metadata])
89
+ handle_before_event(ev)
90
+ yield(to_message(ev))
91
+ ev
92
+ end
87
93
  rescue StandardError => e
88
94
  error = e
89
95
  raise Sequel::Rollback
90
- end
96
+ end.compact!
91
97
  else
92
98
  events.map!(&method(:to_message))
93
99
  end
@@ -148,6 +154,16 @@ module Tobox
148
154
  data
149
155
  end
150
156
 
157
+ def try_insert_inbox(event)
158
+ return yield unless @inbox_table && @inbox_column
159
+
160
+ ret = @db[@inbox_table].insert_conflict.insert(@inbox_column => event[@inbox_column])
161
+
162
+ return unless ret
163
+
164
+ yield
165
+ end
166
+
151
167
  def handle_before_event(event)
152
168
  @logger.debug do
153
169
  log_message("outbox event (type: \"#{event[:type]}\", attempts: #{event[:attempts]}) starting...")
data/lib/tobox/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Tobox
4
- VERSION = "0.2.0"
4
+ VERSION = "0.3.0"
5
5
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: tobox
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 0.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - HoneyryderChuck
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2022-12-05 00:00:00.000000000 Z
11
+ date: 2022-12-21 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: sequel