tobox 0.4.5 → 0.5.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 0be3e9919cb080bf7e57127356e00cc5a13e5d8af6943ee87f688e515771d9ec
4
- data.tar.gz: 215095033e4763f2b5b4bd166f4dcc66fb741424244a4c4a13989974f5dfbb14
3
+ metadata.gz: '0481835fffcdbaf85fb53ebf116a6b117ad267cfb4a4efbcbae05b2f87058951'
4
+ data.tar.gz: 39589407d92d28fc538aebb59a039a606d833aeb0eb26f82b7cf96228e73d5b3
5
5
  SHA512:
6
- metadata.gz: 914972ea18b05b0542bc091cd3cdd602f420c3551f24c598b9d6711ff879076e6e0a03879f27419da6526a449b7e4364ce8e6e0aebd70a698dc250696a0c54f5
7
- data.tar.gz: 419bfcd4c5604c72ace312acb8deebfa3081546b137c7f7c5ee267be5e0f49a8f21a745147e17b99ee002b2e95db7c48a03d84cd578dc3ca72bd452ad8581280
6
+ metadata.gz: 8a1137598cc58cb22573ccbec80f448aa55db3505a97b741cc146eaef6f16392ba3159b9db9f61498cea530dba33c431bba4f9c847325064a770a96553d9ec20
7
+ data.tar.gz: 274be354130a8ae5ab59188ad98c4de208adac78cd7c9720b569db98de75d2ec0514a2560db05c080b9118e6967b573b7ea1fd11b0297edf9c07f05478acbefe
data/CHANGELOG.md CHANGED
@@ -1,5 +1,21 @@
1
1
  ## [Unreleased]
2
2
 
3
+ ## [0.5.0] - 2024-09-16
4
+
5
+ ### Features
6
+
7
+ A new `:progress` plugin can be used in order to release database transactions before handling events (useful for when event handling times vary and may cause transaction bookkeeping overhead in the database).
8
+
9
+ **Note**: This may become the default behaviour in a future release.
10
+
11
+ ### Improvements
12
+
13
+ * The event grouping and inbox capabilities were converted into plugins (respectively, `:event_grouping` and `:inbox`).
14
+
15
+ ### Bugfixes
16
+
17
+ * exponential backoff calculation was broken.
18
+ * behaviour fixed for databases which do not support `ON CONFLICT` or `UPDATE ... RETURNING` (like MySQL).
3
19
 
4
20
  ## [0.4.5] - 2024-02-28
5
21
 
data/README.md CHANGED
@@ -13,10 +13,10 @@ Simple, data-first events processing framework based on the [transactional outbo
13
13
  - [Usage](#usage)
14
14
  - [Configuration](#configuration)
15
15
  - [Event](#event)
16
- - [Features](#features)
17
- - [Ordered event processing](#ordered-event-processing)
18
- - [Inbox](#inbox)
19
16
  - [Plugins](#plugins)
17
+ - [Progress](#progress)
18
+ - [Event Grouping](#event-grouping)
19
+ - [Inbox](#inbox)
20
20
  - [Zeitwerk](#zeitwerk)
21
21
  - [Sentry](#sentry)
22
22
  - [Datadog](#datadog)
@@ -204,7 +204,17 @@ table :outbox
204
204
  Maximum number of times a failed attempt to process an event will be retried (`10` by default).
205
205
 
206
206
  ```ruby
207
- concurrency 4
207
+ max_attempts 4
208
+ ```
209
+
210
+ **Note**: the new attempt will be retried in `n ** 4`, where `n` is the number of past attempts for that event.
211
+
212
+ ### `exponential_retry_factor`
213
+
214
+ Factor by which the number of seconds until an event can be retried will be exponentially calculated, i.e. 2 seconds on first attempt, then 4, then 8, then 16 (`2` by default).
215
+
216
+ ```ruby
217
+ exponential_retry_factor 2
208
218
  ```
209
219
 
210
220
  **Note**: the new attempt will be retried in `n ** 4`, where `n` is the number of past attempts for that event.
@@ -320,17 +330,6 @@ Overrides the internal logger (an instance of `Logger`).
320
330
 
321
331
  Overrides the default log level ("info" when in "production" environment, "debug" otherwise).
322
332
 
323
- ### group_column
324
-
325
- Defines the column to be used for event grouping, when [ordered processing of events is a requirement](#ordered-event-processing).
326
-
327
- ### inbox table
328
-
329
- Defines the name of the table to be used for inbox, when [inbox usage is a requirement](#inbox).
330
-
331
- ### inbox column
332
-
333
- Defines the column in the outbox table which references the inbox table, when one is set.
334
333
 
335
334
  <a id="markdown-event" name="event"></a>
336
335
  ## Event
@@ -345,36 +344,85 @@ The event is composed of the following properties:
345
344
 
346
345
  (*NOTE*: The event is also composed of other properties which are only relevant for `tobox`.)
347
346
 
348
- <a id="markdown-features" name="features"></a>
349
- ## Features
350
347
 
351
- There are a few extra features you can run on top a "vanilla" transactional outbox implementation. This is how you can accomplish them using `tobox`.
348
+ <a id="markdown-plugins" name="plugins"></a>
349
+ ## Plugins
350
+
351
+ `tobox` ships with a very simple plugin system. (TODO: add docs).
352
+
353
+ Plugins can be loaded in the config via `plugin`:
354
+
355
+ ```ruby
356
+ # tobox.rb
357
+ plugin(:plugin_name)
358
+ ```
359
+
360
+ <a id="markdown-progress" name="progress"></a>
361
+ ### Progress
362
+
363
+ By default, the database transaction used to consume the event is kept open while the event is handled. While this ensures atomic event consumption, it may also cause overhead related to transaction management given enough load, particularly in cases where event handling time varies (i.e. throttled HTTP requests).
364
+
365
+ The `:progress` plugin fixes this by releasing the databaase transaction after fetching the event. It does so by making the fetched event "invisible" for a certain period, during which the event must be successfully handled.
366
+
367
+ Here's how to use it:
368
+
369
+ ```ruby
370
+ # in your tobox.rb
371
+ plugin :progress
372
+
373
+ visibility_timeout 90 # default: 30
374
+ ```
375
+
376
+ 3. insert related outbox events with the same group id
377
+
378
+ ```ruby
379
+ order = Order.new(
380
+ item_id: item.id,
381
+ price: 20_20,
382
+ currency: "EUR"
383
+ )
384
+ DB.transaction do
385
+ order.save
386
+ DB[:outbox].insert(event_type: "order_created", group_id: order.id, data_after: order.to_hash)
387
+ DB[:outbox].insert(event_type: "billing_event_started", group_id: order.id, data_after: order.to_hash)
388
+ end
389
+
390
+ # "order_created" will be processed first
391
+ # "billing_event_created" will only start processing once "order_created" finishes
392
+ ```
393
+
394
+ #### Configuration
352
395
 
353
- <a id="markdown-ordered-event-processing" name="ordered-event-processing"></a>
354
- ### Ordered event processing
396
+ ##### `visibility_timeout`
397
+
398
+ Timeout (in seconds) after which a previously marked-for-consumption event can be retried (default: `30`)
399
+
400
+ <a id="markdown-event-grouping" name="event-grouping"></a>
401
+ ### Event grouping
355
402
 
356
403
  By default, events are taken and processed from the "outbox" table concurrently by workers, which means that, while worker A may process the most recent event, and worker B takes the following, worker B may process it faster than worker A. This may be an issue if the consumer expects events from a certain context to arrive in a certain order.
357
404
 
358
- One solution is to have a single worker processing the "outbox" events. Another is to use the `group_column` configuration.
405
+ One solution is to have a single worker processing the "outbox" events. Another is to use the `:event_grouping` plugin.
359
406
 
360
- What you have to do is:
407
+ All you have to do is:
361
408
 
362
409
  1. add a "group id" column to the "outbox" table
363
410
 
364
411
  ```ruby
365
412
  create_table(:outbox) do
366
413
  primary_key :id
367
- column :group_id, :integer
368
- # The type is irrelevant, could also be :string, :uuid...
414
+ column :group_id, :integer # The type is irrelevant, could also be :string, :uuid...
369
415
  # ..
416
+ index :group_id
370
417
  ```
371
418
 
372
- 2. set the "group_column" configuration
419
+ 2. Enable the plugin
373
420
 
374
421
  ```ruby
375
422
  # in your tobox.rb
376
- group_column :group_id
377
- index :group_id
423
+ plugin :event_grouping
424
+
425
+ group_column :group_id # by default already `:group_id`
378
426
  ```
379
427
 
380
428
  3. insert related outbox events with the same group id
@@ -394,25 +442,28 @@ end
394
442
  # "order_created" will be processed first
395
443
  # "billing_event_created" will only start processing once "order_created" finishes
396
444
  ```
445
+
446
+ #### Configuration
447
+
448
+ ##### `group_column`
449
+
450
+ Defines the database column to be used for event grouping (`:group_id` by default).
451
+
397
452
  <a id="inbox" name="inbox"></a>
398
453
  ### Inbox
399
454
 
400
- `tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether).
455
+ Via the `:inbox` plugin, `tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether).
401
456
 
402
457
  In order to do so, you'll have to:
403
458
 
404
- 1. add an "inbox" table in the database
459
+ 1. add an "inbox" table in the database and the unique id reference in the outbox table:
405
460
 
406
461
  ```ruby
407
462
  create_table(:inbox) do
408
463
  column :inbox_id, :varchar, null: true, primary_key: true # it can also be a uuid, you decide
409
464
  column :created_at, "timestamp without time zone", null: false, default: Sequel::CURRENT_TIMESTAMP
410
465
  end
411
- ```
412
466
 
413
- 2. add the unique id reference in the outbox table:
414
-
415
- ```ruby
416
467
  create_table(:outbox) do
417
468
  primary_key :id
418
469
  column :type, :varchar, null: false
@@ -421,15 +472,17 @@ create_table(:outbox) do
421
472
  foreign_key :inbox_id, :inbox
422
473
  ```
423
474
 
424
- 3. reference them in the configuration
475
+ 2. Load the plugin and reference them in the configuration
425
476
 
426
477
  ```ruby
427
478
  # tobox.rb
428
- inbox_table :inbox
429
- inbox_column :inbox_id
479
+ plugin :inbox
480
+
481
+ inbox_table :inbox # :inbox by default already
482
+ inbox_column :inbox_id # :inbox_id by default already
430
483
  ```
431
484
 
432
- 4. insert related outbox events with an inbox id
485
+ 3. insert related outbox events with an inbox id
433
486
 
434
487
  ```ruby
435
488
  order = Order.new(
@@ -446,19 +499,19 @@ end
446
499
  # assuming this bit above runs two times in two separate workers, each will be processed by tobox only once.
447
500
  ```
448
501
 
449
- **NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again.
502
+ #### Configuration
450
503
 
451
- <a id="markdown-plugins" name="plugins"></a>
452
- ## Plugins
453
504
 
454
- `tobox` ships with a very simple plugin system. (TODO: add docs).
505
+ ##### inbox table
455
506
 
456
- Plugins can be loaded in the config via `plugin`:
507
+ Defines the name of the table to be used for inbox (`:inbox` by default).
457
508
 
458
- ```ruby
459
- # tobox.rb
460
- plugin(:plugin_name)
461
- ```
509
+ ##### inbox column
510
+
511
+ Defines the column in the outbox table which references the inbox table (`:inbox_id` by default).
512
+
513
+
514
+ **NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again.
462
515
 
463
516
  It ships with the following integrations.
464
517
 
@@ -576,7 +629,7 @@ end
576
629
  <a id="markdown-supported-rubies" name="supported-rubies"></a>
577
630
  ## Supported Rubies
578
631
 
579
- All Rubies greater or equal to 2.6, and always latest JRuby and Truffleruby.
632
+ All Rubies greater or equal to 2.7, and always latest JRuby and Truffleruby.
580
633
 
581
634
 
582
635
  <a id="markdown-rails-support" name="rails-support"></a>
@@ -7,7 +7,8 @@ module Tobox
7
7
  class Configuration
8
8
  extend Forwardable
9
9
 
10
- attr_reader :handlers, :lifecycle_events, :arguments_handler, :default_logger, :database
10
+ attr_reader :plugins, :handlers, :lifecycle_events, :arguments_handler, :default_logger, :database, :fetcher_class,
11
+ :config
11
12
 
12
13
  def_delegator :@config, :[]
13
14
 
@@ -18,11 +19,9 @@ module Tobox
18
19
  database_uri: nil,
19
20
  database_options: nil,
20
21
  table: :outbox,
21
- group_column: nil,
22
- inbox_table: nil,
23
- inbox_column: nil,
22
+ created_at_column: nil,
24
23
  max_attempts: 10,
25
- exponential_retry_factor: 4,
24
+ exponential_retry_factor: 2,
26
25
  wait_for_events_delay: 5,
27
26
  shutdown_timeout: 10,
28
27
  concurrency: 4, # TODO: CPU count
@@ -45,6 +44,7 @@ module Tobox
45
44
  @handlers = {}
46
45
  @message_to_arguments = nil
47
46
  @plugins = []
47
+ @fetcher_class = Class.new(Fetcher)
48
48
 
49
49
  if block
50
50
  case block.arity
@@ -141,6 +141,8 @@ module Tobox
141
141
 
142
142
  extend(plugin::ConfigurationMethods) if defined?(plugin::ConfigurationMethods)
143
143
 
144
+ @fetcher_class.__send__(:include, plugin::FetcherMethods) if defined?(plugin::FetcherMethods)
145
+
144
146
  plugin.configure(self, **options, &block) if plugin.respond_to?(:configure)
145
147
  end
146
148
 
@@ -157,7 +159,7 @@ module Tobox
157
159
  private
158
160
 
159
161
  def method_missing(meth, *args, &block)
160
- if DEFAULT_CONFIGURATION.key?(meth) && args.size == 1
162
+ if @config.key?(meth) && args.size == 1
161
163
  @config[meth] = args.first
162
164
  elsif /\Aon_(.*)\z/.match(meth) && args.empty?
163
165
  on(Regexp.last_match(1).to_sym, &block)
@@ -167,8 +169,8 @@ module Tobox
167
169
  end
168
170
 
169
171
  def respond_to_missing?(meth, *args)
170
- super(meth, *args) ||
171
- DEFAULT_CONFIGURATION.key?(meth) ||
172
+ super ||
173
+ @config.key?(meth) ||
172
174
  /\Aon_(.*)\z/.match(meth)
173
175
  end
174
176
  end
data/lib/tobox/fetcher.rb CHANGED
@@ -13,14 +13,10 @@ module Tobox
13
13
  @db = configuration.database
14
14
 
15
15
  @table = configuration[:table]
16
- @group_column = configuration[:group_column]
17
16
  @exponential_retry_factor = configuration[:exponential_retry_factor]
18
17
 
19
18
  max_attempts = configuration[:max_attempts]
20
19
 
21
- @inbox_table = configuration[:inbox_table]
22
- @inbox_column = configuration[:inbox_column]
23
-
24
20
  @ds = @db[@table]
25
21
 
26
22
  run_at_conds = [
@@ -32,6 +28,8 @@ module Tobox
32
28
  .where(run_at_conds)
33
29
  .order(Sequel.desc(:run_at, nulls: :first), :id)
34
30
 
31
+ @mark_as_fetched_params = { attempts: Sequel[@table][:attempts] + 1, last_error: nil }
32
+
35
33
  @before_event_handlers = Array(@configuration.lifecycle_events[:before_event])
36
34
  @after_event_handlers = Array(@configuration.lifecycle_events[:after_event])
37
35
  @error_event_handlers = Array(@configuration.lifecycle_events[:error_event])
@@ -39,75 +37,28 @@ module Tobox
39
37
 
40
38
  def fetch_events(&blk)
41
39
  num_events = 0
42
- @db.transaction(savepoint: false) do
43
- if @group_column
44
- group = @pick_next_sql.for_update
45
- .skip_locked
46
- .limit(1)
47
- .select(@group_column)
48
-
49
- # get total from a group, to compare to the number of future locked rows.
50
- total_from_group = @ds.where(@group_column => group).count
51
-
52
- event_ids = @ds.where(@group_column => group)
53
- .order(Sequel.desc(:run_at, nulls: :first), :id)
54
- .for_update.skip_locked.select_map(:id)
55
-
56
- if event_ids.size != total_from_group
57
- # this happens if concurrent workers locked different rows from the same group,
58
- # or when new rows from a given group have been inserted after the lock has been
59
- # acquired
60
- event_ids = []
61
- end
62
-
63
- # lock all, process 1
64
- event_ids = event_ids[0, 1]
65
- else
66
- event_ids = @pick_next_sql.for_update
67
- .skip_locked
68
- .limit(1).select_map(:id) # lock starts here
69
- end
40
+ events_tr do
41
+ event_ids = EMPTY
70
42
 
71
- events = nil
72
- error = nil
73
- unless event_ids.empty?
74
- @db.transaction(savepoint: true) do
75
- events = @ds.where(id: event_ids).returning.delete
76
-
77
- if blk
78
- num_events = events.size
79
-
80
- events.map! do |ev|
81
- try_insert_inbox(ev) do
82
- ev[:metadata] = try_json_parse(ev[:metadata])
83
- handle_before_event(ev)
84
- yield(to_message(ev))
85
- ev
86
- end
87
- rescue StandardError => e
88
- error = e
89
- raise Sequel::Rollback
90
- end.compact!
91
- else
92
- events.map!(&method(:to_message))
93
- end
94
- end
43
+ event_ids_tr do
44
+ event_ids = fetch_event_ids
45
+ mark_as_fetched(event_ids)
95
46
  end
96
47
 
97
- return blk ? 0 : [] if events.nil?
48
+ evts = nil
98
49
 
99
- return events unless blk
50
+ unless event_ids.empty?
51
+ with_events(event_ids) do |events|
52
+ evts = events
53
+ num_events = events.count
100
54
 
101
- if events
102
- events.each do |event|
103
- if error
104
- event.merge!(mark_as_error(event, error))
105
- handle_error_event(event, error)
106
- else
107
- handle_after_event(event)
55
+ evts = events.filter_map do |ev|
56
+ prepare_event(ev, &blk)
108
57
  end
109
58
  end
110
59
  end
60
+
61
+ return 0 if evts.nil?
111
62
  end
112
63
 
113
64
  num_events
@@ -115,19 +66,86 @@ module Tobox
115
66
 
116
67
  private
117
68
 
69
+ def prepare_event(event)
70
+ event[:metadata] = try_json_parse(event[:metadata])
71
+ handle_before_event(event)
72
+ yield(to_message(event))
73
+ event
74
+ end
75
+
76
+ def fetch_event_ids
77
+ @pick_next_sql.for_update
78
+ .skip_locked
79
+ .limit(1).select_map(:id) # lock starts here
80
+ end
81
+
82
+ def mark_as_fetched(event_ids)
83
+ @ds.where(id: event_ids).update(@mark_as_fetched_params) unless event_ids.empty?
84
+ end
85
+
86
+ def events_tr(&block)
87
+ @db.transaction(savepoint: false, &block)
88
+ end
89
+
90
+ def event_ids_tr
91
+ yield
92
+ end
93
+
94
+ def with_events(event_ids, &blk)
95
+ events, error = yield_events(event_ids, &blk)
96
+
97
+ events.each do |event|
98
+ if error
99
+ event.merge!(mark_as_error(event, error))
100
+ handle_error_event(event, error)
101
+ else
102
+ handle_after_event(event)
103
+ end
104
+ end
105
+ end
106
+
107
+ def yield_events(event_ids)
108
+ events_ds = @ds.where(id: event_ids)
109
+ events = EMPTY
110
+ error = nil
111
+
112
+ begin
113
+ events = events_ds.all
114
+
115
+ yield events
116
+
117
+ events_ds.delete
118
+ rescue StandardError => e
119
+ error = e
120
+ end
121
+
122
+ [events, error]
123
+ end
124
+
118
125
  def log_message(msg)
119
126
  "(worker: #{@label}) -> #{msg}"
120
127
  end
121
128
 
122
129
  def mark_as_error(event, error)
123
- @ds.where(id: event[:id]).returning.update(
124
- attempts: Sequel[@table][:attempts] + 1,
130
+ update_params = {
125
131
  run_at: Sequel.date_add(Sequel::CURRENT_TIMESTAMP,
126
- seconds: event[:attempts] + (1**@exponential_retry_factor)),
132
+ seconds: @exponential_retry_factor**(event[:attempts] - 1)),
127
133
  # run_at: Sequel.date_add(Sequel::CURRENT_TIMESTAMP,
128
134
  # seconds: Sequel.function(:POWER, Sequel[@table][:attempts] + 1, 4)),
129
135
  last_error: "#{error.message}\n#{error.backtrace.join("\n")}"
130
- ).first
136
+ }
137
+
138
+ set_event_retry_attempts(event, update_params)
139
+ end
140
+
141
+ def set_event_retry_attempts(event, update_params)
142
+ ds = @ds.where(id: event[:id])
143
+ if @ds.supports_returning?(:update)
144
+ ds.returning.update(update_params).first
145
+ else
146
+ ds.update(update_params)
147
+ ds.first
148
+ end
131
149
  end
132
150
 
133
151
  def to_message(event)
@@ -148,16 +166,6 @@ module Tobox
148
166
  data
149
167
  end
150
168
 
151
- def try_insert_inbox(event)
152
- return yield unless @inbox_table && @inbox_column
153
-
154
- ret = @db[@inbox_table].insert_conflict.insert(@inbox_column => event[@inbox_column])
155
-
156
- return unless ret
157
-
158
- yield
159
- end
160
-
161
169
  def handle_before_event(event)
162
170
  @logger.debug do
163
171
  log_message("outbox event (type: \"#{event[:type]}\", attempts: #{event[:attempts]}) starting...")
@@ -0,0 +1,47 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Tobox
4
+ module Plugins
5
+ module EventGrouping
6
+ def self.configure(conf)
7
+ conf.config[:group_column] = :group_id
8
+ end
9
+
10
+ module FetcherMethods
11
+ def initialize(_, configuration)
12
+ super
13
+
14
+ @group_column = configuration[:group_column]
15
+ end
16
+
17
+ private
18
+
19
+ def fetch_event_ids
20
+ group = @pick_next_sql.for_update
21
+ .skip_locked
22
+ .limit(1)
23
+ .select(@group_column)
24
+
25
+ # get total from a group, to compare to the number of future locked rows.
26
+ total_from_group = @ds.where(@group_column => group).count
27
+
28
+ event_ids = @ds.where(@group_column => group)
29
+ .order(Sequel.desc(:run_at, nulls: :first), :id)
30
+ .for_update.skip_locked.select_map(:id)
31
+
32
+ if event_ids.size != total_from_group
33
+ # this happens if concurrent workers locked different rows from the same group,
34
+ # or when new rows from a given group have been inserted after the lock has been
35
+ # acquired
36
+ event_ids = []
37
+ end
38
+
39
+ # lock all, process 1
40
+ event_ids[0, 1]
41
+ end
42
+ end
43
+ end
44
+
45
+ register_plugin :event_grouping, EventGrouping
46
+ end
47
+ end
@@ -0,0 +1,46 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Tobox
4
+ module Plugins
5
+ module Inbox
6
+ def self.configure(conf)
7
+ conf.config[:inbox_table] = :inbox
8
+ conf.config[:inbox_column] = :unique_id
9
+ end
10
+
11
+ module FetcherMethods
12
+ def initialize(_, configuration)
13
+ super
14
+
15
+ inbox_table = configuration[:inbox_table]
16
+
17
+ @inbox_ds = @db[inbox_table]
18
+ @inbox_column = configuration[:inbox_column]
19
+ end
20
+
21
+ private
22
+
23
+ def prepare_event(event, &blk)
24
+ try_insert_inbox(event) { super }
25
+ end
26
+
27
+ def try_insert_inbox(event)
28
+ if @inbox_ds.respond_to?(:supports_insert_conflict?) && @inbox_ds.supports_insert_conflict?
29
+ ret = @inbox_ds.insert_conflict.insert(@inbox_column => event[@inbox_column])
30
+
31
+ return event unless ret
32
+ else
33
+ begin
34
+ @inbox_ds.insert(@inbox_column => event[@inbox_column])
35
+ rescue Sequel::UniqueConstraintViolation
36
+ return event
37
+ end
38
+ end
39
+
40
+ yield
41
+ end
42
+ end
43
+ end
44
+ register_plugin :inbox, Inbox
45
+ end
46
+ end
@@ -0,0 +1,33 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Tobox
4
+ module Plugins
5
+ module Progress
6
+ def self.configure(conf)
7
+ conf.config[:visibility_timeout] = 30
8
+ end
9
+
10
+ module FetcherMethods
11
+ private
12
+
13
+ def initialize(_, configuration)
14
+ super
15
+
16
+ @mark_as_fetched_params[:run_at] = Sequel.date_add(
17
+ Sequel::CURRENT_TIMESTAMP,
18
+ seconds: configuration[:visibility_timeout]
19
+ )
20
+ end
21
+
22
+ def events_tr
23
+ yield
24
+ end
25
+
26
+ def event_ids_tr(&block)
27
+ @db.transaction(savepoint: false, &block)
28
+ end
29
+ end
30
+ end
31
+ register_plugin :progress, Progress
32
+ end
33
+ end
@@ -25,6 +25,8 @@ module Tobox
25
25
 
26
26
  config = @config
27
27
 
28
+ plugins = config.plugins.map(&:name)
29
+
28
30
  interval = config.stats_interval_seconds
29
31
  @stats_handlers = Array(config.lifecycle_events[:stats])
30
32
 
@@ -34,14 +36,24 @@ module Tobox
34
36
 
35
37
  @max_attempts = config[:max_attempts]
36
38
 
39
+ @created_at_column = config[:created_at_column]
40
+
37
41
  @db = Sequel.connect(config.database.opts.merge(max_connections: 1))
42
+ @db.loggers = config.database.loggers
38
43
  Array(config.lifecycle_events[:database_connect]).each { |cb| cb.call(@db) }
39
44
 
40
- @outbox_table = config[:table]
41
- @outbox_ds = @db[@outbox_table]
45
+ outbox_table = config[:table]
46
+ @outbox_ds = @db[outbox_table]
47
+
48
+ if plugins.include?("Tobox::Plugins::Inbox")
49
+ inbox_table = config[:inbox_table]
50
+ @inbox_ds = @db[inbox_table]
51
+ end
42
52
 
43
- inbox_table = config[:inbox_table]
44
- @inbox_ds = @db[inbox_table] if inbox_table
53
+ if @created_at_column
54
+ # discard already handled events
55
+ @oldest_event_age_ds = @outbox_ds.where(last_error: nil, run_at: nil).order(Sequel.asc(:id))
56
+ end
45
57
 
46
58
  logger = config.default_logger
47
59
 
@@ -104,6 +116,13 @@ module Tobox
104
116
  stats[:failed_count] ||= 0
105
117
 
106
118
  stats[:inbox_count] = @inbox_ds.count if @inbox_ds
119
+
120
+ if @oldest_event_age_ds
121
+ created_at = @oldest_event_age_ds.get(@created_at_column)
122
+ age = created_at ? (Time.now - created_at).to_i : 0
123
+ stats[:oldest_event_age_in_seconds] = age
124
+ end
125
+
107
126
  stats
108
127
  end
109
128
  end
@@ -56,6 +56,8 @@ module Tobox
56
56
  if worker.finished? && @running
57
57
  idx = @workers.index(worker)
58
58
 
59
+ raise Error, "worker not found" unless idx
60
+
59
61
  subst_worker = Worker.new(worker.label, @configuration)
60
62
  @workers[idx] = subst_worker
61
63
  subst_thread = start_thread_worker(subst_worker)
data/lib/tobox/pool.rb CHANGED
@@ -1,9 +1,9 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Tobox
4
- class Pool
5
- class KillError < Interrupt; end
4
+ class KillError < Interrupt; end
6
5
 
6
+ class Pool
7
7
  def initialize(configuration)
8
8
  @configuration = configuration
9
9
  @logger = @configuration.default_logger
data/lib/tobox/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Tobox
4
- VERSION = "0.4.5"
4
+ VERSION = "0.5.0"
5
5
  end
data/lib/tobox/worker.rb CHANGED
@@ -8,7 +8,7 @@ module Tobox
8
8
  @label = label
9
9
  @wait_for_events_delay = configuration[:wait_for_events_delay]
10
10
  @handlers = configuration.handlers || {}
11
- @fetcher = Fetcher.new(label, configuration)
11
+ @fetcher = configuration.fetcher_class.new(label, configuration)
12
12
  @finished = false
13
13
 
14
14
  return unless (message_to_arguments = configuration.arguments_handler)
data/lib/tobox.rb CHANGED
@@ -9,6 +9,8 @@ require "mutex_m"
9
9
  module Tobox
10
10
  class Error < StandardError; end
11
11
 
12
+ EMPTY = [].freeze
13
+
12
14
  module Plugins
13
15
  @plugins = {}
14
16
  @plugins.extend(Mutex_m)
@@ -34,8 +36,8 @@ module Tobox
34
36
  end
35
37
  end
36
38
 
37
- require_relative "tobox/configuration"
38
39
  require_relative "tobox/fetcher"
39
40
  require_relative "tobox/worker"
40
41
  require_relative "tobox/pool"
41
42
  require_relative "tobox/application"
43
+ require_relative "tobox/configuration"
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: tobox
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.4.5
4
+ version: 0.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - HoneyryderChuck
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2024-02-28 00:00:00.000000000 Z
11
+ date: 2024-09-16 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: sequel
@@ -45,6 +45,9 @@ files:
45
45
  - lib/tobox/plugins/datadog/configuration.rb
46
46
  - lib/tobox/plugins/datadog/integration.rb
47
47
  - lib/tobox/plugins/datadog/patcher.rb
48
+ - lib/tobox/plugins/event_grouping.rb
49
+ - lib/tobox/plugins/inbox.rb
50
+ - lib/tobox/plugins/progress.rb
48
51
  - lib/tobox/plugins/sentry.rb
49
52
  - lib/tobox/plugins/stats.rb
50
53
  - lib/tobox/plugins/zeitwerk.rb
@@ -71,14 +74,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
71
74
  requirements:
72
75
  - - ">="
73
76
  - !ruby/object:Gem::Version
74
- version: 2.6.0
77
+ version: 2.7.0
75
78
  required_rubygems_version: !ruby/object:Gem::Requirement
76
79
  requirements:
77
80
  - - ">="
78
81
  - !ruby/object:Gem::Version
79
82
  version: '0'
80
83
  requirements: []
81
- rubygems_version: 3.4.10
84
+ rubygems_version: 3.5.3
82
85
  signing_key:
83
86
  specification_version: 4
84
87
  summary: Transactional outbox pattern implementation in ruby