tobox 0.1.6 → 0.3.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 9e47d46ec5d5f20a122b8667f00227517002ea4503348a253e3e16cd6cabe281
4
- data.tar.gz: 93fbcdbd7280e21249d4f798b7d86dac70cc8ff9847873d6a8368bda371ccf4d
3
+ metadata.gz: 33a40c07b2d7f36933540d390287aeeb7d4e0aa39a7d19a439eaca6a2f8fc23f
4
+ data.tar.gz: 939699d801afdb03ed85c0048eee3b00c50a786a345b0731c6dba910fafb51ad
5
5
  SHA512:
6
- metadata.gz: bc3f385c64603c6a9b8f95f32e9c38d2974080b0df50a6530505c3d5438d3d60a3ecf66b06200beedce9de23838f0e6dbb6e2aca65233180107e362538905853
7
- data.tar.gz: 5cb0d36a97a6bc173dfbfe01fc834a483848faf4157482b982551ffeb1eec3129dccd59b83cf65d9eab034c4560a7809699557a62dece8046f1944e55bcd9ee4
6
+ metadata.gz: 5f5567a9f8e43fbc0f22e10465a0026a4984932183ae7d70c6fa95cdcb092995cbedd44422fa8a65b6c2d321be3ae15a0b72e905edd43c96427fbaac7cc73e34
7
+ data.tar.gz: f7823d88490ab6edc02ed6a8fd846f7e6cf40ce042fb046e502e3024c26f24ced5ee143de80669ffad49d5077dc97e338c363eda1e0ea1d3196479447c6aa7f6
data/CHANGELOG.md CHANGED
@@ -1,36 +1,99 @@
1
1
  ## [Unreleased]
2
2
 
3
- ## [0.1.6] - 2002-10-06
3
+ ## [0.3.0] - 2022-12-12
4
+
5
+ ### Features
6
+
7
+ #### Inbox
8
+
9
+ Implementation of the "inbox pattern", which ensures that events are processed to completion only once.
10
+
11
+ ```ruby
12
+ # create an inbox table and reference it
13
+ create_table(:inbox) do
14
+ column :id, :varchar, null: true, primary_key: true
15
+ # ...
16
+ create_table(:outbox) do
17
+ column :inbox_id, :varchar
18
+ foreign_key :inbox_id, :inbox
19
+ # ...
20
+
21
+ # tobox.rb
22
+ inbox_table :inbox
23
+ inbox_column :inbox_id
24
+
25
+ # event production
26
+ DB[:outbox].insert(event_type: "order_created", inbox_id: "order_created_#{order.id}", ....
27
+ DB[:outbox].insert(event_type: "billing_event_started", inbox_id: "billing_event_started_#{order.id}", ....
28
+ ```
29
+
30
+ ## [0.2.0] - 2022-12-05
31
+
32
+ ### Features
33
+
34
+ #### Ordered event processing
35
+
36
+ When the outbox table contains a `:group_id` table (and the producer fills up events with it), then a group of events with the same `:group_id` will be processed one by one, by order of insertion.
37
+
38
+ ```ruby
39
+ # migration
40
+ create_table(:outbox) do
41
+ column :message_group_id, :integer
42
+
43
+ # tobox.rb
44
+ message_group_column :group_id
45
+
46
+ # event production
47
+ DB[:outbox].insert(event_type: "order_created", message_group_id: order.id, ....
48
+ DB[:outbox].insert(event_type: "billing_event_started", message_group_id: order.id, ....
49
+
50
+ # order_created handled first, billing_event_started only after
51
+ ```
52
+
53
+ #### on_error_worker callback
54
+
55
+ The config option `on_error_worker { |error| }` gets called when an error happens in a worker **before** events are processed (p.ex. when the database connection becomes unhealthy). You can use it to report such errors to an error reporting system (the `sentry` plugin relies on it).
56
+
57
+ ```ruby
58
+ # tobox.rb
59
+ on_error_worker { |error| Sentry.capture_exception(error, hint: { background: false }) }
60
+ ```
61
+
62
+ ### Bugfixes
63
+
64
+ Thread workers: when errors happen which bring down the workers (such as database becoming unresponsive), workers will be restarted.
65
+
66
+ ## [0.1.6] - 2022-10-06
4
67
 
5
68
  ### Bugfixes
6
69
 
7
70
  Allow passing datadog options, initialize tracing from plugin.
8
71
 
9
- ## [0.1.5] - 2002-10-06
72
+ ## [0.1.5] - 2022-10-06
10
73
 
11
74
  ### Bugfixes
12
75
 
13
76
  Fixing datadog plugin name.
14
77
 
15
- ## [0.1.4] - 2002-10-06
78
+ ## [0.1.4] - 2022-10-06
16
79
 
17
80
  ### Bugfixes
18
81
 
19
82
  Actual fix for missing datadog constants.
20
83
 
21
- ## [0.1.3] - 2002-10-06
84
+ ## [0.1.3] - 2022-10-06
22
85
 
23
86
  ### Bugfixes
24
87
 
25
88
  Datadog constants unproperly namespaced.
26
89
 
27
- ## [0.1.2] - 2002-09-14
90
+ ## [0.1.2] - 2022-09-14
28
91
 
29
92
  ### Bugfixes
30
93
 
31
94
  Actual fix for foregoing json parsing.
32
95
 
33
- ## [0.1.1] - 2002-09-14
96
+ ## [0.1.1] - 2022-09-14
34
97
 
35
98
  ### Chore
36
99
 
data/README.md CHANGED
@@ -1,11 +1,34 @@
1
1
  # Tobox: Transactional outbox pattern implementation in ruby
2
2
 
3
3
  [![Gem Version](https://badge.fury.io/rb/tobox.svg)](http://rubygems.org/gems/tobox)
4
- [![pipeline status](https://gitlab.com/honeyryderchuck/tobox/badges/master/pipeline.svg)](https://gitlab.com/honeyryderchuck/tobox/pipelines?page=1&scope=all&ref=master)
5
- [![coverage report](https://gitlab.com/honeyryderchuck/tobox/badges/master/coverage.svg?job=coverage)](https://honeyryderchuck.gitlab.io/tobox/#_AllFiles)
4
+ [![pipeline status](https://gitlab.com/os85/tobox/badges/master/pipeline.svg)](https://gitlab.com/os85/tobox/pipelines?page=1&scope=all&ref=master)
5
+ [![coverage report](https://gitlab.com/os85/tobox/badges/master/coverage.svg?job=coverage)](https://os85.gitlab.io/tobox/#_AllFiles)
6
6
 
7
7
  Simple, data-first events processing framework based on the [transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html).
8
8
 
9
+ <!-- TOC -->
10
+
11
+ - [Requirements](#requirements)
12
+ - [Installation](#installation)
13
+ - [Usage](#usage)
14
+ - [Configuration](#configuration)
15
+ - [Event](#event)
16
+ - [Features](#features)
17
+ - [Ordered event processing](#ordered-event-processing)
18
+ - [Inbox](#inbox)
19
+ - [Plugins](#plugins)
20
+ - [Zeitwerk](#zeitwerk)
21
+ - [Sentry](#sentry)
22
+ - [Datadog](#datadog)
23
+ - [Supported Rubies](#supported-rubies)
24
+ - [Rails support](#rails-support)
25
+ - [Why?](#why)
26
+ - [Development](#development)
27
+ - [Contributing](#contributing)
28
+
29
+ <!-- /TOC -->
30
+
31
+ <a id="markdown-requirements" name="requirements"></a>
9
32
  ## Requirements
10
33
 
11
34
  `tobox` requires integration with RDBMS which supports `SKIP LOCKED` functionality. As of today, that's:
@@ -15,6 +38,7 @@ Simple, data-first events processing framework based on the [transactional outbo
15
38
  * Oracle
16
39
  * Microsoft SQL Server
17
40
 
41
+ <a id="markdown-installation" name="installation"></a>
18
42
  ## Installation
19
43
 
20
44
  Add this line to your application's Gemfile:
@@ -22,7 +46,7 @@ Add this line to your application's Gemfile:
22
46
  ```ruby
23
47
  gem "tobox"
24
48
 
25
- # You'll also need to aadd the right database client gem for the target RDBMS
49
+ # You'll also need to add the right database client gem for the target RDBMS
26
50
  # ex, for postgresql:
27
51
  #
28
52
  # gem "pg"
@@ -37,6 +61,8 @@ Or install it yourself as:
37
61
 
38
62
  $ gem install tobox
39
63
 
64
+
65
+ <a id="markdown-usage" name="usage"></a>
40
66
  ## Usage
41
67
 
42
68
  1. create the `outbox` table in your application's database:
@@ -80,11 +106,14 @@ end
80
106
  on("user_updated") do |event|
81
107
  # ...
82
108
  end
109
+ on("user_created", "user_updated") do |event|
110
+ # ...
111
+ end
83
112
  ```
84
113
 
85
114
  3. Start the `tobox` process
86
115
 
87
- ```
116
+ ```bash
88
117
  > bundle exec tobox -C path/to/tobox.rb -r path/to/file_requiring_application_code.rb
89
118
  ```
90
119
 
@@ -135,11 +164,12 @@ CREATE TRIGGER order_created_outbox_event
135
164
  EXECUTE PROCEDURE order_created_outbox_event();
136
165
  ```
137
166
 
167
+ <a id="markdown-configuration" name="configuration"></a>
138
168
  ## Configuration
139
169
 
140
170
  As mentioned above, configuration can be set in a particular file. The following options are configurable:
141
171
 
142
- ### `environment``
172
+ ### `environment`
143
173
 
144
174
  Sets the application environment (either "development" or "production"). Can be set directly, or via `APP_ENV` environment variable (defaults to "development").
145
175
 
@@ -228,6 +258,15 @@ callback executed when an exception was raised while processing an event.
228
258
  on_error_event { |event, exception| Sentry.capture_exception(exception) }
229
259
  ```
230
260
 
261
+ ### `on_error_worker { |error| }`
262
+
263
+ callback executed when an exception was raised in the worker, before processing events.
264
+
265
+
266
+ ```ruby
267
+ on_error_worker { |exception| Sentry.capture_exception(exception) }
268
+ ```
269
+
231
270
  ### `message_to_arguments { |event| }`
232
271
 
233
272
  if exposing raw data to the `on` handlers is not what you'd want, you can always override the behaviour by providing an alternative "before/after fetcher" implementation.
@@ -257,6 +296,19 @@ Overrides the internal logger (an instance of `Logger`).
257
296
 
258
297
  Overrides the default log level ("info" when in "production" environment, "debug" otherwise).
259
298
 
299
+ ### group_column
300
+
301
+ Defines the column to be used for event grouping, when [ordered processing of events is a requirement](#ordered-event-processing).
302
+
303
+ ### inbox table
304
+
305
+ Defines the name of the table to be used for inbox, when [inbox usage is a requirement](#inbox).
306
+
307
+ ### inbox column
308
+
309
+ Defines the column in the outbox table which references the inbox table, when one is set.
310
+
311
+ <a id="markdown-event" name="event"></a>
260
312
  ## Event
261
313
 
262
314
  The event is composed of the following properties:
@@ -269,20 +321,110 @@ The event is composed of the following properties:
269
321
 
270
322
  (*NOTE*: The event is also composed of other properties which are only relevant for `tobox`.)
271
323
 
272
- ## Rails support
324
+ <a id="markdown-features" name="features"></a>
325
+ ## Features
273
326
 
274
- Rails is supported out of the box by adding the [sequel-activerecord_connection](https://github.com/janko/sequel-activerecord_connection) gem into your Gemfile, and requiring the rails application in the `tobox` cli call:
327
+ There are a few extra features you can run on top a "vanilla" transactional outbox implementation. This is how you can accomplish them using `tobox`.
275
328
 
276
- ```bash
277
- > bundle exec tobox -C path/to/tobox.rb -r path/to/rails_app/config/environment.rb
329
+ <a id="markdown-ordered-event-processing" name="ordered-event-processing"></a>
330
+ ### Ordered event processing
331
+
332
+ By default, events are taken and processed from the "outbox" table concurrently by workers, which means that, while worker A may process the most recent event, and worker B takes the following, worker B may process it faster than worker A. This may be an issue if the consumer expects events from a certain context to arrive in a certain order.
333
+
334
+ One solution is to have a single worker processing the "outbox" events. Another is to use the `group_column` configuration.
335
+
336
+ What you have to do is:
337
+
338
+ 1. add a "group id" column to the "outbox" table
339
+
340
+ ```ruby
341
+ create_table(:outbox) do
342
+ primary_key :id
343
+ column :group_id, :integer
344
+ # The type is irrelevant, could also be :string, :uuid...
345
+ # ..
278
346
  ```
279
347
 
280
- In the `tobox` config, you can set the environment:
348
+ 2. set the "group_column" configuration
281
349
 
282
350
  ```ruby
283
- environment Rails.env
351
+ # in your tobox.rb
352
+ group_column :group_id
353
+ index :group_id
354
+ ```
355
+
356
+ 3. insert related outbox events with the same group id
357
+
358
+ ```ruby
359
+ order = Order.new(
360
+ item_id: item.id,
361
+ price: 20_20,
362
+ currency: "EUR"
363
+ )
364
+ DB.transaction do
365
+ order.save
366
+ DB[:outbox].insert(event_type: "order_created", group_id: order.id, data_after: order.to_hash)
367
+ DB[:outbox].insert(event_type: "billing_event_started", group_id: order.id, data_after: order.to_hash)
368
+ end
369
+
370
+ # "order_created" will be processed first
371
+ # "billing_event_created" will only start processing once "order_created" finishes
372
+ ```
373
+ <a id="inbox" name="inbox"></a>
374
+ ### Inbox
375
+
376
+ `tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether).
377
+
378
+ In order to do so, you'll have to:
379
+
380
+ 1. add an "inbox" table in the database
381
+
382
+ ```ruby
383
+ create_table(:inbox) do
384
+ column :inbox_id, :varchar, null: true, primary_key: true # it can also be a uuid, you decide
385
+ column :created_at, "timestamp without time zone", null: false, default: Sequel::CURRENT_TIMESTAMP
386
+ end
387
+ ```
388
+
389
+ 2. add the unique id reference in the outbox table:
390
+
391
+ ```ruby
392
+ create_table(:outbox) do
393
+ primary_key :id
394
+ column :type, :varchar, null: false
395
+ column :inbox_id, :varchar, null: true
396
+ # ...
397
+ foreign_key :inbox_id, :inbox
398
+ ```
399
+
400
+ 3. reference them in the configuration
401
+
402
+ ```ruby
403
+ # tobox.rb
404
+ inbox_table :inbox
405
+ inbox_column :inbox_id
406
+ ```
407
+
408
+ 4. insert related outbox events with an inbox id
409
+
410
+ ```ruby
411
+ order = Order.new(
412
+ item_id: item.id,
413
+ price: 20_20,
414
+ currency: "EUR"
415
+ )
416
+ DB.transaction do
417
+ order.save
418
+ DB[:outbox].insert(event_type: "order_created", inbox_id: "ord_crt_#{order.id}", data_after: order.to_hash)
419
+ DB[:outbox].insert(event_type: "billing_event_started", inbox_id: "bil_evt_std_#{order.id}", data_after: order.to_hash)
420
+ end
421
+
422
+ # assuming this bit above runs two times in two separate workers, each will be processed by tobox only once.
284
423
  ```
285
424
 
425
+ **NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again.
426
+
427
+ <a id="markdown-plugins" name="plugins"></a>
286
428
  ## Plugins
287
429
 
288
430
  `tobox` ships with a very simple plugin system. (TODO: add docs).
@@ -296,6 +438,7 @@ plugin(:plugin_name)
296
438
 
297
439
  It ships with the following integrations.
298
440
 
441
+ <a id="markdown-zeitwerk" name="zeitwerk"></a>
299
442
  ### Zeitwerk
300
443
 
301
444
  (requires the `zeitwerk` gem.)
@@ -310,6 +453,7 @@ zeitwerk_loader do |loader|
310
453
  end
311
454
  ```
312
455
 
456
+ <a id="markdown-sentry" name="sentry"></a>
313
457
  ### Sentry
314
458
 
315
459
  (requires the `sentry-ruby` gem.)
@@ -321,6 +465,7 @@ Plugin for the [sentry](https://github.com/getsentry/sentry-ruby) ruby SDK for e
321
465
  plugin(:sentry)
322
466
  ```
323
467
 
468
+ <a id="markdown-datadog" name="datadog"></a>
324
469
  ### Datadog
325
470
 
326
471
  (requires the `ddtrace` gem.)
@@ -337,10 +482,28 @@ end
337
482
  plugin(:datadog)
338
483
  ```
339
484
 
485
+ <a id="markdown-supported-rubies" name="supported-rubies"></a>
340
486
  ## Supported Rubies
341
487
 
342
488
  All Rubies greater or equal to 2.6, and always latest JRuby and Truffleruby.
343
489
 
490
+
491
+ <a id="markdown-rails-support" name="rails-support"></a>
492
+ ## Rails support
493
+
494
+ Rails is supported out of the box by adding the [sequel-activerecord_connection](https://github.com/janko/sequel-activerecord_connection) gem into your Gemfile, and requiring the rails application in the `tobox` cli call:
495
+
496
+ ```bash
497
+ > bundle exec tobox -C path/to/tobox.rb -r path/to/rails_app/config/environment.rb
498
+ ```
499
+
500
+ In the `tobox` config, you can set the environment:
501
+
502
+ ```ruby
503
+ environment Rails.env
504
+ ```
505
+
506
+ <a id="markdown-why" name="why"></a>
344
507
  ## Why?
345
508
 
346
509
  ### Simple and lightweight, framework (and programming language) agnostic
@@ -375,10 +538,12 @@ By using the database as the message broker, `tobox` can rely on good old transa
375
538
 
376
539
  (The actual processing may change this to "at least once", as issues may happen before the event is successfully deleted from the outbox. Still, "at least once" is acceptable and solvable using idempotency mechanisms).
377
540
 
541
+ <a id="markdown-development" name="development"></a>
378
542
  ## Development
379
543
 
380
544
  After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
381
545
 
546
+ <a id="markdown-contributing" name="contributing"></a>
382
547
  ## Contributing
383
548
 
384
- Bug reports and pull requests are welcome on GitHub at https://gitlab.com/honeyryderchuck/tobox.
549
+ Bug reports and pull requests are welcome on GitHub at https://gitlab.com/os85/tobox.
@@ -5,10 +5,6 @@ module Tobox
5
5
  def initialize(configuration)
6
6
  @configuration = configuration
7
7
  @running = false
8
- end
9
-
10
- def start
11
- return if @running
12
8
 
13
9
  worker = @configuration[:worker]
14
10
 
@@ -17,7 +13,12 @@ module Tobox
17
13
  when :fiber then FiberPool
18
14
  else worker
19
15
  end.new(@configuration)
16
+ end
17
+
18
+ def start
19
+ return if @running
20
20
 
21
+ @pool.start
21
22
  @running = true
22
23
  end
23
24
 
@@ -17,6 +17,9 @@ module Tobox
17
17
  log_level: nil,
18
18
  database_uri: nil,
19
19
  table: :outbox,
20
+ group_column: nil,
21
+ inbox_table: nil,
22
+ inbox_column: nil,
20
23
  max_attempts: 10,
21
24
  exponential_retry_factor: 4,
22
25
  wait_for_events_delay: 5,
@@ -60,8 +63,10 @@ module Tobox
60
63
  freeze
61
64
  end
62
65
 
63
- def on(event, &callback)
64
- (@handlers[event.to_sym] ||= []) << callback
66
+ def on(*events, &callback)
67
+ events.each do |event|
68
+ (@handlers[event.to_sym] ||= []) << callback
69
+ end
65
70
  self
66
71
  end
67
72
 
@@ -80,6 +85,11 @@ module Tobox
80
85
  self
81
86
  end
82
87
 
88
+ def on_error_worker(&callback)
89
+ (@lifecycle_events[:error_worker] ||= []) << callback
90
+ self
91
+ end
92
+
83
93
  def message_to_arguments(&callback)
84
94
  @arguments_handler = callback
85
95
  self
data/lib/tobox/fetcher.rb CHANGED
@@ -19,10 +19,14 @@ module Tobox
19
19
  @db.loggers << @logger unless @configuration[:environment] == "production"
20
20
 
21
21
  @table = configuration[:table]
22
+ @group_column = configuration[:group_column]
22
23
  @exponential_retry_factor = configuration[:exponential_retry_factor]
23
24
 
24
25
  max_attempts = configuration[:max_attempts]
25
26
 
27
+ @inbox_table = configuration[:inbox_table]
28
+ @inbox_column = configuration[:inbox_column]
29
+
26
30
  @ds = @db[@table]
27
31
 
28
32
  run_at_conds = [
@@ -33,9 +37,6 @@ module Tobox
33
37
  @pick_next_sql = @ds.where(Sequel[@table][:attempts] < max_attempts) # filter out exhausted attempts
34
38
  .where(run_at_conds)
35
39
  .order(Sequel.desc(:run_at, nulls: :first), :id)
36
- .for_update
37
- .skip_locked
38
- .limit(1)
39
40
 
40
41
  @before_event_handlers = Array(@configuration.lifecycle_events[:before_event])
41
42
  @after_event_handlers = Array(@configuration.lifecycle_events[:after_event])
@@ -44,8 +45,34 @@ module Tobox
44
45
 
45
46
  def fetch_events(&blk)
46
47
  num_events = 0
47
- @db.transaction do
48
- event_ids = @pick_next_sql.select_map(:id) # lock starts here
48
+ @db.transaction(savepoint: false) do
49
+ if @group_column
50
+ group = @pick_next_sql.for_update
51
+ .skip_locked
52
+ .limit(1)
53
+ .select(@group_column)
54
+
55
+ # get total from a group, to compare to the number of future locked rows.
56
+ total_from_group = @ds.where(@group_column => group).count
57
+
58
+ event_ids = @ds.where(@group_column => group)
59
+ .order(Sequel.desc(:run_at, nulls: :first), :id)
60
+ .for_update.skip_locked.select_map(:id)
61
+
62
+ if event_ids.size != total_from_group
63
+ # this happens if concurrent workers locked different rows from the same group,
64
+ # or when new rows from a given group have been inserted after the lock has been
65
+ # acquired
66
+ event_ids = []
67
+ end
68
+
69
+ # lock all, process 1
70
+ event_ids = event_ids[0, 1]
71
+ else
72
+ event_ids = @pick_next_sql.for_update
73
+ .skip_locked
74
+ .limit(1).select_map(:id) # lock starts here
75
+ end
49
76
 
50
77
  events = nil
51
78
  error = nil
@@ -56,14 +83,17 @@ module Tobox
56
83
  if blk
57
84
  num_events = events.size
58
85
 
59
- events.each do |ev|
60
- ev[:metadata] = try_json_parse(ev[:metadata])
61
- handle_before_event(ev)
62
- yield(to_message(ev))
86
+ events.map! do |ev|
87
+ try_insert_inbox(ev) do
88
+ ev[:metadata] = try_json_parse(ev[:metadata])
89
+ handle_before_event(ev)
90
+ yield(to_message(ev))
91
+ ev
92
+ end
63
93
  rescue StandardError => e
64
94
  error = e
65
95
  raise Sequel::Rollback
66
- end
96
+ end.compact!
67
97
  else
68
98
  events.map!(&method(:to_message))
69
99
  end
@@ -124,6 +154,16 @@ module Tobox
124
154
  data
125
155
  end
126
156
 
157
+ def try_insert_inbox(event)
158
+ return yield unless @inbox_table && @inbox_column
159
+
160
+ ret = @db[@inbox_table].insert_conflict.insert(@inbox_column => event[@inbox_column])
161
+
162
+ return unless ret
163
+
164
+ yield
165
+ end
166
+
127
167
  def handle_before_event(event)
128
168
  @logger.debug do
129
169
  log_message("outbox event (type: \"#{event[:type]}\", attempts: #{event[:attempts]}) starting...")
@@ -130,6 +130,10 @@ module Tobox
130
130
  config.on_after_event(&event_handler.method(:on_finish))
131
131
  config.on_error_event(&event_handler.method(:on_error))
132
132
 
133
+ config.on_error_worker do |error|
134
+ ::Sentry.capture_exception(error, hint: { background: false })
135
+ end
136
+
133
137
  ::Sentry::Configuration.attr_reader(:tobox)
134
138
  ::Sentry::Configuration.add_post_initialization_callback do
135
139
  @tobox = Plugins::Sentry::Configuration.new
@@ -5,12 +5,9 @@ require "fiber_scheduler"
5
5
 
6
6
  module Tobox
7
7
  class FiberPool < Pool
8
- class KillError < Interrupt; end
9
-
10
8
  def initialize(_configuration)
11
9
  Sequel.extension(:fiber_concurrency)
12
10
  super
13
- @error_handlers = Array(@configuration.lifecycle_events[:error])
14
11
  end
15
12
 
16
13
  def start
@@ -19,14 +16,7 @@ module Tobox
19
16
 
20
17
  FiberScheduler do
21
18
  @workers.each_with_index do |wk, _idx|
22
- Fiber.schedule do
23
- wk.work
24
- rescue KillError
25
- # noop
26
- rescue Exception => e # rubocop:disable Lint/RescueException
27
- @error_handlers.each { |hd| hd.call(:tobox_error, e) }
28
- raise e
29
- end
19
+ Fiber.schedule { do_work(wk) }
30
20
  end
31
21
  end
32
22
  end
@@ -1,32 +1,22 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "monitor"
4
+
3
5
  module Tobox
4
6
  class ThreadedPool < Pool
5
- class KillError < Interrupt; end
6
-
7
7
  def initialize(_configuration)
8
+ @parent_thread = Thread.main
8
9
  @threads = []
10
+ @threads.extend(MonitorMixin)
9
11
  super
10
- @error_handlers = Array(@configuration.lifecycle_events[:error])
11
12
  end
12
13
 
13
14
  def start
14
- @workers.each_with_index do |wk, idx|
15
- th = Thread.start do
16
- Thread.current.name = "tobox-worker-#{idx}"
17
-
18
- begin
19
- wk.work
20
- rescue KillError
21
- # noop
22
- rescue Exception => e # rubocop:disable Lint/RescueException
23
- @error_handlers.each { |hd| hd.call(:tobox_error, e) }
24
- raise e
25
- end
26
-
27
- @threads.delete(Thread.current)
15
+ @workers.each do |wk|
16
+ th = start_thread_worker(wk)
17
+ @threads.synchronize do
18
+ @threads << th
28
19
  end
29
- @threads << th
30
20
  end
31
21
  end
32
22
 
@@ -51,5 +41,30 @@ module Tobox
51
41
  th.value # waits
52
42
  end
53
43
  end
44
+
45
+ private
46
+
47
+ def start_thread_worker(wrk)
48
+ Thread.start(wrk) do |worker|
49
+ Thread.current.name = worker.label
50
+
51
+ do_work(worker)
52
+
53
+ @threads.synchronize do
54
+ @threads.delete(Thread.current)
55
+
56
+ if worker.finished? && @running
57
+ idx = @workers.index(worker)
58
+
59
+ subst_worker = Worker.new(worker.label, @configuration)
60
+ @workers[idx] = subst_worker
61
+ subst_thread = start_thread_worker(subst_worker)
62
+ @threads << subst_thread
63
+ end
64
+ # all workers went down abruply, we need to kill the process.
65
+ # @parent_thread.raise(Interrupt) if wk.finished? && @threads.empty? && @running
66
+ end
67
+ end
68
+ end
54
69
  end
55
70
  end
data/lib/tobox/pool.rb CHANGED
@@ -2,17 +2,39 @@
2
2
 
3
3
  module Tobox
4
4
  class Pool
5
+ class KillError < Interrupt; end
6
+
5
7
  def initialize(configuration)
6
8
  @configuration = configuration
9
+ @logger = @configuration.default_logger
7
10
  @num_workers = configuration[:concurrency]
8
11
  @workers = Array.new(@num_workers) do |idx|
9
12
  Worker.new("tobox-worker-#{idx}", configuration)
10
13
  end
11
- start
14
+ @worker_error_handlers = Array(@configuration.lifecycle_events[:error_worker])
15
+ @running = true
12
16
  end
13
17
 
14
18
  def stop
19
+ return unless @running
20
+
15
21
  @workers.each(&:finish!)
22
+ @running = false
23
+ end
24
+
25
+ def do_work(wrk)
26
+ wrk.work
27
+ rescue KillError
28
+ # noop
29
+ rescue Exception => e # rubocop:disable Lint/RescueException
30
+ wrk.finish!
31
+ @logger.error do
32
+ "(worker: #{wrk.label}) -> " \
33
+ "crashed with error\n" \
34
+ "#{e.class}: #{e.message}\n" \
35
+ "#{e.backtrace.join("\n")}"
36
+ end
37
+ @worker_error_handlers.each { |hd| hd.call(e) }
16
38
  end
17
39
  end
18
40
 
data/lib/tobox/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Tobox
4
- VERSION = "0.1.6"
4
+ VERSION = "0.3.0"
5
5
  end
data/lib/tobox/worker.rb CHANGED
@@ -2,7 +2,10 @@
2
2
 
3
3
  module Tobox
4
4
  class Worker
5
+ attr_reader :label
6
+
5
7
  def initialize(label, configuration)
8
+ @label = label
6
9
  @wait_for_events_delay = configuration[:wait_for_events_delay]
7
10
  @handlers = configuration.handlers || {}
8
11
  @fetcher = Fetcher.new(label, configuration)
@@ -13,6 +16,10 @@ module Tobox
13
16
  define_singleton_method(:message_to_arguments, &message_to_arguments)
14
17
  end
15
18
 
19
+ def finished?
20
+ @finished
21
+ end
22
+
16
23
  def finish!
17
24
  @finished = true
18
25
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: tobox
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.6
4
+ version: 0.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - HoneyryderChuck
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2022-10-06 00:00:00.000000000 Z
11
+ date: 2022-12-21 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: sequel
@@ -52,15 +52,15 @@ files:
52
52
  - lib/tobox/pool/threaded_pool.rb
53
53
  - lib/tobox/version.rb
54
54
  - lib/tobox/worker.rb
55
- homepage: https://gitlab.com/honeyryderchuck/tobox
55
+ homepage: https://gitlab.com/os85/tobox
56
56
  licenses: []
57
57
  metadata:
58
- homepage_uri: https://gitlab.com/honeyryderchuck/tobox
58
+ homepage_uri: https://gitlab.com/os85/tobox
59
59
  allowed_push_host: https://rubygems.org
60
- source_code_uri: https://gitlab.com/honeyryderchuck/tobox
61
- bug_tracker_uri: https://gitlab.com/honeyryderchuck/tobox/issues
62
- documentation_uri: https://gitlab.com/honeyryderchuck/tobox
63
- changelog_uri: https://gitlab.com/honeyryderchuck/tobox/-/blob/master/CHANGELOG.md
60
+ source_code_uri: https://gitlab.com/os85/tobox
61
+ bug_tracker_uri: https://gitlab.com/os85/tobox/issues
62
+ documentation_uri: https://gitlab.com/os85/tobox
63
+ changelog_uri: https://gitlab.com/os85/tobox/-/blob/master/CHANGELOG.md
64
64
  rubygems_mfa_required: 'true'
65
65
  post_install_message:
66
66
  rdoc_options: []