tobox 0.1.6 → 0.3.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +69 -6
- data/README.md +177 -12
- data/lib/tobox/application.rb +5 -4
- data/lib/tobox/configuration.rb +12 -2
- data/lib/tobox/fetcher.rb +50 -10
- data/lib/tobox/plugins/sentry.rb +4 -0
- data/lib/tobox/pool/fiber_pool.rb +1 -11
- data/lib/tobox/pool/threaded_pool.rb +33 -18
- data/lib/tobox/pool.rb +23 -1
- data/lib/tobox/version.rb +1 -1
- data/lib/tobox/worker.rb +7 -0
- metadata +8 -8
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 33a40c07b2d7f36933540d390287aeeb7d4e0aa39a7d19a439eaca6a2f8fc23f
|
4
|
+
data.tar.gz: 939699d801afdb03ed85c0048eee3b00c50a786a345b0731c6dba910fafb51ad
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5f5567a9f8e43fbc0f22e10465a0026a4984932183ae7d70c6fa95cdcb092995cbedd44422fa8a65b6c2d321be3ae15a0b72e905edd43c96427fbaac7cc73e34
|
7
|
+
data.tar.gz: f7823d88490ab6edc02ed6a8fd846f7e6cf40ce042fb046e502e3024c26f24ced5ee143de80669ffad49d5077dc97e338c363eda1e0ea1d3196479447c6aa7f6
|
data/CHANGELOG.md
CHANGED
@@ -1,36 +1,99 @@
|
|
1
1
|
## [Unreleased]
|
2
2
|
|
3
|
-
## [0.
|
3
|
+
## [0.3.0] - 2022-12-12
|
4
|
+
|
5
|
+
### Features
|
6
|
+
|
7
|
+
#### Inbox
|
8
|
+
|
9
|
+
Implementation of the "inbox pattern", which ensures that events are processed to completion only once.
|
10
|
+
|
11
|
+
```ruby
|
12
|
+
# create an inbox table and reference it
|
13
|
+
create_table(:inbox) do
|
14
|
+
column :id, :varchar, null: true, primary_key: true
|
15
|
+
# ...
|
16
|
+
create_table(:outbox) do
|
17
|
+
column :inbox_id, :varchar
|
18
|
+
foreign_key :inbox_id, :inbox
|
19
|
+
# ...
|
20
|
+
|
21
|
+
# tobox.rb
|
22
|
+
inbox_table :inbox
|
23
|
+
inbox_column :inbox_id
|
24
|
+
|
25
|
+
# event production
|
26
|
+
DB[:outbox].insert(event_type: "order_created", inbox_id: "order_created_#{order.id}", ....
|
27
|
+
DB[:outbox].insert(event_type: "billing_event_started", inbox_id: "billing_event_started_#{order.id}", ....
|
28
|
+
```
|
29
|
+
|
30
|
+
## [0.2.0] - 2022-12-05
|
31
|
+
|
32
|
+
### Features
|
33
|
+
|
34
|
+
#### Ordered event processing
|
35
|
+
|
36
|
+
When the outbox table contains a `:group_id` table (and the producer fills up events with it), then a group of events with the same `:group_id` will be processed one by one, by order of insertion.
|
37
|
+
|
38
|
+
```ruby
|
39
|
+
# migration
|
40
|
+
create_table(:outbox) do
|
41
|
+
column :message_group_id, :integer
|
42
|
+
|
43
|
+
# tobox.rb
|
44
|
+
message_group_column :group_id
|
45
|
+
|
46
|
+
# event production
|
47
|
+
DB[:outbox].insert(event_type: "order_created", message_group_id: order.id, ....
|
48
|
+
DB[:outbox].insert(event_type: "billing_event_started", message_group_id: order.id, ....
|
49
|
+
|
50
|
+
# order_created handled first, billing_event_started only after
|
51
|
+
```
|
52
|
+
|
53
|
+
#### on_error_worker callback
|
54
|
+
|
55
|
+
The config option `on_error_worker { |error| }` gets called when an error happens in a worker **before** events are processed (p.ex. when the database connection becomes unhealthy). You can use it to report such errors to an error reporting system (the `sentry` plugin relies on it).
|
56
|
+
|
57
|
+
```ruby
|
58
|
+
# tobox.rb
|
59
|
+
on_error_worker { |error| Sentry.capture_exception(error, hint: { background: false }) }
|
60
|
+
```
|
61
|
+
|
62
|
+
### Bugfixes
|
63
|
+
|
64
|
+
Thread workers: when errors happen which bring down the workers (such as database becoming unresponsive), workers will be restarted.
|
65
|
+
|
66
|
+
## [0.1.6] - 2022-10-06
|
4
67
|
|
5
68
|
### Bugfixes
|
6
69
|
|
7
70
|
Allow passing datadog options, initialize tracing from plugin.
|
8
71
|
|
9
|
-
## [0.1.5] -
|
72
|
+
## [0.1.5] - 2022-10-06
|
10
73
|
|
11
74
|
### Bugfixes
|
12
75
|
|
13
76
|
Fixing datadog plugin name.
|
14
77
|
|
15
|
-
## [0.1.4] -
|
78
|
+
## [0.1.4] - 2022-10-06
|
16
79
|
|
17
80
|
### Bugfixes
|
18
81
|
|
19
82
|
Actual fix for missing datadog constants.
|
20
83
|
|
21
|
-
## [0.1.3] -
|
84
|
+
## [0.1.3] - 2022-10-06
|
22
85
|
|
23
86
|
### Bugfixes
|
24
87
|
|
25
88
|
Datadog constants unproperly namespaced.
|
26
89
|
|
27
|
-
## [0.1.2] -
|
90
|
+
## [0.1.2] - 2022-09-14
|
28
91
|
|
29
92
|
### Bugfixes
|
30
93
|
|
31
94
|
Actual fix for foregoing json parsing.
|
32
95
|
|
33
|
-
## [0.1.1] -
|
96
|
+
## [0.1.1] - 2022-09-14
|
34
97
|
|
35
98
|
### Chore
|
36
99
|
|
data/README.md
CHANGED
@@ -1,11 +1,34 @@
|
|
1
1
|
# Tobox: Transactional outbox pattern implementation in ruby
|
2
2
|
|
3
3
|
[![Gem Version](https://badge.fury.io/rb/tobox.svg)](http://rubygems.org/gems/tobox)
|
4
|
-
[![pipeline status](https://gitlab.com/
|
5
|
-
[![coverage report](https://gitlab.com/
|
4
|
+
[![pipeline status](https://gitlab.com/os85/tobox/badges/master/pipeline.svg)](https://gitlab.com/os85/tobox/pipelines?page=1&scope=all&ref=master)
|
5
|
+
[![coverage report](https://gitlab.com/os85/tobox/badges/master/coverage.svg?job=coverage)](https://os85.gitlab.io/tobox/#_AllFiles)
|
6
6
|
|
7
7
|
Simple, data-first events processing framework based on the [transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html).
|
8
8
|
|
9
|
+
<!-- TOC -->
|
10
|
+
|
11
|
+
- [Requirements](#requirements)
|
12
|
+
- [Installation](#installation)
|
13
|
+
- [Usage](#usage)
|
14
|
+
- [Configuration](#configuration)
|
15
|
+
- [Event](#event)
|
16
|
+
- [Features](#features)
|
17
|
+
- [Ordered event processing](#ordered-event-processing)
|
18
|
+
- [Inbox](#inbox)
|
19
|
+
- [Plugins](#plugins)
|
20
|
+
- [Zeitwerk](#zeitwerk)
|
21
|
+
- [Sentry](#sentry)
|
22
|
+
- [Datadog](#datadog)
|
23
|
+
- [Supported Rubies](#supported-rubies)
|
24
|
+
- [Rails support](#rails-support)
|
25
|
+
- [Why?](#why)
|
26
|
+
- [Development](#development)
|
27
|
+
- [Contributing](#contributing)
|
28
|
+
|
29
|
+
<!-- /TOC -->
|
30
|
+
|
31
|
+
<a id="markdown-requirements" name="requirements"></a>
|
9
32
|
## Requirements
|
10
33
|
|
11
34
|
`tobox` requires integration with RDBMS which supports `SKIP LOCKED` functionality. As of today, that's:
|
@@ -15,6 +38,7 @@ Simple, data-first events processing framework based on the [transactional outbo
|
|
15
38
|
* Oracle
|
16
39
|
* Microsoft SQL Server
|
17
40
|
|
41
|
+
<a id="markdown-installation" name="installation"></a>
|
18
42
|
## Installation
|
19
43
|
|
20
44
|
Add this line to your application's Gemfile:
|
@@ -22,7 +46,7 @@ Add this line to your application's Gemfile:
|
|
22
46
|
```ruby
|
23
47
|
gem "tobox"
|
24
48
|
|
25
|
-
# You'll also need to
|
49
|
+
# You'll also need to add the right database client gem for the target RDBMS
|
26
50
|
# ex, for postgresql:
|
27
51
|
#
|
28
52
|
# gem "pg"
|
@@ -37,6 +61,8 @@ Or install it yourself as:
|
|
37
61
|
|
38
62
|
$ gem install tobox
|
39
63
|
|
64
|
+
|
65
|
+
<a id="markdown-usage" name="usage"></a>
|
40
66
|
## Usage
|
41
67
|
|
42
68
|
1. create the `outbox` table in your application's database:
|
@@ -80,11 +106,14 @@ end
|
|
80
106
|
on("user_updated") do |event|
|
81
107
|
# ...
|
82
108
|
end
|
109
|
+
on("user_created", "user_updated") do |event|
|
110
|
+
# ...
|
111
|
+
end
|
83
112
|
```
|
84
113
|
|
85
114
|
3. Start the `tobox` process
|
86
115
|
|
87
|
-
```
|
116
|
+
```bash
|
88
117
|
> bundle exec tobox -C path/to/tobox.rb -r path/to/file_requiring_application_code.rb
|
89
118
|
```
|
90
119
|
|
@@ -135,11 +164,12 @@ CREATE TRIGGER order_created_outbox_event
|
|
135
164
|
EXECUTE PROCEDURE order_created_outbox_event();
|
136
165
|
```
|
137
166
|
|
167
|
+
<a id="markdown-configuration" name="configuration"></a>
|
138
168
|
## Configuration
|
139
169
|
|
140
170
|
As mentioned above, configuration can be set in a particular file. The following options are configurable:
|
141
171
|
|
142
|
-
### `environment
|
172
|
+
### `environment`
|
143
173
|
|
144
174
|
Sets the application environment (either "development" or "production"). Can be set directly, or via `APP_ENV` environment variable (defaults to "development").
|
145
175
|
|
@@ -228,6 +258,15 @@ callback executed when an exception was raised while processing an event.
|
|
228
258
|
on_error_event { |event, exception| Sentry.capture_exception(exception) }
|
229
259
|
```
|
230
260
|
|
261
|
+
### `on_error_worker { |error| }`
|
262
|
+
|
263
|
+
callback executed when an exception was raised in the worker, before processing events.
|
264
|
+
|
265
|
+
|
266
|
+
```ruby
|
267
|
+
on_error_worker { |exception| Sentry.capture_exception(exception) }
|
268
|
+
```
|
269
|
+
|
231
270
|
### `message_to_arguments { |event| }`
|
232
271
|
|
233
272
|
if exposing raw data to the `on` handlers is not what you'd want, you can always override the behaviour by providing an alternative "before/after fetcher" implementation.
|
@@ -257,6 +296,19 @@ Overrides the internal logger (an instance of `Logger`).
|
|
257
296
|
|
258
297
|
Overrides the default log level ("info" when in "production" environment, "debug" otherwise).
|
259
298
|
|
299
|
+
### group_column
|
300
|
+
|
301
|
+
Defines the column to be used for event grouping, when [ordered processing of events is a requirement](#ordered-event-processing).
|
302
|
+
|
303
|
+
### inbox table
|
304
|
+
|
305
|
+
Defines the name of the table to be used for inbox, when [inbox usage is a requirement](#inbox).
|
306
|
+
|
307
|
+
### inbox column
|
308
|
+
|
309
|
+
Defines the column in the outbox table which references the inbox table, when one is set.
|
310
|
+
|
311
|
+
<a id="markdown-event" name="event"></a>
|
260
312
|
## Event
|
261
313
|
|
262
314
|
The event is composed of the following properties:
|
@@ -269,20 +321,110 @@ The event is composed of the following properties:
|
|
269
321
|
|
270
322
|
(*NOTE*: The event is also composed of other properties which are only relevant for `tobox`.)
|
271
323
|
|
272
|
-
|
324
|
+
<a id="markdown-features" name="features"></a>
|
325
|
+
## Features
|
273
326
|
|
274
|
-
|
327
|
+
There are a few extra features you can run on top a "vanilla" transactional outbox implementation. This is how you can accomplish them using `tobox`.
|
275
328
|
|
276
|
-
|
277
|
-
|
329
|
+
<a id="markdown-ordered-event-processing" name="ordered-event-processing"></a>
|
330
|
+
### Ordered event processing
|
331
|
+
|
332
|
+
By default, events are taken and processed from the "outbox" table concurrently by workers, which means that, while worker A may process the most recent event, and worker B takes the following, worker B may process it faster than worker A. This may be an issue if the consumer expects events from a certain context to arrive in a certain order.
|
333
|
+
|
334
|
+
One solution is to have a single worker processing the "outbox" events. Another is to use the `group_column` configuration.
|
335
|
+
|
336
|
+
What you have to do is:
|
337
|
+
|
338
|
+
1. add a "group id" column to the "outbox" table
|
339
|
+
|
340
|
+
```ruby
|
341
|
+
create_table(:outbox) do
|
342
|
+
primary_key :id
|
343
|
+
column :group_id, :integer
|
344
|
+
# The type is irrelevant, could also be :string, :uuid...
|
345
|
+
# ..
|
278
346
|
```
|
279
347
|
|
280
|
-
|
348
|
+
2. set the "group_column" configuration
|
281
349
|
|
282
350
|
```ruby
|
283
|
-
|
351
|
+
# in your tobox.rb
|
352
|
+
group_column :group_id
|
353
|
+
index :group_id
|
354
|
+
```
|
355
|
+
|
356
|
+
3. insert related outbox events with the same group id
|
357
|
+
|
358
|
+
```ruby
|
359
|
+
order = Order.new(
|
360
|
+
item_id: item.id,
|
361
|
+
price: 20_20,
|
362
|
+
currency: "EUR"
|
363
|
+
)
|
364
|
+
DB.transaction do
|
365
|
+
order.save
|
366
|
+
DB[:outbox].insert(event_type: "order_created", group_id: order.id, data_after: order.to_hash)
|
367
|
+
DB[:outbox].insert(event_type: "billing_event_started", group_id: order.id, data_after: order.to_hash)
|
368
|
+
end
|
369
|
+
|
370
|
+
# "order_created" will be processed first
|
371
|
+
# "billing_event_created" will only start processing once "order_created" finishes
|
372
|
+
```
|
373
|
+
<a id="inbox" name="inbox"></a>
|
374
|
+
### Inbox
|
375
|
+
|
376
|
+
`tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether).
|
377
|
+
|
378
|
+
In order to do so, you'll have to:
|
379
|
+
|
380
|
+
1. add an "inbox" table in the database
|
381
|
+
|
382
|
+
```ruby
|
383
|
+
create_table(:inbox) do
|
384
|
+
column :inbox_id, :varchar, null: true, primary_key: true # it can also be a uuid, you decide
|
385
|
+
column :created_at, "timestamp without time zone", null: false, default: Sequel::CURRENT_TIMESTAMP
|
386
|
+
end
|
387
|
+
```
|
388
|
+
|
389
|
+
2. add the unique id reference in the outbox table:
|
390
|
+
|
391
|
+
```ruby
|
392
|
+
create_table(:outbox) do
|
393
|
+
primary_key :id
|
394
|
+
column :type, :varchar, null: false
|
395
|
+
column :inbox_id, :varchar, null: true
|
396
|
+
# ...
|
397
|
+
foreign_key :inbox_id, :inbox
|
398
|
+
```
|
399
|
+
|
400
|
+
3. reference them in the configuration
|
401
|
+
|
402
|
+
```ruby
|
403
|
+
# tobox.rb
|
404
|
+
inbox_table :inbox
|
405
|
+
inbox_column :inbox_id
|
406
|
+
```
|
407
|
+
|
408
|
+
4. insert related outbox events with an inbox id
|
409
|
+
|
410
|
+
```ruby
|
411
|
+
order = Order.new(
|
412
|
+
item_id: item.id,
|
413
|
+
price: 20_20,
|
414
|
+
currency: "EUR"
|
415
|
+
)
|
416
|
+
DB.transaction do
|
417
|
+
order.save
|
418
|
+
DB[:outbox].insert(event_type: "order_created", inbox_id: "ord_crt_#{order.id}", data_after: order.to_hash)
|
419
|
+
DB[:outbox].insert(event_type: "billing_event_started", inbox_id: "bil_evt_std_#{order.id}", data_after: order.to_hash)
|
420
|
+
end
|
421
|
+
|
422
|
+
# assuming this bit above runs two times in two separate workers, each will be processed by tobox only once.
|
284
423
|
```
|
285
424
|
|
425
|
+
**NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again.
|
426
|
+
|
427
|
+
<a id="markdown-plugins" name="plugins"></a>
|
286
428
|
## Plugins
|
287
429
|
|
288
430
|
`tobox` ships with a very simple plugin system. (TODO: add docs).
|
@@ -296,6 +438,7 @@ plugin(:plugin_name)
|
|
296
438
|
|
297
439
|
It ships with the following integrations.
|
298
440
|
|
441
|
+
<a id="markdown-zeitwerk" name="zeitwerk"></a>
|
299
442
|
### Zeitwerk
|
300
443
|
|
301
444
|
(requires the `zeitwerk` gem.)
|
@@ -310,6 +453,7 @@ zeitwerk_loader do |loader|
|
|
310
453
|
end
|
311
454
|
```
|
312
455
|
|
456
|
+
<a id="markdown-sentry" name="sentry"></a>
|
313
457
|
### Sentry
|
314
458
|
|
315
459
|
(requires the `sentry-ruby` gem.)
|
@@ -321,6 +465,7 @@ Plugin for the [sentry](https://github.com/getsentry/sentry-ruby) ruby SDK for e
|
|
321
465
|
plugin(:sentry)
|
322
466
|
```
|
323
467
|
|
468
|
+
<a id="markdown-datadog" name="datadog"></a>
|
324
469
|
### Datadog
|
325
470
|
|
326
471
|
(requires the `ddtrace` gem.)
|
@@ -337,10 +482,28 @@ end
|
|
337
482
|
plugin(:datadog)
|
338
483
|
```
|
339
484
|
|
485
|
+
<a id="markdown-supported-rubies" name="supported-rubies"></a>
|
340
486
|
## Supported Rubies
|
341
487
|
|
342
488
|
All Rubies greater or equal to 2.6, and always latest JRuby and Truffleruby.
|
343
489
|
|
490
|
+
|
491
|
+
<a id="markdown-rails-support" name="rails-support"></a>
|
492
|
+
## Rails support
|
493
|
+
|
494
|
+
Rails is supported out of the box by adding the [sequel-activerecord_connection](https://github.com/janko/sequel-activerecord_connection) gem into your Gemfile, and requiring the rails application in the `tobox` cli call:
|
495
|
+
|
496
|
+
```bash
|
497
|
+
> bundle exec tobox -C path/to/tobox.rb -r path/to/rails_app/config/environment.rb
|
498
|
+
```
|
499
|
+
|
500
|
+
In the `tobox` config, you can set the environment:
|
501
|
+
|
502
|
+
```ruby
|
503
|
+
environment Rails.env
|
504
|
+
```
|
505
|
+
|
506
|
+
<a id="markdown-why" name="why"></a>
|
344
507
|
## Why?
|
345
508
|
|
346
509
|
### Simple and lightweight, framework (and programming language) agnostic
|
@@ -375,10 +538,12 @@ By using the database as the message broker, `tobox` can rely on good old transa
|
|
375
538
|
|
376
539
|
(The actual processing may change this to "at least once", as issues may happen before the event is successfully deleted from the outbox. Still, "at least once" is acceptable and solvable using idempotency mechanisms).
|
377
540
|
|
541
|
+
<a id="markdown-development" name="development"></a>
|
378
542
|
## Development
|
379
543
|
|
380
544
|
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
381
545
|
|
546
|
+
<a id="markdown-contributing" name="contributing"></a>
|
382
547
|
## Contributing
|
383
548
|
|
384
|
-
Bug reports and pull requests are welcome on GitHub at https://gitlab.com/
|
549
|
+
Bug reports and pull requests are welcome on GitHub at https://gitlab.com/os85/tobox.
|
data/lib/tobox/application.rb
CHANGED
@@ -5,10 +5,6 @@ module Tobox
|
|
5
5
|
def initialize(configuration)
|
6
6
|
@configuration = configuration
|
7
7
|
@running = false
|
8
|
-
end
|
9
|
-
|
10
|
-
def start
|
11
|
-
return if @running
|
12
8
|
|
13
9
|
worker = @configuration[:worker]
|
14
10
|
|
@@ -17,7 +13,12 @@ module Tobox
|
|
17
13
|
when :fiber then FiberPool
|
18
14
|
else worker
|
19
15
|
end.new(@configuration)
|
16
|
+
end
|
17
|
+
|
18
|
+
def start
|
19
|
+
return if @running
|
20
20
|
|
21
|
+
@pool.start
|
21
22
|
@running = true
|
22
23
|
end
|
23
24
|
|
data/lib/tobox/configuration.rb
CHANGED
@@ -17,6 +17,9 @@ module Tobox
|
|
17
17
|
log_level: nil,
|
18
18
|
database_uri: nil,
|
19
19
|
table: :outbox,
|
20
|
+
group_column: nil,
|
21
|
+
inbox_table: nil,
|
22
|
+
inbox_column: nil,
|
20
23
|
max_attempts: 10,
|
21
24
|
exponential_retry_factor: 4,
|
22
25
|
wait_for_events_delay: 5,
|
@@ -60,8 +63,10 @@ module Tobox
|
|
60
63
|
freeze
|
61
64
|
end
|
62
65
|
|
63
|
-
def on(
|
64
|
-
|
66
|
+
def on(*events, &callback)
|
67
|
+
events.each do |event|
|
68
|
+
(@handlers[event.to_sym] ||= []) << callback
|
69
|
+
end
|
65
70
|
self
|
66
71
|
end
|
67
72
|
|
@@ -80,6 +85,11 @@ module Tobox
|
|
80
85
|
self
|
81
86
|
end
|
82
87
|
|
88
|
+
def on_error_worker(&callback)
|
89
|
+
(@lifecycle_events[:error_worker] ||= []) << callback
|
90
|
+
self
|
91
|
+
end
|
92
|
+
|
83
93
|
def message_to_arguments(&callback)
|
84
94
|
@arguments_handler = callback
|
85
95
|
self
|
data/lib/tobox/fetcher.rb
CHANGED
@@ -19,10 +19,14 @@ module Tobox
|
|
19
19
|
@db.loggers << @logger unless @configuration[:environment] == "production"
|
20
20
|
|
21
21
|
@table = configuration[:table]
|
22
|
+
@group_column = configuration[:group_column]
|
22
23
|
@exponential_retry_factor = configuration[:exponential_retry_factor]
|
23
24
|
|
24
25
|
max_attempts = configuration[:max_attempts]
|
25
26
|
|
27
|
+
@inbox_table = configuration[:inbox_table]
|
28
|
+
@inbox_column = configuration[:inbox_column]
|
29
|
+
|
26
30
|
@ds = @db[@table]
|
27
31
|
|
28
32
|
run_at_conds = [
|
@@ -33,9 +37,6 @@ module Tobox
|
|
33
37
|
@pick_next_sql = @ds.where(Sequel[@table][:attempts] < max_attempts) # filter out exhausted attempts
|
34
38
|
.where(run_at_conds)
|
35
39
|
.order(Sequel.desc(:run_at, nulls: :first), :id)
|
36
|
-
.for_update
|
37
|
-
.skip_locked
|
38
|
-
.limit(1)
|
39
40
|
|
40
41
|
@before_event_handlers = Array(@configuration.lifecycle_events[:before_event])
|
41
42
|
@after_event_handlers = Array(@configuration.lifecycle_events[:after_event])
|
@@ -44,8 +45,34 @@ module Tobox
|
|
44
45
|
|
45
46
|
def fetch_events(&blk)
|
46
47
|
num_events = 0
|
47
|
-
@db.transaction do
|
48
|
-
|
48
|
+
@db.transaction(savepoint: false) do
|
49
|
+
if @group_column
|
50
|
+
group = @pick_next_sql.for_update
|
51
|
+
.skip_locked
|
52
|
+
.limit(1)
|
53
|
+
.select(@group_column)
|
54
|
+
|
55
|
+
# get total from a group, to compare to the number of future locked rows.
|
56
|
+
total_from_group = @ds.where(@group_column => group).count
|
57
|
+
|
58
|
+
event_ids = @ds.where(@group_column => group)
|
59
|
+
.order(Sequel.desc(:run_at, nulls: :first), :id)
|
60
|
+
.for_update.skip_locked.select_map(:id)
|
61
|
+
|
62
|
+
if event_ids.size != total_from_group
|
63
|
+
# this happens if concurrent workers locked different rows from the same group,
|
64
|
+
# or when new rows from a given group have been inserted after the lock has been
|
65
|
+
# acquired
|
66
|
+
event_ids = []
|
67
|
+
end
|
68
|
+
|
69
|
+
# lock all, process 1
|
70
|
+
event_ids = event_ids[0, 1]
|
71
|
+
else
|
72
|
+
event_ids = @pick_next_sql.for_update
|
73
|
+
.skip_locked
|
74
|
+
.limit(1).select_map(:id) # lock starts here
|
75
|
+
end
|
49
76
|
|
50
77
|
events = nil
|
51
78
|
error = nil
|
@@ -56,14 +83,17 @@ module Tobox
|
|
56
83
|
if blk
|
57
84
|
num_events = events.size
|
58
85
|
|
59
|
-
events.
|
60
|
-
|
61
|
-
|
62
|
-
|
86
|
+
events.map! do |ev|
|
87
|
+
try_insert_inbox(ev) do
|
88
|
+
ev[:metadata] = try_json_parse(ev[:metadata])
|
89
|
+
handle_before_event(ev)
|
90
|
+
yield(to_message(ev))
|
91
|
+
ev
|
92
|
+
end
|
63
93
|
rescue StandardError => e
|
64
94
|
error = e
|
65
95
|
raise Sequel::Rollback
|
66
|
-
end
|
96
|
+
end.compact!
|
67
97
|
else
|
68
98
|
events.map!(&method(:to_message))
|
69
99
|
end
|
@@ -124,6 +154,16 @@ module Tobox
|
|
124
154
|
data
|
125
155
|
end
|
126
156
|
|
157
|
+
def try_insert_inbox(event)
|
158
|
+
return yield unless @inbox_table && @inbox_column
|
159
|
+
|
160
|
+
ret = @db[@inbox_table].insert_conflict.insert(@inbox_column => event[@inbox_column])
|
161
|
+
|
162
|
+
return unless ret
|
163
|
+
|
164
|
+
yield
|
165
|
+
end
|
166
|
+
|
127
167
|
def handle_before_event(event)
|
128
168
|
@logger.debug do
|
129
169
|
log_message("outbox event (type: \"#{event[:type]}\", attempts: #{event[:attempts]}) starting...")
|
data/lib/tobox/plugins/sentry.rb
CHANGED
@@ -130,6 +130,10 @@ module Tobox
|
|
130
130
|
config.on_after_event(&event_handler.method(:on_finish))
|
131
131
|
config.on_error_event(&event_handler.method(:on_error))
|
132
132
|
|
133
|
+
config.on_error_worker do |error|
|
134
|
+
::Sentry.capture_exception(error, hint: { background: false })
|
135
|
+
end
|
136
|
+
|
133
137
|
::Sentry::Configuration.attr_reader(:tobox)
|
134
138
|
::Sentry::Configuration.add_post_initialization_callback do
|
135
139
|
@tobox = Plugins::Sentry::Configuration.new
|
@@ -5,12 +5,9 @@ require "fiber_scheduler"
|
|
5
5
|
|
6
6
|
module Tobox
|
7
7
|
class FiberPool < Pool
|
8
|
-
class KillError < Interrupt; end
|
9
|
-
|
10
8
|
def initialize(_configuration)
|
11
9
|
Sequel.extension(:fiber_concurrency)
|
12
10
|
super
|
13
|
-
@error_handlers = Array(@configuration.lifecycle_events[:error])
|
14
11
|
end
|
15
12
|
|
16
13
|
def start
|
@@ -19,14 +16,7 @@ module Tobox
|
|
19
16
|
|
20
17
|
FiberScheduler do
|
21
18
|
@workers.each_with_index do |wk, _idx|
|
22
|
-
Fiber.schedule
|
23
|
-
wk.work
|
24
|
-
rescue KillError
|
25
|
-
# noop
|
26
|
-
rescue Exception => e # rubocop:disable Lint/RescueException
|
27
|
-
@error_handlers.each { |hd| hd.call(:tobox_error, e) }
|
28
|
-
raise e
|
29
|
-
end
|
19
|
+
Fiber.schedule { do_work(wk) }
|
30
20
|
end
|
31
21
|
end
|
32
22
|
end
|
@@ -1,32 +1,22 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require "monitor"
|
4
|
+
|
3
5
|
module Tobox
|
4
6
|
class ThreadedPool < Pool
|
5
|
-
class KillError < Interrupt; end
|
6
|
-
|
7
7
|
def initialize(_configuration)
|
8
|
+
@parent_thread = Thread.main
|
8
9
|
@threads = []
|
10
|
+
@threads.extend(MonitorMixin)
|
9
11
|
super
|
10
|
-
@error_handlers = Array(@configuration.lifecycle_events[:error])
|
11
12
|
end
|
12
13
|
|
13
14
|
def start
|
14
|
-
@workers.
|
15
|
-
th =
|
16
|
-
|
17
|
-
|
18
|
-
begin
|
19
|
-
wk.work
|
20
|
-
rescue KillError
|
21
|
-
# noop
|
22
|
-
rescue Exception => e # rubocop:disable Lint/RescueException
|
23
|
-
@error_handlers.each { |hd| hd.call(:tobox_error, e) }
|
24
|
-
raise e
|
25
|
-
end
|
26
|
-
|
27
|
-
@threads.delete(Thread.current)
|
15
|
+
@workers.each do |wk|
|
16
|
+
th = start_thread_worker(wk)
|
17
|
+
@threads.synchronize do
|
18
|
+
@threads << th
|
28
19
|
end
|
29
|
-
@threads << th
|
30
20
|
end
|
31
21
|
end
|
32
22
|
|
@@ -51,5 +41,30 @@ module Tobox
|
|
51
41
|
th.value # waits
|
52
42
|
end
|
53
43
|
end
|
44
|
+
|
45
|
+
private
|
46
|
+
|
47
|
+
def start_thread_worker(wrk)
|
48
|
+
Thread.start(wrk) do |worker|
|
49
|
+
Thread.current.name = worker.label
|
50
|
+
|
51
|
+
do_work(worker)
|
52
|
+
|
53
|
+
@threads.synchronize do
|
54
|
+
@threads.delete(Thread.current)
|
55
|
+
|
56
|
+
if worker.finished? && @running
|
57
|
+
idx = @workers.index(worker)
|
58
|
+
|
59
|
+
subst_worker = Worker.new(worker.label, @configuration)
|
60
|
+
@workers[idx] = subst_worker
|
61
|
+
subst_thread = start_thread_worker(subst_worker)
|
62
|
+
@threads << subst_thread
|
63
|
+
end
|
64
|
+
# all workers went down abruply, we need to kill the process.
|
65
|
+
# @parent_thread.raise(Interrupt) if wk.finished? && @threads.empty? && @running
|
66
|
+
end
|
67
|
+
end
|
68
|
+
end
|
54
69
|
end
|
55
70
|
end
|
data/lib/tobox/pool.rb
CHANGED
@@ -2,17 +2,39 @@
|
|
2
2
|
|
3
3
|
module Tobox
|
4
4
|
class Pool
|
5
|
+
class KillError < Interrupt; end
|
6
|
+
|
5
7
|
def initialize(configuration)
|
6
8
|
@configuration = configuration
|
9
|
+
@logger = @configuration.default_logger
|
7
10
|
@num_workers = configuration[:concurrency]
|
8
11
|
@workers = Array.new(@num_workers) do |idx|
|
9
12
|
Worker.new("tobox-worker-#{idx}", configuration)
|
10
13
|
end
|
11
|
-
|
14
|
+
@worker_error_handlers = Array(@configuration.lifecycle_events[:error_worker])
|
15
|
+
@running = true
|
12
16
|
end
|
13
17
|
|
14
18
|
def stop
|
19
|
+
return unless @running
|
20
|
+
|
15
21
|
@workers.each(&:finish!)
|
22
|
+
@running = false
|
23
|
+
end
|
24
|
+
|
25
|
+
def do_work(wrk)
|
26
|
+
wrk.work
|
27
|
+
rescue KillError
|
28
|
+
# noop
|
29
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
30
|
+
wrk.finish!
|
31
|
+
@logger.error do
|
32
|
+
"(worker: #{wrk.label}) -> " \
|
33
|
+
"crashed with error\n" \
|
34
|
+
"#{e.class}: #{e.message}\n" \
|
35
|
+
"#{e.backtrace.join("\n")}"
|
36
|
+
end
|
37
|
+
@worker_error_handlers.each { |hd| hd.call(e) }
|
16
38
|
end
|
17
39
|
end
|
18
40
|
|
data/lib/tobox/version.rb
CHANGED
data/lib/tobox/worker.rb
CHANGED
@@ -2,7 +2,10 @@
|
|
2
2
|
|
3
3
|
module Tobox
|
4
4
|
class Worker
|
5
|
+
attr_reader :label
|
6
|
+
|
5
7
|
def initialize(label, configuration)
|
8
|
+
@label = label
|
6
9
|
@wait_for_events_delay = configuration[:wait_for_events_delay]
|
7
10
|
@handlers = configuration.handlers || {}
|
8
11
|
@fetcher = Fetcher.new(label, configuration)
|
@@ -13,6 +16,10 @@ module Tobox
|
|
13
16
|
define_singleton_method(:message_to_arguments, &message_to_arguments)
|
14
17
|
end
|
15
18
|
|
19
|
+
def finished?
|
20
|
+
@finished
|
21
|
+
end
|
22
|
+
|
16
23
|
def finish!
|
17
24
|
@finished = true
|
18
25
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: tobox
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.3.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- HoneyryderChuck
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2022-
|
11
|
+
date: 2022-12-21 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: sequel
|
@@ -52,15 +52,15 @@ files:
|
|
52
52
|
- lib/tobox/pool/threaded_pool.rb
|
53
53
|
- lib/tobox/version.rb
|
54
54
|
- lib/tobox/worker.rb
|
55
|
-
homepage: https://gitlab.com/
|
55
|
+
homepage: https://gitlab.com/os85/tobox
|
56
56
|
licenses: []
|
57
57
|
metadata:
|
58
|
-
homepage_uri: https://gitlab.com/
|
58
|
+
homepage_uri: https://gitlab.com/os85/tobox
|
59
59
|
allowed_push_host: https://rubygems.org
|
60
|
-
source_code_uri: https://gitlab.com/
|
61
|
-
bug_tracker_uri: https://gitlab.com/
|
62
|
-
documentation_uri: https://gitlab.com/
|
63
|
-
changelog_uri: https://gitlab.com/
|
60
|
+
source_code_uri: https://gitlab.com/os85/tobox
|
61
|
+
bug_tracker_uri: https://gitlab.com/os85/tobox/issues
|
62
|
+
documentation_uri: https://gitlab.com/os85/tobox
|
63
|
+
changelog_uri: https://gitlab.com/os85/tobox/-/blob/master/CHANGELOG.md
|
64
64
|
rubygems_mfa_required: 'true'
|
65
65
|
post_install_message:
|
66
66
|
rdoc_options: []
|