tobox 0.3.2 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: cb81b3eab97621080f2cf55ba787943d26a11a38c0e807313a810adc8c0d8136
4
- data.tar.gz: 0556f5ba59e7b2feb8a47b04d53e00869d041e747a6789c7f4fe4067f17ad795
3
+ metadata.gz: f162a3a57381f1e5e15ddb1034e82bdd7fcddc720f6fa38264351c6e04407aa9
4
+ data.tar.gz: 7b4ca98af244fff5ee76eb38141047b1c97ee40dd79c942cdb0e82480b8f8439
5
5
  SHA512:
6
- metadata.gz: d109a8b77dafed4066d60cff257541e3eb1a7a6950efa7e94ec75e6e68fb1cf60e5684edc46b9ec99f0f56573807f354b16b7ee5c188b94a848cc52423528c30
7
- data.tar.gz: f8d87bd79921b774326e7db4afd532696814c2b38d5e45deae582eaacdd7e7f161c92b81d627ab2b1c17c51bad0a8a0a1be000839cd675da885a78107e0f4d16
6
+ metadata.gz: 673ead552e6419b25e876e13134fc4b9b04f9505544f04f572570c2bc8d7495f8e5e511047d4545c4bd6ff03a3b0e472c65ec69b8a40ffe355931e3bdb907a4d
7
+ data.tar.gz: 52ba20bddfaaafb13555fc4e72860feea2bb9377480191dcce5be291d136831559f3579467fc44f032cd94ac833094938264a193af558a334b51d2b6550801bd
data/CHANGELOG.md CHANGED
@@ -1,13 +1,49 @@
1
1
  ## [Unreleased]
2
2
 
3
+ ## [0.4.0] - 2023-05-19
3
4
 
4
- ## [0.3.2] - 2022-03-06
5
+ ### Features
6
+
7
+ #### `:stats` plugin
8
+
9
+ The `:stats` plugin collects statistics related with the outbox table periodically, and exposes them to app code (which can then relay them to a statsD collector, or similar tool).
10
+
11
+
12
+ ```ruby
13
+ plugin(:stats)
14
+ on_stats(5) do |stats_collector| # every 5 seconds
15
+ stats = stats_collector.collect
16
+ StatsD.gauge('outbox_pending_backlog', stats[:pending_count])
17
+ end
18
+ ```
19
+
20
+ Read more about it in [the project README](https://gitlab.com/os85/tobox#stats).
21
+
22
+ #### on_start/on_stop callbacks
23
+
24
+ The `on_start` and `on_stop` callbacks can now be defined in `tobox` configuration:
25
+
26
+ ```ruby
27
+ # tobox.rb
28
+ on_start do
29
+ puts "tobox is starting..."
30
+ end
31
+ on_stop do
32
+ puts "tobox is stopping..."
33
+ end
34
+ ```
35
+
36
+ ### Bugfixes
37
+
38
+ * tobox configuration file is now only loaded after everything else, so access to application code is guaranteed.
39
+
40
+ ## [0.3.2] - 2023-03-06
5
41
 
6
42
  ### Bugfixes
7
43
 
8
44
  * allow sentry error capture if `report_after_retries` option is turned off.
9
45
 
10
- ## [0.3.1] - 2022-03-03
46
+ ## [0.3.1] - 2023-03-03
11
47
 
12
48
  ### Bugfixes
13
49
 
data/README.md CHANGED
@@ -20,6 +20,7 @@ Simple, data-first events processing framework based on the [transactional outbo
20
20
  - [Zeitwerk](#zeitwerk)
21
21
  - [Sentry](#sentry)
22
22
  - [Datadog](#datadog)
23
+ - [Stats](#stats)
23
24
  - [Supported Rubies](#supported-rubies)
24
25
  - [Rails support](#rails-support)
25
26
  - [Why?](#why)
@@ -482,6 +483,69 @@ end
482
483
  plugin(:datadog)
483
484
  ```
484
485
 
486
+ <a id="markdown-datadog" name="stats"></a>
487
+ ### Stats
488
+
489
+ The `stats` plugin collects statistics related with the outbox table periodically, and exposes them to app code (which can then relay them to a statsD collector, or similar tool).
490
+
491
+ ```ruby
492
+ plugin(:stats)
493
+ on_stats(5) do |stats_collector| # every 5 seconds
494
+ stats = stats_collector.collect
495
+ #
496
+ # stats => {
497
+ # pending_count: number of new events in the outbox table
498
+ # failing_count: number of events which have failed processing but haven't reached the threshold
499
+ # failed_count: number of events which have failed the max number of tries
500
+ # inbox_count: (if used) number of events marked as received in the inbox table
501
+ # }
502
+ #
503
+ # now you can send them to your statsd collector
504
+ #
505
+ StatsD.gauge('outbox_pending_backlog', stats[:pending_count])
506
+ end
507
+ ```
508
+
509
+ #### Bring your own leader election
510
+
511
+ The stats collection runs on every `tobox` initiated. If you're launching it in multiple servers / containers / pods, this means you'll be collecting statistics about the same database on all of these instances. This may not be desirable, and you may want to do this collection in a single instance. This is not a problem that `tobox` can solve by itself, so you'll have to take care of that yourself. Still, here are some cheap recommendations.
512
+
513
+ ##### Postgres advisory locks
514
+
515
+ If your database is PostgreSQL, you can leverage session-level [advisory locks](https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS) to ensure single-instance access to this functionality. `tobox` also exposes the database instance to the `on_stats` callback:
516
+
517
+ ```ruby
518
+ c.on_stats(5) do |stats_collector, db|
519
+ if db.get(Sequel.function(:pg_try_advisory_lock, 1))
520
+ stats = stats_collector.collect
521
+ StatsD.gauge('outbox_pending_backlog', stats[:pending_count])
522
+ end
523
+ end
524
+ ```
525
+
526
+ If a server goes down, one of the remaining ones will acquire the lock and ensure stats processing.
527
+
528
+ ##### Redis distributed locks
529
+
530
+ If you're already using [redis](https://redis.io/), you can use its distributed lock feature to achieve the goal:
531
+
532
+ ```ruby
533
+ # using redlock
534
+ c.on_stats(5) do |stats_collector, db|
535
+ begin
536
+ lock_info = lock_manager.lock("outbox", 5000)
537
+
538
+ stats = stats_collector.collect
539
+ StatsD.gauge('outbox_pending_backlog', stats[:pending_count])
540
+
541
+ # extend to hold the lock for the next loop
542
+ lock_info = lock_manager.lock("outbox", 5000, extend: lock_info)
543
+ rescue Redlock::LockError
544
+ # some other server already has the lock, try later
545
+ end
546
+ end
547
+ ```
548
+
485
549
  <a id="markdown-supported-rubies" name="supported-rubies"></a>
486
550
  ## Supported Rubies
487
551
 
@@ -6,18 +6,23 @@ module Tobox
6
6
  @configuration = configuration
7
7
  @running = false
8
8
 
9
- worker = @configuration[:worker]
9
+ @on_start_handlers = Array(configuration.lifecycle_events[:on_start])
10
+ @on_stop_handlers = Array(configuration.lifecycle_events[:on_stop])
11
+
12
+ worker = configuration[:worker]
10
13
 
11
14
  @pool = case worker
12
15
  when :thread then ThreadedPool
13
16
  when :fiber then FiberPool
14
17
  else worker
15
- end.new(@configuration)
18
+ end.new(configuration)
16
19
  end
17
20
 
18
21
  def start
19
22
  return if @running
20
23
 
24
+ @on_start_handlers.each(&:call)
25
+
21
26
  @pool.start
22
27
  @running = true
23
28
  end
@@ -25,6 +30,8 @@ module Tobox
25
30
  def stop
26
31
  return unless @running
27
32
 
33
+ @on_stop_handlers.each(&:call)
34
+
28
35
  @pool.stop
29
36
 
30
37
  @running = false
data/lib/tobox/cli.rb CHANGED
@@ -18,15 +18,15 @@ module Tobox
18
18
  def run
19
19
  options = @options
20
20
 
21
+ # boot
22
+ options.fetch(:require).each(&method(:require))
23
+
21
24
  config = Configuration.new do |c|
22
25
  c.instance_eval(File.read(options.fetch(:config_file)), options.fetch(:config_file), 1)
23
26
  end
24
27
 
25
28
  logger = config.default_logger
26
29
 
27
- # boot
28
- options.fetch(:require).each(&method(:require))
29
-
30
30
  # signals
31
31
  pipe_read, pipe_write = IO.pipe
32
32
  %w[INT TERM].each do |sig|
@@ -7,7 +7,7 @@ module Tobox
7
7
  class Configuration
8
8
  extend Forwardable
9
9
 
10
- attr_reader :handlers, :lifecycle_events, :arguments_handler, :default_logger
10
+ attr_reader :handlers, :lifecycle_events, :arguments_handler, :default_logger, :database
11
11
 
12
12
  def_delegator :@config, :[]
13
13
 
@@ -60,6 +60,22 @@ module Tobox
60
60
  @default_logger = @config[:logger] || Logger.new(STDERR, formatter: DEFAULT_LOG_FORMATTER) # rubocop:disable Style/GlobalStdStream
61
61
  @default_logger.level = @config[:log_level] || (env == "production" ? Logger::INFO : Logger::DEBUG)
62
62
 
63
+ @database = if @config[:database_uri]
64
+ database_opts = {}
65
+ database_opts[:max_connections] = @config[:concurrency] if @config[:worker] == :thread
66
+ Sequel.connect(@config[:database_uri].to_s, database_opts)
67
+ else
68
+ Sequel::DATABASES.first
69
+ end
70
+ raise Error, "no database found" unless @database
71
+
72
+ if @database.frozen?
73
+ raise "#{@database} must have the :date_arithmetic extension loaded" unless Sequel.respond_to?(:date_add)
74
+ else
75
+ @database.extension :date_arithmetic
76
+ @database.loggers << @default_logger unless @config[:environment] == "production"
77
+ end
78
+
63
79
  freeze
64
80
  end
65
81
 
@@ -70,6 +86,16 @@ module Tobox
70
86
  self
71
87
  end
72
88
 
89
+ def on_start(&callback)
90
+ (@lifecycle_events[:on_start] ||= []) << callback
91
+ self
92
+ end
93
+
94
+ def on_stop(&callback)
95
+ (@lifecycle_events[:on_stop] ||= []) << callback
96
+ self
97
+ end
98
+
73
99
  def on_before_event(&callback)
74
100
  (@lifecycle_events[:before_event] ||= []) << callback
75
101
  self
@@ -116,6 +142,7 @@ module Tobox
116
142
  @handlers.each_value(&:freeze).freeze
117
143
  @lifecycle_events.each_value(&:freeze).freeze
118
144
  @plugins.freeze
145
+ @database.freeze
119
146
  super
120
147
  end
121
148
 
data/lib/tobox/fetcher.rb CHANGED
@@ -10,13 +10,7 @@ module Tobox
10
10
 
11
11
  @logger = @configuration.default_logger
12
12
 
13
- database_uri = @configuration[:database_uri]
14
- @db = database_uri ? Sequel.connect(database_uri.to_s) : Sequel::DATABASES.first
15
- raise Error, "no database found" unless @db
16
-
17
- @db.extension :date_arithmetic
18
-
19
- @db.loggers << @logger unless @configuration[:environment] == "production"
13
+ @db = configuration.database
20
14
 
21
15
  @table = configuration[:table]
22
16
  @group_column = configuration[:group_column]
@@ -30,16 +30,16 @@ module Tobox
30
30
 
31
31
  scope = ::Sentry.get_current_scope
32
32
 
33
- scope.set_contexts(
34
- id: event[:id],
35
- type: event[:type],
36
- attempts: event[:attempts],
37
- created_at: event[:created_at],
38
- run_at: event[:run_at],
39
- last_error: event[:last_error]&.byteslice(0..1000),
40
- version: Tobox::VERSION,
41
- db_adapter: @db_scheme
42
- )
33
+ scope.set_contexts(tobox: {
34
+ id: event[:id],
35
+ type: event[:type],
36
+ attempts: event[:attempts],
37
+ created_at: event[:created_at],
38
+ run_at: event[:run_at],
39
+ last_error: event[:last_error]&.byteslice(0..1000),
40
+ version: Tobox::VERSION,
41
+ db_adapter: @db_scheme
42
+ })
43
43
  scope.set_tags(
44
44
  outbox: @db_table,
45
45
  event_id: event[:id],
@@ -0,0 +1,121 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Tobox
4
+ module Plugins
5
+ module Stats
6
+ module ConfigurationMethods
7
+ attr_reader :stats_interval_seconds
8
+
9
+ def on_stats(stats_interval_seconds, &callback)
10
+ @stats_interval_seconds = stats_interval_seconds
11
+
12
+ (@lifecycle_events[:stats] ||= []) << callback
13
+ self
14
+ end
15
+ end
16
+
17
+ class StatsEmitter
18
+ def initialize(config)
19
+ @config = config
20
+ @running = false
21
+ end
22
+
23
+ def start
24
+ return if @running
25
+
26
+ config = @config
27
+
28
+ interval = config.stats_interval_seconds
29
+ @stats_handlers = Array(config.lifecycle_events[:stats])
30
+
31
+ return if @stats_handlers.empty?
32
+
33
+ @error_handlers = Array(config.lifecycle_events[:error_worker])
34
+
35
+ @max_attempts = config[:max_attempts]
36
+
37
+ @db = Sequel.connect(config.database.opts.merge(single_threaded: true))
38
+ @outbox_table = config[:table]
39
+ @outbox_ds = @db[@outbox_table]
40
+
41
+ inbox_table = config[:inbox_table]
42
+ @inbox_ds = @db[inbox_table] if inbox_table
43
+
44
+ logger = config.default_logger
45
+
46
+ stats = method(:collect_event_stats)
47
+ stats.instance_eval do
48
+ alias collect call
49
+ end
50
+
51
+ @th = Thread.start do
52
+ Thread.current.name = "outbox-stats"
53
+
54
+ loop do
55
+ logger.debug { "stats worker: sleep for #{interval}s..." }
56
+ sleep interval
57
+
58
+ begin
59
+ emit_event_stats(stats)
60
+ rescue RuntimeError => e
61
+ @error_handlers.each { |hd| hd.call(e) }
62
+ end
63
+
64
+ break unless @running
65
+ end
66
+ end
67
+
68
+ @running = true
69
+ end
70
+
71
+ def stop
72
+ return unless @running
73
+
74
+ @th.terminate
75
+
76
+ @db.disconnect
77
+
78
+ @running = false
79
+ end
80
+
81
+ private
82
+
83
+ def emit_event_stats(stats)
84
+ @stats_handlers.each do |hd|
85
+ hd.call(stats, @db)
86
+ end
87
+ end
88
+
89
+ def collect_event_stats
90
+ stats = @outbox_ds.group_and_count(
91
+ Sequel.case([
92
+ [{ last_error: nil }, "pending_count"],
93
+ [Sequel.expr([:attempts]) < @max_attempts, "failing_count"]
94
+ ],
95
+ "failed_count").as(:status)
96
+ )
97
+ stats = stats.as_hash(:status, :count).transform_keys(&:to_sym)
98
+
99
+ # fill it in
100
+ stats[:pending_count] ||= 0
101
+ stats[:failing_count] ||= 0
102
+ stats[:failed_count] ||= 0
103
+
104
+ stats[:inbox_count] = @inbox_ds.count if @inbox_ds
105
+ stats
106
+ end
107
+ end
108
+
109
+ class << self
110
+ def configure(config)
111
+ emitter = StatsEmitter.new(config)
112
+
113
+ config.on_start(&emitter.method(:start))
114
+ config.on_stop(&emitter.method(:stop))
115
+ end
116
+ end
117
+ end
118
+
119
+ register_plugin :stats, Stats
120
+ end
121
+ end
data/lib/tobox/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Tobox
4
- VERSION = "0.3.2"
4
+ VERSION = "0.4.0"
5
5
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: tobox
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.2
4
+ version: 0.4.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - HoneyryderChuck
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2023-03-06 00:00:00.000000000 Z
11
+ date: 2023-05-19 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: sequel
@@ -46,6 +46,7 @@ files:
46
46
  - lib/tobox/plugins/datadog/integration.rb
47
47
  - lib/tobox/plugins/datadog/patcher.rb
48
48
  - lib/tobox/plugins/sentry.rb
49
+ - lib/tobox/plugins/stats.rb
49
50
  - lib/tobox/plugins/zeitwerk.rb
50
51
  - lib/tobox/pool.rb
51
52
  - lib/tobox/pool/fiber_pool.rb
@@ -77,7 +78,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
77
78
  - !ruby/object:Gem::Version
78
79
  version: '0'
79
80
  requirements: []
80
- rubygems_version: 3.3.7
81
+ rubygems_version: 3.4.6
81
82
  signing_key:
82
83
  specification_version: 4
83
84
  summary: Transactional outbox pattern implementation in ruby